I recently changed jobs and inherited an infrastructure built with Terraform with an application which deploys to EC2 via AWS CodeDeploy. The ASG’s and target groups were all built and managed by Terraform, and CodeDeploy used In-Place deployments. In-place deployments means that the deployment automation does this: Stop the application on the existing servers, which brings the website down for customers Install the new version of the application on to the same existing servers Start the application Obviously, stopping the application and bringing the website down is a huge anti-pattern, so one of the first tasks assigned to me at the new job was to implement blue/green deployments, roughly like this: ...

I am a huge fan of high taxes (for the rich), and a common argument I hear against it is “If taxes are too high, nobody will open businesses! Why should someone do a bunch of work if they can’t reap the rewards?”. And I guess it sounds right intuitively, but it’s also factually dead wrong. Tons of companies got started when tax rates were way higher than they are today. In this blog post I’ll combine some data from various sources into a big table which shows the highest marginal tax rates when various companies got started. ...

Recently at work we started using AWS Network Firewall to meet a compliance objective, and in doing so I learned some of the tricks and gotchas around how to set up your network routing in various network architectures. AWS has some of this documented pretty well and some of it documented pretty terribly. In this post I’ll explain what I learned and hopefully the next person who Googles it will end up here. ...

The other day at work I had a need to use the AWS CLI to get a list of RDS snapshots (aws rds describe-db-snapshots), but I only wanted the snapshots where Status == "available". In true DevOps fashion, I googled for a one-liner that I could copy/paste instead of actually learning the query syntax myself, but apparently nobody has posted it anywhere on the internet. So if you’re in the same spot and you’re googling for “aws rds describe-db-snapshots status available” or similar, here it is: ...

AWS CodePipeline and AWS CodeBuild are a great, serverless way to build your Docker containers and deploy them to ECS (or wherever). You can even use an IAM role with CodeBuild to let your build pipeline access other services like S3, etc. AWS CodeBuild already provisions a container for you to use, and that’s where it runs all the commands in your buildspec file. But if you’re using CodeBuild to build a Docker container, then you’ll be running a container (the one you’re building) inside another container (the CodeBuild one, where your buildspec stuff runs). ...

We started using Concourse at $job, as a replacement for Jenkins. One common thing that almost every Concourse job/pipeline/task will need is access to our git repository, which means Concourse needs an SSH key for our repos. I googled around for examples of other people using Concourse with SSH keys, and wasn’t able to find any example of anyone storing their SSH keys in AWS SSM Parameter Store. So I’ll post my own example and hopefully help the next person who googles it. ...

At $job, we use a version of chef-client which is woefully out of date. We’ll get around to fixing it, but until we do, I ran into a unique problem with encrypted data bags that I hadn’t seen documented anywhere on the internet. Hence this blog post. My version of knife is: knife --version Chef: 14.1.12 When I create an encrypted data bag, it uses version 3 of Chef’s data bag encryption. When I try to use that data bag with our ancient version of Chef, it results in: ...

At $job, we’ve started using AWS CodePipeline and AWS CodeBuild to build and deploy our Docker images to ECS. Once we got the pipeline working, we could git push and a few minutes later a container build would kickoff. This was mostly great, but the problem was that our builds took too long. Developers would push their changes and expect to see them live, but it took about half an hour for the build to complete and go live. ...

At $job, we use an SSH bastion host to connect to our cloud environment. This made using Test Kitchen a bit annoying. My previous way of getting Test Kitchen to work involved putting a ProxyCommand into my ~/.ssh/config, which worked, sometimes, but was still annoying. Eventually I stumbled into a completely undocumented function of test kitchen which solves this exact problem. ssh_gateway Turns out that kitchen has support for SSH bastion hosts built in, it’s just not documented anywhere. Here’s what you need in your .kitchen.yml: ...

I recently started a new job at a startup. It’s a remote job at a company I had never previously heard of, and I found the job listing through Hacker News. If you already read Hacker News, you already know that it’s a great resource for startups, nerds, nerds at startups, and more. There’s a monthly jobs thread where tons of startups will post jobs, usually seeking software engineers, big data experts, and DevOps cloud nerds like myself. Some of the jobs have a requirement of physically coming to an office, but many of the jobs are remote. Here is a link to this month’s jobs thread (June 2018). ...