Tag: CI

Using ECR images with Elastic Beanstalk and CodeBuild

As I mentioned previously, I’ve been using CodePipeline and CodeBuild for continuous integration and delivery on a new project. For the most part I’ve been happy with it, but ran into an issue last week that took some experimenting to figure out.

In addition to CodePipeline and CodeBuild, we’re also using Elastic Beanstalk for deploying our application. I first experimented with Elastic Beanstalk when it was released, and at the time it had quite a few opinions about how to build an application. When I took another look a few months ago, though, I found that it has grown up considerably. The default model is still dumping your code on an EC2 host and managing the autoscaling group, but it also supports deploying arbitrary Docker images — both single images and ECS clusters. The ability to use CodeBuild with Elastic Beanstalk means you can utilize the same CI pipeline regardless of whether you’re deploying as the result of a web hook or as a one off from the command line.

This was working great until we started using a custom Docker image, hosted in the Elastic Container Registry (ECR), for our CodeBuild jobs. When we moved to that image, one-off builds broke with an error indicating CodeBuild couldn’t pull the Docker image from ECR (“Perhaps you need to login?” The error message helpfully suggested. Yes, perhaps.) This was pretty confusing, since I was able to confirm that both one-off and pipeline builds were using the same IAM Role, which had permission to fetch images from ECR (and was working from the pipeline). The AWS documentation iterates the permissions you need in order to use ECR with CodeBuild, and indeed, that role had the proper permissions assigned.

As I experimented, though, I discovered that the issue was not the CodeBuild role, but rather the ECR Repository Policy. When executing CodeBuild as the result of eb deploy, the ECR repository must be configured to allow image pulls from the CodeBuild service. I suspect this is because eb deploy doesn’t execute as your existing CodeBuild project: it creates a new one with the source and output set to S3, runs the build, and tears it down after collecting the artifacts.

I was able to apply the policy with the following Terraform fragment:

data "aws_iam_policy_document" "ecr_policy" {
  statement {
    sid    = "CodebuildPullPolicy"
    effect = "Allow"
    actions = [
      "ecr:GetDownloadUrlForLayer",
      "ecr:BatchGetImage",
      "ecr:BatchCheckLayerAvailability",
    ]

    principals {
      type = "Service"
      identifiers = [
        "codebuild.amazonaws.com",
      ]
    }
  }
}

resource "aws_ecr_repository_policy" "build" {
  repository = "${aws_ecr_repository.build.name}" # the name of your ECR repository
  policy = "${data.aws_iam_policy_document.ecr_policy.json}"
}

Once this was applied, Pipeline and command line builds both were able to pull the image.

Continuous Integration with CodeBuild and CodePipeline

Last week I started on a new project — venture, perhaps — which means I’m in the thick of figuring out all the things you take for granted when you join an established project. I’m trying to balance pragmatism and perfectionism: for example, I don’t need to scale to 1MM users yet (I probably don’t need to scale to 10 yet), so spinning up a Kubernetes cluster doesn’t make sense. And, I’m not an animal, so it’s not like I’m going to SSH into a machine and build it by hand. So I’m trying to look at it from the perspective of “what do we need for the next 3 months?”

One of the questions was about CI/CD. I’ve previously used Travis and Circle CI, and they’re both fine. They work, they have the features you want, they have outages when you curse them, etc. As I was looking around, though, I came across the AWS offerings: the confusingly named Code Build and Code Pipeline. (There’s also a Code Deploy; who could say why you’d chose a Deploy over, say, a Pipeline?)

It took me a while to figure out how the two are different, and how they work together. My present understanding is that Code Build provides “continuous integration”, and Code Pipeline provides “continuous delivery”. In my past experience these have been squished together into a single thing. Which is fine, except that they’re not quite the same thing.

Continuous Integration (CI) ensures that your code builds, tests pass, etc whenever you make a change. In other words, it ensures that your change integrates with the existing code base.

Continuous Delivery (CD) takes your code and gets it onto servers. This may involve deploying Docker containers, pushing Serverless code to Lambda, etc.

In the Code Build / Code Pipeline world, the Build is defined in the source repository (much like the Circle or Travis configuration I’ve seen before), and the Pipeline is configured as infrastructure, outside of the code. I’m using Terraform, which means I can also treat this as code, albeit in a different language.

The Build is defined by a buildspec.yml file, which defines commands to run for each phase. Unlike CircleCI, where you can define arbitrary steps, the phases are more rigid here: install, pre-build, build, post-build. The build spec also defines the artifacts that the build produces, and the artifacts are what downstream processes can consume.

We’re only two weeks in, but one thing that I liked about CodeBuild is that the pricing required very little thought: you pay per minute, regardless of how many things you run. This is in contrast to CircleCI, where you have to think about parallelism: it’s free for one-at-a-time, but after that you pay per parallel build. More specifically, you pay per potential parallel build.

In our case the Pipeline takes the artifact from Code Build, and deploys it to Elastic Beanstalk. I suspect Beanstalk is the first thing we’ll throw away infrastructure wise, but it’s nice for now: we’re able to configure an EC2 instance with everything running on it we need, and scale that up behind an ALB. Because we’re trying to describe all of our infrastructure with Terraform, the hardest part was getting it to use an ALB rather than a classic ELB. This required an Option to be defined in the Environment declaration:


setting {
namespace = "aws:elasticbeanstalk:environment"
name = "LoadBalancerType"
value = "application"
}

Overall I’m pretty happy with CodeBuild and CodePipeline. We’ve decided to re-evaluate early decisions in a month or so, and I’ll post an update about whether we stick with it then.

CI at CC

I wrote about our roll-out of Hudson on the CC Labs blog. I wanted to note a few things about deploying that, primarily for my own reference. Hudson has some great documentation, but I found Joe Heck’s step by step instructions on using Hudson for Python projects particularly helpful. We’re using nose for most of our projects, and buildout creates a nosetest script wrapper that Hudson runs to generate pass/fail reports.

Setting up coverage is on the todo list, but it appears that our particular combination of libraries has at least one strange issue: when cc.license uses Jinja2 to load a template, coverage thinks it’s a Python source file (maybe it uses an import hook or something? haven’t looked) and tries to tokenize it when generating the xml report. Ka-boom. (This has apparently already been reported.)

Another item in the “maybe/someday” file is using Tox to run the tests using multiple versions of Python (example configuration for Tox + Hudson exists). I can see that this is a critical part of the process when releasing libraries for others to consume. We have slightly less surface area — all the servers run the same version of Python — but it’d be great to know exactly what our possible deployment parameters are.

Overall Hudson already feels like it’s adding to our sanity. I just received my copy of Continuous Delivery, so I think this is the start of something wonderful.

date:2010-08-20 10:37:43
wordpress_id:1734
layout:post
slug:ci-at-cc
comments:
category:cc, development
tags:cc, CI, coverage, Hudson, python, sanity