Deploying to EC2 from GitHub via S3

I won’t get into the reasoning behind it but I recently found myself looking to set up continuous code deployment from a private Git repository at GitHub to an EC2 instance at Amazon Web Services.

There are some documented ways to do this that combine GitHub Actions and AWS CodeDeploy but most of what I saw required logging into a GitHub account via AWS and stashing those credentials, then using them to pull from the repo during the deployment process.  For various reasons, that wasn’t going to work for me.  Deployment keys were also an option that had to be ruled out.  (I’m not saying this was a typical use case, don’t worry.)

What I ended up doing instead was adding a step in the middle using AWS S3.  Essentially, the code gets staged there and then deployed to the EC2 instance, all triggered by GitHub Actions.

It works like this…

The EC2 instance (project-instance [obviously not the actual name I used but I’m going to be obfuscating names with horribly creative replacements]) needs to have the CodeDeploy Agent installed.  This instance is Ubuntu 22.04 and the automated tooling for installing the agent on 22.04 is broken so I followed some manual steps.

sudo apt-get update
sudo apt-get install ruby-full ruby-webrick wget -y
cd /tmp
wget https://aws-codedeploy-us-east-1.s3.us-east-1.amazonaws.com/releases/codedeploy-agent_1.3.2-1902_all.deb
mkdir codedeploy-agent_1.3.2-1902_ubuntu22
dpkg-deb -R codedeploy-agent_1.3.2-1902_all.deb codedeploy-agent_1.3.2-1902_ubuntu22
sed 's/Depends:.*/Depends:ruby3.0/' -i ./codedeploy-agent_1.3.2-1902_ubuntu22/DEBIAN/control
dpkg-deb -b codedeploy-agent_1.3.2-1902_ubuntu22/
sudo dpkg -i codedeploy-agent_1.3.2-1902_ubuntu22.deb
systemctl list-units --type=service | grep codedeploy
sudo service codedeploy-agent status

I restarted the instance at this point but I’m not sure if that’s necessary.  It’s a little-used service so I could get away with that.

With that out of the way, we bring  up the AWS Management Console and head over to S3, mostly because it’s the easiest part to take care of.  There, we create a bucket (referred to henceforth as deploy-bucket) with default settings.  It’s not strictly necessary but I added a lifecycle rule to expire all objects in the bucket after two days to keep storage costs down.

Staying in AWS, we then move over to IAM, where we’ve got a handful of things to do.

We create a new user (deploy-user).  While creating it, we add a new group (CodeDeploy) and give that group the AWSCodeDeployDeployerAccess and AmazonS3FullAccess permissions policies.  It seems like AmazonS3FullAccess might be overkill, that access could be limited to deploy-bucket, but I’m going off of some other docs and not really experimenting.  We add deploy-user to this new group.  We also give it programmatic access and save the generated keys for later.

We need the EC2 instance to have a proper role.  If it already has one, it can be edited to include the permissions that follow.  Otherwise a new role with those permissions can be created.

If we’re creating the new role, the entity is an AWS Service and the use case is EC2.  We’ll call it ec2-instance-role.  It gets AmazonEC2RoleForAWSCodeDeploy permissions.

Now we need a service role.  Once again, the entity is an AWS Service and the use case is EC2.  Permissions are AWSCodeDeployRole and we’ll call it code-deploy-service.

Now we edit code-deploy-service so that the trust relationships policy is as follows:

{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Effect": "Allow",
			"Principal": {
				"Service": "codedeploy.amazonaws.com"
			},
			"Action": "sts:AssumeRole"
		}
	]
}

Since we created a new role for the EC2 instance, we need to edit the instance to use that role.  To do that, we head over to EC2, open the instance, click the “Actions” drop-down, select “Security” and “Modify IAM Role.”  From there, we select the newly-created ec2-instance-role and save the change.

Now we move over to AWS CodeDeploy, where we create a new application (new-application), set to EC2/On-premises.

Then we create a deployment group inside new-application.  Name it something like new-application-deploy.  For the Service Role, copy over the ARN for code-deploy-service.  The deployment type is in-place.  We set the environment configuration as Amazon EC2 instances, then select from the tag group where the Key is Name and the Value is project-instance.  If we didn’t install the CodeDeploy Agent earlier, we have the option to do it at this point, using the default settings.  Then we set the deployment settings to AllAtOnce and click Create Deployment Group.

Now we head over to GitHub and, under Settings, go to Secrets and Variables.  Here we add new secrets for the previously, saved AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

Finally, we add some definitions to our repository.  In appspec.yml in the root directory, we can do something simple like the following:

version: 0.0
os: linux
files:
  - source: /
    destination: <path-to-destination>

This would deploy the entire repo into the specified destination path.  I’ve actually got some hooks defined, too, but that’s a whole other thing.

The last piece is .github/workflows/workflow.yml.  It looks as follows:

name: CI/CD Pipeline
on:
  push:
    branches: [ main ]
jobs:
  continuous-deployment:
    runs-on: ubuntu-latest
    env:
      AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
      AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
      AWS_DEFAULT_REGION: us-east-2
    strategy:
      matrix:
        appname: ['new-application']
        deploy-group: ['new-application-deploy']
        s3-bucket: ['deploy-bucket']
        s3-filename: ['${{ github.repository }}/${{ github.ref_name }}/${{ github.sha }}']
    steps:
      # Step 1
      - uses: actions/checkout@v3
      - name: Push Deployment to S3
        id: push
        run: |
          aws deploy push \
            --application-name ${{ matrix.appname }} \
            --s3-location s3://${{ matrix.s3-bucket }}/${{ matrix.s3-filename }}.zip \
            --source .
      # Step 2
      - name: Create CodeDeploy Deployment
        id: deploy
        run: |
          aws deploy create-deployment \
            --application-name ${{ matrix.appname }} \
            --deployment-group-name ${{ matrix.deploy-group }} \
            --deployment-config-name CodeDeployDefault.OneAtATime \
            --file-exists-behavior OVERWRITE \
            --s3-location bucket=${{ matrix.s3-bucket }},key=${{ matrix.s3-filename }}.zip,bundleType=zip

We define some configuration variables, then two steps to the deployment.  The first step pushes the repository code to the previously-created S3 bucket as a zip file, named after the repository, the branch, and the commit SHA.  This naming convention is to allow for future projects to also use this S3 bucket.

The second step then creates a deployment using that zip file as the source.

The end result is that, on every commit to the repository’s main branch, GitHub Actions will fire and the repo’s code will be zipped up and sent to S3, with a new deployment created.  Then, over on the CodeDeploy side of things, that zip file will be pulled in and deployed to the EC2 instance.

There’s probably a way to only push edited files but, in this specific case, the repo is small enough that it doesn’t hurt to deploy the whole thing over and over again.