TL;DR: AWS Lambda’s are much simpler and cheaper than managing a server. For nontrivial AWS Lambda functions, consider using the serverless CLI tool.

Non-Existent Problems

My personal website is a static website generated via Hugo. You would think a static website wouldn’t need much on the side of dynamic computing but I found a way!

I use AWS Lambda (which is AWS’s serverless offering) to generate the static website. The Lambda is triggered by a GitHub webhook which sends a HTTP request when a new commit is pushed to the website repo. After setting up the webhook, I realized this freed me up to edit the website anywhere since I no longer needed to run the build step manually.

That’s when I found netlify-cms which creates an admin UI to edit/update static websites that are stored on GitHub/Gitlab/Bitbucket. When you make updates in the admin UI, it will push commits to your git repo (or create PRs depending on the setting). In order to do this, the UI will need to authenticate you via GitHub. Due to GitHub’s design this step requires a server side component. Netlify offers an authentication server for free but I didn’t want to use a third-party service here so I decided to create another Lambda to do this authentication for me.

So this means I created two different Lambda projects. I created and configured one manually and used the serverless CLI tool for the other. The experience was very different for each so I want to talk about it a bit more.

Manually Configured Webhook

I created a new project azeemba/github-webhook-aws-lambda-to-S3 to handle a GitHub webhook. There was already an example project in the serverless/example repo but at this point I was worried about all the permissions that the cli tool required.

Design of the webhook

As the above image shows, when a new commit is pushed to GitHub it will send an HTTP request to the url that I have configured as webhook. The URL is an AWS API Gateway endpoint which is configured to trigger the AWS Lambda. Then the AWS Lambda clones the repo and checks if there is any regeneration needed, if there is then it will regenerate the website and push it back to GitHub. Then it will check if there are any changes that need to be pushed to S3 (where the website is stored), if there are then it will update S3.

The linux image that Lambda uses is missing git (even though their documentation says otherwise). Since I needed git to push back the updated website, I used a feature of Lambda called layers. Layers are intended to allow you to share/reuse code but in this particular instance it means I can use a layer where someone has done the work to set up git already. I tried compiling git in a portable way myself but it turned out to be pretty difficult. Just adding this layer in the AWS UI was much easier.

In addition to adding the layer in the AWS UI, I also had to configure the following things manually:

  • Creating an API Gateway that creates an endpoint/url that will trigger a Lambda. This is the URL I gave to GitHub webhook.
  • Create a user that the Lambda runs as
  • Create a Lambda hooked to the API Gateway
  • Create a script that generates a zip file from my code and pushes the zip to AWS Lambda

Note that the zip file contains the node_modules folder as well since everything your code needs must be packaged in the zip file. This means if I added any “devDependenices” like ESLint, my packaging script will upload that as well.

The Lambda works really well but the manual set up leaves some things to be desired:

  • The API Gateway, the user and the Lambda are configured separately.
    • If I ever wanted to delete this project, I will have to remember to delete all three things.
    • If I wanted to recreate this project, I will have to remember each step and redo the whole process.
  • No way of testing code locally before pushing it out
  • Adding support for dev dependencies that are not pushed will require me to make my packaging script a lot more complex

That’s why for the authentication server I decided to try out serverless CLI.

Automatically Configured Authentication Service

Someone had already written an OAuth server for netlify-cms using serverless here. Since this was my first time using serverless tool, I made some changes in the serverless.yml to simplify it by hardcoding some options like stage and region.

Almost all the configuration for this project is declared within the serverless.yml file. The only data outside of it are the encrypted parameters stored in Amazon KMS. This allows you to deploy secrets without committing to the repo.

The serverless tool, just like my export script, relies on the aws cli being preconfigured. However, the amount of permissions the configured tool needs to have is very different. For my export script, the aws tool needs to be configured with only Lambda access. But for it to work with serverless, it needs a ton of more permissions. The official serverless docs links to this gist which lists out all the permissions needed. The docs also guide you through how you can create a user with those permissions and how you can configure aws cli with that user’s keys. Once this is set up, you can use serverless for your project!

The really cool thing about this setup is that I can test the lambda locally by running serverless offline. Additionally, when I finally deploy this lambda, the serverless optimize plugin will make sure that the dev dependencies are not deployed and it will also significantly compress the scripts being deployed as well. Compared to my export script, the size difference is at least an order of magnitude.

Takeaways

Serverless/AWS Lambda offerings are great. For needs that don’t require excellent response times like hobby projects or non-interactive systems, a Lambda fits the needs much better than a full server like EC2. You don’t have to worry about managing a server and you don’t have to pay for the server when its sitting there just idle and waiting.

If your Lambda is short (30-50 lines) and doesn’t require any additional dependencies, just edit your Lambda in the browser and call it a day. If your Lambda is more complex, consider using the serverless CLI too. The examples repo has loads of examples that you can use to get your started.