- Terraform expects a deployment package before it can provision a lambda function
- The deployment package can actually be any non-empty zip file
- Once the lambda has been created you should deploy actual code using a CI/CD pipeline
- Consider using a dedicated serverless framework, but define clear boundaries
- Use a Terraform module for provisioning lambda functions to reduce effort
Terraform is a great infrastructure-as-code tool which we love at Amido, but effectively implementing the aws_lambda_function resource in the real world can be a little bit challenging. I am going to explain an approach for provisioning lambda functions without needing a pre-built deployment package.
Terraform and lambda – chicken or the egg?
The lambda function resource is a bit special, as it requires a suitable deployment package containing your function code to exist before it can go ahead and create the function. This contrasts to typical Terraform resources, and infrastructure as code in general, where you can stand-up resources in advance. The necessary deployment package can be in the form of a local file or an s3 object.
(This is requirement comes from the underlying AWS CreateFunction API action.)
Local file vs s3 object
If you have a function which is using an interpreted language, and the code is never going to change, then storing a pre-packaged deployment package alongside your Terraform and using the local “filename” option is probably going to suffice. In which case, move along, nothing to see here.
However, functions which using a compiled language, which get built and packaged in a CI pipeline are going to require a more creative solution.
What Terraform wants you to do
What Terraform wants you to do feels something like this:
Great, except providing actual code to Terraform just feels wrong for many reasons:
- Last time I checked, Terraform was an Infrastructure-as-code tool. Infrastructure! Since when did you have to provide application code to create infrastructure?
- There is a circular dependency between the function code CI/CD pipeline and Terraform
- You need to ensure Terraform always references the correct version of the deployment package for each environment, this might not always be the latest
- How do you ensure Terraform doesn’t subsequently “update” the function code outside of your CI/CD process?
Surely there must be a better way….
What I want to do
I want to create the infrastructure (specifically, an empty shell of a lambda function, with it’s corresponding IAM role, triggers, log configuration, permissions, etc) first, so that a separate CI/CD pipeline can then build, package, and deploy the function code. Like this
My first plan was to simply trick Terraform into provisioning my lambda function by giving it an empty zip file, so I whipped out the trusty archive_file provider and pointed it to an empty dir and fired away:
Not so fast! Looks like someone has thought of this already, and the AWS API swiftly rejects my file with the following message:
OK then, so let’s just add some content into the zip file:
It worked! Now obviously any invocations of the lambda function will fail at this point, but it’s now super easy to deploy your actual code to the function from a CI/CD pipeline completely independently from Terraform, for example:
aws lambda update-function-code --function-name $function_name --zip-file fileb://$source_path –publish
Now that you are ready to create the lambda function, You probably need a whole host of other resources to complement it into an actual working
- IAM (aws_iam_role / aws_iam_role_policy)
- event source mapping for SQS, DynamoDB, Kinesis (aws_lambda_event_source_mapping)
- bucket notification for s3 (aws_s3_bucket_notification)
- cloudwatch trigger for scheduling (aws_cloudwatch_event_rule, aws_cloudwatch_event_target)
- permissions for invocations from other resources (aws_lambda_permission)
You can encapsulate this collection of resources into a re-usable Terraform module to reduce the effort required. Or you could use one which already exists
There are frameworks available which are designed specifically for implementing serverless architectures. These do a lot of the heavy lifting by assisting with creation of supporting resources and configuration such as IAM roles, triggers, permissions, etc. In addition to provisioning resources, they can also handle packaging and deployment of the function code, so if you have a significant amount of lambda functions, should really check them out:
- Serverless Framework – https://serverless.com/
- AWS Serverless Application Model (SAM) – https://aws.amazon.com/serverless/sam/
- Apex – https://apex.run/
There is overlap between Terraform and the “Serverless” / “SAM frameworks and as these can all provision supporting resources as well as the lambda function itself. Consider how the rest of your infrastructure is going to be managed, as if you end up using Terraform and the Serverless framework, this will cause some fragmentation in tooling. For example, what if you also have a significant container infrastructure provisioned using Terraform, but some containers share resources (s3, sqs, etc) with lambda functions, where should the shared resources get provisioned? Terraform or the Serverless framework?