Configure AWS Lambda function to use latest version of a Layer - amazon-web-services

Question

I have more than 20 lambda functions in a developing application. And a lambda layer that contains a good amount of common code.
A Lambda function, is hook it to a particular version of the layer, and every time I update a layer, it generates a new version. Since it is a developing application, I have a new version of the layer almost every day. That creates a mess on the lambda functions that have to be touched every day - to upgrade the layer version.
I know it is important to freeze code for a lambda function in production, and it is essential to hook one version of the lambda function to a version of the layer.
But, for the development environment, is it possible to prevent generating a new layer version every time a layer is updated? Or configure the lambda function so that the latest lambda version always refers to the latest layer version?

Answer 1

Unfortunately it is currently not possible to reference the latest, and there is no concept of aliases for the layer versions.
The best suggestion would be to automate this, so that whenever you create a new Lambda Layer version it would update all Lambda functions that currently include this Lambda Layer.
To create this event trigger, create a CloudWatch function that uses its event to listen for the PublishLayerVersion event.
Then have it trigger a Lambda that would trigger the update-function-layers function for each Lambda to replace its layer with the new one.

Answer 2

Enhance from #Chris answer, you can also use a lambda-backed Custom Resource in your stack and use this lambda to update the target configuration with the new layer ARN. I note this out in case if there someone have the similar need when I found out this thread couple days ago.
There are some notes on this solution:
The lambda of the customer resource has to send status response back to the trigger CloudFormation (CFN) endpoint, or else the CFN stack will hanging till timeout (about an hour or more, it's a painful process if you have problem on this lambda, be careful with that)
Easy way to send response back, you can use cfnresponse (pythonic way), this lib is available magically when you use CFN lambda inline code (CFN setup this lib when processing CFN with inline code) and must have a line 'import cfnresponse' :D
CFN will not touch to the custom resource after it created, so when you update stack for new layer change, the lambda will not trigger. A trick to make it move is to use custom resource with custom property then you will change this property with something will change each time you execute the stack, layer version arn. So this custom resource will be updated, means the lambda of this resource will be triggered when the stack update.
Not sure why the Logical Name of the lambda layer is changed with AWS::Serverless:Layer so I can't DependOns that layer logical name but I still have !Ref its ARN
Here is a sample code
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
myshared-libraries layer
Resources:
LambdaLayer:
Type: AWS::Serverless::LayerVersion
Properties:
LayerName: !Sub MyLambdaLayer
Description: Shared library layer
ContentUri: my_layer/layerlib.zip
CompatibleRuntimes:
- python3.7
ConsumerUpdaterLambda:
Type: AWS::Serverless::Function
Properties:
FunctionName: consumer-updater
InlineCode: |
import os, boto3, json
import cfnresponse
def handler(event, context):
print('EVENT:[{}]'.format(event))
if event['RequestType'].upper() == 'UPDATE':
shared_layer = os.getenv("DB_LAYER")
lambda_client = boto3.client('lambda')
consumer_lambda_list = ["target_lamda"]
for consumer in consumer_lambda_list:
try:
lambda_name = consumer.split(':')[-1]
lambda_client.update_function_configuration(FunctionName=consumer, Layers=[shared_layer])
print("Updated Lambda function: '{0}' with new layer: {1}".format(lambda_name, shared_layer))
except Exception as e:
print("Lambda function: '{0}' has exception: {1}".format(lambda_name, str(e)))
responseValue = 120
responseData = {}
responseData['Data'] = responseValue
cfnresponse.send(event, context, cfnresponse.SUCCESS, responseData)
Handler: index.handler
Runtime: python3.7
Role: !GetAtt ConsumerUpdaterRole.Arn
Environment:
Variables:
DB_LAYER: !Ref LambdaLayer
ConsumerUpdaterRole:
Type: AWS::IAM::Role
Properties:
Path: /
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service: lambda.amazonaws.com
Action: sts:AssumeRole
ManagedPolicyArns:
- Fn::Sub: arn:${AWS::Partition}:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
Policies:
- PolicyName:
Fn::Sub: updater-lambda-configuration-policy
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- lambda:GetFunction
- lambda:GetFunctionConfiguration
- lambda:UpdateFunctionConfiguration
- lambda:GetLayerVersion
- logs:DescribeLogGroups
- logs:CreateLogGroup
Resource: "*"
ConsumerUpdaterMacro:
DependsOn: ConsumerUpdaterLambda
Type: Custom::ConsumerUpdater
Properties:
ServiceToken: !GetAtt ConsumerUpdaterLambda.Arn
DBLayer: !Ref LambdaLayer
Outputs:
SharedLayer:
Value: !Ref LambdaLayer
Export:
Name: MySharedLayer
Another option is using stack Notification ARN which send all stack events into a defined SNS, where you will use it to trigger your update lambda. In your lambda, you will filter the SNS message body (which is a readable json liked format string) with the AWS::Lambda::Layer resource then grab the PhysicalResourceId for the layer ARN. How to engage the SNS topic to your stack, use CLI sam/cloudformation deploy --notification-arns option. Unfortunately, CodePipeline doesn't support this configuration option so you can only use with CLI only
Sample code for your lambda to extract/filter the SNS message body with resource data
import os, boto3, json
def handler(event, context):
print('EVENT:[{}]'.format(event))
resource_data = extract_subscription_msg(event['Records'][0]['Sns']['Message'])
layer_arn = ''
if len(resource_data) > 0:
if resource_data['ResourceStatus'] == 'CREATE_COMPLETE' and resource_data['ResourceType'] == 'AWS::Lambda::LayerVersion':
layer_arn = resource_data['PhysicalResourceId']
if layer_arn != '':
lambda_client = boto3.client('lambda')
consumer_lambda_list = ["target_lambda"]
for consumer in consumer_lambda_list:
lambda_name = consumer.split(':')[-1]
try:
lambda_client.update_function_configuration(FunctionName=consumer, Layers=[layer_arn])
print("Update Lambda: '{0}' to layer: {1}".format(lambda_name, layer_arn))
except Exception as e:
print("Lambda function: '{0}' has exception: {1}".format(lambda_name, str(e)))
return
def extract_subscription_msg(msg_body):
result = {}
if msg_body != '':
attributes = msg_body.split('\n')
for attr in attributes:
if attr != '':
items = attr.split('=')
if items[0] in ['PhysicalResourceId', 'ResourceStatus', 'ResourceType']:
result[items[0]] = items[1].replace('\'', '')
return result

Answer 3

It is possible to derive the most recent version number of a layer, using an additional data statement, as per https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/lambda_layer_version
So in your definition module, you will have the original layer resource definition
resource "aws_lambda_layer_version" "layer_mylib" {
filename = "layer_mylib.zip"
layer_name = "layer_mylib"
compatible_runtimes = ["python3.6", "python3.7", "python3.8"]
}
and then to obtain the ARN with latest version, use
data "aws_lambda_layer_version" "mylatest" {
layer_name = aws_lambda_layer_version.layer_mylib.layer_name
}
then data.aws_lambda_layer_version.mylatest.arn
will give the reference which includes the latest version number, which can be checked by placing
output {
value = data.aws_lambda_layer_version.mylatest.arn
}
in your common.tf