I have a lambda layer which I keep updating. This lambda layer has multiple versions. How can I find the lambda layer ARN with latest version using aws cli?
I am able to do this using the command listed below -
aws lambda list-layer-versions --layer-name <layer name> --region us-east-1 --query 'LayerVersions[0].LayerVersionArn'
Unfortunately, it's currently not possible (I have encountered the same issue).
You can keep the latest ARN in your own place (like DynamoDB) and update it whenever you publish a new version of the layer.
You can create a custom macro to get the latest lambda layer version and use that as a reference.
The following function gets the latest version from the Lambda Layer stack:
import json
import boto3
def latest_lambdalayer(event, context):
fragment = get_latestversion(event['fragment'])
return {
'requestId': event['requestId'],
'status': 'success',
'fragment': fragment
}
def get_latestversion(fragment):
cloudformation = boto3.resource('cloudformation')
stack = cloudformation.Stack('ticketapp-layer-dependencies')
for o in stack.outputs:
if o['OutputKey']=='TicketAppLambdaDependency':
return o['OutputValue']
#return "arn:aws:lambda:eu-central-1:899885580749:layer:ticketapp-dependencies-layer:16"
And you use this when defining the Lambda layer—here using same global template:
Globals:
Function:
Layers:
- !Transform { "Name" : "LatestLambdaLayer"}
Runtime: nodejs12.x
MemorySize: 128
Timeout: 101
Related
I'm playing with AWS lambda and I am unable to change the default parameters that are used in the lambda. Is there a workaround for this?
Setup:
Lambda "iAmInvoked" is created by a stack in cloudformation which has default parameter values set (I set these defaults thinking that, these will be used in case invoker doesn't provide values for the parameters required and can be overridden). I'm invoking this iAmInvoked lambda asynchronously using a lambda called "iWillInvoke" and providing the payload which contains new values for parameters to be used by iAmInvoked instead of its defaults.
iWillInvoke code:
import json
import boto3
client = boto3.client('lambda')
def lambda_handler(event, context):
payloadForLambda = { 'parameter1' : 'abc,def' , 'parameter2' : '123456' , 'parameter3' : '987654' }
client.invoke(
FunctionName='arn:aws:lambda:us-west-2:123456789:function:iAmInvoked',
InvocationType='Event',
Payload=json.dumps(payloadForLambda)
)
iAmInvoked Code:
AWSTemplateFormatVersion: 2010-09-09
Description: |
"Creates required IAM roles to give permission to get and put SSM parameters and creates lambda function that shares the parameter(s)."
Parameters:
parameter1:
Type: String
Default: parameterValueThatShallBeOverridden1
parameter2:
Type: String
Default: parameterValueThatShallBeOverridden2
parameter3:
Type: String
Default: parameterValueThatShallBeOverridden3
Question/Issue:
Doesn't matter what I provide in the payload of iWillInvoke, iAmInvoked is using its default values. Is there a way I can override the defaults?
iAmInvoked Code is not your function code nor its parameters. Its CloudFormation template and parameters for the template. Using client.invoke does not affect in any form and shape the CloudFormation template.
To work with CloudFormation in boto3, there is cloudformation SDK.
The scenario is like this
I have a microservice which invokes a LAMBDA function whose role will be to delete things from the AWS IOT.
Is there a way I can perform operations in AWS IOT using the lambda function?
Any article, blog regarding this will be a huge help as I'm not able to find any integration document on the web.
I found a way to do several operations in AWS IOT using lambda function by the means of API's.
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/iot.html#IoT.Client.delete_thing_group
The above link has description about API's which will help in this case.
A sample lambda function script to delete a thing from the IOT is
import json
import boto3
def lambda_handler(event, context):
thing_name = event['thing_name']
delete_thing(thing_name=thing_name)
def delete_thing(thing_name):
c_iot = boto3.client('iot')
print(" DELETING {}".format(thing_name))
try:
r_principals = c_iot.list_thing_principals(thingName=thing_name)
except Exception as e:
print("ERROR listing thing principals: {}".format(e))
r_principals = {'principals': []}
print("r_principals: {}".format(r_principals))
for arn in r_principals['principals']:
cert_id = arn.split('/')[1]
print(" arn: {} cert_id: {}".format(arn, cert_id))
r_detach_thing = c_iot.detach_thing_principal(thingName=thing_name, principal=arn)
print("DETACH THING: {}".format(r_detach_thing))
r_upd_cert = c_iot.update_certificate(certificateId=cert_id, newStatus='INACTIVE')
print("INACTIVE: {}".format(r_upd_cert))
r_del_cert = c_iot.delete_certificate(certificateId=cert_id, forceDelete=True)
print(" DEL CERT: {}".format(r_del_cert))
r_del_thing = c_iot.delete_thing(thingName=thing_name)
print(" DELETE THING: {}\n".format(r_del_thing))
And the input for this lambda function will be
{
"thing_name": "MyIotThing"
}
Is it possible to create an Athena view via cloudformation template. I can create the view using the Athena Dashboard but I want to do this programmatically using CF templates. Could not find any details in AWS docs so not sure if supported.
Thanks.
It is possible to create views with CloudFormation, it's just very, very, complicated. Athena views are stored in the Glue Data Catalog, like databases and tables are. In fact, Athena views are tables in Glue Data Catalog, just with slightly different contents.
See this answer for the full description how to create a view programmatically, and you'll get an idea for the complexity: Create AWS Athena view programmatically – it is possible to map that to CloudFormation, but I would not recommend it.
If you want to create databases and tables with CloudFormation, the resources are AWS::Glue::Database and AWS::Glue::Table.
In general, CloudFormation is used for deploying infrastructure in a repeatable manner. This doesn't apply much to data inside a database, which typically persists separately to other infrastructure.
For Amazon Athena, AWS CloudFormation only supports:
Data Catalog
Named Query
Workgroup
The closest to your requirements is Named Query, which (I think) could store a query that can create the View (eg CREATE VIEW...).
See: AWS::Athena::NamedQuery - AWS CloudFormation
Update: #Theo points out that AWS CloudFormation also has AWS Glue functions that include:
AWS::Glue::Table
This can apparently be used to create a view. See comments below.
I think for now the best way to create Athena view from CloudFormation template is to use Custom resource and Lambda. We have to supply methods for View creation and deletion. For example, using crhelper library Lambda could be defined:
from __future__ import print_function
from crhelper import CfnResource
import logging
import os
import boto3
logger = logging.getLogger(__name__)
helper = CfnResource(json_logging=False, log_level='DEBUG', boto_level='CRITICAL', sleep_on_delete=120)
try:
client = boto3.client('athena')
ATHENA_WORKGROUP = os.environ['athena_workgroup']
DATABASE = os.environ['database']
QUERY_CREATE = os.environ['query_create']
QUERY_DROP = os.environ['query_drop']
except Exception as e:
helper.init_failure(e)
#helper.create
#helper.update
def create(event, context):
logger.info("View creation started")
try:
executionResponse = client.start_query_execution(
QueryString=QUERY_CREATE,
QueryExecutionContext={'Database': DATABASE},
WorkGroup='AudienceAthenaWorkgroup'
)
logger.info(executionResponse)
response = client.get_query_execution(QueryExecutionId=executionResponse['QueryExecutionId'])
logger.info(response)
if response['QueryExecution']['Status']['State'] == 'FAILED':
logger.error("Query failed")
raise ValueError("Query failed")
helper.Data['success'] = True
helper.Data['id'] = executionResponse['QueryExecutionId']
helper.Data['message'] = 'query is running'
except Exception as e:
print(f"An exception occurred: {e}")
if not helper.Data.get("success"):
raise ValueError("Creating custom resource failed.")
return
#helper.delete
def delete(event, context):
logger.info("View deletion started")
try:
executionResponse = client.start_query_execution(
QueryString=QUERY_DROP,
QueryExecutionContext={'Database': DATABASE},
WorkGroup='AudienceAthenaWorkgroup'
)
logger.info(executionResponse)
except Exception as e:
print("An exception occurred")
print(e)
#helper.poll_create
def poll_create(event, context):
logger.info("Pol creation")
response = client.get_query_execution(QueryExecutionId=event['CrHelperData']['id'])
logger.info(f"Poll response: {response}")
# There are 3 types of state of query
# if state is failed - we stop and fail creation
# if state is queued - we continue polling in 2 minutes
# if state is succeeded - we stop and succeed creation
if 'FAILED' == response['QueryExecution']['Status']['State']:
logger.error("Query failed")
raise ValueError("Query failed")
if 'SUCCEEDED' == response['QueryExecution']['Status']['State']:
logger.error("Query SUCCEEDED")
return True
if 'QUEUED' == response['QueryExecution']['Status']['State']:
logger.error("Query QUEUED")
return False
# Return a resource id or True to indicate that creation is complete. if True is returned an id
# will be generated
# Return false to indicate that creation is not complete and we need to poll again
return False
def handler(event, context):
helper(event, context)
The Athena queries for view creation/updation/deletion are passed as environmental parameters to Lambda.
In CloudFormation template we have to define the Lambda that invokes mentioned Python code and creates/updates/deletes Athena view. For example
AthenaCommonViewLambda:
Type: 'AWS::Lambda::Function'
DependsOn: [CreateAthenaViewLayer, CreateAthenaViewLambdaRole]
Properties:
Environment:
Variables:
athena_workgroup: !Ref AudienceAthenaWorkgroup
database:
Ref: DatabaseName
query_create: !Sub >-
CREATE OR REPLACE VIEW ${TableName}_view AS
SELECT field1, field2, ...
FROM ${DatabaseName}.${TableName}
query_drop: !Sub DROP VIEW IF EXISTS ${TableName}_common_view
Code:
S3Bucket: !Ref SourceS3Bucket
S3Key: createview.zip
FunctionName: !Sub '${AWS::StackName}_create_common_view'
Handler: createview.handler
MemorySize: 128
Role: !GetAtt CreateAthenaViewLambdaRole.Arn
Runtime: python3.8
Timeout: 60
Layers:
- !Ref CreateAthenaViewLayer
AthenaCommonView:
Type: 'Custom::AthenaCommonView'
Properties:
ServiceToken: !GetAtt AthenaCommonViewLambda.Arn
I am trying to get AWS CloudFormation to create a template that will allow me to attach an event to an existing S3 Bucket that will trigger a Lambda Function whenever a new file is put into a specific directory within the bucket. I am using the following YAML as a base for the CloudFormation template but cannot get it working.
---
AWSTemplateFormatVersion: '2010-09-09'
Resources:
SETRULE:
Type: AWS::S3::Bucket
Properties:
BucketName: bucket-name
NotificationConfiguration:
LambdaConfigurations:
- Event: s3:ObjectCreated:Put
Filter:
S3Key:
Rules:
- Name: prefix
Value: directory/in/bucket
Function: arn:aws:lambda:us-east-1:XXXXXXXXXX:function:lambda-function-trigger
Input: '{ CONFIGS_INPUT }'
I have tried rewriting this template a number of different ways to no success.
Since you have mentioned that those buckets already exists, this is not going to work. You can use CloudFormation in this way but only to create a new bucket, not to modify existing bucket if that bucket was not created via that template in the first place.
If you don't want to recreate your infrastructure, it might be easier to just use some script that will subscribe lambda function to each of the buckets. As long as you have a list of buckets and the lambda function, you are ready to go.
Here is a script in Python3. Assuming that we have:
2 buckets called test-bucket-jkg2 and test-bucket-x1gf
lambda function with arn: arn:aws:lambda:us-east-1:605189564693:function:my_func
There are 2 steps to make this work. First, you need to add function policy that will allow s3 service to execute that function. Second, you will loop through the buckets one by one, subscribing lambda function to each one of them.
import boto3
s3_client = boto3.client("s3")
lambda_client = boto3.client('lambda')
buckets = ["test-bucket-jkg2", "test-bucket-x1gf"]
lambda_function_arn = "arn:aws:lambda:us-east-1:605189564693:function:my_func"
# create a function policy that will permit s3 service to
# execute this lambda function
# note that you should specify SourceAccount and SourceArn to limit who (which account/bucket) can
# execute this function - you will need to loop through the buckets to achieve
# this, at least you should specify SourceAccount
try:
response = lambda_client.add_permission(
FunctionName=lambda_function_arn,
StatementId="allow s3 to execute this function",
Action='lambda:InvokeFunction',
Principal='s3.amazonaws.com'
# SourceAccount="your account",
# SourceArn="bucket's arn"
)
print(response)
except Exception as e:
print(e)
# loop through all buckets and subscribe lambda function
# to each one of them
for bucket in buckets:
print("putting config to bucket: ", bucket)
try:
response = s3_client.put_bucket_notification_configuration(
Bucket=bucket,
NotificationConfiguration={
'LambdaFunctionConfigurations': [
{
'LambdaFunctionArn': lambda_function_arn,
'Events': [
's3:ObjectCreated:*'
]
}
]
}
)
print(response)
except Exception as e:
print(e)
You could write a custom resource to do this, in fact that's what I've ended up doing at work for the same problem. At the simplest level, define a lambda that takes a put bucket notification configuration and then just calls the put bucket notification api with the data that was passed it.
If you want to be able to control different notifications across different cloudformation templates, then it's a bit more complex. Your custom resource lambda will need to read the existing notifications from S3 and then update these based on what data was passed to it from CF.
I have a lambda function which reads from Dynamodb stream. I have the Dynamodb stream ARN exported from another stack in the same AWS account. Now, while adding eventSource in Lambda, it asks from Table construct.
const function = new lambda.Function(...);
function.addEventSource(new DynamoEventSource(table, {
startingPosition: lambda.StartingPosition.TrimHorizon
}));
Ref: https://awslabs.github.io/aws-cdk/refs/_aws-cdk_aws-lambda-event-sources.html#dynamodb-streams
But I have the stream ARN. Is there any way I can make use of this to add the event source. Or I have to export the table itself?
It's currently not possible to import a DynamoDB table with the AWS CDK. Importing a DynamoDB table.
Still you can reach your goal by using the EventSourceMapping class from #aws-cdk/aws-lambda directly:
import iam = require('#aws-cdk/aws-iam');
import lambda = require('#aws-cdk/aws-lambda');
const fn = new lambda.Function(...);
new lambda.EventSourceMapping(this, 'DynamoDBEventSource', {
target: fn,
batchSize: ...,
eventSourceArn: <your stream arn>,
startingPosition: lambda.StartingPosition.TrimHorizon
});
fn.addToRolePolicy(
new iam.PolicyStatement()
.addActions('dynamodb:DescribeStream', 'dynamodb:GetRecords', 'dynamodb:GetShardIterator', 'dynamodb:ListStreams')
.addResource('<your stream arn>/*');
);
As the Stream ARN could change every time it's disabled/enabled, one can look up the ARN dynamically and export it as env variable.
export STREAM_ARN=$(aws dynamodb describe-table --table-name MyFirstTable |jq -r '.Table.LatestStreamArn')
Then this could be used inside CDK Code
const table = dynamodb.Table.fromTableAttributes(this, 'id-for-table', {
tableArn:
"arn:aws:dynamodb:us-east-1:123456789:table/MyFirstTable",
tableStreamArn:process.env.STREAM_ARN
});