So im trying to create a lambda function that will trigger when an S3 PUT occurs for now it seems to be executing the code but im getting a problem
import boto3
def lambda_handler(event, context):
instances = ["i-0b5d926"]
region = 'ap-south-1a'
ec2 = boto3.client('ec2',, region_name=region)
ec2.instances.filter(InstanceIds=instances ).start()
im getting this error
{
"stackTrace": [
[
"/var/task/index.py",
9,
"lambda_handler",
"return ec2.instances.filter(InstanceIds=instaces).start()"
],
[
"/var/runtime/botocore/client.py",
509,
"__getattr__",
"self.__class__.__name__, item)"
]
],
"errorType": "AttributeError",
"errorMessage": "'EC2' object has no attribute 'instances'"
}
Any help would be appreciated
ON A Side Note THE IAMROLE has full Access (S3,EC2,Lambda) will be configured later for specific purpose
ok i understood the error
instance was null
i changed
ec2.instances.filter(InstanceIds=instances ).start()
to
ec2.start_instances(InstanceIds=instances)
and started
Related
I have a Lambda function (east-us-1) that needs to publish messages to SNS topics in both (east-us-1) & (eu-central-1) regions. Is this possible?
Here is my code snippet. Can someone help on how can I achieve this?
I am surprised to see that the boto3 client documentation says "You can publish messages only to topics and endpoints in the same Amazon Web Services Region"
Link Here:
from __future__ import print_function
import json
import urllib
import boto3
print('Loading message function...')
def send_to_sns(message, context):
# This function receives JSON input with three fields: the ARN of an SNS topic,
# a string with the subject of the message, and a string with the body of the message.
# The message is then sent to the SNS topic.
#
# Example:
# {
# "topic": "arn:aws:sns:REGION:123456789012:MySNSTopic",
# "subject": "This is the subject of the message.",
# "message": "This is the body of the message."
# }
sns = boto3.client('sns')
sns.publish(
TopicArn=message['topic'],
Subject=message['subject'],
Message=message['body']
)
return ('Sent a message to an Amazon SNS topic.')
My code throws the following error when SNS is in different region
{
"response": {
"stackTrace": [
[
"/var/task/lambda_function.py",
28,
"send_to_sns",
"Message=message['body']"
],
[
"/var/runtime/botocore/client.py",
357,
"_api_call",
"return self._make_api_call(operation_name, kwargs)"
],
[
"/var/runtime/botocore/client.py",
676,
"_make_api_call",
"raise error_class(parsed_response, operation_name)"
]
],
"errorType": "InvalidParameterException",
"errorMessage": "An error occurred (InvalidParameter) when calling the Publish operation: Invalid parameter: TopicArn"
}
}
you are facing this issue because of region conflict with below code you can check
your current region
my_session = boto3.session.Session()
my_region = my_session.region_name
and to solve your problem just do this i am assuming you are connected to us-east-1 and your sns is present in us-east-2 region
sns = boto3.client('sns',region_name='us-east-2')
Working on starting a new ECS task using Boto3. Here is the snippet:
responseStartTask = client.start_task(
cluster=clstr,
containerInstances=[
"e21200f598xxxx26bd32xxxxxfccxxxx-2355xxxx0",
],
taskDefinition=taskDefinition,
networkConfiguration={
'awsvpcConfiguration': {
'subnets': [
'subnet-06b1exxxxx8eb5312',
'subnet-0ccf0b4xxxxx8df77',
'subnet-0334eaxxxxx441f63'
],
'securityGroups': [
'sg-053356faxxxxxx400',
]
}
})
But this fails with this exception:
An error occurred (InvalidParameterException) when calling the StartTask operation: instanceId length should be one of [32,36]
Could you please help me out in what to pass in for this containerInstances parameter ? (Currently I have passed container runtime ID from the ECS task details)
Compute environments created via boto3 are not displayed in AWS console. I can see them in the batch_client.describe_compute_environments() call response:
{
'computeEnvironmentName': 'name',
'computeEnvironmentArn': 'arn:aws:batch:us-east-1:<ID>:compute-environment/ml-retraining-compute-env-second',
'ecsClusterArn': 'arn:aws:ecs:us-east-1:<ID>:cluster/ml-retraining-compute-env-second_Batch_b18fcd09-8d7e-351b-bc0f-13ffa83a6b15',
'type': 'MANAGED',
'state': 'ENABLED',
'status': 'INVALID',
'statusReason': "CLIENT_ERROR - The security group 'sg-2436d85c' does not exist",
'computeResources': {
'type': 'EC2',
'minvCpus': 0,
'maxvCpus': 512,
'desiredvCpus': 24,
'instanceTypes': [
'optimal'
],
'subnets': [
'subnet-fa22de86'
],
'securityGroupIds': [
'sg-2436d85c'
],
'instanceRole': 'arn:aws:iam::<ID>:instance-profile/ecsInstanceRole',
'tags': {
'component': 'ukai-training-pipeline',
'product': 'Cormorant',
'jira_project_team': 'CORPRJ',
'business_unit': 'Threat Systems Products',
'created_by': 'ml-pipeline'
}
},
'serviceRole': 'arn:aws:iam::<ID>:role/AWSBatchServiceRole'
}
but the Compute Environments table on the Batch page in AWS console UI does not show anything. The table is empty. When I try to create compute environment with the same name again via boto3 call, I get this response:
ERROR - Error setting compute environment: An error occurred
(ClientException) when calling the CreateComputeEnvironment operation: Object already exists.
Based on the comments, the issue was the use of different region in the console.
The solution was to change the region.
I am following this doc https://aws.amazon.com/premiumsupport/knowledge-center/start-glue-job-run-end/ to setup an auto-trigger on lambda when crawler finishes. The event pattern I set on cloudwatch is:
{
"detail": {
"crawlerName": [
"reddit_movie"
],
"state": [
"Succeeded"
]
},
"detail-type": [
"Glue Crawler State Change"
],
"source": [
"aws.glue"
]
}
And I add a lambda function as target for this rule in cloudwatch.
I manually trigger the crawler but it doesn't trigger the lambda after it finished. From the crawler log I can see:
04:36:28
[6c8450a5-970a-4190-bd2b-829a82d67fdf] INFO : Table redditmovies_bb008c32d0d970f0465f47490123f749 in database video has been updated with new schema
04:36:30
[6c8450a5-970a-4190-bd2b-829a82d67fdf] BENCHMARK : Finished writing to Catalog
04:37:37
[6c8450a5-970a-4190-bd2b-829a82d67fdf] BENCHMARK : Crawler has finished running and is in state READY
Does above log mean crawler finished successfully? How do I know why the lambda function is not triggered from crawler?
And how I can debug this issue? which log should i look at?
Following works -
Cloudwatch Event Rule -
{
"source": [
"aws.glue"
],
"detail-type": [
"Glue Crawler State Change"
],
"detail": {
"state": [
"Succeeded"
]
}
}
Sample lambda -
def lambda_handler(event, context):
try:
if event and 'detail' in event and event['detail'] and 'crawlerName' in event['detail']:
crawler_name = event['detail']['crawlerName']
print('Received event from crawlerName - {0}'.format(crawler_name))
crawler = glue.get_crawler(Name=crawler_name)
print('Received crawler from glue - {0}'.format(str(crawler)))
database = crawler['Crawler']['DatabaseName']
except Exception as e:
print('Error handling events from crawler. Details - {0}'.format(e))
raise e
Here is screenshot -
At first, I follow the link https://aws.amazon.com/premiumsupport/knowledge-center/start-glue-job-run-end/ and it doesn't work. I found it is due to the python script lambda in the link is not correct if you paste it directly. Please have a check of your lambda.
The python lambda copied from link
import boto3
client = boto3.client('glue')
def lambda_handler(event, context):
response = client.start_job_run(JobName = 'MyTestJob')
We need to fix it as below:
import boto3
client = boto3.client('glue')
def lambda_handler(event, context):
response = client.start_job_run(JobName = 'MyTestJob')
I've prepared a simple lambda function in AWS to terminate long running EMR clusters after a certain threshold is reached. This code snippet is tested locally and is working perfectly fine. Now I pushed it into a lambda, took care of the library dependencies, so that's also fine. This lambda is triggered from a CloudWatch rule, which is a simple cron schedule. I'm using an existing IAM rule which has these 7 policies attached to it.
SecretsManagerReadWrite
AmazonSQSFullAccess
AmazonS3FullAccess
CloudWatchFullAccess
AWSGlueServiceRole
AmazonSESFullAccess
AWSLambdaRole
I've configured the lambda to be inside the same vpc and security group as that of the emr(s). Still I'm getting this error consistently:
An error occurred (AccessDeniedException) when calling the ListClusters operation: User: arn:aws:sts::xyz:assumed-role/dev-lambda-role/terminate_inactive_dev_emr_clusters is not authorized to perform: elasticmapreduce:ListClusters on resource: *: ClientError
Traceback (most recent call last):
File "/var/task/terminate_dev_emr.py", line 24, in terminator
ClusterStates=['STARTING', 'BOOTSTRAPPING', 'RUNNING', 'WAITING']
File "/var/runtime/botocore/client.py", line 314, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/var/runtime/botocore/client.py", line 612, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (AccessDeniedException) when calling the ListClusters operation: User: arn:aws:sts::xyz:assumed-role/dev-lambda-role/terminate_inactive_dev_emr_clusters is not authorized to perform: elasticmapreduce:ListClusters on resource: *
My lambda function looks something like this:
import pytz
import boto3
from datetime import datetime, timedelta
def terminator(event, context):
''' cluster lifetime limit in hours '''
LIMIT = 7
TIMEZONE = 'Asia/Kolkata'
AWS_REGION = 'eu-west-1'
print('Start cluster check')
emr = boto3.client('emr', region_name=AWS_REGION)
local_tz = pytz.timezone(TIMEZONE)
today = local_tz.localize(datetime.today(), is_dst=None)
lifetimelimit = today - timedelta(hours=LIMIT)
clusters = emr.list_clusters(
CreatedBefore=lifetimelimit,
ClusterStates=['STARTING', 'BOOTSTRAPPING', 'RUNNING', 'WAITING']
)
if clusters['Clusters'] is not None:
for cluster in clusters['Clusters']:
description = emr.describe_cluster(ClusterId=cluster['Id'])
if(len(description['Cluster']['Tags']) == 1
and description['Cluster']['Tags'][0]['Key'] == 'dev.ephemeral'):
print('Terminating Cluster: [{id}] with name [{name}]. It was active since: [{time}]'.format(id=cluster['Id'], name=cluster['Name'], time=cluster['Status']['Timeline']['CreationDateTime'].strftime('%Y-%m-%d %H:%M:%S')))
emr.terminate_job_flows(JobFlowIds=[cluster['Id']])
print('cluster check done')
return
Any help is appreciated.
As error message indicates, lambda does not have permissions to call ListClusters on EMR. As you are working with EMR clusters and would also like to terminate the clusters, you should give lambda function proper IAM role which is having that capability to do that. Create a new IAM policy from AWS console (say EMRFullAccess). here is how it looks like
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": "elasticmapreduce:*",
"Resource": "*"
}
]
}
After creating policy, create a new role from AWS console with lambda as service and attach newly created policy above. After that, attach this role to your lambda function. That should solve issue :-)