I need to invoke a Lambda function that accepts an S3 path. Below sample code of the lambda function.
def lambda_handler(event, context):
bucket = "mybucket"
key = "mykey/output/model.tar.gz"
model = load_model(bucket, key)
somecalc = some_func(model)
result = {'mycalc': json.dumps(somecalc)}
return result
I need to invoke this handler from my client code using boto3. I know I can do a request like below
lambda_client = boto3.client('lambda')
response = lambda_client.invoke(
FunctionName='mylambda_function',
InvocationType='RequestResponse',
LogType='Tail',
ClientContext='myContext',
Payload=b'bytes'|file,
Qualifier='1'
)
But I am not sure how to specify an S3 path in the payload. Looks like it is expecting a JSON.
Any suggestion?
You can specify a payload like so:
payload = json.dumps({ 'bucket': 'myS3Bucket' })
lambda_client = boto3.client('lambda')
response = lambda_client.invoke(
FunctionName='mylambda_function',
InvocationType='RequestResponse',
LogType='Tail',
ClientContext='myContext',
Payload=payload,
Qualifier='1'
)
And access the payload properties in your lamdba handler like so:
def lambda_handler(event, context):
bucket = event['bucket'] # pull from 'event' argument
key = "mykey/output/model.tar.gz"
model = load_model(bucket, key)
somecalc = some_func(model)
result = {'mycalc': json.dumps(somecalc)}
return result
Related
I'm currently developing a lambda that invokes another Lambda with Boto3. However, I need from one statement to retrieve a group of results and send them through the invoke payload to the other Lambda. However, I can't find how to send this function as a parameter to call another Lambda and pass through parameters a function that returns a set of information.
I have implemented this method:
from MysqlConnection import MysqlConnection
from sqlalchemy import text
def make_dataframe(self):
conn = MysqlConnection()
query = text("""select * from queue WHERE estatus = 'PENDING' limit 4;""")
df = pd.read_sql_query(query,conn.get_engine())
return df.to_json()
This is the Lambda handler:
import json
import boto3
from MysqlConnection import MysqlConnection
from Test import Test
client = boto3.client('lambda')
def lambda_handler(event, context):
mydb = MysqlConnection()
print(mydb.get_engine)
df = Test()
df.make_dataframe()
object = json.loads(df.make_dataframe())
response = client.invoke(
FunctionName='arn:aws:lambda:',
InvocationType='RequestResponse'#event
Payload=json.dumps(object)
)
responseJson = json.load(response['Payload'])
print('\n')
print(responseJson)
print('\n')
What you're doing is correct in terms of structuring your call.
I assume the problem is with your payload structure and whether its stringified.
I would try invoke your lambda with an empty payload and see what happens. If it works with empty payload then its your payload serialising, if it doesnt work with empty payload then its something else.
In cloudwatch what do your logs of both your "runner" lambda and your "target" lambda say?
It might also be a permissions thing - you will need to specify and grant execute permissions on your runner lambda.
after days of refactoring and research I am sharing the answer. It is about packing the json.dump object and inside the handler place the method with the response already packed
This a method to parent child
class Test:
def make_dataframe(self):
conn = MysqlConnection()
query = text("""select * from TEST WHERE status'PEN' limit 4;""")
df = pd.read_sql_query(query,conn.get_engine())
lst = df.values.tolist()
obj = json.dumps(lst, cls=CustomJSONEncoder)
return obj
def lambda_handler(event, context):
mydb = MysqlConnection()
df = Test()
response = client.invoke(
FunctionName='arn:aws:lambda:',
InvocationType='RequestResponse',
Payload= df.make_dataframe()
)
responseJson = json.load(response['Payload'])
print('\n')
print(responseJson)
print('\n')
`
I am trying to copy multiple files from one s3 bucket to another s3 bucket using lambda function but it is just copying 2 files in destination s3 bucket.
Here is my code:
# using python and boto3
import json
import boto3
s3_client = boto3.client('s3')
def lambda_handler(event, context):
source_bucket_name = event['Records'][0]['s3']['bucket']['name']
file_name = event['Records'][0]['s3']['object']['key']
destination_bucket_name = 'nishantnkd'
copy_object = {'Bucket': source_bucket_name, 'Key': file_name}
s3_client.copy_object(CopySource=copy_object,
Bucket=destination_bucket_name, Key=file_name)
return {'statusCode': 3000,
'body': json.dumps('File has been Successfully Copied')}
I presume that the Amazon S3 bucket is configured to trigger the AWS Lambda function when a new object is created.
When the Lambda function is triggered, it is possible that multiple event records are sent to the function. Therefore, it should loop through the event records like this:
# using python and boto3
import json
import boto3
s3_client = boto3.client('s3')
def lambda_handler(event, context):
for record in event['Records']: # This loop added
source_bucket_name = record['s3']['bucket']['name']
file_name = urllib.parse.unquote_plus(record['s3']['object']['key']) # Note this change too
destination_bucket_name = 'nishantnkd'
copy_object = {'Bucket': source_bucket_name, 'Key': file_name}
s3_client.copy_object(CopySource=copy_object, Bucket=destination_bucket_name, Key=file_name)
return {'statusCode': 3000,
'body': json.dumps('File has been Successfully Copied')}
I'm working on unit tests for my lambda which is getting some files from S3, processing them and loading data from them to DynamoDB. I created botocore stubbers that are used during tests, but I got botocore.exceptions.NoCredentialsError: Unable to locate credentials
My lambda handler code
s3_client = boto3.client('s3')
ddb_client = boto3.resource('dynamodb', region_name='eu-west-1')
def lambda_handler(event, context):
for record in event['Records']:
s3_event = record.get('s3')
bucket = s3_event.get('bucket', {}).get('name', '')
file_key = s3_event.get('object', {}).get('key', '')
file = s3_client.get_object(Bucket=bucket, Key=file_key)
and tests file:
class TestLambda(unittest.TestCase):
def setUp(self) -> None:
self.session = botocore.session.get_session()
# S3 Stubber Set Up
self.s3_client = self.session.create_client('s3', region_name='eu-west-1')
self.s3_stubber = Stubber(self.s3_client)
# DDB Stubber Set Up
self.ddb_resource = boto3.resource('dynamodb', region_name='eu-west-1')
self.ddb_stubber = Stubber(self.ddb_resource.meta.client)
def test_s3_to_ddb_handler(self) -> None:
event = {}
with self.s3_stubber:
with self.ddb_stubber:
response = s3_to_ddb_handler.s3_to_ddb_handler(event, ANY)
Issue seems to be that actual call to AWS resources is done which shouldnt be the case and stubber should be used, how can I force that?
You need to call .activate() on your Stubber instances: https://botocore.amazonaws.com/v1/documentation/api/latest/reference/stubber.html#botocore.stub.Stubber.activate
I just used below code on Lambda function and length function to calculate listner rules. However it always return the value as 2 even ALB has more than 20 rules.
import json
import boto3
def lambda_handler(event, context):
client = boto3.client('elbv2')
response1 = client.describe_listeners(
ListenerArns=[
'arn:aws:elasticloadbalancing:eu-west-2:account_id:listener/app/my_load_balancer_listener',
],
)
tot = len(response1)
return response1
Get the output like this.
Response as 2
The get the rules, you should use describe_rules:
def lambda_handler(event, context):
client = boto3.client('elbv2')
response1 = client.describe_rules(
ListenerArn='arn:aws:elasticloadbalancing:eu-west-2:account_id:listener/app/my_load_balancer_listener',
)
tot = len(response1['Rules'])
return tot
I have a lambda function that generates a list of untagged buckets in AWS environment. Currently I send the output to a slack channel directly. Instead I would like to have my lambda dump the output to a csv file and send it as a report. Here is the code for it, let me know if you need any other details.
import boto3
from botocore.exceptions import ClientError
import urllib3
import json
http = urllib3.PoolManager()
def lambda_handler(event, context):
#Printing the S3 buckets with no tags
s3 = boto3.client('s3')
s3_re = boto3.resource('s3')
buckets = []
print('Printing buckets with no tags..')
for bucket in s3_re.buckets.all():
s3_bucket = bucket
s3_bucket_name = s3_bucket.name
try:
response = s3.get_bucket_tagging(Bucket=s3_bucket_name)
except ClientError:
buckets.append(bucket)
print(bucket)
for bucket in buckets:
data = {"text": "%s bucket has no tags" % (bucket)}
r = http.request("POST", "https://hooks.slack.com/services/~/~/~",
body = json.dumps(data),
headers = {"Content-Type": "application/json"})