So I have an API that makes calls to AWS services and I am using Boto3 in order to do this within my python application. The question I have deals with Boto3's client vs resource access levels. I think I understand the difference between them (one is low-level access the other is higher-level object-oriented service access) but my question is if it is okay to instantiate both a client and resource? For example, some resource functionality is easier to access using a resource over a client, but there is some functionality only the client has. Is it bad to instantiate both and use the easiest access level when needed or will there be some sort of disconnect when using two separate access levels when connecting to the same resource?
I am not running into any errors with my code to connect to SQS shown below, however I want to make sure that down the line I am not shooting myself in the foot by arbitrarily choosing between the client/resource for the same aws connection.
import boto3
REGION = 'us-east-1'
sqs_r = boto3.resource('sqs', REGION)
sqs_c = boto3.client('sqs', REGION)
def create_queue(queue_name):
queue_attributes = {
'FifoQueue': 'true',
'DelaySeconds': '0',
'MessageRetentionPeriod': '900', # 15 minutes to complete a command, else deleted.
'ContentBasedDeduplication': 'true'
}
try:
queue = sqs_r.get_queue_by_name(QueueName=queue_name)
except:
queue = sqs_r.create_queue(QueueName=queue_name, Attributes=queue_attributes)
def list_all_queues(queue_name_prefix=''):
all_queues = sqs_c.list_queues(QueueNamePrefix=queue_name_prefix)
print(all_queues['QueueUrls'])
print(type(all_queues))
Both of the above function work properly, one creates a queue and the other lists all of the queues at sqs. However, one function uses a resource and the other uses a client. Is this okay?
You can certainly use both.
The resource method actually uses the client method behind-the-scenes, so AWS only sees client-like calls.
In fact, the resource even contains a client. You can access it like this:
import boto3
s3 = boto3.resource('s3')
copy_source = {
'Bucket': 'mybucket',
'Key': 'mykey'
}
s3.meta.client.copy(copy_source, 'otherbucket', 'otherkey')
This example is from the boto3 documentation. It shows how a client is being extracted from a resource, and makes a client call, effectively identical to s3_client.copy().
Both client and resource just create a local object. There is no back-end activity involved.
Related
I have a normal monolith app running on Heroku. I would like to use an SQS queue to process some background jobs.
Monolith -> posts to SQS -> SQS holds event & calls -> Monolith -> Confirms processing and deletes event from SQS
I already have an S3 bucket I have configured via the AWS CDK, now I'm trying to figure out to configure an SQS queue which I can also call from the outside (via access_token/key authentication):
// The user I used for S3 uploads
const uploadUser = new iam.User(this, "upload-user", {
userName: "upload-user",
})
// ...Here is some other code to initialize S3 bucket
const bucket = ...
bucket.grantWrite(uploadUser)
// But when creating a queue this does not work
const queue = new sqs.Queue(this, 'linear-sync.fifo')
queue.grant(uploadUser) // errors
Trying to do this gives me an error:
Argument of type 'User' is not assignable to parameter of type 'IGrantable'
Any idea on how to achieve this?
Edit: I ended up abandoning the idea of using a SQS queue, I might come back to it later, but decided to go for redis, it's easier to set up with our current docker-based stack but I might revisit the idea later
First of all, the Queue.grant method requires you to list which actions you want to grant access to. If you want to post messages to the queue, you would call Queue.grantSend.
Now, the reason User may not be assignable to IGrantable is because you have mismatched versions of CDK installed. Are you using V1? Are all the CDK modules the same version? Make sure you're not using V1 and V2 together.
I'm currently writing a Telegram bot using python-telegram-bot as a wrapper. I want to try and host this on AWS Lambda. However, so far the examples I've seen are simple, dumb bots that are unable to continue a conversation. I'm leveraging ConversationHandler to run the bot's conversations but this doesn't work well on AWS Lambda. I'm not sure how to fix this issue.
bot = MyBot()
def lambda_handler(event=None, context=None):
try:
dispatcher = bot.updater.dispatcher
message = json.loads(event['body'])
print("Incoming:", message)
dispatcher.process_update(Update.de_json(message, bot.updater.bot))
except Exception as e:
print(e)
return {"statusCode": 500}
bot.updater.idle()
return {"statusCode": 200}
How can I get the bot to hold a conversation state throughout?
ConversationHandler stores the state internally, i.e. in memory. I don't know how AWS handles initialization of variables, but if the ConversationHandler is initialized anew on each incoming update, it won't remember which state each conversation was in. If you can use some sort of database/file storage on AWS, you can try to use PTBs persistence setup to store the converstanion states and reload them for each incoming update.
Disclaimer: I'm currently the maintainer of python-telegram-bot
I have a Lambda that requires messages to be sent to another Lambda to perform some action. In my particular case it is passing a message to a Lambda in order for it to perform HTTP requests and refresh cache entries.
Currently I am relying on the AWS SDK to send an SQS message. The mechanics of this are working fine. The concern that I have is that the SQS send method call takes around 50ms on average to complete. Considering I'm in a Lambda, I am unable to perform this in the background and expect for it to complete before the Lambda returns and is frozen.
This is further compounded if I need to make multiple SQS send calls, which is particularly bad as the Lambda is responsible for responding to low-latency HTTP requests.
Are there any alternatives in AWS for communicating between Lambdas that does not require a synchronous API call, and that exhibits more of a fire and forget and asynchronous behavior?
Though there are several approaches to trigger one lambda from another, (in my experience) one of the fastest methods would be to directly trigger the ultimate lambda's ARN.
Did you try invoking one Lambda from the other using AWS SDKs?
(for e.g. in Python using Boto3, I achieved it like this).
See below, the parameter InvocationType = 'Event' helps in invoking target Lambda asynchronously.
Below code takes 2 parameters (name, which can be either your target Lambda function's name or its ARN, params is a JSON object with input parameters you would want to pass as input). Try it out!
import boto3, json
def invoke_lambda(name, params):
lambda_client = boto3.client('lambda')
params_bytes = json.dumps(params).encode()
try:
response = lambda_client.invoke(FunctionName = name,
InvocationType = 'Event',
LogType = 'Tail',
Payload = params_bytes)
except ClientError as e:
print(e)
return None
return response
Hope it helps!
For more, refer to Lambda's Invoke Event on Boto3 docs.
Alternatively, you can use Lambda's Async Invoke as well.
It's difficult to give exact answers without knowing what language are you writing the Lambda function in. To at least make "warm" function invocations faster I would make sure you are creating the SQS client outside of the Lambda event handler so it can reuse the connection. The AWS SDK should use an HTTP connection pool so it doesn't have to re-establish a connection and go through the SSL handshake and all that every time you make an SQS request, as long as you reuse the SQS client.
If that's still not fast enough, I would have the Lambda function handling the HTTP request pass off the "background" work to another Lambda function, via an asynchronous call. Then the first Lambda function can return an HTTP response, while the second Lambda function continues to do work.
You might also try to use Lambda Destinations depending on you use case. With this you don't need to put things in a queue manually.
https://aws.amazon.com/blogs/compute/introducing-aws-lambda-destinations/
But it limits your flexibility. From my point of view chaining lambdas directly is an antipattern and if you would need that, go for step functions
My Lambda function invokes CloudWatch describe-alarms API.
Then those alarms are removed.
I'm using CloudWatch Cron Event as a trigger.
The impression I have is that responses with alarms are cached, that is even if they are deleted are still being appeared.
Is there any caching system within AWS Lambda?
It's your code that's caching the response. Not Lambda.
To fix it, you have to fix your code by making sure that you invoke the API inside your handler and return it without storing it outside your handler function's scope.
For illustration purposes,
Don't
const response = callAnApi()
async function handler(event, context, callback) {
// No matter how many times you call the handler,
// response will be the same
return callback(null, response)
}
Do
async function handler(event, context, callback) {
// API is called each time you call the handler.
const response = await callAnApi()
return callback(null, response)
}
Reference: AWS Lambda Execution Model
Any declarations in your Lambda function code (outside the handler code, see Programming Model) remains initialized, providing additional optimization when the function is invoked again. For example, if your Lambda function establishes a database connection, instead of reestablishing the connection, the original connection is used in subsequent invocations. We suggest adding logic in your code to check if a connection exists before creating one.
There is no caching mechanism in AWS Lambda to my knowledge,
That said, after a (successful) request the container Lambda created is "frozen" to preventing it from doing "async" or "background" work. A subsequent request will reuse the container and pass the new event to your function handler. This container will remain in the cluster, ready to be reused and serve requests so long that it isn’t idle for too long, after which it may be discarded entirely. These details are unspecified by AWS.
Because the container sits around waiting for subsequent requests and the memory allocated to it does not magically disappear each time, we can store data for future requests. (But would not recommend it)
Complementing: If you are reaching your AWS Lambda through the API Gateway, then you can activate cache on the API Gateway level, which is great for speed and reducing costs with Lambda. That caching system allows you to use the params, request headers, etc, as keys for the calls, making it simple and efficient.
Thanks to Noels answer, I was facing similar issue where API GW was using lambda(python runtime) functions(with api gw cache disabled).The problem was that i defined the db connection outside the lambda handler. The result of the code below was old data(even after database table updates) from api.
db = pymysql.connect(host=DB_HOST,user=DB_USER, password=DB_PSWD,database=DB_NAME,cursorclass=pymysql.cursors.DictCursor)
cursor = db.cursor()
def lambda_handler(event, context):
try:
cursor.execute("SELECT id, product FROM repository")
return { "body":json.dumps(cursor.fetchall()), "statusCode":200}
except Exception as e:
return { "body":json.dumps(event), "statusCode":500}
To fix this i moved the db connection inside lambda handler:
def lambda_handler(event, context):
db = pymysql.connect(host=DB_HOST,user=DB_USER, password=DB_PSWD,database=DB_NAME,cursorclass=pymysql.cursors.DictCursor)
cursor = db.cursor()
try:
cursor.execute("SELECT id, product FROM repository")
return { "body":json.dumps(cursor.fetchall()), "statusCode":200}
except Exception as e:
return { "body":json.dumps(event), "statusCode":500}
With Boto3:
I am using put_object() function to upload object in s3. I am using put_object() with customer encryption key parameter for server side encryption.
With Boto:
I am using upload_chunk function to upload object in s3. Here I am using aws managed keys for server side encryption and not customer given as it is not supported in API
So With Boto3 approach my program is taking more memory than Boto approach.
Please tell me how does put_object function works in boto3 for server side encryption .
is it using memory of machine for encryption on which it has called ?
should I explicitly clean data buffer which is passed as Body parameter to put_object function ?
Code:
def put_s3_object(self, target_key_name, data, sse_cust_key, sse_cust_key_md5):
''' description: Upload file as s3 object using SSE with customer key
It will store s3 object in encrypted format
input:
target_key_name (#string) data(in memory string/bytes)
sse_cust_key (#string)
sse_cust_key_md5 (#string)
output: response
'''
if not target_key_name:
raise
try:
response = self.s3_client.put_object(Bucket = self.source_bucket, Body = data, Key = target_key_name, SSECustomerAlgorithm = awsParams.CLOUD_DR_AWS_SSE_ALGO, SSECustomerKey = sse_cust_key, SSECustomerKeyMD5 = sse_cust_key_md5)
del data
except botocore.exceptions.ClientError, fault:
raise
except Exception, fault:
raise
Instead of Boto3 put_object(), We can go with set_contents_to_string() and get_contents_as_string() functions from boto.
These calls also supports server side encryption with customer keys(SSE-C). we just need to give all keys information in headers
For more details
http://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html