I have a handful of Python Lambda functions with tracing enabled, they start like this:
import json
import boto3
from aws_xray_sdk.core import patch
from aws_xray_sdk.core import xray_recorder
patch(['boto3'])
def lambda_handler(event, context):
...
With this tracing works for every Lambda function itself and I can see the subservice calls to DynamoDB or Kinesis done through boto3.
But how can I connect various Lambda functions together in one trace? I'm thinking of generating a unique string in the first function and write it into the message stored in Kinesis. Another function would then pick up the string from the Kinesis' message and trace it again.
How would this be possible in a way to then see the whole connected trace in X-Ray?
If your upstream service which invokes your Lambda functions has tracing enabled, your functions will automatically send traces. From your question, I'm not sure how your functions are invoked. If one function is directly invoking another function, you'll have a single trace for them.
For your approach of invoking lambdas with Kinesis messages, I'm not sure it would achieve what you want due to several reasons.Firstly, Kinesis is not integrated with X-Ray, which means it will not propagate the trace header to downstream lambda. Secondly, the segment and the trace header for a lambda function is not directly accessible from your function's code since it is generated by the lambda runtime upon invocation and is thus immutable. Explicitly overriding the trace id in a lambda function may result in undesired behavior of your service graph.
Thanks.
Related
I have a Lambda that requires messages to be sent to another Lambda to perform some action. In my particular case it is passing a message to a Lambda in order for it to perform HTTP requests and refresh cache entries.
Currently I am relying on the AWS SDK to send an SQS message. The mechanics of this are working fine. The concern that I have is that the SQS send method call takes around 50ms on average to complete. Considering I'm in a Lambda, I am unable to perform this in the background and expect for it to complete before the Lambda returns and is frozen.
This is further compounded if I need to make multiple SQS send calls, which is particularly bad as the Lambda is responsible for responding to low-latency HTTP requests.
Are there any alternatives in AWS for communicating between Lambdas that does not require a synchronous API call, and that exhibits more of a fire and forget and asynchronous behavior?
Though there are several approaches to trigger one lambda from another, (in my experience) one of the fastest methods would be to directly trigger the ultimate lambda's ARN.
Did you try invoking one Lambda from the other using AWS SDKs?
(for e.g. in Python using Boto3, I achieved it like this).
See below, the parameter InvocationType = 'Event' helps in invoking target Lambda asynchronously.
Below code takes 2 parameters (name, which can be either your target Lambda function's name or its ARN, params is a JSON object with input parameters you would want to pass as input). Try it out!
import boto3, json
def invoke_lambda(name, params):
lambda_client = boto3.client('lambda')
params_bytes = json.dumps(params).encode()
try:
response = lambda_client.invoke(FunctionName = name,
InvocationType = 'Event',
LogType = 'Tail',
Payload = params_bytes)
except ClientError as e:
print(e)
return None
return response
Hope it helps!
For more, refer to Lambda's Invoke Event on Boto3 docs.
Alternatively, you can use Lambda's Async Invoke as well.
It's difficult to give exact answers without knowing what language are you writing the Lambda function in. To at least make "warm" function invocations faster I would make sure you are creating the SQS client outside of the Lambda event handler so it can reuse the connection. The AWS SDK should use an HTTP connection pool so it doesn't have to re-establish a connection and go through the SSL handshake and all that every time you make an SQS request, as long as you reuse the SQS client.
If that's still not fast enough, I would have the Lambda function handling the HTTP request pass off the "background" work to another Lambda function, via an asynchronous call. Then the first Lambda function can return an HTTP response, while the second Lambda function continues to do work.
You might also try to use Lambda Destinations depending on you use case. With this you don't need to put things in a queue manually.
https://aws.amazon.com/blogs/compute/introducing-aws-lambda-destinations/
But it limits your flexibility. From my point of view chaining lambdas directly is an antipattern and if you would need that, go for step functions
I currently have a bunch of AWS Lambda functions and would like to spin them each up in containers (preferably Docker) to integrate with the current CICD pipeline. It's totally OK to have these containers spun up in an Amazon service such as Fargate -- the important thing is that it must be spun up in a container. The closest thing I've found is this repo by Amazon that converts images into Lambdas, though this is the opposite of what I'm looking for.
Is this task possible, and if so what's the best way to go about it?
Behind the scenes, a Lambda "function" is a Python script that listens for invocation events and invokes the actual handler function. This means that you can pretty easily invoke the function using the standard Python "script" code:
if __name__ == "__main__":
args = # this is the tricky part
ctx = # some dict that looks like what Lambda expects
handler(args, ctx)
These two parameters are just dicts, so if you can populate them with something that looks right then you can wrap the entire thing in a container and deploy it to ECS or Batch, or your own Docker daemon.
The problem is how you populate them, and that depends on what your lambda functions do -- which you don't say -- and how they're invoked -- which you also don't say.
You could use command-line parameters. For example, if your lambda is intended to respond to an S3 PutObject operation, then all you really need is the bucket and key name. From that you can easily create a dummy S3 invocation event, using fake values for all of the other fields.
If you invoke your Lambda in response to an SQS message it's even easier: your "main" handler enters a loop that polls SQS, and reformats the messages that it pulls off the queue.
Other event sources are going to be much harder. It may make sense (depending on why you're doing this) to use SQS for all invocations, and simply pass a fully-formed event message into the queue.
I have written and saved a lambda function. I see:
Congratulations! Your Lambda function "lambda_name" has been
successfully created. You can now change its code and configuration.
Choose Test to input a test event when you want to test your function.
Now how do I run it? I cannot see a 'run' or 'invoke' button as I would expect
Note
The lambda doesn't accept any arguments (it's extremely simple - for the purposes of this question, please presume it's simply 2 * 2 so when I run it it should not require any inputs and should return 4).
Also note
I can see a tonne of different ways to run the lambda here. I just want the simplest way (preferably a button in the browser)
Sending a test message via the Lambda console will run your Lambda function. The test message that you configure will define what is in the event parameter of your lambda handler function.
Since you are not doing anything with that message, you can send any arbitrary test message and it should work for you. You can just use the default hello world message and give it an arbitrary name.
It should then show you the results: any logs or returned objects right in the AWS Lambda console.
Further reading here
AWS Lambda functions are typically triggered by an event, such as an object being uploaded to Amazon S3 or a message being send to an Amazon SNS topic.
This is because Lambda functions are great at doing a small task very often. Often, Lambda functions only run for a few seconds, or even less than a second! Thus, they are normally triggered in response to something else happening. It's a bit like when somebody rings your phone, which triggers you to answer the phone. You don't normally answer your phone when it isn't ringing.
However, it is also possible to directly invoke an AWS Lambda function using the Invoke() command in the AWS SDK. For convenience, you can also use the AWS Command-Line Interface (CLI) aws lambda invoke command. When directly invoking an AWS Lambda function, you can receive a return value. This is in contrast to situations where a Lambda function is triggered by an event, in which case there is nowhere to 'return' a value since it was not directly invoked.
I would like to chain two lambda function calls in the xray service map to visualize the connections between the function calls and get the overall execution time.
My lambda functions are being called by step functions.
When enabling active tracing I see the individual functions in the service map but there is no connections between them.
_X_AMZN_TRACE_ID: Contains the tracing header, which includes the sampling decision, trace ID, and parent segment ID. (To learn more
about these properties, see Tracing Header.) If Lambda receives a
tracing header when your function is invoked, that header will be used
to populate the _X_AMZN_TRACE_ID environment variable. If a tracing
header was not received, Lambda will generate one for you.
https://docs.aws.amazon.com/lambda/latest/dg/lambda-x-ray.html
_X_AMZN_TRACE_ID seems to contain the relevant information. My question is, how can I use it to establish a connection between two lambda function calls (being called by step functions)?
Step Functions X-Ray trace context propagation isn’t yet supported so there isn’t a way to chain your lambda functions together if they are being called by Step Functions. We are working on this feature but don’t currently have an ETA.
X-Ray integration is added to AWS Step Function:
https://aws.amazon.com/blogs/compute/introducing-aws-x-ray-new-integration-with-aws-step-functions/
You can enable X-Ray tracing when creating a new state machine by selecting Enable X-Ray tracing on the Specify details page.
In the case that state machine gets call from upper service with X-Ray enabled, state machine automatically continue the trace.
And for all Task states in state machine, subsegments will be added tp the trace.
I am trying to ultimately change the a trace-id so tell me if I'm taking a wrong approach.
What I'm currently thinking of is to create a segment, and add the parent-id as the trace_id that I want so one trace can follow another trace. The reason I'm trying to do that is because asynchronous parts of aws such as kinesis streams are not supported in aws x-ray.
const segment = new awsXRay.Segment('1-11111111-111111111111111111111111', '1-11111111-111111111111111111111111', '1-11111111-111111111111111111111111')
console.log(segment)
awsXRay.setSegment(segment)
However, I get the error:
TypeError: segment.resolveLambdaTraceData is not a function
Any ideas why I'm getting the error, or how I can connect two events that happened before and after a kinesis stream by connecting their trace_ids?
When the X-Ray SDK is used in Lambda, Lambda creates it's own segment (and sends it out of band of the Lambda function code) and the SDK creates a "facade" on the context as a placeholder given the parent ID and ID of the segment Lambda created (exposed via process.env._x_amzn_trace_id). The SDK only creates subsegments attached to this facade segment, and when the Lambda function concludes, these subsegments are sent and the service pieces it together. Each time the Lambda function fires, and it wasn't a cold start, the facade segment refreshes itself with the new parent ID and ID as supplied by Lambda (with the call to resolveLambdaTraceData). The SDK does not expect a user created segment in Lambda set on the context. With an manually created segment, the resolveLambdaTraceData function is not available, hence the error, and when set on the context, this fires automatically when it detects a separate Lambda execution.
Can you go into more detail for your use-case?