I would like to chain two lambda function calls in the xray service map to visualize the connections between the function calls and get the overall execution time.
My lambda functions are being called by step functions.
When enabling active tracing I see the individual functions in the service map but there is no connections between them.
_X_AMZN_TRACE_ID: Contains the tracing header, which includes the sampling decision, trace ID, and parent segment ID. (To learn more
about these properties, see Tracing Header.) If Lambda receives a
tracing header when your function is invoked, that header will be used
to populate the _X_AMZN_TRACE_ID environment variable. If a tracing
header was not received, Lambda will generate one for you.
https://docs.aws.amazon.com/lambda/latest/dg/lambda-x-ray.html
_X_AMZN_TRACE_ID seems to contain the relevant information. My question is, how can I use it to establish a connection between two lambda function calls (being called by step functions)?
Step Functions X-Ray trace context propagation isn’t yet supported so there isn’t a way to chain your lambda functions together if they are being called by Step Functions. We are working on this feature but don’t currently have an ETA.
X-Ray integration is added to AWS Step Function:
https://aws.amazon.com/blogs/compute/introducing-aws-x-ray-new-integration-with-aws-step-functions/
You can enable X-Ray tracing when creating a new state machine by selecting Enable X-Ray tracing on the Specify details page.
In the case that state machine gets call from upper service with X-Ray enabled, state machine automatically continue the trace.
And for all Task states in state machine, subsegments will be added tp the trace.
Related
I am trying to x-ray trace a process through a multitude of services.
I have a lambda that is already traced by x-ray, that submits to kinesis, and eventually is processed by a Scala ECS consumer. How do I link the lambda to the consumer?
Is this handled automagically, or do I need to grab and insert the x-amzn-trace-id value?
Thanks
Yes, the idea is right.
get the current Lambda trace context from Lambda function, add it to Kinesis data.
extract that trace context from ECS cunsumer side, and use it as the parent of consumer segment.
One possible technical trouble is, if consumer processes a batch of trace contexts in one operation, you might not be able to set parent for consumer segment because a segment can only have one parent. X-Ray needs to support links feature to support this fan-in scenario.
FYI links: https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/overview.md#links-between-spans
Aws lambdas have 3 steps in their lifecycle:
init
invoke
shutdown
Is there a way to trigger automatically a lambda init without deploying any code?
Update:
I have some actions that are launched during the init: saving the content of a file in a variable. Then during the invoke actions, the content is not downloaded anymore. But I need to launch the action of downloading from time to time. Then, I was wondering if there's a way to trigger the init action.
Why not just call your init() function again?
Take the following Go Lambda for example:
// handleRequest does process every request to the Lambda.
func handleRequest(ctx context.Context) error {
init() // <-- call it whenever you want in your handler again
}
// init does store some data from a file in a variable.
func init() {}
func main() {
init() // <-- called on the first start of the Lambda
lambda.Start(handleRequest)
}
The init function is called in the main() method, which is executed when the Lambda is started for first time (cold start or init). But if you move your "business logic" for the initialisation into a separate function, you can call it whenever you want in your handler function.
Note: I don't think at all that this is a great idea. It is definitely a design "smell" if you need/want to do something like this. So you should rather rethink what you are doing than building workarounds like this.
There is no way you can manually trigger those events as I know however, you can perform some action for those events.
check AWS Extention API if you are interested.
https://docs.aws.amazon.com/lambda/latest/dg/runtimes-context.html
Perhaps you should take a look at lambda layers. This allows you to place some files in advance, so that your invocations can access it at runtime (without having to download it). You can then easily update your layer whenever you need.
See documentation here
I figured out a way to do this in NodeJs Lambdas. I’m not aware of any previous solutions for JS Lambdas. Normally, the lambda container is paused before async operations complete in the init phase.
The key is to leverage an AWS Lambda custom Extension. Then if you enable provisioned concurrency, the Lambdas execute the init phase before a request. You have to determine how many provisioned concurrency instances to keep online because concurrent requests can trigger on-demand instances. Or you can enable auto-scaling with provisioned concurrency to try to scale out ahead of the load.
This article explains the details and includes a link to the GitHub sample repo.
https://barker.tech/aws-lambda-nodejs-async-init-88b5c24af567?source=friends_link&sk=79dbb80045ad97c62f58fa4b74ade68b
https://github.com/chuckbarkertech/nodejs-example-async-init-extension
You still have to deploy code. But you can trigger the init before requests hit the lambda by combining provisioned concurrency with AWS Lambda Extensions like the example info above.
I have a handful of Python Lambda functions with tracing enabled, they start like this:
import json
import boto3
from aws_xray_sdk.core import patch
from aws_xray_sdk.core import xray_recorder
patch(['boto3'])
def lambda_handler(event, context):
...
With this tracing works for every Lambda function itself and I can see the subservice calls to DynamoDB or Kinesis done through boto3.
But how can I connect various Lambda functions together in one trace? I'm thinking of generating a unique string in the first function and write it into the message stored in Kinesis. Another function would then pick up the string from the Kinesis' message and trace it again.
How would this be possible in a way to then see the whole connected trace in X-Ray?
If your upstream service which invokes your Lambda functions has tracing enabled, your functions will automatically send traces. From your question, I'm not sure how your functions are invoked. If one function is directly invoking another function, you'll have a single trace for them.
For your approach of invoking lambdas with Kinesis messages, I'm not sure it would achieve what you want due to several reasons.Firstly, Kinesis is not integrated with X-Ray, which means it will not propagate the trace header to downstream lambda. Secondly, the segment and the trace header for a lambda function is not directly accessible from your function's code since it is generated by the lambda runtime upon invocation and is thus immutable. Explicitly overriding the trace id in a lambda function may result in undesired behavior of your service graph.
Thanks.
I have defined a lambda function that is invoked from API Gateway with proxy integration. Thus, I have defined an eager resource path for it:
And referenced my lambda function:
My lambda is able to process request like GET /myresource, POST /myresource.
I have tried this strategy to keep it warm, described in acloudguru. It consists of setting up a CloudWatch event rule that invokes the lambda every 5 minutes to keep it warm. Unfortunately it isn't working.
This is the behaviour I have seen:
After some period, let's say 20 minutes, I call GET /myresource from API Gateway and it takes around 15 seconds. Subsequent requests last ~30ms. The CloudWatch event is making no difference...
Let's suppose another long period without calling the gateway. If I go to the Lambda console and invoke it directly (test button) it answers right away (less than 1ms) with a 404 (that's normal because my lambda expects GET /myresource or POST /myresource).
Immediately after this lambda console execution I call GET /myresource from API Gateway and it still takes ~20 seconds. That is to say, the function was still cold despite having being invoked from the Lambda console. This might explain why the CloudWatch event doesn't work since it calls the lambda without setting the method/resource-url.
So, how can I make this particular case with API Gateway with proxy integration + Lambda stay warm to prevent those slow first request?
As of now (2019-02-27) [1], A periodic CloudWatch event rule does not deterministically solve the cold start issue. But a periodic CloudWatch event rule will reduce the probability of cold starts.
The reason is it's upto the Lambda server to decide whether to use a new Lambda container instead of an existing container to process an incoming request. Some of the related details regarding how Lambda containers are reused is explained in [1]
In order to reduce the cold start time (not to reduce the number cold starts), can you try followings? 1. increasing the memory allocated to the function, 2. reduce the deployment package size (eg- remove unnecessary dependencies), and 3. use a language like NodeJS, Python instead of Java, .Net
[1]According to reinvent session, (39:50 at https://www.youtube.com/watch?v=QdzV04T_kec), the Lambda team expects to improve the VPC cold start latency in Lambda.
[2] https://aws.amazon.com/blogs/compute/container-reuse-in-lambda/
Denis is quite right about the non deterministic lambda behaviour regarding the number of containers hit by CloudWatch events. I'll follow his advice to improve the startup time.
On the other hand I have managed to make my CloudWatch events hit the lambda function properly, reducing (in many cases) the number of cold starts.
I just had to add an additional controller mapped to "/" with a hardcoded response:
#Controller("/")
class WarmUpController {
private val logger = LoggerFactory.getLogger(javaClass)
#Get
fun warmUp(): String {
logger.info("Warming up")
return """{"message" : "warming up"}"""
}
}
With this in place the default (/) invocation from CloudWatch does keep the container warm most of the time.
I am trying to ultimately change the a trace-id so tell me if I'm taking a wrong approach.
What I'm currently thinking of is to create a segment, and add the parent-id as the trace_id that I want so one trace can follow another trace. The reason I'm trying to do that is because asynchronous parts of aws such as kinesis streams are not supported in aws x-ray.
const segment = new awsXRay.Segment('1-11111111-111111111111111111111111', '1-11111111-111111111111111111111111', '1-11111111-111111111111111111111111')
console.log(segment)
awsXRay.setSegment(segment)
However, I get the error:
TypeError: segment.resolveLambdaTraceData is not a function
Any ideas why I'm getting the error, or how I can connect two events that happened before and after a kinesis stream by connecting their trace_ids?
When the X-Ray SDK is used in Lambda, Lambda creates it's own segment (and sends it out of band of the Lambda function code) and the SDK creates a "facade" on the context as a placeholder given the parent ID and ID of the segment Lambda created (exposed via process.env._x_amzn_trace_id). The SDK only creates subsegments attached to this facade segment, and when the Lambda function concludes, these subsegments are sent and the service pieces it together. Each time the Lambda function fires, and it wasn't a cold start, the facade segment refreshes itself with the new parent ID and ID as supplied by Lambda (with the call to resolveLambdaTraceData). The SDK does not expect a user created segment in Lambda set on the context. With an manually created segment, the resolveLambdaTraceData function is not available, hence the error, and when set on the context, this fires automatically when it detects a separate Lambda execution.
Can you go into more detail for your use-case?