I deployed a lambda with xray is enabled. And I am able to see all trace in XRay console from my lambda. But I can see a warning message in below screenshot. It shows Active tracing requires permissions that are not configured to lambda. But I don't understand what Active tracing mean. I have read article like this https://docs.aws.amazon.com/xray/latest/devguide/xray-services-lambda.html but it doesn't explain very well.
So what does Active tracing mean and does it cost too much?
I also had this warning under "Active tracing." If you click into Edit it gives a bit more explanation, saying it needs permission to send trace data.
You can find the documentation here, but the short version is that you'll want to add the AWSXRayDaemonWriteAccess policy to your lambda function's execution role.
The different levels of x-ray integration with AWS services is explained here:
Active instrumentation – Samples and instruments incoming requests.
Passive instrumentation – Instruments requests that have been sampled by another service.
Request tracing – Adds a tracing header to all incoming requests and propagates it downstream.
Tooling – Runs the X-Ray daemon to receive segments from the X-Ray SDK.
AWS Lambda supports both active and passive instrumentation. So basically you use passive instrumentation if your function handles requests that have been sampled by some other service (e.g. API gateway). In contrast, if your function gets "raw" un-sampled requests, you should use active instrumentation, so that the sampling takes place.
Related
In the AWS Xray documentation it is mentioned that the sdk applies sampling to the requests.
Now I want to implement distributed logging such that any request that comes in to the system can be tracked by using the X-Amzn-Trace-Id or some correlation ID, so that I can later execute a query to fetch all the logs for a given request (across all the microservices).
What is the best possible way to achieve it?
Also, if there is a SNS Topic where I am publishing my events and then a Queue is listening to the Topic, then how can I include that relation in the Xray Map?
This is a common requirement in monitoring system, correlate traces, metrics and logs by keywords. In traces <-> logs case, it is by attaching trace id into logs. Please search topics like OpenTelemetry Logging Instrumentation
Regarding sampling, please check this table. Basically, not sampled still generates trace context for correlating logs.
I have a lambda function written in c# which is also accessible through API Gateway. I have enabled X-Ray tracing for both of them i.e. Lambda and API gateway. To view better traces I have created a sampling rule in AWS Console, As mentioned below image but I am not able to get any data in the graph for that sampling rule.
I have also tried to add sampling rules from code, as in the image below
The JSON file is something like this
I will really appreciate it if you can guide me to get trace data in the sampling rule graph.
A question that I have in mind, If I create a new sampling rule, do I have to make changes to the code as well? If yes, what will be the required changes for lambda (C#) code.
AWS Lambda will adopt the default sampling rule, which is 1 request per second and 5 percent of additional requests and the sampling rule in Lambda can't be configured at this moment. https://docs.aws.amazon.com/lambda/latest/dg/services-xray.html
I am using AWS IoT and using the JITP for device/thing registration.
https://aws.amazon.com/blogs/iot/setting-up-just-in-time-provisioning-with-aws-iot-core/
When I carry out the load testing, some of the things are not registered. But I don't see any error either.
I feel this might be happening because of the AWS IoT Throttling Limits.
I suspect one of the below APIs calls, throttling is happening.
CreateThing
AttachPolicy
AttachThingPrincipal
How can I set an alarm to check if for any of this API calls, throttling is happening?
I don't believe there's a way to do this currently.
Some alternatives:
Don't use JITP and call the APIs yourself.
The response of the APIs (including throttling) will be returned directly to you. Then you can log when this happens.
Connect your devices within the API request limits .
Provision your Things (CreateThing, etc.) before starting the load test.
I have an AWS Lambda that was triggered by SNS message. Many time, it has reached the max duration allowed by AWS, and AWS killed it immediately.
I have to either dig into the Lambda logs or the lambda duration chart to find out about the error.
Are there a better way to report this kind of errors?
Yes, there are some 3rd party tools that help you monitor your environment and provide exactly that - filter on specific errors and drill down to what happened there (the input event, the outgoing HTTP requests etc.).
Moreover, you can also configure alerts on specific errors that you will get via slack/mail.
Disclosure: I work for Lumigo, a company that does exactly that.
I've gone over the documentation and cannot find a clear statement regarding how much latency is X-ray tracing supposed to add to Lambda function executions (and to other services as well). It should be minimal, but since it's sending out traces, some latency is expected.
Does anyone have the numbers?
AWS X-Ray SDKs which you use in your application do not send trace segments to X-Ray service directly. The segments are transmitted over UDP to xray daemon running on localhost. So the latency involved is only for in memory updates to the segment data. Only when the segments are complete, they are sent over UDP to localhost. Hence, you should expect minimal possible overhead on your application. Also the daemon which runs in a separate process does not send segments to the service immediately. It buffers the segments for a short period and periodically sends them in batches using the PutTraceSegments API call.
If you are interested to dig further, most AWS X-Ray SDKs are open sourced on GitHub. Java SDK for example https://github.com/aws/aws-xray-sdk-java