According to the docs,
the X-Ray SDK applies a sampling algorithm to determine which requests get traced. By default, the X-Ray SDK records the first request each second, and five percent of any additional requests.
Is it possible to trace all requests?
It is possible to set the sampling rate to 100%.
However, as noted in the FAQs:
X-Ray should not be used as an audit or compliance tool because it does not guarantee data completeness.
Related
In the AWS Xray documentation it is mentioned that the sdk applies sampling to the requests.
Now I want to implement distributed logging such that any request that comes in to the system can be tracked by using the X-Amzn-Trace-Id or some correlation ID, so that I can later execute a query to fetch all the logs for a given request (across all the microservices).
What is the best possible way to achieve it?
Also, if there is a SNS Topic where I am publishing my events and then a Queue is listening to the Topic, then how can I include that relation in the Xray Map?
This is a common requirement in monitoring system, correlate traces, metrics and logs by keywords. In traces <-> logs case, it is by attaching trace id into logs. Please search topics like OpenTelemetry Logging Instrumentation
Regarding sampling, please check this table. Basically, not sampled still generates trace context for correlating logs.
I have a lambda function written in c# which is also accessible through API Gateway. I have enabled X-Ray tracing for both of them i.e. Lambda and API gateway. To view better traces I have created a sampling rule in AWS Console, As mentioned below image but I am not able to get any data in the graph for that sampling rule.
I have also tried to add sampling rules from code, as in the image below
The JSON file is something like this
I will really appreciate it if you can guide me to get trace data in the sampling rule graph.
A question that I have in mind, If I create a new sampling rule, do I have to make changes to the code as well? If yes, what will be the required changes for lambda (C#) code.
AWS Lambda will adopt the default sampling rule, which is 1 request per second and 5 percent of additional requests and the sampling rule in Lambda can't be configured at this moment. https://docs.aws.amazon.com/lambda/latest/dg/services-xray.html
I deployed a lambda with xray is enabled. And I am able to see all trace in XRay console from my lambda. But I can see a warning message in below screenshot. It shows Active tracing requires permissions that are not configured to lambda. But I don't understand what Active tracing mean. I have read article like this https://docs.aws.amazon.com/xray/latest/devguide/xray-services-lambda.html but it doesn't explain very well.
So what does Active tracing mean and does it cost too much?
I also had this warning under "Active tracing." If you click into Edit it gives a bit more explanation, saying it needs permission to send trace data.
You can find the documentation here, but the short version is that you'll want to add the AWSXRayDaemonWriteAccess policy to your lambda function's execution role.
The different levels of x-ray integration with AWS services is explained here:
Active instrumentation – Samples and instruments incoming requests.
Passive instrumentation – Instruments requests that have been sampled by another service.
Request tracing – Adds a tracing header to all incoming requests and propagates it downstream.
Tooling – Runs the X-Ray daemon to receive segments from the X-Ray SDK.
AWS Lambda supports both active and passive instrumentation. So basically you use passive instrumentation if your function handles requests that have been sampled by some other service (e.g. API gateway). In contrast, if your function gets "raw" un-sampled requests, you should use active instrumentation, so that the sampling takes place.
In most of the official documents to express throttling limits, AWS uses metrics like Requests per second or Requests per client. e.g. here. But for AWS IOT API throttling limit, there are using a metric called Transactions per seconds. Is there an actual difference between "Transactions per Second" and "Requests per second" metrics or they are just the same?
They mean the same thing — the rate in which you're allowed to call the API. It seems there's no standard for this term; it's chosen at the discretion of the writers. Some services only state a plain number, i.e. 1000, others use requests, and a few use transactions.
My cloudwatch bill has a $400 spike from August to September.
$0.01 per 1,000 requests(blended rate: 0.000010)161,204,177 Requests
$0.01 per 1,000 requests(blended rate: 0.000010)206,899,565 Requests
how can we check why requests have increased so much and determine their end points
or where the requests are coming from
This is likely a monitoring tool, DataDog is one example, but any tool that queries AWS continually to get stats on all of your servers. Are you importing your data to any third party tools, or internally built tools?
CloudTrail does NOT log this request, per the answer below from AWS to a similar question - This would be my answer too in general, this is a special case :-(
"Unfortunately, there is no tool from your side to check the source of GetMetricData. The reason being, this API call is not logged in CloudTrail." -- akash-aws from forums
Additional Info: https://forums.aws.amazon.com/thread.jspa?messageID=865061
Good luck finding the cause, I'm looking into a similar problem for a client and tracked down when it started, now to hopefully find more details.
See if you're getting billed for GetMetricData requests. If yes, see if you're running any 3rd party monitoring solutions that get metrics from your AWS account. Finally, to confirm it's caused by the 3rd party monitoring tool, turn off the tool if you're allowed to, at least for an hour or so, to see if there's indeed a dent in your bill at that hour.
I wrote detailed steps here: https://ystoneman.medium.com/a-hidden-cost-to-monitoring-aws-with-3rd-party-tools-e23dd9e38c75
You can Log Amazon CloudWatch API Calls with AWS CloudTrail - Amazon CloudWatch:
Amazon CloudWatch is integrated with AWS CloudTrail, a service that provides a record of actions taken by a user, role, or an AWS service in CloudWatch. CloudTrail captures API calls made by or on behalf of your AWS account. The calls captured include calls from the CloudWatch console and code calls to the CloudWatch API operations.
If you create a trail, you can enable continuous delivery of CloudTrail events to an Amazon S3 bucket, including events for CloudWatch. If you don't configure a trail, you can still view the most recent events in the CloudTrail console in Event history.
Using the information collected by CloudTrail, you can determine the request that was made to CloudWatch, the IP address from which the request was made, who made the request, when it was made, and additional details.