AWS X-Ray ERROR:aws_xray_sdk.core.context:cannot find the current segment/subsegment - amazon-web-services

Question

We recently added X-Ray to our code by having:
from aws_xray_sdk.core import xray_recorder
from aws_xray_sdk.core import patch_all
patch_all()
While this runs fine on AWS Lambda, but when trying to run locally during calling ElasticSearch we got the following exception:
ERROR:aws_xray_sdk.core.context:cannot find the current segment/subsegment, please make sure you have a segment open
queryCustomers - DEBUG - Caught exception for <function search_customer at 0x10bfcf0d0>
Traceback (most recent call last):
File "/Users/jameslin/virtualenvs/test-project/lib/python3.6/site-packages/chalice/app.py", line 659, in _get_view_function_response
response = view_function(**function_args)
File "/Users/jameslin/projects/test-project/src/app.py", line 57, in search_customer
return query[0:size].execute().to_dict()['hits']['hits']
File "/Users/jameslin/virtualenvs/test-project/lib/python3.6/site-packages/elasticsearch_dsl/search.py", line 639, in execute
**self._params
File "/Users/jameslin/virtualenvs/test-project/lib/python3.6/site-packages/elasticsearch/client/utils.py", line 73, in _wrapped
return func(*args, params=params, **kwargs)
File "/Users/jameslin/virtualenvs/test-project/lib/python3.6/site-packages/elasticsearch/client/__init__.py", line 632, in search
doc_type, '_search'), params=params, body=body)
File "/Users/jameslin/virtualenvs/test-project/lib/python3.6/site-packages/elasticsearch/transport.py", line 312, in perform_request
status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
File "/Users/jameslin/virtualenvs/test-project/lib/python3.6/site-packages/elasticsearch/connection/http_requests.py", line 71, in perform_request
prepared_request = self.session.prepare_request(request)
File "/Users/jameslin/virtualenvs/test-project/lib/python3.6/site-packages/aws_xray_sdk/ext/requests/patch.py", line 38, in _inject_header
inject_trace_header(headers, xray_recorder.current_subsegment())
File "/Users/jameslin/virtualenvs/test-project/lib/python3.6/site-packages/aws_xray_sdk/core/recorder.py", line 251, in current_subsegment
entity = self.get_trace_entity()
File "/Users/jameslin/virtualenvs/test-project/lib/python3.6/site-packages/aws_xray_sdk/core/recorder.py", line 316, in get_trace_entity
return self.context.get_trace_entity()
File "/Users/jameslin/virtualenvs/test-project/lib/python3.6/site-packages/aws_xray_sdk/core/context.py", line 93, in get_trace_entity
return self.handle_context_missing()
File "/Users/jameslin/virtualenvs/test-project/lib/python3.6/site-packages/aws_xray_sdk/core/context.py", line 118, in handle_context_missing
raise SegmentNotFoundException(MISSING_SEGMENT_MSG)
aws_xray_sdk.core.exceptions.exceptions.SegmentNotFoundException: cannot find the current segment/subsegment, please make sure you have a segment open
I have no idea what his means and how to get rid of it, my google attempts gives not many relevant results and I also tried running the x-ray daemon locally but still having the same problem:
./xray_mac -o -n ap-southeast-2

Answer 1

When your code is running on AWS Lambda with tracing enabled, Lambda container will generate a segment representing the whole function invocation. It also sets the context as the environment variable so the SDK can link any subsegment created inside the function back to the parent segment.
If you run the same code locally the SDK still tries to create subsegments for the actual function code but it can't find any context, thus throwing the error you posted.
To solve this you will need to setup some environment variables to make sure the SDK has the same information as it were running on an actual Lambda container.
Make sure the SDK thinks it is running on a Lambda container by setting LAMBDA_TASK_ROOT with whatever value you'd like (only the presence of the key matters). You can see the source code here: https://github.com/aws/aws-xray-sdk-python/blob/master/aws_xray_sdk/core/lambda_launcher.py
Setting LAMBDA_TRACE_HEADER_KEY so the function has a tracing context. The value must be a trace header and you can see more details here: https://docs.aws.amazon.com/lambda/latest/dg/lambda-x-ray.html
This workaround is not ideal as it requires extra code changes from user side. We would like to provide better customer experience for testing X-Ray instrumented Lambda function locally. Do you mind sharing more details about how you are doing local testing and how you expect X-Ray tracing works in such testing environment, so we can have better improvement for your use case?