I have a Flask app running as an AWS Lambda Function deployed with Zappa and would like to activate X-Ray to get more information for the different functions.
Activating X-Ray with Zappa was easy enough - it only requires adding this line in the zappa-settings.json:
"xray_tracing": true
Further, I installed the AWS X-Ray Python SDK and added a few decorators to some functions, like this:
#xray_recorder.capture()
When I deploy this as a Lambda function, it all works well. The problem is using the system locally, both when running tests and when running the Flask in a local server instead of as a lambda function.
When I use any of the functions that are decorated either in a test or through the local server, the following exception is thrown:
aws_xray_sdk.core.exceptions.exceptions.SegmentNotFoundException: cannot find the current segment/subsegment, please make sure you have a segment open
Which of course makes sense, because AWS Lambda handles the creation of segments.
Are there any good ways to deactivate capturing locally? This would be useful e.g. for running unit tests locally on functions that I would like to watch in X-Ray.
One of the feature request of this SDK is to have a "disabled" global flag so everything becomes no-ops https://github.com/aws/aws-xray-sdk-python/issues/26.
However, it still depends on what you are testing against. It's good practice to test what actually will be run on Lambda. You can set some environment variables so the SDK thinks it is running on Lambda.
You can see the SDK is looking for two env vars https://github.com/aws/aws-xray-sdk-python/blob/master/aws_xray_sdk/core/lambda_launcher.py. One is LAMBDA_TASK_ROOT set to true so it knows to switch to lambda-mode. The other one is _X_AMZN_TRACE_ID which contains the tracing context normally passed by lambda container.
If you just want to test non-XRay code you can set AWS_XRAY_CONTEXT_MISSING to LOG_ERROR so the SDK doesn't complain on context missing and simply give up capturing wrapped functions. This will run much less code path than mimic lambda behaviors. Ideally it would be better for the lambda local testing tool to be X-Ray friendly. Are you using https://github.com/awslabs/aws-sam-cli? There is already an open issue for this feature https://github.com/awslabs/aws-sam-cli/issues/217
Related
I am looking for a language / framework or a method by which I can build API / web application code such that it can run on Serverless compute's like aws lambda and the same code runs on a dedicated compute system like lightsail or EC2.
First I thought of using Docker to do this but AWS Lambda entry point is a specific function signature which is very different than Spring Controllers. Is there a solution available currently?
So basically when I run it in lambda - it will have cold start issue, later when the app is ready or get popular I would like to move it to a EC2 instance for better performance and higher traffic load.
I want to start right in this situation so that later it can be easy to port and resolve the performance issue's
I'd say; no this is not possible easily.
When you are building an api that you'd want to run on lambda's you most likely will be using an API Gateway which takes care of your routing to different lambda functions (best practice). So the moment you would me working on an api like this migrating to EC2 would be a nightmare as you would need to rebuild the whole application a more of a monolith application which could run on EC2.
I would honestly commit to either run it on EC2/Containers or run it on Lambda, if cold start is your main issue with Lambda's you might wanna look into Lambda Snapstart for Java or use another language like Typescript/Python.
After some correct keywords in google I finally got what I was looking for, checkout this blog and code library shared by AWS which helps you convert the request and response of the request as per the framework required http request
Running APIs Written in Java on AWS Lambda: https://aws.amazon.com/blogs/opensource/java-apis-aws-lambda/
Repo Code: https://github.com/awslabs/aws-serverless-java-container
Thanks Ricardo for your response - will do check out Lambda Snapstart for sure and try it as well. I have not tested out this completely but it looks promising to some extent.
edit: Turns out the solution is in the docs. I had bog standard normal 'sam' installed but I needed what they call the 'public preview version' AKA 'sam-beta-cdk'. With this installed the API can be started locally with sam-betacdk start-api and works well. While I appreciate the answers which suggest that development should be done using purely TDD I feel there is also value in this more interactive, manual mode as it permits quicker exploration of the problem space.
I'm trying to build my first app with CDK + Typescript using API Gateway, Lambdas and DynamoDB. I have built a couple of Lambdas and deployed them and they work fine live on the web. However I don't want a minute long deploy cycle and various associated AWS costs as part of my workflow. What I want is to be able to test my API locally.
I have struggled to find docs on how to do this. Amazon seem to recommend using the SAM CLI here so that is what I've been trying.
The docs claim running sam local xyz runs cdk synth to make a "could assembly" in ./aws-sam/build but I see no evidence of this. Instead what I get is a complaint that sam could not find a 'template.yml'. So I manually run cdk synth > template.yml which creates one in the root folder. Then I run sam local start-api and it seems happy to start up.
Then I try and hit my test lambda using CURL: curl 'http://127.0.0.1:3000/test' I get {"message":"Internal server error"} and a huge ugly stack trace in the console that is running sam local start-api
The lambda is this...
exports.handler = async function() {
console.log("WooHoo! Test handler ran")
return {statusCode: 200, headers: {"Content-Type": "application/json"}, body: "Test handler ran!"}
}
Start of the huge ugly stack trace...
Mounting /home/user/code/image-cache/asset.beeaa749e012b5921018077f0a5e4fc3ab271ef1c191bd12a82aa9a92148782e as /var/task:ro,delegated inside runtime container
START RequestId: 99f53642-b294-4ce5-a1b4-8c967db80ce1 Version: $LATEST
2021-09-15T12:33:37.086Z undefined ERROR Uncaught Exception {"errorType":"Runtime.ImportModuleError","errorMessage":"Error: Cannot find module 'test'\nRequire stack:\n- /var/runtime/UserFunction.js\n- /var/runtime/index.js","stack":["Runtime.ImportModuleError: Error: Cannot find module 'test'","Require stack:","- /var/runtime/UserFunction.js","- /var/runtime/index.js"," at _loadUserApp (/var/runtime/UserFunction.js:100:13)"," at Object.module.exports.load (/var/runtime/UserFunction.js:140:17)",
The end of the huge ugly stack trace...
Invalid lambda response received: Lambda response must be valid json
So it would seem sam local start-api can't find test and throws and error which means the API gateway doesn't get a valid 'lambda response'. So far this has not helped me chase down the problem :/ It certainly seems aware that test is a route, as trying to hit other endpoints gives the classic {"message":"Missing Authentication Token"} but it chokes hard trying to fulfill it despite me having both functions/test.ts and the compiled functions/test.js present.
I have the test route and handler defined in my CDK stack definition like so...
const testLambda = new lambda.Function(this, "testLambdaHandler", {
runtime: lambda.Runtime.NODEJS_14_X,
code: lambda.Code.fromAsset("functions"),
handler: "test.handler"
})
api.root
.resourceForPath("test")
.addMethod("GET", new apigateway.LambdaIntegration(testLambda))
I considered posting my template.yml but that is even longer than the big ugly error message so I haven't.
So I have three questions (well actually a million but I don't want to be too cheeky!)
Is this actually the canonical way of locally testing apps made with CDK
If so, where am I going wrong?
If not, what is the better/proper way?
Lambda handlers are just functions. They do not need any special environment to function - they are called at a specific point in the Lambda Invocation process, and provided an event (a json object) and a context (another json object)
You can (and should!) unit test them just like any other individual function in your language/testing framework.
As #Lucasz mentioned, you should rely on the fact that, if set up properly, API gateway and Lambda will interact the same way every time. Once you have run one end to end test and you know that the basics work, any further implementation can be done trough unit testing
There are numerous libraries for mocking AWS service calls in unit testing, and there are plenty of good practice work arounds for the services that are more difficult to mock (ie: its difficult to mock a Lambda call from inside another lambda - but if you wrap that lambda call in its own function, you can mock the function itself to return whatever you want it to - and this is good practice for testing as well!)
using jest, in a coded unit test, you can call the lambda handler, give it stubbed or mocked event json, as well as a context json (probably just blank as youre not using it) and the lambda handler will act just like any other function with two parameters you've ever written, including returning what you want it to return.
You must be doing something wrong with your file directory. Where is your index.js located? If you generate the template.json, is the directory correct?
Also in what directory do you execute the Sam local command?
The thing with testing your serverless application is you don't have to test your full application. You need to count on AWS that API gateway, dynamodb and lambda is perfectly working.
The only thing you need to test is the logic you implemented.
In here you make sure your function prints out something and returns a 200. That's all you have to do.
Look into 'jest' for testing js.
If you want to test cdk you should into https://docs.aws.amazon.com/cdk/latest/guide/testing.html
Also "running Aws locally" is not good practice. it's never the same as how it's running in real life aka the cloud. You use plugins for this, tools for that... Local is not the same as in the cloud.
If you have any more questions, feel free to ask.
Is there a way to invoke lambda with X-Ray by using sam invoke local?
According to the idea which PaulMaddox mentioned,
I have tried the step below, and I don't know whether I misunderstood :
Run a X-Ray Daemon locally (0.0.0.0:2000) by following the document
In my lambda's template.yaml set the ENV AWS_XRAY_DAEMON_ADDRESS: 0.0.0.0:2000
Invoke the function, but still got error Missing AWS Lambda trace data for X-Ray. Expected _X_AMZN_TRACE_ID to be set
Here is part of template.yaml setting, I used the environment variable to set AWS_XRAY_DAEMON_ADDRESS
It would be nice if you could provide more information.
I'm not too familiar with SAM, but...
You need to set the _X_AMZN_TRACE_ID environment variable. Currently, the X-Ray Node SDK works by cross communicating between the Lambda runtime start-up code and the user code.
Lambda starts the segment in their start-up code, records information such as timing and exceptions, and sends the segment to the X-Ray service. Then, it forwards the trace ID/parent ID/sampling decision to the user code via setting the _X_AMZN_TRACE_ID environment variable. This allows the SDK to create a separate subsegment, inferring a connection to the original segment, which gets "weaved" into the original on the service end without actually being directly related. Both are sent out of band, asynchronously from each other.
The _X_AMZN_TRACE_ID variable fits the same format as the tracing header here as discussed here: https://docs.aws.amazon.com/xray/latest/devguide/xray-concepts.html#xray-concepts-tracingheader
If you want to send traces through the Daemon to the X-Ray service, you'll need to figure out how to get SAM to construct this Lambda segment initially and set the _X_AMZN_TRACE_ID prior to importing the SDK.
Since the SDK auto-detects the presence of Lambda (which as I understand, SAM mimics), you'll have to set the _X_AMZN_TRACE_ID variable before importing in the SDK. Which, is kind of a catch-22, because you need to import the SDK (in the non-Lambda mode) to construct the Lambda segment before you can populate the _X_AMZN_TRACE_ID.
The problem lies here: https://github.com/aws/aws-xray-sdk-node/blob/master/packages/core/lib/aws-xray.js#L361
If you flip the SDK into LOG_ERROR mode (ignoring the Lambda errors), create and send the Lambda segment (just manually create a segment, load the generated ID/Parent ID/Sampling into _X_AMZN_TRACE_ID then close the segment) and then clear cache/re-import the SDK after, then that should work.
Otherwise, I suspect there may be some work on the SAM end to have this built in. But, hopefully this work around works.
I have a serverless framework service with (say)five aws lambda functions using python. By using github I have created a CodePipeline for CI/CD.
When I push the code changes, it deploys all the functions even only function is changed.
I want to avoid the deployment of all functions and the CI/CD should determine the changed function and deploy it. Rest of functions should not be deployed again.
Moreover, is there anyway to deal with such problems using AWS SAM, as at this stage I have an option to switch towards SAM by quitting serverless framework
Unfortunately there is no "native" way to do it. You would need to write a bash that will loop through the changed files and call sls deploy -s production -f for each one of them.
I was also faced this issue, and eventually it drove me to create an alternative.
Rocketsam takes advantage of sam local to allow deploying only changed functions instead of the entire microservice.
It also supports other cool features such as:
Fetching live logs for each function
Sharing code between functions
Template per function instead of one big template file
Hope it solves your issue :)
There are a set of Docker images available called docker-lambda that aim to replicate the AWS Lambda environment as closely as possible. It's third party, but it's somewhat endorsed by Amazon and used in their SAM Local offering. I wrote a Lambda function that invokes an ELF binary, and it works when I run it in a docker-lambda container with the same runtime and kernel as the AWS Lambda environment. When I run the same function on AWS Lambda, the method mysteriously crashes with no output. I've given the function more RAM than the process could possibly use, and the dependencies are clearly there because the I know that it's not really possible to debug this without access to the binaries, but are there any common/possible reasons why this might be happening?
Additionally, are there any techniques to debug something like this? When running in Docker, I can add a --cap-add SYS_PTRACE flag to invoke a command using strace and see what might be causing an error. I'm not aware of an equivalent debugging technique on AWS Lambda.
Willing to bet that the AWS Lambda execution environment isn't playing nicely with that third party image. I've had no problems with Amazon's amazon/aws-lambda-python:3.8 image.
Only other thing I can think of is tweaking the timeout of your lambda.