How to compile and run a c, c++, python, java code on aws lambda function. I am working on a project that user can upload their code and It will compile and run the code and return the output of the program to the user through aws lambda function. For compiling the codes I have to install the compilers in aws lambda function but I don't know how to do that. Is that possible to install in lambda function?
Thanks,
It isn't possible to install additional operating system dependencies like the gcc compiler. Additionally, even if you could I don't think you'd want to do that as it would dramatically increase the startup time of the Lambda. Remember that you may have thousands of Lamda's running and they may be re-initiated at any time.
A better way might be to have a Docker container that has all of the compilers available to your customers and deploy that to ECS. That way you could keep the isolation of Lambda but customize the runtime environment more.
Related
AWS publishes a lot of their source code on GitHub, but so far I haven't been able to find the source for the dotnetcore3.1 Lambda Runtime.
I expect this source will be a console application responsible for startup and IoC (using the Lambda configuration to determine the handler class and function, using reflection to instantiate the handler, as well as the serializer specified on the lambda's assembly attribute). So far I have found a number of repositories containing things like NuGet libraries and dotnet CLI tooling--but haven't located the runtime itself.
Is the AWS Lambda dotnetcore3.1 runtime source publically available? Can you point me to the repo and .csproj? Thanks!
I am not 100% sure, but there is probably nothing like a dedicated ".NET Core Runtime" that is publicly available.
Let's explore how Lambda works to explain why I think that:
First there is an EC2 instance with a host operating system. That's the bare metal your resources are coming from (vCPU, RAM, disk, etc.).
Second, there is a thin management layer for microVMs. For AWS Lambda this is a project called Firecracker.
The microVMs that are started use a micro kernel. In case of AWS Lambda it is the OSv micro kernel. The micro kernel already has some support for runtimes:
OSv supports many managed language runtimes including unmodified JVM, Python 2 and 3, Node.JS, Ruby, Erlang as well as languages compiling directly to native machine code like Golang and Rust.
.NET Core is compiled so I think this falls into the same category. It is a "native" binary, that is self-contained and just started.
You mentioned reflection etc. to run the binary. I think it is not that complicated. This is were the last puzzle piece comes into place: environment variables.
There are a bunch of environment variables thar are set by the "runtime". You can find the full list in the AWS documentation: Defined runtime environment variables.
Every Lambda has a environment variable called _HANDLER which points to the handler. In your case this would be the binary. This would allow the runtime to just "run" your handler.
When the handler runs the underlying AWS SDK is starting some kind of webserver/socket (I think it depends on the framework/language) which exposes everything the "standard" runtime needs to communicate with your code.
So as far as I can tell there is not really a dedicated .NET Core runtime. It is pretty generic all the way down, which simplifies development for the folks at AWS I guess.
The source code for Lambda's dotnet runtime(s) is within the aws/aws-lambda-dotnet repository--it doesn't appear to have a dedicated netcoreapp3.1 library, or a preserved branch, but presumably the 3.1 runtime is contained within this repository's history.
The Amazon.Lambda.RuntimeSupport console application appears to be the entrypoint, which in turn instantiates and invokes the published Handler.
I want to make a bot that makes other bots on Telegram platform. I want to use AWS infrastructure, look like their Lamdba functions are perfect fit, pay for them only when they are active. In my concept, each bot equal to one lambda function, and they all share the same codebase.
At the starting point, I thought to make each new Lambda function programmatically, but this will bring me problems later I think, like need to attach many services programmatically via AWS SDK: Gateway API, DynamoDB. But the main problem, how I will update the codebase for these 1000+ functions later? I think that bash script is a bad idea here.
So, I moved forward and found SAM (AWS Serverless Application Model) and CloudFormatting, which should help me I guess. But I can't understand the concept. I can make a stack with all the required resources, but how will I make new bots from this one stack? Or should I build a template and make new stacks for each new bot programmatically via AWS SDK from this template?
Next, how to update them later? For example, I want to update all bots that have version 1.1 to version 1.2. How I will replace them? Should I make a new stack or can I update older ones? I don't see any options in UI of CloudFormatting or any related methods in AWS SDK for that.
Thanks
But the main problem, how I will update the codebase for these 1000+ functions later?
You don't. You use lambda alias. This allows you to fully decouple your lambda versions from your clients. This works because you are using an alias of your function in your client's code (or api gateway). The alias is fixed and does not change.
However, alias is like a pointer - it can point to different versions of your lambda function. Therefore, when you publish a new lambda version you just point alias to it. Its fully transparent from your clients and their alias does not require any change.
I agree with #Marcin. Also it would be worth checking serverless? Seems like you are still experimenting so most likely you are deploying using bash scripts with AWS SDK/SAM commands. This is fine but once you start getting the gist of what your architecture looks like, I think you will appreciate what serverless can offer. You can deploy/teardown cloudformation stacks in matter of seconds. Also you can use serverless-offline so that you can have a local build of your AWS lambda architecture on your local machine.
All this has saved me hours of grunt work.
I would like to debug an AWS Lambda function written in C++ locally. Ideally, I want to be able to step through my code (like you do with GDB or any decent IDE). The way to do this with AWS Lambdas would usually include AWS SAM.
Idea 1, debug using SAM:
Unfortunately, SAM does not enable you do debug provided runtimes, since it "only" supports Node.js, Python and Go (compare documentation).
Question 1:
Is there a way to somehow attach GDB to the my compiled lambda function?
Idea 2, run using docker-lambda:
You can also start your function using lambci/docker-lambda. If you would like to get some debugging output, you could do strace docker run --rm -v my/build/dir:/var/task lambci/lambda:provided handler '{"some": "event"}'. This works, but is not really the quality of debugging that I am looking for.
Question 2:
Can I somehow attach a to a lambda function that is running in docker-lambda?
Idea 3: remote:
Be desperate and go for anything you can get (this is NOT what I wanna do).
Question 3:
If there is really no way to do this locally, how should I do it on AWS?
When I want to launch some code serverless, I use AWS Lambda. However, this time my deployment package is greater than 250MB.
So I can't deploy it on a Lambda...
I want to know what are the alternatives in this case?
I'd question your architecture. If you are running into problems with how AWS has designed a service (i.e. lambda 250mb max size) its likely you are using the service in a way it wasn't intended.
An anti-pattern I often see is people stuffing all their code into one function. Similar to how you'd deploy all your code to a single server. This is not really the use case for AWS lambda.
Does your function do one thing? If not, refactor it out into different functions doing different things. This may help remove dependencies when you split into multiple functions.
Another thing you can look at is can you code the function in a different language (another reason to keep functions small). I once had a lambda function in python that went over 250mb. When I looked at solving the same problem with node.js, my function size dropped to 20mb.
One thing you can do is before run the lambda function you can download the dependencies to /tmp folder from s3 bucket and then add it to python path, it would give you extra 512MB, although you need to take into consideration the download time for some of the lambda invocations
There are a set of Docker images available called docker-lambda that aim to replicate the AWS Lambda environment as closely as possible. It's third party, but it's somewhat endorsed by Amazon and used in their SAM Local offering. I wrote a Lambda function that invokes an ELF binary, and it works when I run it in a docker-lambda container with the same runtime and kernel as the AWS Lambda environment. When I run the same function on AWS Lambda, the method mysteriously crashes with no output. I've given the function more RAM than the process could possibly use, and the dependencies are clearly there because the I know that it's not really possible to debug this without access to the binaries, but are there any common/possible reasons why this might be happening?
Additionally, are there any techniques to debug something like this? When running in Docker, I can add a --cap-add SYS_PTRACE flag to invoke a command using strace and see what might be causing an error. I'm not aware of an equivalent debugging technique on AWS Lambda.
Willing to bet that the AWS Lambda execution environment isn't playing nicely with that third party image. I've had no problems with Amazon's amazon/aws-lambda-python:3.8 image.
Only other thing I can think of is tweaking the timeout of your lambda.