If I have an application that runs solely as lambda functions within AWS, is there a way I can setup the logging to tell me how my lambda was executed? For example, I only want the application to be able to execute the lambda based on triggers, but I want to be able to detect if someone logged in and executed one by hand, or even worse, if someone externally was able to remotely execute a lambda.
I understand that I can lock these things down, and they are, and there are guardrails to help prevent external access; but on top of this, I still want to be able to detect and verify that only the application is executing the lambda. Ideally, there's something that I can trace in the logging that shows me an execution IP that I can verify comes from the lambda service, or a log that states how the lambda was executed, then I could trace that back to an executing application or service.
You can use CloudTrail logging to retrospectively evaluate how your Lambda was invoked. If you go with this option you will need to enable Lambda logging as this is disabled by default.
It will push logs into S3 which you could retrospectively parse and evaluate if this happened.
You can add restrictions to invoking the Lambda via its function policy. By using conditions you can tighten exactly which resources can invoke.
Related
Thinking that I wanted to clear out old logs, I made the mistake of deleting my Lambda's "Log Stream" on CloudWatch.
The result, as I should have expected if I was awake, is that now CloudWatch isn't getting the Lambda's console logs at all. Oops.
The log group still exists.
I can see how to create a new log stream.
What I haven't been able to find on the web is clear instructions to get the existing Lambda to output to this new stream... ie, to repair what I did.
Can someone provide instructions or a pointer to them, please? I'm sure I'm not the only one who's made this mistake, so I think it's an answer worth having on tap.
UPDATE: Decided to try recovering by creating an entirely new Lambda, running the same code and configured the same way, expecting that it would Just Work; my understanding was that a new Lambda binds to a CloudWatch group automagically.
Then I ran my test, clicked the twist-arrow to see the end of the output, and hit "Click here to view the corresponding CloudWatch log group.". It opened Cloudwatch looking at the expected log group name -- with a big red warning that this group did not exist. Clicking "(Logs)" at the top of the test output gave the same behavior.
I tried creating the group manually, but now I'm back where I was -- lambda runs, I get local log output, but the logs are not reaching CloudWatch.
So it looks like there's something deeper wrong. CloudWatch is still getting logs from the critical lambda (the one driving my newly-released Alexa skill), and the less-critical one (scheduled update for the skill's database) is running OK so I don't absolutely need its logs right now -- but I need to figure this out so I can read them if that background task ever breaks.
Since this is now looking like real Unexpected Behavior rather than user error, I'll take it to the AWS forums and post here if they come up with an answer. On that system, the question is now at https://repost.aws/questions/QUDzF2c_m0TPCwl3Ufa527Wg/lambda-logging-to-cloud-watch-seems-to-be-broken
Programmer's mantra: "If it was easy, they wouldn't need us..."
After a Lambda function is executed, you can go to the Monitoring tab and click View logs in CloudWatch -- it will take you to the location where the logs should be present.
If you know that the function has executed but no logs are appearing, then confirm that your Lambda function has the AWSLambdaBasicExecutionRole assigned to the IAM Role being used by the Lambda function. This grants permission for the Lambda function to write to CloudWatch Logs.
See: AWS Lambda execution role - AWS Lambda
Here's what I know, or think I know.
In AWS Lambda, the first time you call a function is commonly called a "cold start" -- this is akin to starting up your program for the first time.
If you make a second function invocation relatively quickly after your first, this cold start won't happen again. This is colloquially known as a "warm start"
If a function is idle for long enough, the execution environment goes away, and the next request will need to cold start again.
It's also possible to have a single AWS Lambda function with multiple triggers. Here's an example of a single function that's handling both API Gateway requests and SQS messages.
My question: Will AWS Lambda reuse (warm start) an execution environment when different event triggers come in? Or will each event trigger have it's own cold start? Or is this behavior that's not guaranteed by Lambda?
Yes, different triggers will use the same containers since the execution environment is the same for different triggers, the only difference is the event that is passed to your Lambda.
You can verify this by executing your Lambda with two types of triggers (i.e. API Gateway and simply the Test function on the Lambda Console) and looking at the CloudWatch logs. Each Lambda container creates its own Log Stream inside of your Lambda's Log Group. You should see both event logs going to the same Log Stream which means the 2nd event is successfully using the warm container created by the first event.
I have a variety of functions, all in Node.js, in AWS Lambda. They're triggered by certain events like S3 triggers, API Gateway methods, or sometimes just called manually. I create them by pasting code in the console or uploading a zip file I've built locally.
On rare occasion, a function will fail. To detect failures, I've set up a CloudWatch alarm that looks like this:
This works, to an extent: when a function anywhere in my account fails, I get an email. The problem is the email just states that the alarm got tripped. It doesn't state what Lambda function actually failed so I have to dig through Lambda to find which function actually caused the alarm.
I've considered the following:
Setting up a CloudWatch alarm per function. This is the most obvious solution but is also the most tedious and highest maintenance.
Building a CI/CD pipeline for my Lambda functions instead of entering the code or uploading zips in the console. I can then add a step that sets up a CloudWatch alert for the function automatically. This is better than the first option but also is a lot of infrastructure to set up for potentially a simple problem.
Using another Lambda function to custom handle the alert. The problem is, best I can tell, the SNS message that CloudWatch publishes doesn't contain any more data than the email; it just says in essence "your alarm named X tripped" but not why.
Any ideas on how to achieve this?
We handle it internally. When there is a problem, the Lambda attempts to handle it, and sends an alert. The CloudWatch metric is only for truly unhandled exceptions. Remember Lambda automatically retries if a function has an error, which can be undesirable for certain situations. So it may be preferable to handle any exceptions internal to the Lambda function.
How can I check the running Lambda functions running using the aws cli?
It seems that there is no a command to check it:
aws lambda XXXX
I have several scripts running, and I'd like to monitor the situation.
It is enough to show how many functions are running.
Thank you
Watching or monitoring the cloud watch logs would be the best way to monitor if a lambda is running or not. These logs are not real time, but may be near real time enough for your needs. You could ask CloudWatch for the last X minutes of log for a particular lambda and monitor the timing of the log statements. As Aniket Chopade stated though, knowing why you're trying to do this could help someone provide a better solution.
You will not see any state such as "running" in CLI ouput.
From users perspective, Lambda functions are always up running (actually right word is invokable) because they respond to triggers.
There is concept of keeping them "warm" , that means keeping one physical instance alive having containers running in it. But again these level of details are hidden from lambda's users.
I am curious why you want to know such state for lambda functions.
I'm serving static JS files over from my S3 Bucket over CloudFront and I want to monitor whoever accesses them, and I don't want it to be done over CloudWatch and such, I want to log it on my own.
For every request to the CloudFront I'd like to trigger a lambda function that inserts data about the request to my MySQL RDS instance.
However, CloudFront limits Viewer Request Viewer Response triggers too much, such as 1-second timeout (which is too little to connect to MySQL), no VPC configuration to the lambda (therefore I can't even access the RDS subnet) and such.
What is the most optimal way to achieve that? Setup an API Gateway and how would I send a request to there?
The typical method to process static content (or any content) accessed from CloudFront is to enable logging and then process the log files.
To enable CloudFront Edge events, which can include processing and changing an event, look into Lambda#Edge.
Lambda#Edge
I would enable logging first and monitor the traffic for a while. When the bad actors hit your web site (CloudFront Distribution) they will generate massive traffic. This could result in some sizable bills using Lambda Edge. I would also recommend looking in Amazon WAF to help mitigate Denial of Service attacks which may help with the amount of Lambda processing.
This seems like a suboptimal strategy, since CloudFront suspends request/response processing while the trigger code is running -- the Lambda code in a Lambda#Edge trigger has to finish executing before processing of the request or response continues, hence the short timeouts.
CloudFront provides logs that are dropped multiple times per hour (depending on the traffic load) into a bucket you select, which you can capture from an S3 event notification, parse, and insert into your database.
However...
If you really need real-time capture, your best bet might be to create a second Lambda function, inside your VPC, that accepts the data structures provided to the Lambda#Edge trigger.
Then, inside the code for the viewer request or viewer response trigger, all you need to do is use the built-in AWS SDK to invoke your second Lambda function asynchronously, passing the event to it.
That way, the logging task is handed off, you don't wait for a response, and the CloudFront processing can continue.
I would suggest that if you really want to take this route, this will be the best alternative. One Lambda function can easily invoke a second one, even if the second function is not in the same account, region, or VPC, because the invocation is done by communicating with the Lambda service's endpoint API.
But, there's still room for some optimization, because you have to take another aspect of Lambda#Edge into account, and it's indirectly related to this:
no VPC configuration to the lambda
There's an important reason for this. Your Lambda#Edge trigger code is run in the region closest to the edge location that is handling traffic for each specific viewer. Your Lambda#Edge function is provisioned in us-east-1, but it's then replicated to all the regions, ready to run if CloudFront needs it.
So, when you are calling that 2nd Lambda function mentioned above, you'll actually be reaching out to the Lambda API in the 2nd function's region -- from whichever region is handling the Lambda#Edge trigger for this particular request.
This means the delay will be more, the further apart the two regions are.
This your truly optimal solution (for performance purposes) is slightly more complex: instead of the L#E function invoking the 2nd Lambda function asynchronously, by making a request to the Lambda API... you can create one SNS topic in each region, and subscribe the 2nd Lambda function to each of them. (SNS can invoke Lambda functions across regional boundaries.) Then, your Lambda#Edge trigger code simply publishes a message to the SNS topic in its own region, which will immediately return a response and asynchronously invoke the remote Lambda function (the 2nd function, which is in your VPC in one specific region). Within your Lambda#Edge code, the environment variable process.env.AWS_REGION gives you the region where you are currently running, so you can use this to identify how to send the message to the correct SNS topic, with minimal latency. (When testing, this is always us-east-1).
Yes, it's a bit convoluted, but it seems like the way to accomplish what you are trying to do without imposing substantial latency on request processing -- Lambda#Edge hands off the information as quickly as possible to another service that will assume responsibility for actually generating the log message in the database.
Lambda and relational databases pose a serious challenge around concurrency, connections and connection pooling. See this Lambda databases guide for more information.
I recommend using Lambda#Edge to talk to a service built for higher concurrency as the first step of recording access. For example you could have your Lambda#Edge function write access records to SQS, and then have a background worker read from SQS to RDS.
Here's an example of Lambda#Edge interacting with STS to read some config. It could easily be refactored to write to SQS.