Since past week we recorded irregular deletions of the trigger of AWS Lambda.
We would like to find out when this exactly happened to determine the reason/cause of deletion. We tried looking for the entries in Cloudtrail, but not sure what to look for exactly ?
How to find the root cause and reasons for the deletion ?
thanks Marcin and ydaetskcoR. We found the problem. The Lambda trigger is a property of the S3 bucket. We had different lambda trigger in different projects (with different terraform states). So every time one (terraform) project will be applied, the trigger of the other project will be overwritten, because the terraform state is not aware of it. We saw PutBucketNotifications in cloudtrail,but didn't recognize the connections...
You can troubleshoot operational and security incidents over the past 90 days in the CloudTrail console by viewing Event history. You can look up events related to creation, modification, or deletion of resources. To view Events in logs follow this ;
https://docs.aws.amazon.com/awscloudtrail/latest/userguide/get-and-view-cloudtrail-log-files.html
Related
I found a really weird bug on aws js sdk.
If I try to delete/add LifecycleRules in my s3Bucket (using putBucketLifecycleConfiguration) and fetch theses rules just after this call using getBucketLifecycleConfiguration I can receive more or less rules than that I've put with putBucketLifecycleConfiguration.
If I keep calling getBucketLifecycleConfiguration I will continue to receive more or less rules that I've put, it seems to be a random behavior...
Do you know if it's a known bug or the reason of this behavior?
NOTE: It seems it has the same behavior with aws s3api get-bucket-lifecycle-configuration AND also in the AWS Management Console.
Maybe we have to wait a moment for AWS servers to replicate the data on all servers?
Thanks!
It seems to be the normal behavior of aws, look at this link:
https://docs.aws.amazon.com/AmazonS3/latest/userguide/how-to-set-lifecycle-configuration-intro.html
Propagation delay
When you add an S3 Lifecycle configuration to a bucket, there is
usually some lag before a new or updated Lifecycle configuration is
fully propagated to all the Amazon S3 systems. Expect a delay of a
few minutes before the configuration fully takes effect.
This delay can also occur when you delete an S3 Lifecycle configuration.
I am looking for a way to monitor any changes that occur to my production envrionment. Such as security group changes, ec2 create/stop/deletes, database changes, s3 bucket changes, route table changes, subnet changes, etc... I was looking at using cloudtrail for this and monitoring all api calls. However, when testing, my subscribed SNS topic was not receiving any notifications when i was making some changes for a test. Curious if anyone else has a work around for this or if I am missing something? Maybe lambda? Just looking for the easiest way to receive email notifications when any changes are made within my prod environment. Thank you.
If you're looking to audit the entire event history of AWS API calls then you would use CloudTrail, remembering to create a trail and enabling the options if you want to audit S3 or Lambda API calls.
By itself CloudTrail will provide auditing, but it can be combined with CloudWatch/EventBridge to automate actions based on specific API calls such as triggering a Lambda or triggering an SNS topic.
Regarding your own implementation so far using SNS always ensure you've accepted the subscription first on the subscriber(s).
In addition you can use AWS Config with many resources in AWS providing 2 benefits to you. You will be able to maintain a history of changes to you resources, whilst also being able to configure compliance and resolution rules for your resources.
I need to start a Lambda Function when an object has been created on an S3 Bucket. I found 2 solutions to do this.
Using AWS::S3::Bucket NotificationConfiguration.
Using a CloudWatch AWS::Events::Rule.
They both seem to do exactly the same thing, which is to track specific changes and launch a Lambda Function when it happens. I could not find any information on which one should be used. I'm using Cloud Formation Template to provision the Lambda, the S3 Bucket and the trigger.
Which one should I use to call a Lambda on Object level changes and why?
Use the 1st one because of
A push model is much better than a pull model. Push means you send data when you get it instead of polling onto something for some set of interval. This is an era for push notifications all over us. You don't go to facebook to check every 5 minutes if someone has liked your picture or not OR someone has replied to your comment, etc.
In terms of cost and efforts also, S3 event notification wins the race.
Cloudwatch was the best option if you didn't have S3 notification but since you have it, that's the best. Plus if you have a feature in the service itself then why will you go for an alternative solution like Cloudwatch rules.
I have a few java lambda functions that depend on dynamoDB. The issue I am having is that my tables are being automatically deleted from AWS for no reason. Has anyone had this happen to them ?
Enable CloudTrail logs and you will be able to see who deleted the tables next time this happens. There is a tutorial on how to do this here.
Btw, double check that you're in the right region because sometimes I find all my AWS resources missing and then I notice I'm in the wrong region.
We have 20 AWS accounts and we create resources in 10 regions in each account. We want to ensure that AWS resources - ELB, AMI and EBS snapshots are properly tagged. We want to have a service that runs periodically to scan the accounts and delete any of the above mentioned resource that is not properly tagged. We want this to be serverless and we were looking at using Lambda. However, there are 2 issues with Lambda:
Lambda timeout - currently it is 5 mins.
Throttling errors
We need to ensure that we process the next account after the first account processing is completed (we could put a hard sleep for a few minutes and then start processing the next account).
Has someone faced a similar scenario and if so, how was it achieved?
Worst case scenario: we will use ECS.
First, can your innermost task complete in under 5 minutes reliably? If so Lambda is a good fit. Your situation looks to be a good fit.
Next, throttling is easily raised by requesting a higher limit through a support ticket.
Finally, try breaking this up into several smaller functions. Maybe something like this:
delete-resource -- Deletes a single untagged resource
get-untagged-resources -- gets untagged resources in an account and invokes "delete-resource" in an async.each loop
get-accounts -- gets list of accounts and invokes "get-untagged-resources" in an async.each loop
I actually prefer having my functions triggered by SNS rather than invoking them directly, but you get the idea. Hope this helps.