I have a stack with:
API Gateway
Lambda
Kinesis
When deleting this CloudFormation stack from the AWS console, the process is very slow.
Everything works fine until you execute the exclusion of 'AWS :: Lambda :: Function' -> 'CloudFormation is waiting for NetworkInterfaces associated with the Lambda Function to be cleaned up.'
The time in this process takes about 30 minutes.
Has anyone had the same problem?
To prevent this from blocking the stack deletion, you could set a DeletionPolicy property to Retain for that specific Lambda and have another scheduled process that would clean up each day the orphaned Lambdas.
Check that the lambda function assigned Role has delete permissions for the network interface, ie all of these:
- ec2:CreateNetworkInterface
- ec2:DescribeNetworkInterfaces
- ec2:DeleteNetworkInterface
If it is unable to delete the interface the Cloudformation template might hang, as you have experienced.
If that doesn't work you might have to script something to delete the ENIs on the lambda while the lambda is trying to teardown.
Related
Couldn't delete lambda stack via aws cdk:
Lambda was unable to delete lambda because it is a replicated function.
When deleting a CloudFormation Stack with CloudFront resource and associated Lambda#edge - CloudFormation first initiate a delete request for the CloudFront resource and the links to lambda#edge, since the lambda#edge is replicated to edges locations it takes up to a few hours to complete the links deletion.
My workaround is to run the complete clean-up in two phases -
Delete the stack and ignore the DELETE_FAILED status if just the lambda was failed to be deleted (and the other resources were deleted successfully)
Re-delete the DELETE_FAILED stack after a few hours (Or do it automatically using a cleanup lambda triggered by cron event to clean-up the CloudFormation stacks with DELETE_FAILED status)
If you are able to get in such a situation then yo most likely have freshly deleted a stack that did have lambda edge functions. In that case replicated functions get revealed (you dont see them while the edge function is inplace/being used). You dont need to do anything, just maybe to be patient - the replicated function will disappear after some time.
I am trying a scenario where cloud formation has to wait until an object is created in the specified bucket (where the object creation happens outside the scope of cloud formation by an external application).
I tried enabling bucket event notifications and hook a lambda function (so whenever an object is created in the bucket, lambda function is triggered). But I am not sure how to make cloud formation wait until this hooked lambda function execution is invoked.
Kindly let me if there any ideas on how to achieve this scenario.
I think the following should work:
Create WaitConditionHandle
Create a lambda function and pass !Ref to the wait condition handle created as an environment variable. When you !Ref a wait condition you get an url address. The lambda has only one job - to call the url when invoked.
Create WaitCondition and associate it with the wait handle created in step 1.
Add DependsOn attribute to the WaitCondition so that the condition gets created after the last resource to be created before CFN should pause and wait.
Use the S3 notification (as you already wrote in your question) to invoke lambda created in Step 2 when you get your object. Lambda gets invoked, calls the url, wait conditions stops waiting, and CFN should continue.
With the above there are no loops or long running processes, such as calling a lambda every 2 minutes.
Max timeout for the WaitCondition is 12 hours. You should adjust it 40 minutes or 1h for instance.
Try using a wait condition to solve this: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-waitcondition.html
You could try using Custom CloudFormation resources: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/template-custom-resources-sns.html. This would require you can send a http request to a S3-url provided through an SNS notification.
You would create file based on the SNS-notification (using lambda?) and then send a request back to cloudformation.
I have a Lambda that I have created following the example given by the aws docs (https://docs.aws.amazon.com/codepipeline/latest/userguide/actions-invoke-lambda-function.html), but I am invoking my Lambda from within a VPC and it seems that the CodePipeline never successfully talks to the Lambda (it gets a timeout and never seems to enter the Lambda as CloudWatch has none of my console.logs); this is despite the fact that I have created a CodePipeline Endpoint from within the VPC and associated the private subnet from which I launch the Lambda out to codepipeline.
I can give the Lambda an API Gateway endpoint and fire it manually just fine from Postman; it takes ~1 second to run. My Cloudwatch logs just have "Task timed out after 20.02 seconds." I'm not sure what else I can try; what else might prevent CodePipeline from talking to the Lambda?
After additional logging, I discovered that I actually had the VPC set up correctly and that the Lambda was being invoked; the Lambda was failing to get to S3 and was hanging on getting objects. I created another Endpoint for S3 for the VPC and was able to move passed the initial issue.
Lets say I have a CloudFormation stack running, which creates and deploys an Lambda function. In the AWS Console, if I connect my Lambda function to an API in API Gateway, will my CloudFormation Template be updated immediately if the Lambda function successfully integrates with the API?
It's a one way traffic from Cloudformation to resources.
Meaning if you modify your Cloudformation template and update the stack then the resources that were created by Cloudformation get modified/updated. However the other way is not true. Meaning if you modify your resources the Cloudformation template does not get updated.
Moreover, as a good practice you should avoid modifying the resources directly because you may end up breaking the Cloudformation's update stack functionality for that that stack
I'd like to run some code using Lambda on the event that I create a new EC2 instance. Looking the blueprint config-rule-change-triggered I have the ability to run code depending on various configuration changes, but not when one is created. Is there a way to do what I want? Or have I misunderstood the use case of Lambda?
We had similar requirements couple of days back(Users were supposed to get emails whenever a new instance gets launched)
1) Go to cloudwatch, then select Rules
2) Select service name (its ec2 for your case) then select "Ec2 instance state-change notification"
3) Then select pending in "Specific state" dropdown
4) Click on Add target option and select your lambda function.
That's it, whenever a new instance gets launched, Cloudwatch will trigger your lambda function.
Hope it helps !!
You could do this by inserting code into your EC2 instance launch userdata and have that code explicitly invoke a Lambda function, but that's not the best way to do it.
A better way is to use a combination of CloudTrail and Lambda. If you enable CloudTrail logging (every a/c should have this enabled, all the time, in all regions) then CloudTrail will log to S3 all of the API calls made in your account. You then connect this to Lambda by configuring S3 to publish events to Lambda. Your Lambda function will receive an S3 event, can then retrieve the API logs, find RunInstances API calls, and then do whatever work you need to as a consequence of the new instance being launched.
Some helpful references here and here.
I don't see a notification trigger for instance startup, however what you can do is write a startup script and pass that in via userdata. That startup script would need to download and install the AWS CLI and then authenticate to SNS and publish a message to a pre-configured topic. The startup script would authenticate to SNS and whatever other AWS services are needed via your IAM Role, so you would need to give the IAM Role permission to do whatever you want the script to do. This can be done in the IAM console.
That topic would then have your Lambda function subscribed to it, which would execute. Similar to the below article (though the author is doing something similar for shutdown, not startup).
http://rogueleaderr.com/post/48795010760/how-to-notifyemail-yourself-when-an-ec2-instance
If you are putting the EC2 instances into an autoscale group, I believe there is a trigger that gets fired when the autoscale group launches a new instance, so you could take advantage of that.
I hope that helps.
If you have CloudTrail enabled, then you can have S3 PutObject/TrailBucket trigger a Lambda function. Lambda function parses the object that is passed to it and if it finds RunInstances event, then run your code.
I do the exact same thing to notify certain users when a new instance is launched. With Lambda/Python, it is ~20 lines of code.