I have a multi-account pipeline using AWS CodePipeline that is failing in the prod account. The pipeline will work in dev and test but fails in prod and the Cloudformation stack gives the error: "The Following Resources Failed to Update:" and lists several Lambda functions. Anyone know how to fix this? I've checked the permissions and compared them to the other accounts and they seem to match. From my understanding the resources are under the control of CF and should be able to be changed via CDK?
The CDK definitions of the Lambda functions seem to match their current configurations in AWS, So I am having trouble understanding why it no longer works.
The resources also were "drifted" and to fix that I deleted the resource from the stack and made sure to include the "DeletionPolicy: "Retain" before deleting. I of course Imported those resources but I am wondering if that could be the source of the issue?
Really racking my brain with this one
Related
I am creating a project in cloud9 with AWS Lambda and Amazon EFS resources so that users can use a machine learning model.
I was able to build the project successfully, but no matter what I do, sam deploy --guided fail and error message is:
Error: Failed to create/update the stack: sam4-app, Waiter StackCreateComplete failed: Waiter encountered a terminal failure state: For expression "Stacks[].StackStatus" we matched expected path: "ROLLBACK_COMPLETE" at least once.
In the aws cloudFormation console, I found that the creation of my two functions failed
I am sure I am doing something wrong, yet after so much struggle I have not been able to identify my mistake.
Do I need to make any changes inside the template.yml in order to use AWS Lambda?
It is important to mention that I am taking as reference multiple machine learning models for inference on AWS Lambda and Amazon EFS
Also I have dumb question, I am using windows inside cloud9, could this be causing this error?
You are trying to use too much memory. You are trying to provision 5000MB for your function. While a limit of 10GB is now possible in some regions, it isn't supported in all regions. In the regions where 10GB is not supported, the old 3008MB limit applies.
I'm new to a large AWS deployment where stuff is mostly deployed through CloudFormation (and some through Terraform). But there are always cases where something has been deployed manually and not through code. Is there a reliable way to quickly figure out if a resource (say, an EC2 instance) already existing in the deployment was deployed through IaC or manually? A CloudFormation-specific answer will be good enough for now.
Going through literally hundreds of CloudFormation stacks manually and looking for the resource is not an option.
You can identify the resources created by cloudformation. Cloudformation applies few default tags as mentioned here
aws:cloudformation:logical-id
aws:cloudformation:stack-id
aws:cloudformation:stack-name
You can run a script to check whether the resource contain one/all of these tags to update your count.
Offical documentation on resource tags
Unfortunately looking at an AWS resource you don't see how it got created. While some resources might have been tagged by CloudFormation indicating that they got created by a CloudFormation stack, that's only valid for a subset of resources.
The only reliable way to figure out whether or not a resource got created via a CloudFormation stack is to go through all CloudFormation stacks and check whether or not the resource in question is a part of it. While that might be cumbersome when doing manually, it's also something you can automate using the AWS CLI.
I follow the tutorial on http://docs.aws.amazon.com/lambda/latest/dg/automating-deployment.html
The tutorial demonstrate how to automatically deploy a lambda and an API gateway using AWS cloudformation.
After some time I was able to complete the tutorial with success. This means that when I push a commit to the github repository linked to the AWS CodePipeline the changed code is uploaded/packaged to AWS -> build -> and deployed (i.e. i can see the code change)
My problem is that I tried to delete the lambda function and then invoke the Codepipeline by pushing a git commit. This trickered the codepipeline and I could watch source, build and staging steps complete successfully. However, I cannot find the lambda? I thought that cloudformation would recreate the application ? Can you help?
If you deleted the function manually then you're most likely running into this issue:
Resources that are created as part of an AWS CloudFormation stack must be managed from the same stack. Modifications to a resource must be done by a stack update. If a resource is deleted, a stack update is also necessary to remove the resource from the template. If a resource has been accidentally or purposely manually deleted, you can encounter errors when attempting to perform a stack update.
https://aws.amazon.com/premiumsupport/knowledge-center/failing-stack-updates-deleted/
You can resolve this by manually recreating the resource with the same name, then allowing CloudFormation to manage the resource in future.
The reason why I did not see any lambda function was because I only created the change set ("create or update change set") and missed to add the actual deploy stage "execute change set".
Has anyone come across the below error before?
The service role arn:aws:iam::20011470201:role/deploy doesn't have permission to perform the following operation: autoscaling:DescribeLifecycleHooks
I have code-deploy set-up between by bit-bucket account and my Amazon AWS instance.
I am able to deploy to the test server everyday without issue.
But when i try to add the instance of our production server to the list of instances, i get the above error
Note: I have added this instance and successfully deployed the code in the past , i'm not sure why i get this error now.
Any directions/hints on how to solve this would be appreciated.
Not sure how i missed it, but the policy i had defined was missing the "autoscaling:DescribeLifecycleHooks", once i added this to the existing permission everything worked fine.
Then again, the policy has not changed in well over a year, not sure why aws did not complain about this earlier
We currently rolled out a fix for permission issues between CodeDeploy and AutoScaling. Previously CodeDeploy doesn't require autoscaling:DescribeLifecycleHooks to describe or create a lifecycle hook to AutoScaling, when customer's deplyoment group contains AutoScaling groups. But now we started to require this permission, which is actually the right way and also expected. Adding the proper permission fixes the problem.
Thanks,
Binbin
I see that you fixed this. Can you paste an example config here so noobs like me know just how to place this bit of code? Oh, and I can't comment on your accepted solution yet, not enough points...
I would like to make an automated call to a custom program API as soon as CloudFormation has completed the entire stack creation (deployment of instances, setup of VPC, Puppet scripts, etc.).
What is the correct way to go about this?
After some research, it seems a good option would be to launch an AWS Lambda function triggered by the event that stack creation has been completed successfully, but I have no idea how to approach this.
Any ideas or advice would be appreciated.
You can provision and coordinate a lot of what you're talking about (setup of VPC, etc.) with CloudFormation, the DependsOn attribute and nested CloudFormation stacks. This way you can order the execution of the CloudFormation stacks so that, for example, your VPC is created first followed by launching your EC2 instance(s) followed by the deployment of the software on the instance(s).
You can also coordinate the execution of the other behavior you mentioned (deployment [on] instances, [calling] Puppet scripts, etc.) using AWS::CloudFormation::Init. This way, you can call out to your Puppet scripts from your EC2 instance within the CloudFormation template. The actual execution of your Puppet scripts occurs on the EC2 instance(s).
If you want to see an example of calling out to a configuration management tool from CloudFormation (in this case, we're using Chef Solo), see app-instance.json.
If you'd like to see an example of using nested stacks, see dromedary-master.json.
There's also some examples of using Lambda on our blog as well (Stelligent), but it doesn't seem like you need to use Lambda in this case based on the problem you're trying to solve.
P.S. You don't have to use nested stacks either, but it can make things a little cleaner. But, you do want to control the creation order of the resources so the DependsOn attribute will help you in doing so.