I accidentally created an instance of AWS::ApiGateway::GatewayResponse with a bad value - DEFAULT_400 instead of DEFAULT_4XX. The instance got created, or at least "partially created", although the Cloudformation stack in which it was contained returned an error. Now I can't delete it - every time I try I get
1 validation error detected: Value 'DEFAULT_400' at 'responseType' failed to satisfy constraint: Member must satisfy enum value set: [REQUEST_TOO_LARGE, RESOURCE_NOT_FOUND, AUTHORIZER_CONFIGURATION_ERROR, {...}]
How can I force deletion of this instance, ideally via the AWS CLI ?
PS it appears to have been "orphaned", ie the parent RestApi it belonged to has been deleted
There are few ways of deleting stacks which can be automatically removed. They are listed in AWS docs:
How do I delete an AWS CloudFormation stack that's stuck in DELETE_FAILED status?
Related
I am trying to create or update Lambdas on AWS using the Terraform CDKTF. During deployment, I am getting the error of
"An event source mapping with SQS arn (\" arn:aws:sqs:eu-west-2:*******:*****-*****-******** \") and function (\" ******-******-****** \") already exists. Please update or delete the existing mapping with UUID *******-****-****-****-***********"
**** are sensitive info I have swapped out.
Some of our Lambdas are called via SQS, which is what this mapping is referring to. I assumed the first fix would be to remove the mappings that might already exist (on a previous deployment that might have partly gone through), but I am unsure where to find them, nor if they are even available to delete. I originally assumed by calling cdktf deploy it would update these mappings and not throw the err at all.
Does anyone have any advice?
Your diagnosis seems right, there might be some stray resources left behind due to an aborted / unfinished Terraform run. You should be able to clean up after these runs by running terraform destroy in the stack directory ./cdktf.out/stacks/..../. That should delete all previously existing resources created through this Terraform stack.
I'm just starting my serverless/cloudformation/AWS SAM journey. I've created a stack that has a resource of type AWS::CloudFormation::Stack, and I've separated some of my resources into that child stack.
When I do sam build and then sam deploy, I get the following error:
Embedded stack arn:aws:cloudformation:us-
west-2:111111111111:stack/ParentStack-
ChildStack-1QK94LXRA71CS/f9885e30-631c-11eb-
bfd8-021cb123b7ed was not successfully created: The
following resource(s) failed to create: [DynamoDBTable].
-
The following resource(s) failed to create:
[ChildStack].
Of course, what I really want to know is which resource in the nested stack failed to create, and why. When I copy/paste the resources from the child stack into the parent .yaml file and rebuild/redeploy, I see:
One or more parameter values were invalid: Some index key
attributes are not defined in AttributeDefinitions. Keys:
[userID], AttributeDefinitions: [userId] (Service:
AmazonDynamoDBv2; Status Code: 400; Error Code:
ValidationException; Request ID:
SMJDHUT0CQKM8IBQJVMAIJM4RRVV4KQNSO5AEMVJF66Q9ASUAAJG;
Proxy: null)
This is what I want to see in the output when I build the parent stack: the errors that caused the child stack to fail.
This has led me to use a rather tortuous workflow: build the resources in the main stack, then separate them to an independent stack when they build properly. There's got to be a better way, and I'm sure the community knows something here that I don't.
How do y'all debug child stacks when you're on the CloudFormation train?
This is normal behaviour you have to take help from the AWS Console or use AWSW CLI in this case.
Deploy error reporting is not showing the reason of failure when using nested stacks. #5974
Feature Requests for nested stacks
Why doesn't the error tell me what's wrong?:
CloudFormation passes the creation and updating of a resource to the service responsible for those resources. When a resource fails to create/update, the resource's backing service returns a reason to the stack, which gets logged as the Status Reason within the events. Child Stack is a CloudFormation::Stack resource, so it's being created by CloudFormation. As far as CloudFormation knows, it didn't run into an error trying to actually do anything. Everything it did relate to the CloudFormation side of things was fine; the blame is with the Service resource, which failed to create it. Therefore, Child Stack tells Parent-Stack that it only failed because Service failed to create, not because of a problem on the CloudFormation Service side of things.
you can read more about it here CloudFormation troubleshooting
When using CloudFormation templates, I find the "Custom Resource" feature, with its Lambda backing function implementation, very useful to handle all kinds of tasks that CloudFormation does not provide good support for.
Usually, I use custom resources to setup things during stack creation (such as looking up AMI names) or clean up things during deletion (such as removing objects from S3 or Route53 that would block deletion) - and this works great.
But when I try to actually use a "custom resource" to manage an actual custom resource, that has to be created during stack creation, deleted during stack deletion, and - this is where the problem lies - sometimes updated with new values during a stack update, the CloudFormation integration behaves unexpectedly and causes the custom resource to fail.
The problem seems to be that during a stack update where one of the custom resource properties has changed, during the stack's UPDATE_IN_PROGRESS stage, CloudFormation sends an update event to the backing Lambda function, with all values set correctly and a copy of the old values sent as well. But after the update completes, CloudFormation starts the UPDATE_COMPLETE_CLEANUP_IN_PROGRESS stage and sends the backing Lambda function a delete event (RequestType set to Delete).
When that happens, the backing lambda function assumes the stack is being deleted and removes the custom resource. The result is that after an update the custom resource is gone.
I've looked at the request data in the logs, and the "cleanup delete" looks identical to a real "delete" event:
Cleanup Delete:
{
RequestType: 'Delete',
ServiceToken: 'arn:aws:lambda:us-east-2:1234567890:function:stackname-resname-J0LWT56QSPIA',
ResponseURL: 'https://cloudformation-custom-resource-response-useast2.s3.us-east-2.amazonaws.com/arn%3Aaws%3Acloudformation%3Aus-east-2%3A1234567890%3Astack/stackname/3cc80cf0-5415-11e8-b6dc-503f3157b0d1%7Cresnmae%7C15521ba8-1a3c-4594-9ea9-18513efb6e8d?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20180511T140259Z&X-Amz-SignedHeaders=host&X-Amz-Expires=7199&X-Amz-Credential=AKISOMEAWSKEYID%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Signature=3abc68e1f8df46a711a2f6084debaf2a16bd0acf7f58837b9d02c805975df91b',
StackId: 'arn:aws:cloudformation:us-east-2:1234567890:stack/stackname/3cc80cf0-5415-11e8-b6dc-503f3157b0d1',
RequestId: '15521ba8-1a3c-4594-9ea9-18513efb6e8d',
LogicalResourceId: 'resname',
PhysicalResourceId: '2018/05/11/[$LATEST]28bad2681fb84c0bbf80990e1decbd97',
ResourceType: 'Custom::Resource',
ResourceProperties: {
ServiceToken: 'arn:aws:lambda:us-east-2:1234567890:function:stackname-resname-J0LWT56QSPIA',
VpcId: 'vpc-35512e5d',
SomeValue: '4'
}
}
Real Delete:
{
RequestType: 'Delete',
ServiceToken: 'arn:aws:lambda:us-east-2:1234567890:function:stackname-resname-J0LWT56QSPIA',
ResponseURL: 'https://cloudformation-custom-resource-response-useast2.s3.us-east-2.amazonaws.com/arn%3Aaws%3Acloudformation%3Aus-east-2%3A1234567890%3Astack/stackname/3cc80cf0-5415-11e8-b6dc-503f3157b0d1%7Cresname%7C6166ff92-009d-47ac-ac2f-c5be2c1a7ab2?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20180524T154453Z&X-Amz-SignedHeaders=host&X-Amz-Expires=7200&X-Amz-Credential=AKISOMEAWSKEYID%2F20180524%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Signature=29ca1d0dbdbe9246f7f82c1782726653b2aac8cd997714479ab5a080bab03cac',
StackId: 'arn:aws:cloudformation:us-east-2:123456780:stack/stackname/3cc80cf0-5415-11e8-b6dc-503f3157b0d1',
RequestId: '6166ff92-009d-47ac-ac2f-c5be2c1a7ab2',
LogicalResourceId: 'resname',
PhysicalResourceId: '2018/05/11/[$LATEST]c9494122976b4ef3a4102628fafbd1ec',
ResourceType: 'Custom::Resource',
ResourceProperties: {
ServiceToken: 'arn:aws:lambda:us-east-2:1234567890:function:stackname-resname-J0LWT56QSPIA',
VpcId: 'vpc-35512e5d',
SomeValue: '0'
}
}
The only interesting request field that I can see is the physical resource ID is different, but I don't know what to correlate that to, to detect if it is the real delete or not.
The problem seems to be the sample implementation of the sendResponse() function that is used to send the custom resource completion event back to CloudFormation. This method is responsible for setting the custom resource's physical resource ID. As far as I understand, this value represents the globally unique identifier of the "external resource" that is managed by the Lambda function backing the CloudFormation custom resource.
As can be seen in the CloudFormation's "Lambda-backed Custom Resource" sample code, as well as in the cfn-response NPM module's send() and the CloudFormation's built-in cfn-response module, this method has a default behavior for calculating the physical resource ID, if not provided as a 5th parameter, and it uses the CloudWatch Logs' log stream that is handling logging for the request being processed:
var responseBody = JSON.stringify({
...
PhysicalResourceId: context.logStreamName,
...
})
Because CloudFormation (or the AWS Lambda runtime?) occasionally changes the log stream to a new one, the physical resource ID generated by sendResponse() is changing unexpectedly from time to time, and confuses CloudFormation.
As I understand it, CloudFormation managed entities sometimes need to be replaced during an update (a good example is RDS::DBInstance that needs replacing for almost any change). CloudFormation policy is that if a resource needs replacing, the new resource is created during the "update stage" and the old resource is deleted during the "cleanup stage".
So using the default sendResponse() physical resource ID calculation, the process looks like this:
A stack is created.
A new log stream is created to handle the custom resource logging.
The backing Lambda function is called to create the resource and the default behavior set its resource ID to be the log stream ID.
Some time passes
The stack gets updated with new parameters for the custom resource.
A new log stream is created to handle the custom resource logging, with a new ID.
The backing Lambda function is called to update the resource and the default behavior set a new resource ID to the new log stream ID.
CloudFormation understands that a new resource was created to replace the old resource and according to the policy it should delete the old resource during the "cleanup stage".
CloudFormation reaches the "cleanup stage" and sends a delete request with the old physical resource ID.
The solution, at least in my case where I never "replace the external resource" is to fabricate a unique identifier for the managed resource, provide it as the 5th parameter to the send response routine, and then stick to it - keep sending the same physical resource ID received in the update request, in the update response. CloudFormation will then never send a delete request during the "cleanup stage".
My implemenation (in JavaScript) looks something like this:
var resID = event.PhysicalResourceId || uuid();
...
sendResponse(event, context, status, resData, resID);
Another alternative - which would probably only make sense if you actually need to replace the external resource and want to adhere to the CloudFormation model of removing the old resource during cleanup - is to use the actual external resource ID as the physical resource ID, and when receiving a delete request - to use the provided physical resource ID to delete the old external resource. That is what CloudFormation designers probably had in mind in the first place, but their default sample implementation causes a lot of confusion - probably because the sample implementation doesn't manage a real resource and has no update functionality. There is also zero documentation in CloudFormation to explain the design and reasoning.
It’s important to understand the custom resource life cycle, to prevent your data from being deleted.
A very interesting and important thing to know is that CloudFormation
compares the physical resource id you returned by your Lambda function
to the one you returned previously. If the IDs are different,
CloudFormation assumes the resource has been replaced with a new
resource. Then something interesting happens.
When the resource update logic completes successfully, a Delete
request is sent with the old physical resource id. If the stack update
fails and a rollback occurs, the new physical resource id is sent in
the Delete event.
You can read more here about custom resource life cycle and other best practices
I am running a AWS CloudFormation stack that takes in some parameters and launches EC2 instances along with other AWS resources. The parameters are fed into the user data of the EC2 instance and based on that changes are made dynamically to the web application residing on the EC2 instance.
UserData:
Fn::Base64:
Fn::Join:
- ""
-
- "#!/bin/bash \n"
- "sh website-conf/website_mysql_config.sh "
- " -c \""
-
Ref: "CompanyName"
As shown in the example above, CompanyName is one of the many parameters passed to the userdata script. The problem is, when any one or multiple of parameters are updated, CloudFormation does not detect that and instead throws this error.
So, in order to update the stack, I have to edit the stack and make changes to the ASG so that CloudFormation 'sees' the changes and executes the stack update.
Is there a way to force CFN to update the stack when the parameters are updated?
CloudFormation will not update the stack unless there is a change in properties of the resources already created in the stack.
For example:
Consider I have a simple template to create a database where I need to pass 2 parameters:
db-name
region
Assume that I am using db-name passing it as value to DBInstanceIdentifier.
Also assume that I am not using the input parameter region for any purpose in creation of resources (or its properties) of the stack in any way.It is more of a dummy parameter I keep for readability purpose.
I passed (TEST-DB1, us-east-1) as input parameters to the CloudFormation template and successfully created the resources.
Scenario-1:
Now if I update the stack(still using the existing template) and just change the input parameters to (TEST-DB2, us-east-1). ie: changing just the db-name and not the region. Then CloudFormation will detect that, this parameter update, results in change in properties of running resource(s) of the stack and will compute and display the modifications as a change set.
Scenario-2:
Suppose I make another update(still using the existing template) property and just change the input parameters to (TEST-DB1, us-east-2). ie: changing just the region and not the db-name. Then CloudFormation will detect that, this parameter update, result in NO change in properties of running resource(s) of the stack will show the Error creating change set.
Bottomline:
Your change in input parameter must result in an update/replacement of any resources(or its attributes like security-groups,port etc..) of the stack. Then AWS CloudFormation will display them as Change Sets for your review. Also, the method (update or replacement) AWS CloudFormation uses depends on which property you update for a given resource type.
Your parameter "CompanyName" is not making any changes to the running
resources of the stack. Hence it is reporting as Error creating
change set. You need to use it to create any resource/resource properties of the stack. Then CloudFormation will detect the change-sets when you modify it. The same applies for any other input-parameters which you use.
Use the AWS CLI Update-Stack command. If you use the AWS CLI you can inject parameters into your stack so any change to any of the parameters result in a new stack. I do this myself to inject the Git/version commit ID into UserData so simply committing changes to the stack's JSON/Yaml to Git will allow stack updates. Any change to the parameters file will allow stack updates, even just a comment. I reference my Git commit ID in UserData the same way you are referencing Ref:CompanyName so when I change the Git commit ID the userData section is updated on stack updates.
Update Stack Command
aws cloudformation update-stack --stack-name MyStack --template-body file:///Users/Documents/Git/project/cloudformation/stack.json --parameters file:///Users/Documents/Git/project/cloudformation/parameters/stack-parameters.dev.json --capabilities CAPABILITY_IAM
Process
With this approach you make your parameters changes to the parameters json or yaml file then check it into version control. Now if you use a build server you can update your stack by checking out master and just running that one line above. Using AWS CodeBuild makes this easy so you don't need jenkins.
The answer of your problem is already answered with this state, CloudFormation will not update the stack unless there is a change in properties of the resources already created in the stack.
And for the answer for your question, please check the explanation below.
There is a way to force Cloudformation to update the stack using the AWS::CloudFormation::Init.
By using cfn-init, each instance can update itself when it detect the change that made by AWS::CloudFormation::Init in metadata.
There is a concept that we must understand first, that is the difference between UserData and metadata, at least under the AWS::CloudFormation::Init case.
Userdata: Will be only called once when the instance is being launch for the first time (this including update that need the instance to be replaced). So, if you update the stack (not creating a new one), even if you change the parameter value, it won't change anything if you call the parameter under UserData.
Metadata: Can be updated anytime. To make it works, you have to make sure that the daemon that detect the metadata changed is running (the daemon is called the cfn-hup)
If you already use the Metadata and AWS::CloudFormation::Init, the data is not immediately being updated. As far I know, here is the condition the data to be change after change the Metadata value.
Reboot the instance
Run cfn-init command again with it's parameter
Waiting about 15 minutes, because the daemon to check the change in Metadata is checking the change once in 15 minutes.
I have a cloudFormation stack which is in an active state. I executed a change set on the specific stack.
Once executed, the change no longer appears as a value on the CloudFormation console. However if I do a describe-change-set operation with the change set ARN, I get the details of the changeset.
CloudFormation has an list-stacks API which lists deleted and active stacks. Is there any API to list inactive/expired/deleted change sets? Is that even possible?
No, there doesn't appear to be any API that can list inactive (but not deleted) CloudFormation Change Sets.
The ListChangeSets API is described as follows:
Returns the ID and status of each active change set for a stack.
Once a Change Set is executed, it is not deleted, but enters the EXECUTE_COMPLETE state. The Change Set is still referenced by the stack in the Stack.ChangeSetId property returned by the DescribeStacks API, as used by the CloudFormation Console's Change Sets tab, though it no longer appears in the ListChangeSets output.
Beyond that, since Change Sets become unusable as soon as the stack has been updated, there's not really any other use for them. I'm also surprised they are still retained (indefinitely?). I wouldn't be surprised if a more explicit/controllable lifecycle for inactive Change Sets is eventually added, since this feature is still less than a year old.