Cloudformation: The resource you requested does not exist - amazon-web-services

I have a cloudformation stack which has a Lambda function that is mapped as a trigger to an SQS queue.
What happened was that I had to delete the mapping and create it again manually cos I wanted to change the batch size. Now when I want to update the mapping the cloudformation throws an error with The resource you requested does not exist. message.
The resource mapping code looks like this:
"EventSourceMapping":{
"Properties":{
"BatchSize":5,
"Enabled":"true",
"EventSourceArn":{
"Fn::GetAtt":[
"ProcessorQueue",
"Arn"
]
},
"FunctionName":{
"Fn::GetAtt":[
"ProcessorLambda",
"Arn"
]
}
},
"Type":"AWS::Lambda::EventSourceMapping"
}
I know that I've deleted the mapping cloudformation created initially and added it manually which is causing the issue. How do I fix this? Cos I cannot push any update now.
Please help

What you did, from my perspective, it is a mistake. When you use Cloud Formation you are not suppose to apply changes manually. You can, and maybe that's fine since one may don't care about the stack once is created. But since you are trying to update the stack, this tells me that you want to keep the stack and update it on a time basis.
To narrow down your problem, first let make clear that the manually-created mapping is out of sync with your cloud formation stack. So, from a cloud formation perspective, it doesn't matter if you keep that mapping or not. I'm wondering, what would happen if you keep the manually-created mapping and create a new from Cloud Formation? Maybe it will complain, since you would have repeated mappings for the same pair of (lambda,queue). Try this:
Create a change for your stack, where you completely remove the EventSourceMapping resource from your script. This step is to basically clean loosing references. Apply the change set.
Then, and this is where I think you may get some kind of issue, add back again EventSourceMapping to your stack.
If you get errors in the step 2, like "this mapping already exists", you will have to remove the manually-created mapping from the console. And then try again step 2.

You probably know now that you should not have removed the resource manually. If you change the CF, you can update it without changing resources which did not change in CF. You can try to replace the resource with the exact same physical name https://aws.amazon.com/premiumsupport/knowledge-center/failing-stack-updates-deleted/ The other option is to remove the resource from CF, update, and then add it back and update again - from the same doc.

While comments above are valid, I found it interesting, that no one mentioned much simpler option: using SAM commands (sam build/sam deploy). It's understandable that during the development process and designing the architecture, there might be flaws and situations where manual input in the console is necessary, therefore there's something I reference to every time I have similar issue.
Simply comment out the chunk of code that is creating troubles, run sam build/deploy on top of it, CloudFormation stack will recognize that the resource no longer in the template and will delete it.
Now, since the resource is no longer in the architecture anyway(removed manually prior), it will have no issues passing the step and successfully updating the stack.
Then simply uncomment, make any necessary changes (if any) and deploy.
Works every time.

Related

Errors during deployment to AWS using Terraform (cdktf)

I am trying to create or update Lambdas on AWS using the Terraform CDKTF. During deployment, I am getting the error of
"An event source mapping with SQS arn (\" arn:aws:sqs:eu-west-2:*******:*****-*****-******** \") and function (\" ******-******-****** \") already exists. Please update or delete the existing mapping with UUID *******-****-****-****-***********"
**** are sensitive info I have swapped out.
Some of our Lambdas are called via SQS, which is what this mapping is referring to. I assumed the first fix would be to remove the mappings that might already exist (on a previous deployment that might have partly gone through), but I am unsure where to find them, nor if they are even available to delete. I originally assumed by calling cdktf deploy it would update these mappings and not throw the err at all.
Does anyone have any advice?
Your diagnosis seems right, there might be some stray resources left behind due to an aborted / unfinished Terraform run. You should be able to clean up after these runs by running terraform destroy in the stack directory ./cdktf.out/stacks/..../. That should delete all previously existing resources created through this Terraform stack.

Export/Outputs that don't exist preventing stack from updating/deleting

Using serverless to deploy to AWS.
I created a Cognito user pool via serverless then realised I wanted to change it's attributes.
I couldn't deploy because you can't update attributes on an existing user pool.
"No problem - I'll just delete it and make it again" I thought. So I did.
But I had created two Outputs that referencing Client ID and Pool ID so now I get this:
Export alpha-UserPoolId cannot be deleted as it is in use by alpha-Stack
I can't see any way to remove theese references manually via the AWS console.
Anyone know what I can do to remove these dead references?
There's no option to manually remove an Output and I tried editing the template but it didn't seem to actually do anything.
Thanks
[EDIT: Check comments for full details on solution]
You have to edit the importing stack to not rely on these values, afterwards you can remove them.
As long as there is an Fn::ImportValue somewhere, it won't let you delete the export.
From the docs:
The following restrictions apply to cross stack references
...
You can't delete a stack if another stack references one of its outputs.
You can't modify or remove an output value that is referenced by another stack.

User X not authorized to perform cloudformation:CreateChangeSet on resource arn:cloudformation:ap-xx-x:transform:Serverless-2016-10-31

Beginning with a new stack I get the error message as in the title.
I am using SAM, and I am confused, why it wants to update the macro.
I thought, this macro is provided by aws and I wonder why it is requesting to modify it.
My template spins up a lambda, a database and a REST api, but does even try to touch existing macros.
My template did contain the TableName tag for a DynamoDb.
As I am aware, named tables cannot be updated, if resource replacement required. I was not trying to do updates on that resource though.
The table existed before I cloudformed that new stack though.

How to conditionally update a resource in Terraform

Seems it's common practice to make use of count on a resource to conditionally create it in Terraform using a ternary statement.
I'd like to conditionally update an AWS Route 53 entry based on a push_to_prod variable. Meaning I don't want to delete the resource if I'm not pushing to production, I only want to update it, or leave the CNAME value as it is.
Has anyone done something like this before in Terraform?
Currently as it stands interpolation syntax isn't supported in lifecycle tags. You can read more here. Which will make this harder because you could use the "Prevent Destroy". However, without more specifics I am going to take my best guess on how to get your there.
I would use the allow_overwrite property on the Route53 record and set that based on your flag. That way if you are pushing to prod you can set it it false. Which should trigger creating a new one. I haven't tested that.
Also note that if you don't make any changes to the Route53 resource it should trigger any changes in Terraform to be applied. So updating any part of the record will trigger the deployment.
You may want to combine this with some lifecycle events, but I don't have enough time to dig into that specific resource and how it happens.
Two examples I can think of are:
type = "${var.push_to_prod == "true" ? "CNAME" : var.other_value}" - this will have a fixed other_value, there is no way to have terraform "ignore" the resource once it's being managed by terraform.
or
type = "${var.aws_route53_record_type}" and you can have dev.tfvars and prod.tfvars, with aws_route53_record_type defined as whatever you want for dev and CNAME for prod.
The thing is with what you're trying to do, "I only want to update it, or leave the CNAME value as it is.", that's not how terraform works. Terraform either manages the resource for you or it doesn't. If it's managing it, it'll update the resource based on the config you've defined in your .tf file. If it's not managing the resource it won't modify it. It sounds like what you're really after is the second solution where you pass in two different configs from your .tfvars file into your .tf file and based off the different configs, different resources are created. You can couple this with count to determine if a resource should be created or not.

How do I force a CloudFormation stack to update when the parameter is updated?

I am running a AWS CloudFormation stack that takes in some parameters and launches EC2 instances along with other AWS resources. The parameters are fed into the user data of the EC2 instance and based on that changes are made dynamically to the web application residing on the EC2 instance.
UserData:
Fn::Base64:
Fn::Join:
- ""
-
- "#!/bin/bash \n"
- "sh website-conf/website_mysql_config.sh "
- " -c \""
-
Ref: "CompanyName"
As shown in the example above, CompanyName is one of the many parameters passed to the userdata script. The problem is, when any one or multiple of parameters are updated, CloudFormation does not detect that and instead throws this error.
So, in order to update the stack, I have to edit the stack and make changes to the ASG so that CloudFormation 'sees' the changes and executes the stack update.
Is there a way to force CFN to update the stack when the parameters are updated?
CloudFormation will not update the stack unless there is a change in properties of the resources already created in the stack.
For example:
Consider I have a simple template to create a database where I need to pass 2 parameters:
db-name
region
Assume that I am using db-name passing it as value to DBInstanceIdentifier.
Also assume that I am not using the input parameter region for any purpose in creation of resources (or its properties) of the stack in any way.It is more of a dummy parameter I keep for readability purpose.
I passed (TEST-DB1, us-east-1) as input parameters to the CloudFormation template and successfully created the resources.
Scenario-1:
Now if I update the stack(still using the existing template) and just change the input parameters to (TEST-DB2, us-east-1). ie: changing just the db-name and not the region. Then CloudFormation will detect that, this parameter update, results in change in properties of running resource(s) of the stack and will compute and display the modifications as a change set.
Scenario-2:
Suppose I make another update(still using the existing template) property and just change the input parameters to (TEST-DB1, us-east-2). ie: changing just the region and not the db-name. Then CloudFormation will detect that, this parameter update, result in NO change in properties of running resource(s) of the stack will show the Error creating change set.
Bottomline:
Your change in input parameter must result in an update/replacement of any resources(or its attributes like security-groups,port etc..) of the stack. Then AWS CloudFormation will display them as Change Sets for your review. Also, the method (update or replacement) AWS CloudFormation uses depends on which property you update for a given resource type.
Your parameter "CompanyName" is not making any changes to the running
resources of the stack. Hence it is reporting as Error creating
change set. You need to use it to create any resource/resource properties of the stack. Then CloudFormation will detect the change-sets when you modify it. The same applies for any other input-parameters which you use.
Use the AWS CLI Update-Stack command. If you use the AWS CLI you can inject parameters into your stack so any change to any of the parameters result in a new stack. I do this myself to inject the Git/version commit ID into UserData so simply committing changes to the stack's JSON/Yaml to Git will allow stack updates. Any change to the parameters file will allow stack updates, even just a comment. I reference my Git commit ID in UserData the same way you are referencing Ref:CompanyName so when I change the Git commit ID the userData section is updated on stack updates.
Update Stack Command
aws cloudformation update-stack --stack-name MyStack --template-body file:///Users/Documents/Git/project/cloudformation/stack.json --parameters file:///Users/Documents/Git/project/cloudformation/parameters/stack-parameters.dev.json --capabilities CAPABILITY_IAM
Process
With this approach you make your parameters changes to the parameters json or yaml file then check it into version control. Now if you use a build server you can update your stack by checking out master and just running that one line above. Using AWS CodeBuild makes this easy so you don't need jenkins.
The answer of your problem is already answered with this state, CloudFormation will not update the stack unless there is a change in properties of the resources already created in the stack.
And for the answer for your question, please check the explanation below.
There is a way to force Cloudformation to update the stack using the AWS::CloudFormation::Init.
By using cfn-init, each instance can update itself when it detect the change that made by AWS::CloudFormation::Init in metadata.
There is a concept that we must understand first, that is the difference between UserData and metadata, at least under the AWS::CloudFormation::Init case.
Userdata: Will be only called once when the instance is being launch for the first time (this including update that need the instance to be replaced). So, if you update the stack (not creating a new one), even if you change the parameter value, it won't change anything if you call the parameter under UserData.
Metadata: Can be updated anytime. To make it works, you have to make sure that the daemon that detect the metadata changed is running (the daemon is called the cfn-hup)
If you already use the Metadata and AWS::CloudFormation::Init, the data is not immediately being updated. As far I know, here is the condition the data to be change after change the Metadata value.
Reboot the instance
Run cfn-init command again with it's parameter
Waiting about 15 minutes, because the daemon to check the change in Metadata is checking the change once in 15 minutes.