AWS Nested Stacks - Referencing a Parent Stack's Resource - amazon-web-services

I'm trying to pass resources (ApiGatewayRestApi and a custom authorizer) to a nested stack through stack parameters, however, they continually fail with Embedded stack <stack_name> was not successfully created: The following resource(s) failed to create. Here's my set up in Serverless:
Parent Stack
{
...
"NestedStack": {
"Type": "AWS::CloudFormation::Stack",
"Properties": {
"Parameters": {
"ServerlessDeploymentBucket": {
"Ref": "ServerlessDeploymentBucket"
},
"ApiGatewayRestApi": {
"Ref": "ApiGatewayRestApi"
},
"AuthDashjwtApiGatewayAuthorizer": {
"Ref": "AuthDashjwtApiGatewayAuthorizer"
},
},
"TemplateURL": "..."
}
},
}
Nested Stack
{
"AWSTemplateFormatVersion": "2010-09-09",
"Description": "Nested Stack",
"Parameters": {
"ServerlessDeploymentBucket": { "Type": "String" },
"ApiGatewayRestApi": {
"Description": "Rest API",
"Type": "String"
},
"AuthDashjwtApiGatewayAuthorizer": { "Type": "String" },
},
"Resources": {
"ApiGatewayMethodEventsEventidVarStreamsPost": {
"Type": "AWS::ApiGateway::Method",
"Properties": {
"HttpMethod": "POST",
"RequestParameters": {},
"ResourceId": { "Ref": "ApiGatewayResourceEventsEventidVarStreams" },
"RestApiId": { "Ref": "ApiGatewayRestApi" },
"AuthorizationType": "CUSTOM",
"AuthorizerId": { "Ref": "AuthDashjwtApiGatewayAuthorizer" },
...
}
...
}
...
}
Am I not referencing or passing in parameters correctly?
Update based on comments
Unless I'm missing something, the only error message in the CF section of the console is:
Embedded stack <stac_name> was not successfully created: The
following resource(s) failed to create: [PatchDasheventLogGroup,
PostDashstreamLogGroup, GetDashstreamsLogGroup, GetDasheventsLogGroup,
ApiGatewayRestApi, GetDasheventLogGroup, PostDasheventLogGroup,
AuthDashjwtApiGatewayAuthorizer]
as far as log groups go, they look like this:
"GetDasheventLogGroup": {
"Type": "AWS::Logs::LogGroup",
"Properties": {
"LogGroupName": "/aws/lambda/live-api-local-get-event"
}
}
Update 2
The log group issue was due to these logs being moved from the parent stack to the nested stack and needing a new name. In the LogGroup docs I found:
If you specify a name, you cannot perform updates that require replacement of this resource. You can perform updates that require no or some interruption. If you must replace the resource, specify a new name.
This looks like it may have solved the issue... Some more testing is needed to confirm!

The comment from #speshak eventually lead me to the answer. I didn't need to filter by Failed state, but rather Deleted. This allowed me to see the logs for the nested stack that was created, then deleted with more specific messaging.
What this ended up showing me is that the update-stack process was applying the nested stacks to my current set up before removing all the resources from, what would become, the root stack. So the true problem was that I was accidentally trying to create duplicate resources -- AWS saw a resource in a nested stack that matched the root stack, and kicked out with a validation error even though the resource would have been removed from the root stack... eventually.

Related

How do I successfully retrieve an ALB ListenerArn with CloudFormation to setup ListenerRules?

I'm starting to think there is a fundamental flaw in AWS Cloudformation Template validation/resource lookup related to "Type": "AWS::ElasticLoadBalancingV2::ListenerRule", resources.
Specifically, every time I try to create a new ListenerRule for known working Listeners, Cloudformation errors out with
Unable to retrieve ListenerArn attribute for AWS::ElasticLoadBalancingV2::Listener, with error message One or more listeners not found (Service: ElasticLoadBalancingV2, Status Code: 400, Request ID: c6914f71-074c-4367-983a-bcf1d8fd1350, Extended Request ID: null)
Upon testing, I can make it work by hardcoding the ListenArn attribute in my template, but that's not a solution since the template is used for multiple Stacks with different resources.
Below are the relevant parts of the template:
"WLBListenerHttp": {
"Type": "AWS::ElasticLoadBalancingV2::Listener",
"Properties": {
"DefaultActions": [{
"Type": "forward",
"TargetGroupArn": { "Ref": "WLBTargetGroupHttp" }
}],
"LoadBalancerArn": { "Ref": "WebLoadBalancer" },
"Port": 80,
"Protocol": "HTTP"
}
},
"ListenerRuleHttp": {
"DependsOn": "WLBListenerHttp",
"Type": "AWS::ElasticLoadBalancingV2::ListenerRule",
"Properties": {
"Actions": [{
"Type": "fixed-response",
"FixedResponseConfig": { "StatusCode": "200" }
}],
"Conditions": [{
"Field": "host-header",
"HostHeaderConfig": { "Values": ["domain*"] }
}, {
"Field": "path-pattern",
"PathPatternConfig": { "Values": ["/path/to/respond/to"] }
}],
"ListenerArn": { "Fn::GetAtt": ["WLBListenerHttp", "ListenerArn"] },
"Priority": 1
}
},
Per the documentation on listeners, Fn::GetAtt or Ref should both return the ListenerARN:
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-elasticloadbalancingv2-listener.html
"Return values
Ref
When you pass the logical ID of this resource to the intrinsic Ref function, Ref returns the Amazon Resource Name (ARN) of the listener.
For more information about using the Ref function, see Ref.
Fn::GetAtt
The Fn::GetAtt intrinsic function returns a value for a specified attribute of this type. The following are the available attributes and sample return values.
For more information about using the Fn::GetAtt intrinsic function, see Fn::GetAtt.
ListenerArn
The Amazon Resource Name (ARN) of the listener."
I've tried both "ListenerArn": { "Fn::GetAtt": ["WLBListenerHttp", "ListenerArn"] }, and "ListenerArn": { "Ref": "WLBListenerHttp"}, with no success, resulting in the error noted. If I hardcode the Arn "ListenerArn": "arn::", with the full Arn, it works fine.
As it turns out, my syntax was perfectly fine. However, what I didn't realize is that while the WLBListenerHttp resource existed, it was not actually the same ARN as the one created by CloudFormation. Apparently, someone accidentally deleted it at some point without telling us and then manually recreated it. This left the account in a broken state where CloudFormation had an ARN recorded for the listener from when it was created, but it was truly no longer valid since the new resource had a new ARN.
The solution to this was to delete the offending resource manually, then change the name of it slightly in our CloudFormation templates so it would create a new one.

Updating an AWS::EC2::EIPAssociation via change set

I keep running into the following error when attempting to update an EIPAssociation via a change set:
Interface: [eni-04740b9f34d1d75dc] in use. (Service: AmazonEC2; Status Code: 400; Error Code: InvalidNetworkInterface.InUse; Request ID: 3bfa13f2-abe5-4bb5-80aa-183605d3dfde)
Basically, I changed my resource to reference a value passed in as a parameter instead of an EIP that is created in the same CFT.
Old:
"EIPAssociation": {
"Type": "AWS::EC2::EIPAssociation",
"Properties": {
"AllocationId": {
"Fn::GetAtt": [
"EIP",
"AllocationId"
]
},
"NetworkInterfaceId": {
"Ref": "NetworkInterface"
}
},
"DependsOn": "EC2Instance"
},
New:
"EIPAssociation": {
"Type": "AWS::EC2::EIPAssociation",
"Properties": {
"AllocationId": {
"Ref": "EIPAllocationID"
},
"NetworkInterfaceId": {
"Ref": "NetworkInterface"
}
},
"DependsOn": "EC2Instance"
},
In both cases, "NetworkInterface" is created within the CFT while in the first example "EIP" is created within the CFT and the second example "EIPAllocationID" is passed into the CFT as a parameter. I can log into the console, unassociate the old EIP and associate the new one no problem. But when I attempt to do these via change set I get the annoying ENI in use error. Happens whether the instance is on or off.
Any idea how I proceed with this or what I am doing incorrectly?

AWS Cloudformation "include" transform error

I have some template and other partial definition I'd like to include in main template definition. The sample is below (main template).
{
"AWSTemplateFormatVersion": "2010-09-09",
"Description": "",
"Parameters": {
"Environment": {
"Type": "String",
"Description": "Specify Environment: prod | dev ",
"AllowedValues": [ "prod", "dev" ],
"Default": "dev"
}
},
"Transform": {
"Name": "AWS::Include",
"Parameters": {
"Location": "s3://some-s3-local-bucket/part-1.json"
}
},
"Resources": {
},
"Outputs": {
}
}
Below is a definition of the part to include in main template
{
"AWSTemplateFormatVersion": "2010-09-09",
"Description": "",
"Resources": {
"hellobucket": {
"Type": "AWS::S3::Bucket",
"Properties": {
"BucketName": { "Fn::Sub": "testbucket-${Environment}" }
}
}
},
"Outputs": {
}
}
When I try to create stack based on such definitions I receive strange error like "Template parameters modified by transform". I don't know/see any reason any parameter could be considered as "modified".
I don't want to create many "nested" stacks, because there is a limit on aws in number of stacks I can create so the goal is to split stack definition into many (well managable) smaller files and based on them to create ONE stack with all the related resources.
How to properly decompose bigger stack definition into smaller files?
I haven't done this before but it might be because you are using the transform to pull in a template that creates an s3 bucket BUT the template you are pulling into the original one has all it's parameter fields etc. empty. I think this is what the error message - Template parameters modified by transform - is relating to. Try removing the empty parameters entry from the S3 template to see if that helps.

Stack is hung using CloudFormation with SNS-backed CustomResources

I'm trying to learn working of CustomResources in CloudFormation Template. Created simple template to create s3 bucket. But on creating stack, it remains in Create in progress state for long time and no bucket is created.
Is there anything, I'm missing in below validated template:
{
"AWSTemplateFormatVersion" : "2010-09-09",
"Description" : "Building A bucket With customeResources in CloudFormation",
"Parameters" : {
"NewBucket": {
"Default": "",
"Description": "S3 bucket containing customer assets",
"Type": "String"
}
},
"Conditions": {
"NewBucket": {
"Fn::Not": [
{
"Fn::Equals": [
{
"Ref": "NewBucket"
},
""
]
}
]
}
},
"Resources" : {
"CustomResource": {
"Properties": {
"S3Bucket": {
"Ref": "NewBucket"
},
"ServiceToken": "SNS topic ARN"
},
"Type": "AWS::CloudFormation::CustomResource"
}
},
"Outputs": {
"BucketName": {
"Value": {
"Fn::GetAtt": [ "CustomResource", {"Ref": "NewBucket"} ]
}
}
}
}
It would appear that your SNS-backed custom resource is not sending a response back to cloud formation, and it is stuck waiting for that response.
From Amazon Simple Notification Service-backed Custom Resources:
The custom resource provider processes the data sent by the template
developer and determines whether the Create request was successful.
The resource provider then uses the S3 URL sent by AWS CloudFormation
to send a response of either SUCCESS or FAILED.
When the request is made to the SNS service provider, it include the following object:
{
"RequestType": "Create",
"ServiceToken": "arn:aws:sns:us-west-2:2342342342:Critical-Alerts-development",
"ResponseURL": "https:\/\/cloudformation-custom-resource-response-uswest2.s3-us-west-2.amazonaws.com\/arn%3Aaws%3Acloudformation%3Aus-west-2%3A497903502641%3Astack\/custom-resource\/6bf07a80-d44a-11e7-84df-503aca41a029%7CCustomResource%7C5a695f41-61d7-475b-9110-cdbaec04ee55?AWSAccessKeyId=AKIAI4KYMPPRGIACET5Q&Expires=1511887381&Signature=WmHQVqIDCBwQSfcBMpzTfiWHz9I%3D",
"StackId": "arn:aws:cloudformation:us-west-2:asdasdasd:stack\/custom-resource\/6bf07a80-d44a-11e7-84df-503aca41a029",
"RequestId": "5a695f41-61d7-475b-9110-cdbaec04ee55",
"LogicalResourceId": "CustomResource",
"ResourceType": "AWS::CloudFormation::CustomResource",
"ResourceProperties": {
"ServiceToken": "arn:aws:sns:us-west-2:234234234:Critical-Alerts-development",
"S3Bucket": "test-example-com"
}
}
You will need to send a success/fail response to the ResponseURL provided in the event for Cloud Formation to continue processing.
I would also note that the bucket will not be created unless your custom service provider creates it. The Custom Resource function is only sending the request to the provider.

How to check if specific resource already exists in CloudFormation script

I am using cloudformation to create a stack which inlcudes an autoscaled ec2 instance and an S3 bucket. For the S3 bucket I have DeletionPolicy set to Retain, which works fine, until I want to rerun my cloudformation script again. Since on previous runs, the script created the S3 bucket, it fails on subsequent runs saying my S3 bucket already exists. None of the other resources of course get created as well. My question is how do I check if my S3 bucket exists first inside the cloudformation script, and if it does, then skip creating that resources. I've looked in conditions in the AWS, but it seems all parameter based, I have yet to find a function which checks from existing resources.
There is no obvious way to do this, unless you create the template dynamically with an explicit check. Stacks created from the same template are independent entities, and if you create a stack that contains a bucket, delete the stack while retaining the bucket, and then create a new stack (even one with the same name), there is no connection between this new stack and the bucket created as part of the previous stack.
If you want to use the same S3 bucket for multiple stacks (even if only one of them exists at a time), that bucket does not really belong in the stack - it would make more sense to create the bucket in a separate stack, using a separate template (putting the bucket URL in the "Outputs" section), and then referencing it from your original stack using a parameter.
Update November 2019:
There is a possible alternative now. On Nov 13th AWS launched CloudFormation Resource Import. With that feature you can now creating a stack from existing resources. Currently not many resource types are supported by this feature, but S3 buckets are.
In your case you'd have to do it in two steps:
Create a template that only contains the preexisting S3 bucket using the "Create stack" > "With existing resources (import resources)" (this is the --change-set-type IMPORT flag in the CLI) (see docs)
Update the the template to include all resources that don't already exist.
As they note in their documentation; this feature is very versatile. So it opens up a lot of possibilities. See docs for more info.
Using cloudformation you can use Conditions
I created an input parameter "ShouldCreateBucketInputParameter" and then using CLI you just need to set "true" or "false"
Cloudformation json file:
{
"AWSTemplateFormatVersion": "2010-09-09",
"Transform": "AWS::Serverless-2016-10-31",
"Description": "",
"Parameters": {
"ShouldCreateBucketInputParameter": {
"Type": "String",
"AllowedValues": [
"true",
"false"
],
"Description": "If true then the S3 bucket that will be proxied will be created with the CloudFormation stack."
}
},
"Conditions": {
"CreateS3Bucket": {
"Fn::Equals": [
{
"Ref": "ShouldCreateBucketInputParameter"
},
"true"
]
}
},
"Resources": {
"SerialNumberBucketResource": {
"Type": "AWS::S3::Bucket",
"Condition": "CreateS3Bucket",
"Properties": {
"AccessControl": "Private"
}
}
},
"Outputs": {}
}
And then (I am using CLI do deploy the stack)
aws cloudformation deploy --template ./s3BucketWithCondition.json --stack-name bucket-stack --parameter-overrides ShouldCreateBucketInputParameter="true" S3BucketNameInputParameter="BucketName22211"
Just add an input parameter to the CloudFormation template to indicate that an existing bucket should be used.... unless you don't already know at the time when you are going to use the template? Then you can either add a new resource or not based on the parameter value.
If you do updates, (potentially of stacks within stacks aka Nested Stacks), the unchanged parts don't get updated.
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-stack.html?icmpid=docs_cfn_console_designer
You can then set policies as mentioned to prevent deletion. [remember 'cancel update' permissions for rollbacks]
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/protect-stack-resources.html
There is also Cross-Stack Output to be aware of by adding Export Names to the Stack Outputs.
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/outputs-section-structure.html
Walkthrough...
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/walkthrough-crossstackref.html
Then you need to use Fn::ImportValue ...
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/intrinsic-function-reference-importvalue.html
It implies one could use a network stack name parameter.
Unfortunately you get an error like this when you try them in Conditions.
Template validation error: Template error: Cannot use Fn::ImportValue
in Conditions.
Or in the Parameters?
Template validation error: Template format error: Every Default member
must be a string.
Also this can happen while trying...
Template format error: Output ExportOut is malformed. The Name field
of Export must not depend on any resources, imported values, or
Fn::GetAZs.
So you can't stop it making the existing resource again from the same file. Only when putting it into another stack and using the export import reference.
But if you separate the two then there is a dependency that will stop and rollback for instance a dependency's deletion, thanks to the reference via the ImportValue function.
Example Given here is:
First Make a Group Template
{
"AWSTemplateFormatVersion": "2010-09-09",
"Metadata": {
"AWS::CloudFormation::Designer": {
"6927bf3d-85ec-449d-8ee1-f3e1804d78f7": {
"size": {
"width": 60,
"height": 60
},
"position": {
"x": -390,
"y": 130
},
"z": 0,
"embeds": []
},
"6fe3a2b8-16a1-4ce0-b412-4d4f87e9c54c": {
"source": {
"id": "ac295134-9e38-4425-8d20-2c50ef0d51b3"
},
"target": {
"id": "6927bf3d-85ec-449d-8ee1-f3e1804d78f7"
},
"z": 1
}
}
},
"Resources": {
"TestGroup": {
"Type": "AWS::IAM::Group",
"Properties": {},
"Metadata": {
"AWS::CloudFormation::Designer": {
"id": "6927bf3d-85ec-449d-8ee1-f3e1804d78f7"
}
},
"Condition": ""
}
},
"Parameters": {},
"Outputs": {
"GroupNameOut": {
"Description": "The Group Name",
"Value": {
"Ref": "TestGroup"
},
"Export": {
"Name": "Exported-GroupName"
}
}
}
}
Then make a User Template that needs the group.
{
"AWSTemplateFormatVersion": "2010-09-09",
"Metadata": {
"AWS::CloudFormation::Designer": {
"ac295134-9e38-4425-8d20-2c50ef0d51b3": {
"size": {
"width": 60,
"height": 60
},
"position": {
"x": -450,
"y": 130
},
"z": 0,
"embeds": [],
"isrelatedto": [
"6927bf3d-85ec-449d-8ee1-f3e1804d78f7"
]
},
"6fe3a2b8-16a1-4ce0-b412-4d4f87e9c54c": {
"source": {
"id": "ac295134-9e38-4425-8d20-2c50ef0d51b3"
},
"target": {
"id": "6927bf3d-85ec-449d-8ee1-f3e1804d78f7"
},
"z": 1
}
}
},
"Resources": {
"TestUser": {
"Type": "AWS::IAM::User",
"Properties": {
"UserName": {
"Ref": "UserNameParam"
},
"Groups": [
{
"Fn::ImportValue": "Exported-GroupName"
}
]
},
"Metadata": {
"AWS::CloudFormation::Designer": {
"id": "ac295134-9e38-4425-8d20-2c50ef0d51b3"
}
}
}
},
"Parameters": {
"UserNameParam": {
"Default": "testerUser",
"Description": "Username For Test",
"Type": "String",
"MinLength": "1",
"MaxLength": "16",
"AllowedPattern": "[a-zA-Z][a-zA-Z0-9]*",
"ConstraintDescription": "must begin with a letter and contain only alphanumeric characters."
}
},
"Outputs": {
"UserNameOut": {
"Description": "The User Name",
"Value": {
"Ref": "TestUser"
}
}
}
}
You will get
No export named Exported-GroupName found. Rollback requested by user.
if running User with no Group found Exported.
You could then use the Nested stack approach.
{
"AWSTemplateFormatVersion": "2010-09-09",
"Metadata": {
"AWS::CloudFormation::Designer": {
"66470873-b2bd-4a5a-af19-5d54b11f48ef": {
"size": {
"width": 60,
"height": 60
},
"position": {
"x": -815,
"y": 169
},
"z": 0,
"embeds": []
},
"ed1de011-f1bb-4788-b63e-dcf5494d10d1": {
"size": {
"width": 60,
"height": 60
},
"position": {
"x": -710,
"y": 170
},
"z": 0,
"dependson": [
"66470873-b2bd-4a5a-af19-5d54b11f48ef"
]
},
"c978f2d9-3fb2-4420-b255-74941f10a28a": {
"source": {
"id": "ed1de011-f1bb-4788-b63e-dcf5494d10d1"
},
"target": {
"id": "66470873-b2bd-4a5a-af19-5d54b11f48ef"
},
"z": 1
}
}
},
"Resources": {
"GroupStack": {
"Type": "AWS::CloudFormation::Stack",
"Properties": {
"TemplateURL": "https://s3-us-west-2.amazonaws.com/cf-templates-x-TestGroup.json"
},
"Metadata": {
"AWS::CloudFormation::Designer": {
"id": "66470873-b2bd-4a5a-af19-5d54b11f48ef"
}
}
},
"UserStack": {
"Type": "AWS::CloudFormation::Stack",
"Properties": {
"TemplateURL": "https://s3-us-west-2.amazonaws.com/cf-templates-x-TestUserFindsGroup.json"
},
"Metadata": {
"AWS::CloudFormation::Designer": {
"id": "ed1de011-f1bb-4788-b63e-dcf5494d10d1"
}
},
"DependsOn": [
"GroupStack"
]
}
}
}
Unfortunately you can still delete the User stack even though it was made by MultiStack in this example but with deletion policies and other things it just might help.
Then you are only Updating the various stacks it creates, and you won't do the Multi Stack if you're for instance reusing a Bucket.
Otherwise you'll be looking at APIs and scripts in various flavors.
If you're trying to incorporate some existing resources into CF, it is unfortunately not possible. If you just want a set of resources to be part of your template or not depending on the value of some parameters, you can use Conditions. But they don't change the nature of CF itself, and only work to determine which resources are desired, not what actions will be taken, and cannot see whether a resource exists or not beforehand.
Something not explicitly stated. If your first deployment fails, resources will be deleted unless you have at retention policy. In this case it is safe to delete the resource in question manually. Next deployment will recreate it without generating error that resource already exists.