Is there anyway to make CloudFormation parameter dynamic? I know about the System Manager Parameter, but again I have to change its value manually. I want to use somehow the result of the API call or script(Bash, python) in my CloudFormation resources
for example, as part of the parameter, run a API call to get back some data (any data) and then use/reference the result into the resources, and all in one template.
You can use Cloudformation Custom resource to achieve similar effect, with some caveats.
As an example we can use AWS CDK, which provides a module to create custom resources, and even has a wrapper specifically designed to call AWS API and return the results: https://docs.aws.amazon.com/cdk/api/latest/docs/#aws-cdk_custom-resources.AwsSdkCall.html
Some things to remember:
Custom resource needs to return value in form {'PhysicalResourceId': ..., Data: {"MyAttribute": ...}} in order to support using !GetAtt MyResource.MyAttribute style of reference
Like any other CF resource, Custom resource is not triggered on every update, only if one of the parameters of the resource has changed. So if you supplied some parameter to your API call on stack creation, unless you change value, no update will happen and API call will not be triggered.
Related
I'm building a contact flow which creates a block of tasks in pairs. Ideally, one task in a pair should include a reference to its partner in its description.
I've almost achieved this: When creating the second task, I add a reference type URL and I'm using the $.Task.ContactId attribute, prefixed with the access URL for my instance, i.e.
https://<myurl>.my.connect.aws/connect/contact-trace-records/details/$.Task.ContactID
I'd like to deploy this in more than one Connect instance without having to keep manually editing the contact flow. Is there any way I can specify the access URL as a parameter?
For dynamic attributes like this, you need to call a lambda. The lambda can return the appropriate value and then you can use that returned value to set your reference URL attribute.
If I was writing the lambda I'd send through the $.Task.ContactID then use Details.ContactData.InstanceARN from the event that gets passed to the lambda to lookup the instance alias via the Connect API DescribeInstance function. You can then build the url in the lambda and send it back.
More info about using lambdas in a contact flow can be found here
In my stack definition I pull in a number of parameters from SSM Parameter Store...
const p1 = ssm.StringParameter.fromStringParameterAttributes( ... )
const p2 = ssm.StringParameter.fromStringParameterAttributes( ... )
I then pass them along to the relevant lambdas as environment vars...
environment: {
PARAM_ONE: p1.stringValue
PARAM_TWO: p2.stringValue
}
However I don't want all of those parameters to be mandatory. I would like the ones that exist to be passed in as env vars, and the ones that don't to just remain undefined as my app has defaults for them anyway. However, trying to inspect the value of p1.stringValue just gives me a Token, not a value, so I can't do any logic based on it's presence or absence: https://docs.aws.amazon.com/cdk/latest/guide/tokens.html
If I ask for the parameter and it is not defined in SSM Parameter Store I then get an error that I can't catch or ignore when it tries to build the changeset and the deployment fails...
MyApp: creating CloudFormation changeset...
❌ MyAppStack failed: Error [ValidationError]: Unable to fetch parameters [/myapp/param1,/myapp/param2] from parameter store for this account.
So how can I deal with SSM parameters which may or may not exist at deploy time?
I assume you are only grabbing the manager in your import, not the actual values inside your secrets. If this is the case, then your best bet is to leverage the SDK to do this for you - a simple call using the SDK (which will be run during the synth stage of a cdk deploy or cdk synth) to see if said SMM fields/groups exist. If they do, go ahead and import them.
I do something very similar with Layers - the from methods for layers require the version number - that may change at any time. So i have a small function that gets the latest version number of a given layer using the SDK and i can then use that to import the layer definition into my stack.
If you are trying to get the actual secret inside the secret manager parameter ... that is better suited to outside the CDK for most scenarios - done in the exact location you need the secrets so you dont end up with secret value in plain text somewhere.
Beginning with a new stack I get the error message as in the title.
I am using SAM, and I am confused, why it wants to update the macro.
I thought, this macro is provided by aws and I wonder why it is requesting to modify it.
My template spins up a lambda, a database and a REST api, but does even try to touch existing macros.
My template did contain the TableName tag for a DynamoDb.
As I am aware, named tables cannot be updated, if resource replacement required. I was not trying to do updates on that resource though.
The table existed before I cloudformed that new stack though.
I am running a AWS CloudFormation stack that takes in some parameters and launches EC2 instances along with other AWS resources. The parameters are fed into the user data of the EC2 instance and based on that changes are made dynamically to the web application residing on the EC2 instance.
UserData:
Fn::Base64:
Fn::Join:
- ""
-
- "#!/bin/bash \n"
- "sh website-conf/website_mysql_config.sh "
- " -c \""
-
Ref: "CompanyName"
As shown in the example above, CompanyName is one of the many parameters passed to the userdata script. The problem is, when any one or multiple of parameters are updated, CloudFormation does not detect that and instead throws this error.
So, in order to update the stack, I have to edit the stack and make changes to the ASG so that CloudFormation 'sees' the changes and executes the stack update.
Is there a way to force CFN to update the stack when the parameters are updated?
CloudFormation will not update the stack unless there is a change in properties of the resources already created in the stack.
For example:
Consider I have a simple template to create a database where I need to pass 2 parameters:
db-name
region
Assume that I am using db-name passing it as value to DBInstanceIdentifier.
Also assume that I am not using the input parameter region for any purpose in creation of resources (or its properties) of the stack in any way.It is more of a dummy parameter I keep for readability purpose.
I passed (TEST-DB1, us-east-1) as input parameters to the CloudFormation template and successfully created the resources.
Scenario-1:
Now if I update the stack(still using the existing template) and just change the input parameters to (TEST-DB2, us-east-1). ie: changing just the db-name and not the region. Then CloudFormation will detect that, this parameter update, results in change in properties of running resource(s) of the stack and will compute and display the modifications as a change set.
Scenario-2:
Suppose I make another update(still using the existing template) property and just change the input parameters to (TEST-DB1, us-east-2). ie: changing just the region and not the db-name. Then CloudFormation will detect that, this parameter update, result in NO change in properties of running resource(s) of the stack will show the Error creating change set.
Bottomline:
Your change in input parameter must result in an update/replacement of any resources(or its attributes like security-groups,port etc..) of the stack. Then AWS CloudFormation will display them as Change Sets for your review. Also, the method (update or replacement) AWS CloudFormation uses depends on which property you update for a given resource type.
Your parameter "CompanyName" is not making any changes to the running
resources of the stack. Hence it is reporting as Error creating
change set. You need to use it to create any resource/resource properties of the stack. Then CloudFormation will detect the change-sets when you modify it. The same applies for any other input-parameters which you use.
Use the AWS CLI Update-Stack command. If you use the AWS CLI you can inject parameters into your stack so any change to any of the parameters result in a new stack. I do this myself to inject the Git/version commit ID into UserData so simply committing changes to the stack's JSON/Yaml to Git will allow stack updates. Any change to the parameters file will allow stack updates, even just a comment. I reference my Git commit ID in UserData the same way you are referencing Ref:CompanyName so when I change the Git commit ID the userData section is updated on stack updates.
Update Stack Command
aws cloudformation update-stack --stack-name MyStack --template-body file:///Users/Documents/Git/project/cloudformation/stack.json --parameters file:///Users/Documents/Git/project/cloudformation/parameters/stack-parameters.dev.json --capabilities CAPABILITY_IAM
Process
With this approach you make your parameters changes to the parameters json or yaml file then check it into version control. Now if you use a build server you can update your stack by checking out master and just running that one line above. Using AWS CodeBuild makes this easy so you don't need jenkins.
The answer of your problem is already answered with this state, CloudFormation will not update the stack unless there is a change in properties of the resources already created in the stack.
And for the answer for your question, please check the explanation below.
There is a way to force Cloudformation to update the stack using the AWS::CloudFormation::Init.
By using cfn-init, each instance can update itself when it detect the change that made by AWS::CloudFormation::Init in metadata.
There is a concept that we must understand first, that is the difference between UserData and metadata, at least under the AWS::CloudFormation::Init case.
Userdata: Will be only called once when the instance is being launch for the first time (this including update that need the instance to be replaced). So, if you update the stack (not creating a new one), even if you change the parameter value, it won't change anything if you call the parameter under UserData.
Metadata: Can be updated anytime. To make it works, you have to make sure that the daemon that detect the metadata changed is running (the daemon is called the cfn-hup)
If you already use the Metadata and AWS::CloudFormation::Init, the data is not immediately being updated. As far I know, here is the condition the data to be change after change the Metadata value.
Reboot the instance
Run cfn-init command again with it's parameter
Waiting about 15 minutes, because the daemon to check the change in Metadata is checking the change once in 15 minutes.
I have a Lambda function invoked by S3 put events, which in turn needs to process the objects and write to a database on RDS. I want to test things out in my staging stack, which means I have a separate bucket, different database endpoint on RDS, and separate IAM roles.
I know how to configure the lambda function's event source and IAM stuff manually (in the Console), and I've read about lambda aliases and versions, but I don't see any support for providing operational parameters (like the name of the destination database) on a per-alias basis. So when I make a change to the function, right now it looks like I need a separate copy of the function for staging and production, and I would have to keep them in sync manually. All of the logic in the code would be the same, and while I get the source bucket and key as a parameter to the function when it's invoked, I don't currently have a way to pass in the destination stuff.
For the destination DB information, I could have a switch statement in the function body that checks the originating S3 bucket and makes a decision, but I hate making every function have to keep that mapping internally. That wouldn't work for the DB credentials or IAM policies, though.
I suppose I could automate all or most of this with the SDK. Has anyone set something like this up for a continuous integration-style deployment with Lambda, or is there a simpler way to do it that I've missed?
I found a workaround using Lambda function aliases. Given the context object, I can get the invoked_function_arn property, which has the alias (if any) at the end.
arn_string = context.invoked_function_arn
alias = arn_string.split(':')[-1]
Then I just use the alias as an index into a dict in my config.py module, and I'm good to go.
config[alias].host
config[alias].database
One thing I'm not crazy about is that I have to invoke my function from an alias every time, and now I can't use aliases for any other purpose without affecting this scheme. It would be nice to have explicit support for user parameters in the context object.