I need to modify the CloudFormation template : Video on Demand on AWS CloudFormation template
When I deploy the main template without any modifications and upload video in the source S3 bucket, then the folders that are getting created in the destination S3 bucket are having their names as guid of Dynamodb item as shown in the below picture,
In my case, the requirement is that those folders in the destination S3 bucket should get created with some meaningful names.
To resolve this issue, where exactly do I need to modify the template
Steps for modifying a CloudFormation stack can be found at:
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-updating-stacks-get-template.html
In general, Stacks support passed parameters of the form:
aws cloudformation create-stack --stack-name mystack
--template-body file://mypath/mystack.json
--parameters ParameterKey=KeyPairName
So one of the passed parameters could be the desired output filename.
Regarding file naming within MediaConvert, the service supports a set of time+date variables for naming output files, which can found at: https://docs.aws.amazon.com/mediaconvert/latest/ug/list-of-settings-variables-with-examples.html
Alternatively, you could rename the files after output using a Lambda Function triggered by S3 File Events. This would allow you to generate or retrieve a name conducive to your workflows, as well as set Tags and other object metadata.
Examples of this workflow are documented here: https://docs.aws.amazon.com/lambda/latest/dg/with-s3-example.html
I hope this information helps you tune your workflow. Good luck!
Related
My CF is having Step Function state machine configuration which definition is placed at some S3 location.
Now if there is any change in the definition, i.e. change only in the S3 file, in this case updating CF is failing.
Solution I tried
I Have declared a parameter in my CF(say buildVersion) and every change in the S3 will lead to new build version and this build version I am sending though parameters
aws cloudformation --region "xyz" update-stack --stack-name "my stack name" --timeout-in-minutes 15 --template-body "CF file path" --parameters "parameter-file path"
This is how my parameter file look like, So before calling this command I am updating the parameter file with this buildVersion parameter.
[
{"ParameterKey":"EnvironmentName", "ParameterValue":"dev"},
{"ParameterKey":"BuildVersion", "ParameterValue":"0.0"}
]
But it is not solving the purpose and update command is failing, If I am doing the same thing with the AWS console i.e. updating parameter and click on update it is working fine.
Any suggestions will be highly appreciated.
When you're updating your template. It will not look into the file referenced on S3 to see if anything has changed. So it will simply see that the file location has not changed and conclude that there is no change in the template itself.
Since your question is not really complete, I'll have to answer based on some assumptions. You could enable versioning on the S3 bucket that holds your definition and pass in the S3 version identifier as a parameter to your stack to use as a property of your StepFunction S3 Location declaration. If you do this, you can upload a new version of the state machine declaration to S3 and use that version identifier as a parameter for updating your CloudFormation stack.
You could automate the entire process of updating the CloudFormation stack by using a CodePipeline or a Lambda trigger on S3 to automatically update the CloudFormation stack when a new StateMachine definition is uploaded.
The issue with BuildVersion approach was, this parameter was not participating in the AWS resource creation and updating this parameter though CLI does not affecting any thing in the change set.
I have added this parameter in the 'Tag' section of the state machine Resource and it worked for me.
I'm trying to get this repo going here - https://github.com/mydatastack/google-analytics-to-s3.
A link is provided to launch the AWS CloudFormation stack but it is no longer working as the S3 bucket containing the template is no longer active.
I have 2 questions about getting data pipeline running:
My first question would be what is 631216aef6ab2824fc63572d1d3d5e6c.template and can I create it through the 3 .yml files in the CloudFormation folder?
I've tried to create a template through CloudFormation designer , collector-ga.yml but it fails. I think its because the Resources within the yml aren't available when creating a template just from collector-ga. I've also tried uploading the repo to s3 and creating a template from there but that was also unsuccessful.
How can I launch the stack from the repo? I've found very little information online so an explanation or a pointer to some relevant resources would be appreciated.
This repository doesn't use the "standard" CloudFormation resources, but it uses AWS SAM. You'll have to install the SAM CLI tool and use that to deploy the CloudFormation stack. If you run sam deploy --guided it will help you with the setup of the necessary S3 bucket etc on your AWS account. SAM will upload the necessary files, resolve the internal local links between the templates by updating them with the S3 URLs and construct a packaged.yml template which it will use to deploy the stack.
Also, check out the AWS SAM user guide for more information.
I created an S3 bucket but now I would like to update/add lifecycle policies to it using CDK.
Currently I can import this bucket to a new stack file.
const testBucket = s3.Bucket.fromBucketAttributes(this, 'TestBucket', {
bucketArn: 'arn:aws:s3:::xxxxxx'});
How can I use AwsCustomResource to update/add lifecycle policies? For example, for prefix = long, I want those objects to expire in 7 days, and for prefix = short, I want them to expire in 3 days.
Or is there a general way of updating an existing S3 bucket in a new stack with CDK?
The best option for this is probably to add it to the stack using resource importing. Don't confuse a custom resource with a CDK construct. A custom resource involves deploying a lambda function and calling that function when the CloudFormation custom resource is present in your stack. A CDK construct is used to generate a CloudFormation template. It allows you to combine different resources into a single bit of code, and allows some logical decisions based on the input values.
Steps to import:
Make sure your current CDK is what is deployed. You cannot import resources when there are other changes.
Add the S3 bucket to your CDK code as if you were creating it for the first time. Make sure that all the settings are the same as what is currently deployed. This is important because the import process doesn't validate that what you have configured matches what you are importing.
Run cdk synth to generate the CloudFormation template that includes the S3 bucket.
In the CloudFormation console locate the stack that represents the CDK stack you are working with.
Select Import resources into stack from the Stack actions menu.
Follow the prompts. When asked for the template select Upload a template file and select the template created by the cdk synth command (hint: it will be in cdk-out). You will be prompted to add the bucket name. This should be the bucket that you are wanting to add to this stack.
Once that is done you can modify the bucket as if it were created by your stack.
One thing to note. CDK includes some meta-data in the CloudFormation template. This value must be the same as what is currently deploy or it will be seen as a change and you won't be able to perform the import. You can copy the values from what is currently deployed and manually edit the template created by cdk synth to match that.
Need to access the CfnBucket reference of the testBucket and add lifecycle rules to it.
const testBucket = s3.Bucket.fromBucketAttributes(this, 'TestBucket', {
bucketArn: 'arn:aws:s3:::xxxxxx')
testBucket.node.root.addLifecycleRule({prefix: 'short', expiration:3, enabled:true})
testBucket.node.root.addLifecycleRule({prefix: 'long', expiration:7, enabled:true})
I have created an S3 Bucket, with the cloud formation, Lets Say Bucket Name is S3Bucket,
I don't want this bucket getting deleted if I delete stack, so added Deletion Policy to Retain,
Now the problem here is, If run the stack again, it complains S3Bucket name already exists.
If a bucket already exists, it should not complain.
What to do for this.
Please help
I faced this in the past and what i did in order to resolve this is that i created a common AWS cloudformation template/stack which will create all our common resources which are static(Handle it like a bootstrap template).
Usually i am adding in this template the creation of s3 buckets,VPC, networking, databases creation, etc.
Then you can create other AWS cloudformation templates/stacks for your rest resources which are dynamic and changing usually like lambdas,ec2, api gateway etc.
S3 names are globally unique. (e.g if I have s3 bucket in my AWS account s3-test, you cannot have a bucket with the same name).
The only way to use same name is to delete the bucket, or retype your cloud formation template and use new cloud formation feature to import resource:
https://aws.amazon.com/blogs/aws/new-import-existing-resources-into-a-cloudformation-stack/
I am trying to create serverless image handler on AWS using the this template
I use the management console for that and even though I rename the bucket name in the configuration before creating the stack, it adds "demowebsitebucket-" and something random at the bucket name.
Here is the input where I put "imagehandler" as the origin S3 Bucket:
However the stack creates the bucket with this arn:
arn:aws:s3:::imagehandler-demowebsitebucket-iafi34.....
I downloaded and tried to find the renaming in the template but I could not figure out where the renaming is done. I also want to get rid of the random generated naming at the end if possible.
I solved it by renaming all occurrences of Demo Website Bucket to something else under Resources in the template file.
Demo Website Bucket was used there as a variable and it can be renamed to anything.