I'm creating a bunch of application resources with AWS CloudFormation, and when the resources are created, CloudFormation adds a hash at the end of the name to make it unique.
i.e. If you wanted to create a Kinesis stream names MyStream, the actually name would be something like my-stack-MyStream-1F8ISNCLP0W4O.
I want to be able to programmatically access the resources without having to know the hash, without having to query AWS for my resources to match the names myself, and without manual steps. Does anybody know a convenient way to use AWS resources in your application programmatically and predictably?
Here are the less ideal options I can think of:
Set a tag on the resource (i.e. name -> MyStream) and query AWS to get the actual resource name.
Query AWS for a list of resources names and look for a partial match on the expected name.
After you create your resources, manually copy the actual names into your config file (probably the sanest of these options)
You can use the CloudFormation API to get a list of resources in your stack. This will give you a list of logical ids (i.e. the name in your CloudFormation template without the hash) and matching physical ids (with the stack name and hash). Using the AWS CLI, this will show a mapping between the two ids:
aws cloudformation describe-stack-resources
--query StackResources[].[LogicalResourceId,PhysicalResourceId]
--stack-name <my-stack>
CloudFormation APIs to do the same query are provided in all the various language SDKs provided by Amazon.
You can use this as an alternative to #1, by querying CloudFormation at runtime, or #3, by querying CloudFormation at buildtime and embedding the results in a config file. I don't see any advantage to using your own tags over simply querying the CF API. #2 will cause problems if you want two or more stacks from the same template to coexist.
I've used both the runtime and build time approaches. The build time approach lets you remove the dependency on or knowledge of CloudFormation, but needs stack specific information in your config file. I like the runtime approach to allow the same build to be deployed to multiple stacks and all it needs is the stack name to find all the related resources.
Related
I'm working with a CloudFormation template which is defining a lot of parameters for static values out of the scope of the template.
For example, the template is creating some EC2, and it has parameters for each VPC subnet. If this was Terraform, I would just remove all of these parameters and use data to fetch the information.
Is it possible to do that with CloudFormation?
Notice that I'm not talking about referencing another resource created within the same template, but about a resource that already exists in the account that could have been created by different means (manual, Terraform, CloudFormation, whatever...)
No, CloudFormation does not have any native ability to look up existing resources. You can, however, achieve this using a Cloudformation macro.
A CloudFormation macro leverages a lambda function, which you can implement with whatever logic you need (e.g. using boto3) so that it returns the value you're after. You can even pass parameters to it.
Once the macro has been created, you can then consume it in your existing template.
You can find a full example on how to implement a macro, and on how to consume it, here: https://stackoverflow.com/a/70475459/3390419
I need to generate a report of all AWS Services that were provisioned after a certain date (say last 3 months).
AWS Service Catalog seems relevant here; but can this be used only if the services were provisioned using CloudFormation Templates?
We did our provisioning using Terraform - can AWS Service Catalog still be used to generate an inventory?
If not, is there an alternate way to generate this report?
You can try to use the Resource Groups for that https://eu-central-1.console.aws.amazon.com/resource-groups/home?region=eu-central-1#
There you will find the Tag Editor https://eu-central-1.console.aws.amazon.com/resource-groups/tag-editor/find-resources?region=eu-central-1 and list all of your resources.
If you have tagged your resources, you can filter by them. Alternative solution would be to tag all resources with the current date...wait one day...search again and find resources without the specific date tag. So you will find the differences.
To automate this, you can use e.g. https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/resourcegroupstaggingapi.html#client
To get a full solution, you can use Tag Editor, get all resources and request the resources itself with the specific API of each resource, e.g. EC2, Lambda, RDS, etc.
This could be time consuming, so maybe a solution like from aquasec could fit your needs.
We have a very complex api-gateway api that has been constructed manually through the console. I would like to create a cloudformation template from this existing api, so it can be managed instead in code.
The "create stack from existing resources" seems to require all of the resources to be pre-defined in a template. However this is exactly the bit I'm trying to avoid. Due to the complexity of the existing api, it would take a very long time to manually work through all of the api resources to create all the definitions in a template.
Is there some way I can have CloudFormation automatically scan through the existing api resources and create the template definitions from it?
There is popular open-source tool called former2 which can generated CFN templates from existing resources.
Other then former2 there is nothing (CloudFormer is not maintained nor supported by AWS anymore). You would have to manually per-populate entire template before importing resources to CloudFormation.
If given the correct permissions, Lambda functions will automatically create CloudWatch log groups to hold log output from the Lambda. Same for LambdaEdge functions with the addition that the log groups are created in each region in which the LambdaEdge function has run and the name of the log group includes the name of the region. The problem is that the retention time is set to forever and there is no way to change that unless you wait for the log group to be created and then change the retention config after the fact.
To address this, I would like to create those log groups preemptively in Terraform. The problem is that the region would need to be set in the provider meta argument or passed in the providers argument to a module. I had originally thought that I could get the set of all AWS regions using the aws_regions data source and then dynamically create a provider for each region. However, there is currently no way to dynamically generate providers (see https://github.com/hashicorp/terraform/issues/24476).
Has anyone solved this or a similar problem in some other way? Yes, I could create a script using the AWS CLI to do this, but I'd really like to keep everything in Terraform. Using Terragrunt is also an option, but I wanted to see if there were any solutions using pure Terraform before I go that route.
I'm creating a stack with CloudFormation. When you create log groups, it automatically adds prefixes and suffixes to my log group names. For example, if I try to create log group MyLogGroup, it creates log group my-stack-name-MyLogGroup-EEWJYSCJRK2V.
I understand that for a lot of use cases, this might be desired to differentiate the same resources for different stacks. However, my team has different accounts for our different stacks, so there will be no overlap. Having dynamic prefixes and suffixes makes it hard to reference log groups from static files (i.e. CloudWatch Logs agent config file).
Is there a way to make sure that resources get named EXACTLY what I put and not add a prefix or suffix?
We have run into this same issue with our AWS ecosystem and after speaking to several folks at AWS, this is by design and is not modifiable right now.
Depending on the complexity of what you are trying to do, I would recommend replacing CloudFormation with some Lambda functions to manage the resources (can be done cross account with sts:AssumeRole).
Yes its possible. For instance in our cloud formation template we create the cloud watch conf file with log_group_name and log_stream_name parameter set to combination of different parameters. Our log groups are created without prefix and postfix. See following example:
"log_group_name = "MyLogGroup\n",
"log_stream_name = {instance_id}/", "MyLogGroup", ".log\n",