I would like to create a custom resource in cloudformation that will contain sub resources so that I can prevision application resources for high value clients.
For example. I have an application that has an SNS::Subscription, SQS::Queue, ElasticBeanstalk::Environment, ElasticBeanstalk::Application and ElasticBeanstalk::ApplicationVersion.
Copying this for any client I want to provision dedicated resources for is a hassle. Keeping them all up to date is a problem.
I would like to define these once, and use a custom resource to generate them with custom variables provided. Like-
GeneralPurpose:
Type: COM::MyApplication
Properties:
QueueName: general
InstanceType: t3.micro
Instances: 30
AcmeClient:
Type: COM::MyApplication
Properties:
QueueName: acme
InstanceType: t5.medium
Instances: 10
SnsFilterValue: acme
These custom properties would be used in the custom resource template I defined and generate all the resources required for both the general and acme application.
If I need to change the default visibility timeout for the application queue, I can do it in one place and update all resources that used this custom resource.
Is this possible?
Consider using a nested stacks instead of a custom resource. You can define parameterized templates to use as the nested stacks. This will be much easier to maintain.
Related
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: hello
Resources:
ApiGatewayApi:
Type: AWS::Serverless::Api
Properties:
StageName: stage
TracingEnabled: true
FunctionA:
...
Environment:
Variables:
TEST: !Ref ApiGatewayApi
Events:
GetUsers:
Type: Api
Properties:
Path: /account
Method: get
RestApiId:
Ref: ApiGatewayApi
FunctionB:
...
Environment:
Variables:
API_URL: !GetAtt ApiGatewayApi.RootResourceId
Events:
OrderEvent:
Type: SQS
Properties:
Queue: !GetAtt OrderServiceQueue.Arn
This leads to a circular dependency. IF I do !Ref in a function that does not have an event with API, it does not complain about it. I read the premium support article from aws, blogs and other stack overflow questions but they are not similar to my question.
FunctionB successfully refers to the API gateway id while FunctionA does not.
I create the api outside the function, so I think it SHOULD !Ref the endpoint in it. Is there something else?
The circular reference is created by how AWS SAM uses the events in order to create the definition of the API. This basically means that it needs the ARNs of the lambda functions to construct this definition before it can create the API. But since you are needing IDs of the API in order to create the lambda, you end up with a circular reference since neither can be created without the other one already existing.
The first way to solve this problem is by deploying your stacks in multiple steps. You could first deploy an empty API, which would allow you to reference the API IDs when adding the lambdas. The significant drawback of this approach is of course if you want to easily replicate this stack on another account or redeploying the API for some reason, which means you'd have to use this trick again each time.
Another way, if you really want to have this value as an environment variable, would be to manually create the definition body for the API (in which you construct the ARNs of the lambda, not reference them) and presumably, you'll also need to manually create the permissions in order to allow your API Gateway resource to execute the lambda functions.
However, a better way I feel would be to use the lambda proxy integration (which is used by default I think, but I could not find any documentation to verify this). When using the lambda proxy integration, the incoming event in lambda contains all the information about the API. You could easily extract the information you need from that event instead of having it as an environment variable, but this depends on your exact use case.
I would like to perform the following operations in order with CloudFormation.
Start up an EC2 instance.
Give it privileges to access the full internet using security group A.
Download particular versions of Java and Python
Remove its internet privileges by removing security group A and adding a security group B.
I observe that there is a DependsOn attribute for specifying the order in which to create resources, but I was unable to find a feature that would allow me to update the security groups on the same EC2 instance twice over the course of creating a stack.
Is this possible with CloudFormation?
Not in CloudFormation natively, but you could launch the EC2 instance with a configured userdata script that itself downloads Java/Python and the awscli, as necessary, and then uses the awscli to switch security groups for the current EC2 instance.
However, if all you need is Java and Python pre-loaded then why not simply create an AMI with them already installed and launch from that AMI?
The best way out is to utilise a Cloudformation custom resource here. You can create a lambda function that does exactly what you need. This lambda function can then be called as a custom resource function in the cloud formation template.
You can pass your new security group ID and instance ID to the lambda function and code the lambda function to use AWS SDK and do the modifications that you need.
I have leveraged it to post an update to my web server about the progress of the cloud formation template. Below is the sample code of the template.
EC2InstanceProfile:
Type: AWS::IAM::InstanceProfile
Properties:
Path: /
Roles: [!Ref 'EC2Role']
MarkInstanceProfileComplete:
Type: 'Custom::EC2InstanceProfileDone'
Version: '1.0'
DependsOn: EC2InstanceProfile
Properties:
ServiceToken: !Ref CustomResourceArn
HostURL: !Ref Host
LoginType: !Ref LoginType
SecretId: !Ref SecretId
WorkspaceId: !Ref WorkspaceId
Event: 2
Total: 3
Here the resource MarkInstanceProfileComplete is a custom resource that calls a Lambda function. It takes the event count and total count as input and processes them to calculate percentage progress. Based on that it sends out a request to my web server. For all we care, this Lambda function can do potentially anything you want it to do.
I'm facing a decision to Use Cross-Stack References to Export Shared Resources or to Use Nested Stacks to Reuse Common Template Patterns following AWS CloudFormation best practices.
However, they seem the same to me apart from a few differences:
cross-stack uses Fn::ImportValue, templates are in one folder.
nested-stack must be in S3, uses type AWS::CloudFormation::Stack and TemplateURL.
There's no clear pros and cons between them as far as I could search.
My goal is to create a parent stack that passes some core variables like stackName to the child stacks, then the child stacks create the resources sharing some variables between them like ARN or Policies, using the stackName to name their resources like stackNameDynamoDBTable.
You should use cross-stack references as it was created for your use case of passing between stacks.
Whereas nested stacks would work, it’s primary purpose is for reuse of modular components, like a template of a resource you use in lots of stacks to save copy pasting and updating the stacks independently.
Nested stacks: if you need to manage your stacks from a single point, you should use nested stacks.
example: assume that you have load balancer configuration that you use for most of your stacks. Instead of copying and pasting the same configurations into your templates you can create a dedicated template for load balancer.
cross-stack : Alternatively, if you need to manage your stacks as separate entities, you should use cross-stack references.(AWS limits the number of VPCs you can create in an AWS region to five.)
example : You might have a network stack that includes a VPC, a security group, and a subnet. You want all public web apps to use these resources. By exporting the resources, you allow all stacks with public web applications to use them.
There is a way to get the best of both worlds. The trick is to use cross-stack resource sharing but make it depend on a parameter that is passed using Nested stack.
Here's an example from how I used this, consider two stacks IAMRoleStack and ComputeStack. The former contains all the necessary IAM roles and the latter contains a bunch of Lambda functions that those roles are applied to.
Resources:
IAMCustomAdminRoleForLambda:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Policies:
Output:
IAMRoleArnForLambda:
Description: Returns the Amazon Resource Name for the newly created IAM Custom
Role for Lambda function
Value: !GetAtt 'IAMCustomAdminRoleForLambda.Arn'
Export:
Name: !Sub '${AWS::StackName}-IAMRoleArnForLambda'
StackName:
Description: Returns name of stack after deployment
Value: !Sub ${AWS::StackName}
As you can see I've exported the IAM role but it's Name depends on the stack name that is calculated once the stack is deployed. You can read more about exporting outputs in the docs.
In the ComputeStack, I use this role by importing it.
Resources:
LambdaForCompute:
Type: AWS::Lambda::Function
Properties:
Role: !ImportValue
Fn::Sub: ${StackNameOfIAMRole}-IAMRoleArnForLambda
The parent stack that "nests" both ComputeStack and IAMRoleStack orchestrates passing the stack name parameter.
Resources:
IAMRoleStack:
Type: AWS::CloudFormation::Stack
Properties:
TemplateURL: !Ref IAMRoleStackURL
ComputeStack:
Type: AWS::CloudFormation::Stack
Properties:
TemplateURL: !Ref ComputeStackURL
Parameters:
StackNameOfIAMRole: !GetAtt IAMRoleStack.Outputs.StackName
I can't attest to best practice but this style allows me to pick and choose where I want orchestrated deployment and where I want to do the deployments individually.
I also want to point out that this kind of modularization based on type of resources is not very feasible for nested stacks. For e.g. in this scenario, if I had 10 different roles for 10 different Lambda functions, I would have to pass each of those 10 roles through parameters. Using this hybrid style, I only need to pass one parameter the stack name.
With cross stacks, you pass a reference to a bunch existing components X to stacks A and B when you want A and B to reuse these very same existing components. With nested stacks, when you nest a nested stack Y in stacks C and D, Y shall create a new set of components Y is describing individually for C and for D.
It is similar to concepts 'passing by reference' and 'passing by value' in programming.
I know the equivalent to AWS::Lambda::Function is aws_lambda_function
But I'm not sure what the equiv for Custom::LoadLambda
I'm trying to translate the below into Terraform:
CreateRsaKey:
Type: Custom::LoadLambda
Properties:
ServiceToken: # This seems to call another lambda
Fn::GetAtt:
- SolutionHelper
- Arn
Region:
Ref: AWS::Region
The Custom::String Resource Type refers to a Custom Resource. This means that what it does depends on the particular implementation of the Lambda function provided to the ServiceToken property (SolutionHelper in your example). When a Custom Resource is used, the Lambda function is invoked with a Request Object specifying a RequestType of Create/Update/Delete.
The Terraform equivalent of a Custom Resource is a Custom Provider plugin. These are packaged and distributed the same as the standard set of Providers, only less officially. They are built as separate binaries (typically Go packages) auto-discovered by the core Terraform process using a filename convention (terraform-<TYPE>-<NAME>), and are invoked in a subprocess using a custom RPC mechanism. The plugin binary provides through RPC a Provider containing a collection of Resources that implement Create/Read/Update/Delete functions for the resource.
So it's possible to re-implement the functionality of a Lambda-backed Custom Resource within a Terraform Provider Plugin by translating the CloudFormation Create/Update/Delete logic in the Lambda function to the Create/Update/Delete functions in the Terraform Resource (and adding an appropriate Read function). However, it's not a very simple or straightforward process.
you can try using this provider
https://github.com/mobfox/terraform-provider-multiverse
I have a use case where I want to create a resource only in test environment using cloud formation. I'm trying to use conditionals to achieve this but it fails.
Resources:
TestClientRole:
Type: AWS::IAM::Role
Condition: NotProdStage
Properties:
AssumeRolePolicyDocument:
Statement:
- Action: <>
Effect: Allow
<>
Version: '2012-10-17'
RoleName:
"some-test-role"
It works fine in non-prod stages, but fails with "Unresolved resource dependencies [TestClientRole] in the Resources block of the template"
How to make cloud formation ignore resources for prod stages?
The error message is indicating that there is a dependency on TestClientRole elsewhere in your CloudFormation template.
The problem is that the Role is not being created in a Prod environment, yet the other resource is saying that it is dependent on the role being created.
Solution: Remove the dependency elsewhere in your template that refers to TestClientRole.
In fact, it is very rare to actually require dependencies because CloudFormation figures out the correct build order based upon references between resources. The only time you need a dependency is when you specifically want one thing to finish building before another starts, such as having an app server wait until a database server is ready. Typically, leave them out unless you have a particular need that is non-obvious.