Specify Array of VPC SubnetID / SecurityGroupIds to aws cli cloudformation deploy - amazon-web-services

I am using aws-cli to deploy my stack across several environments and need to parametrize the subnets / security groups available to my stack.
I have a section in my SAM template defining the subnets and security groups as such:
EnvSubnets:
Description: Define subnet ids
Type: 'List<AWS::EC2::Subnet::Id>'
EnvSecGroups:
Description: Security Groups
Type: 'List<AWS::EC2::SecurityGroup::Id>'
I specify the arguments using `aws cloudformation deploy ... --parameter-overrides file://env.json' but cannot find a single format that passes the arrays to cloudformation.
I keep getting the followign errors:
#/VpcConfig/SecurityGroupIds: expected type: JSONArray, found: String #/VpcConfig/SubnetIds: expected type: JSONArray, found: String
Any hints?

It seems that at the current time this is not supported - I ended-up using a nested template driven by a user-overridable parameter:
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: 'SAM Template for XXXXX XXXXX'
Parameters:
LambdaRole:
Description: Define exiting Lambda role to provide permissions
Type: String
LambdaImage:
Description: Define Lambda image URI
Type: String
LambdaVPCInclude:
Description: S3 URI of the YAML for the S3 VPC section
Type: String
Resources:
FOO:
Type: AWS::Serverless::Function
Properties:
PackageType: Image
ImageUri: !Ref LambdaImage
Architectures:
- x86_64
MemorySize: 1024
Timeout: 900
Role: !Ref LambdaRole
'Fn::Transform':
Name: 'AWS::Include'
Parameters:
Location: !Ref LambdaVPCInclude
Metadata:
SamResourceId: FOO
Outputs:
QuantUniverse:
Description: FOO Lambda Function ARN
Value: !GetAtt FOO.Arn
and in an S3 bucket I have a file with my VPC config:
VpcConfig:
SubnetIds:
- subnet-*****************
- subnet-*****************
- subnet-*****************
SecurityGroupIds:
- sg-*****************
- sg-*****************
and pass the S3 URI of this file as the override for LambdaVPCInclude in aws cloudformation deploy
Hope this helps others.

Related

Passing Security Group Ids and Subnet Ids in a Clould Formation template

Parameters:
ClusterName:
Type: String
ClusterVersion:
Type: Number
AllowedValues: [1.21, 1.20, 1.19, 1.18]
RoleArnValue:
Type: String
ListOfSubnetIDs:
Description: Array of Subnet IDs
Type: List<AWS::EC2::Subnet::Id>
ListOfSecurityGroupIDs:
Description: Array of security group ids
Type: List<AWS::EC2::SecurityGroup::Id>
Resources:
EKSCluster:
Type: AWS::EKS::Cluster
Properties:
Name: !Sub ${ClusterName}
Version: !Sub ${ClusterVersion}
RoleArn: !Sub ${RoleArnValue}
ResourcesVpcConfig:
SecurityGroupIds:
- !Sub ${ListOfSecurityGroupIDs}
SubnetIds:
- !Sub ${ListOfSubnetIDs}
Above is the .yaml clouldformation template I have created so i can spin up eks cluster. Then i am using aws cli to spin up the cluster using the following command.
aws cloudformation deploy --template-file eks.yaml --stack-name cluster-test --parameter-overrides ClusterName=Dev ClusterVersion=1.21 ListOfSubnetIDs=subnet-11111d11b11b011f4,subnet-99999d237f87f11d7,subnet-222222c110c7e4be7,subnet-88888884de8d25176 ListOfSecurityGroupIDs=sg-01111111a21221 RoleArnValue=arn:aws:iam::123456546456:role/cluster-ServiceRole-WMIC72AOWSP0 --capabilities CAPABILITY_NAMED_IAM
I get the following error
An error occurred (ValidationError) when calling the CreateChangeSet operation: Template error: variable ListOfSecurityGroupIDs in Fn::Sub expression does not resolve to a string
I am not sure why. Am i using !sub in correctly? Would really appreciate input on this.
Since you want to reference the parameters you provided the template as they are, you should use the Ref function.
Here's an example of a valid template:
Parameters:
ClusterName:
Type: String
RoleArnValue:
Type: String
ListOfSubnetIDs:
Description: Array of Subnet IDs
Type: List<AWS::EC2::Subnet::Id>
ListOfSecurityGroupIDs:
Description: Array of security group ids
Type: List<AWS::EC2::SecurityGroup::Id>
Resources:
EKSCluster:
Type: AWS::EKS::Cluster
Properties:
Name: !Ref ClusterName
RoleArn: !Ref RoleArnValue
ResourcesVpcConfig:
SecurityGroupIds: !Ref ListOfSecurityGroupIDs
SubnetIds: !Ref ListOfSubnetIDs
and here's how I deployed it:
aws cloudformation deploy --template-file eks.yml --stack-name cluster-test --parameter-overrides ClusterName=Dev ListOfSubnetIDs=subnet-be0a99c4,subnet-c71046ae ListOfSecurityGroupIDs=sg-009690ac6b3bff6df,sg-009a3f1cb63943941 -RoleArnValue=...
Sub should be used when you want to perform string manipulation. Checkout the examples from the documentation.

Instead of referring an existing AWS S3 bucket, Cloud Formation is trying to create the bucket

I'm stuck in a weird issue. I have created an AWS S3 bucket using following cloud formation template:-
AWSTemplateFormatVersion: '2010-09-09'
Metadata:
License: Unlicensed
Description: >
This template creates a global unique S3 bucket in a specific region which is unique.
The bucket name is formed by the environment, account id and region
Parameters:
# https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/parameters-section-structure.html
Environment:
Description: This paramenter will accept the environment details from the user
Type: String
Default: sbx
AllowedValues:
- sbx
- dev
- qa
- e2e
- prod
ConstraintDescription: Invalid environment. Please select one of the given environments only
Resources:
# https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-s3-bucket.html
MyS3Bucket:
Type: AWS::S3::Bucket
DeletionPolicy: Retain
Properties:
BucketName: !Sub 'global-bucket-${Environment}-${AWS::Region}-${AWS::AccountId}' # https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/pseudo-parameter-reference.html
AccessControl: Private
LoggingConfiguration:
DestinationBucketName: !Ref 'LoggingBucket'
LogFilePrefix: 'access-logs'
Tags:
- Key: name
Value: globalbucket
- Key: department
Value: engineering
LoggingBucket:
Type: AWS::S3::Bucket
DeletionPolicy: Retain
Properties:
BucketName: !Sub 'global-loggings-${Environment}-${AWS::Region}-${AWS::AccountId}'
AccessControl: LogDeliveryWrite
Outputs:
GlobalS3Bucket:
Description: A private S3 bucket with deletion policy as retain and logging configuration
Value: !Ref MyS3Bucket
Export:
Name: global-bucket
If you note in the template above then I'm exporting this S3 bucket in the Outputs section by the name called global-bucket.
Now, my intention is to refer to this existing bucket going forward in my AWS account whenever any new resource like Lambda, etc wants an S3 bucket. Here is an example using AWS SAM (Serverless Application Model), I'm trying to create an AWS Lambda and trying to refer to this existing S3 bucket using property !ImportValue and the export name as global-bucket as shown below:-
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
hellolambda
Sample SAM Template for hellolambda
# More info about Globals: https://github.com/awslabs/serverless-application-model/blob/master/docs/globals.rst
Globals:
Function:
Timeout: 3
Resources:
HelloWorldFunction:
Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
Properties:
CodeUri: hello-world/
Handler: app.lambdaHandler
Runtime: nodejs12.x
Events:
HelloLambdaEvent:
Type: S3
Properties:
Bucket: !Ref SrcBucket
Events: s3:ObjectCreated:*
SrcBucket:
Type: AWS::S3::Bucket
Properties:
BucketName: !ImportValue global-bucket
Now, the problem is when I execute the command like sam build and then sam deploy --guided and select the same region (where my previous CloudFormation stack output is present) then I get the following error:-
global-bucket-sbx-ap-southeast-1-088853283839 already exists in stack arn:aws:cloudformation:ap-southeast-1:088853283839:stack/my-s3-global-bucket/aabd20e0-f57d-11ea-80bf-06f1487f6a64
The screenshot below:-
The problem is AWS CloudFormation is trying to create the S3 bucket rather than referring to the existing one.
But, if I try to update this SAM template like and then execute sam deploy, I get the following error:-
Waiting for changeset to be created..
Error: Failed to create changeset for the stack: my-lambda-stack, ex: Waiter ChangeSetCreateComplete failed: Waiter encountered a terminal failure state Status: FAILED. Reason: Transform AWS::Serverless-2016-10-31 failed with: Invalid Serverless Application Specification document. Number of errors found: 1. Resource with id [HelloWorldFunction] is invalid. Event with id [HelloLambdaEvent] is invalid. S3 events must reference an S3 bucket in the same template.
I'm blocked by both ends. I would really appreciate it if someone can assist to guide me writing the SAM template correctly in my Lambda so that I can refer the existing bucket properly instead of creating the new one.
Thank you
Any items listed under the Resources section refer to the resources the stack is responsible for maintaining.
When you list SrcBucket you are asking for CloudFormation to create a new S3 bucket with the name being the value of !ImportValue global-bucket which is the name of an S3 bucket you have already created.
Assuming that this is the bucket name you can simply reference it in your template as shown below.
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
hellolambda
Sample SAM Template for hellolambda
# More info about Globals: https://github.com/awslabs/serverless-application-model/blob/master/docs/globals.rst
Globals:
Function:
Timeout: 3
Resources:
HelloWorldFunction:
Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
Properties:
CodeUri: hello-world/
Handler: app.lambdaHandler
Runtime: nodejs12.x
Events:
HelloLambdaEvent:
Type: S3
Properties:
Bucket: !ImportValue global-bucket
Events: s3:ObjectCreated:*

How to export a resource name and use in different Cloudformation Stackset?

I created a CloudFormation Stackset that deployed AWS Config Rules to two accounts. Now I want to create a stackset that deploys the Remediation. the bottom lines of code work when I have it all in one CFT. but I want to deploy te detection rules in one script first then the remediation rules second. How can I reference the S3BucketEncryptionEnabled Resource from a different scipt?
---------------------Detection --------------------------------------------------------
S3BucketEncryptionEnabled:
Type: AWS::Config::ConfigRule
Properties:
Description: Checks that your Amazon S3 bucket either has S3 default encryption enabled or that the S3 bucket policy explicitly denies put-object requests without server side encryption.
Source:
Owner: AWS
SourceIdentifier: S3_BUCKET_SERVER_SIDE_ENCRYPTION_ENABLED
Scope:
ComplianceResourceTypes:
- AWS::S3::Bucket
DependsOn: ConfigRecorder
----------------------Remediation Script-----------------------------------------------
BasicRemediationConfiguration:
Type: "AWS::Config::RemediationConfiguration"
Properties:
Automatic: True
MaximumAutomaticAttempts: 5
RetryAttemptSeconds: 60
ConfigRuleName: !Ref S3BucketEncryptionEnabled
Parameters:
AutomationAssumeRole:
StaticValue:
Values: [{"Fn::GetAtt" : ["S3Role","Arn"]}]
BucketName:
ResourceValue:
Value: RESOURCE_ID
SSEAlgorithm:
StaticValue:
Values: [AES256]
TargetId: "AWS-EnableS3BucketEncryption"
TargetType: "SSM_DOCUMENT"
TargetVersion: "1"
Normally, in your Detection template you would export the S3BucketEncryptionEnabled in your outputs.
For example:
Outputs:
S3BucketEncryptionEnabled:
Value: !Ref S3BucketEncryptionEnabled
Export:
Name: MyS3BucketEncryptionEnabled
Then in your Remediation template, you would use ImportValue to import the exported value.
For example:
BasicRemediationConfiguration:
Type: "AWS::Config::RemediationConfiguration"
Properties:
Automatic: True
MaximumAutomaticAttempts: 5
RetryAttemptSeconds: 60
ConfigRuleName: !ImportValue MyS3BucketEncryptionEnabled
# remaining properties

Configuration of environment variables for different AWS Lambda aliases in CloudFormation template

I create CloudFormation template for my AWS Lambda function and I need to specify different values of environment variables for different lambda aliases.
My template looks like:
AWSTemplateFormatVersion: "2010-09-09"
Transform: "AWS::Serverless-2016-10-31"
Description: Lambda function configuration
Resources:
EndpointLambda:
Type: "AWS::Lambda::Function"
Properties:
FunctionName: "endpoint-lambda"
Handler: "com.test.aws.RequestHandler::handleRequest"
Runtime: java8
Code:
S3Bucket: "lambda-functions"
S3Key: "test-endpoint-lambda-0.0.1.jar"
Description: Test Lambda function
MemorySize: 256
Timeout: 60
Environment:
Variables:
ES_HOST: test-es-host-url
ES_ON: true
ES_PORT: 443
ES_PROTOCOL: https
REDIS_URL: test-redis-host-url
QaLambdaAlias:
Type: "AWS::Lambda::Alias"
Properties:
FunctionName: !Ref EndpointLambda
FunctionVersion: 1
Name: "QA"
Description: "QA alias"
ProdLambdaAlias:
Type: "AWS::Lambda::Alias"
Properties:
FunctionName: !Ref EndpointLambda
FunctionVersion: 1
Name: "Prod"
Description: "Production alias"
As you see, I have two aliases - QA and Prod and bunch of environment variables. I specified variables with common values in lambda function declaration. But I need to declare for QA alias env. variable's values related to QA, and for Prod alias - values for Prod environment. Any ideas how can I do that?
You can use CloudFormation Parameters to do this. As a quick example:
Parameters:
LambdaRuntime:
Type: String
Default: 'java8'
Description: What Lambda runtime do we use?
Resources:
QaLambdaAlias:
Type: "AWS::Lambda::Alias"
Properties:
FunctionName:
Ref: EndpointLambda
FunctionVersion: 1
Name: "QA"
Description: "QA alias"
Runtime:
Ref: LambdaRuntime
Then, if you want to use a different parameter, when you deploy via CLI, you can override with parameter-overrides like this:
aws cloudformation deploy --stack-name MyStack --template-file \
CloudFormation/MyStack.yaml --capabilities CAPABILITY_IAM \
--parameter-overrides LambdaRuntime=nodejs8.10

Create and Execute an aws lambda function through cloud formation

I am creating my lambda function like following through cloud formation template.
My question is , after creating this lambda resource , I want to pass it few variables as an input and execute it immediately. Is there a way I can do it through cloud formation template?
AWSTemplateFormatVersion: '2010-09-09'
Description: Create a lambda function for chef rds read replica
Parameters:
Environment:
Description: Environment that will be built
Type: String
Default: q1
AllowedValues:
- q1
Mappings:
ChefEnvironmentMap:
q1:
IAMRole: CHEFAWS-RDS
Subnets:
- subnet-***
SecurityGroups:
- sg-***
NetCoreEnvironment: qa
Resources:
ChefRDSReadOnlyReplica:
Type: AWS::Lambda::Function
Properties:
Code:
S3Bucket: cheftestbucket
S3Key: lambda/cheflambda.zip
Description: "Chef rds"
Environment:
Variables:
http_proxy: 'http://**'
https_proxy: 'http://**'
no_proxy: '169.254.169.254,127.0.0.1,localaddress,.localdomain.com'
Handler: createreadreplica.lambda_handler
MemorySize: 128
Role: arn:aws:iam::*****:role/CHEFAWS-RDS
Runtime: python2.7
Timeout: 60
VpcConfig:
SecurityGroupIds: !FindInMap [ChefEnvironmentMap, !Ref Environment, SecurityGroups]
SubnetIds: !FindInMap [ChefEnvironmentMap, !Ref Environment, Subnets]
I followed the approach to create a custom resource to execute the lambda fuction. I launched the custom resource through the cft as well, like this.
Lambdaresource:
DependsOn: ChefRDSReadOnlyReplica
Properties:
sourceregion: "us-east-1"
target_region: "us-west-2"
db_instance_id: "chef-tod-pg-rds"
dbsubnet_groupname: "******"
kms_keyid: "******"
ServiceToken: !GetAtt ChefRDSReadOnlyReplica.Arn
Type: AWS::CloudFormation::CustomResource
However, this lead to more complexity . As cloud formation of custom resource doesnt get notified after it triggers the lambda function. So it remains in “Creating Resource” State. To handle this , I had to create a new function in my lambda Python code that returned the success/failure. As mentioned here: http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-lambda-function-code.html