Cloudformation unable to create resource policy for apigateway - amazon-web-services

The resource policy is working fine when i directly pass it to the console.
Below is resource policy example :-
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": "*",
"Action": "execute-api:Invoke",
"Resource": "arn:aws:execute-api:us-west-2:339159142535:ooxmwl6q4e/*",
"Condition": {
"IpAddress": {
"aws:SourceIp": [
""14.98.8.190/32""
]
}
}
}
]
}
Now how to create a cloudformation template for this to get created and get attached to the apigateway
I tried to create a policy but as per new policy "Principal" is depricated.
I created a role also but no help. Below is role snippet :-
{
"AWSTemplateFormatVersion": "2010-09-09",
"Resources": {
"Apifirewall": {
"Type": "AWS::IAM::Role",
"Properties": {
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": [
"apigateway.amazonaws.com"
]
},
"Action": [
"sts:AssumeRole"
]
}
]
},
"Policies": [
{
"PolicyName": "Apifirewall",
"PolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "*",
"Resource": [
"arn:aws:execute-api:us-west-2:339159142535:ooxmwl6q4e/*"
],
"Condition": {
"IpAddress": {
"aws:SourceIp": [
"14.98.8.190/32"
]
}
}
}
]
}
}
]
}
}
},
"Outputs": {
"Apifirewall": {
"Value": {
"Fn::GetAtt": [
"Apifirewall",
"Arn"
]
}
}
}
}

APIGateway resource policy is not binding to IAM Policy, it's different kind of resource.
So to implement it on your RestApi your should use the Policy parameter on AWS::ApiGateway::RestApi resource on
{
"Type" : "AWS::ApiGateway::RestApi",
"Properties" : {
"ApiKeySourceType" : String,
"BinaryMediaTypes" : [ String, ... ],
"Body" : JSON object,
"BodyS3Location" : S3Location,
"CloneFrom" : String,
"Description" : String,
"EndpointConfiguration" : EndpointConfiguration,
"FailOnWarnings" : Boolean,
"MinimumCompressionSize" : Integer,
"Name" : String,
"Parameters" : { String:String, ... },
"Policy" : JSON object
}
}

Below is the entire CFT for api deployment with lambda integration
{
"AWSTemplateFormatVersion": "2010-09-09",
"Parameters": {
"AppEnv": {
"Type": "String",
"Description": "Application environment, for this deployment"
},
"DeployTag": {
"Type": "String",
"Description": "Distinct deployment tag ex: BLUE, GREEN"
}
},
"Resources": {
"LambdaExecutionRole": {
"Type": "AWS::IAM::Role",
"Properties": {
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": [
"lambda.amazonaws.com"
]
},
"Action": [
"sts:AssumeRole"
]
}
]
},
"ManagedPolicyArns": [
"arn:aws:iam::aws:policy/AWSLambdaFullAccess"
]
}
},
"RecommenderLambda": {
"Type": "AWS::Lambda::Function",
"Properties": {
"Handler": "recommender_field_validation_lambda.lambda_handler",
"FunctionName": "recommenderlambda2",
"Role": {
"Fn::GetAtt": [
"LambdaExecutionRole",
"Arn"
]
},
"Environment": {
"Variables": {
"S3_BUCKET": "belcorp.recommender.test",
"REGION_NAME": "us-west-2",
"TOPIC_ARN": {
"Fn::ImportValue": "RecommenderTopicARN"
},
"TABLE_NAME": {
"Fn::ImportValue": "recommederrequestinfo"
}
}
},
"Code": {
"S3Bucket": "belcorp.recommender.lambdas",
"S3Key": "recommender_field_validation_lambda.zip"
},
"Runtime": "python3.6",
"Timeout": 25
}
},
"LambdaPermission": {
"DependsOn": "RecommenderLambda",
"Type": "AWS::Lambda::Permission",
"Properties": {
"Action": "lambda:invokeFunction",
"FunctionName": "recommenderlambda2",
"Principal": "apigateway.amazonaws.com",
"SourceArn": {
"Fn::Join": [
"",
[
"arn:aws:execute-api:",
{
"Ref": "AWS::Region"
},
":",
{
"Ref": "AWS::AccountId"
},
":",
{
"Ref": "RecommenderApi"
},
"/*"
]
]
}
}
},
"RecommenderApi": {
"Type": "AWS::ApiGateway::RestApi",
"Properties": {
"EndpointConfiguration": {
"Types": [
"EDGE"
]
},
"Description": "RecommenderAPI",
"Name": {
"Fn::Sub": "RecommenderApi-${AppEnv}-${DeployTag}"
},
"Policy": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": "*",
"Action": "execute-api:Invoke",
"Resource": {
"Fn::Sub": "arn:aws:execute-api:${AWS::Region}:${AWS::AccountId}:*/*"
},
"Condition": {
"IpAddress": {
"aws:SourceIp": [
"14.98.8.190/32"
]
}
}
}
]
}
}
},
"ApiGatewayAccount": {
"Type": "AWS::ApiGateway::Account",
"Properties": {
"CloudWatchRoleArn": {
"Fn::ImportValue": "cloudwatchRole"
}
}
},
"ApiDeployment": {
"Type": "AWS::ApiGateway::Deployment",
"DependsOn": [
"OfferPostMethod",
"OrderPostMethod"
],
"Properties": {
"RestApiId": {
"Ref": "RecommenderApi"
},
"StageName": "dev"
}
},
"ProcessInput": {
"Type": "AWS::ApiGateway::Resource",
"Properties": {
"RestApiId": {
"Ref": "RecommenderApi"
},
"ParentId": {
"Fn::GetAtt": [
"RecommenderApi",
"RootResourceId"
]
},
"PathPart": "process-input"
}
},
"OfferLevel": {
"Type": "AWS::ApiGateway::Resource",
"Properties": {
"RestApiId": {
"Ref": "RecommenderApi"
},
"ParentId": {
"Ref": "ProcessInput"
},
"PathPart": "offer-level"
}
},
"OrderLevel": {
"Type": "AWS::ApiGateway::Resource",
"Properties": {
"RestApiId": {
"Ref": "RecommenderApi"
},
"ParentId": {
"Ref": "ProcessInput"
},
"PathPart": "order-level"
}
},
"OfferPostMethod": {
"DependsOn": "RecommenderLambda",
"Type": "AWS::ApiGateway::Method",
"Properties": {
"RestApiId": {
"Ref": "RecommenderApi"
},
"ResourceId": {
"Ref": "OfferLevel"
},
"HttpMethod": "POST",
"AuthorizationType": "NONE",
"Integration": {
"Type": "AWS_PROXY",
"IntegrationHttpMethod": "POST",
"Uri": {
"Fn::Join": [
"",
[
"arn:aws:apigateway:",
{
"Ref": "AWS::Region"
},
":lambda:path/2015-03-31/functions/",
{
"Fn::GetAtt": [
"RecommenderLambda",
"Arn"
]
},
"/invocations"
]
]
},
"IntegrationResponses": [
{
"StatusCode": 200,
"ResponseTemplates": {
"application/json": "$input.json('$.body')"
}
}
]
}
}
},
"OrderPostMethod": {
"DependsOn": "RecommenderLambda",
"Type": "AWS::ApiGateway::Method",
"Properties": {
"RestApiId": {
"Ref": "RecommenderApi"
},
"ResourceId": {
"Ref": "OrderLevel"
},
"HttpMethod": "POST",
"AuthorizationType": "NONE",
"Integration": {
"Type": "AWS_PROXY",
"IntegrationHttpMethod": "POST",
"Uri": {
"Fn::Join": [
"",
[
"arn:aws:apigateway:",
{
"Ref": "AWS::Region"
},
":lambda:path/2015-03-31/functions/",
{
"Fn::GetAtt": [
"RecommenderLambda",
"Arn"
]
},
"/invocations"
]
]
},
"IntegrationResponses": [
{
"StatusCode": 200,
"ResponseTemplates": {
"application/json": "$input.json('$.body')"
}
}
]
}
}
}
},
"Outputs": {
"RootUrl": {
"Description": "Root URL of the API gateway",
"Value": {
"Fn::Join": [
"",
[
"https://",
{
"Ref": "RecommenderApi"
},
".execute-api.",
{
"Ref": "AWS::Region"
},
".amazonaws.com"
]
]
}
},
"OfferUrl": {
"Description": "Root URL of the API gateway",
"Value": {
"Fn::Join": [
"",
[
"https://",
{
"Ref": "RecommenderApi"
},
".execute-api.",
{
"Ref": "AWS::Region"
},
".amazonaws.com",
"/dev/process-input/offer-level"
]
]
}
},
"OrderUrl": {
"Description": "Root URL of the API gateway",
"Value": {
"Fn::Join": [
"",
[
"https://",
{
"Ref": "RecommenderApi"
},
".execute-api.",
{
"Ref": "AWS::Region"
},
".amazonaws.com",
"/dev/process-input/order-level"
]
]
}
}
}
}

Too long for a comment. This is the transformed YAML from this answer, which a commenter pointed out can be done in CloudFormation Designer:
AWSTemplateFormatVersion: 2010-09-09
Parameters:
AppEnv:
Type: String
Description: 'Application environment, for this deployment'
DeployTag:
Type: String
Description: 'Distinct deployment tag ex: BLUE, GREEN'
Resources:
LambdaExecutionRole:
Type: 'AWS::IAM::Role'
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
Action:
- 'sts:AssumeRole'
ManagedPolicyArns:
- 'arn:aws:iam::aws:policy/AWSLambdaFullAccess'
RecommenderLambda:
Type: 'AWS::Lambda::Function'
Properties:
Handler: recommender_field_validation_lambda.lambda_handler
FunctionName: recommenderlambda2
Role: !GetAtt
- LambdaExecutionRole
- Arn
Environment:
Variables:
S3_BUCKET: belcorp.recommender.test
REGION_NAME: us-west-2
TOPIC_ARN: !ImportValue RecommenderTopicARN
TABLE_NAME: !ImportValue recommederrequestinfo
Code:
S3Bucket: belcorp.recommender.lambdas
S3Key: recommender_field_validation_lambda.zip
Runtime: python3.6
Timeout: 25
LambdaPermission:
DependsOn: RecommenderLambda
Type: 'AWS::Lambda::Permission'
Properties:
Action: 'lambda:invokeFunction'
FunctionName: recommenderlambda2
Principal: apigateway.amazonaws.com
SourceArn: !Join
- ''
- - 'arn:aws:execute-api:'
- !Ref 'AWS::Region'
- ':'
- !Ref 'AWS::AccountId'
- ':'
- !Ref RecommenderApi
- /*
RecommenderApi:
Type: 'AWS::ApiGateway::RestApi'
Properties:
EndpointConfiguration:
Types:
- EDGE
Description: RecommenderAPI
Name: !Sub 'RecommenderApi-${AppEnv}-${DeployTag}'
Policy:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal: '*'
Action: 'execute-api:Invoke'
Resource: !Sub 'arn:aws:execute-api:${AWS::Region}:${AWS::AccountId}:*/*'
Condition:
IpAddress:
'aws:SourceIp':
- 14.98.8.190/32
ApiGatewayAccount:
Type: 'AWS::ApiGateway::Account'
Properties:
CloudWatchRoleArn: !ImportValue cloudwatchRole
ApiDeployment:
Type: 'AWS::ApiGateway::Deployment'
DependsOn:
- OfferPostMethod
- OrderPostMethod
Properties:
RestApiId: !Ref RecommenderApi
StageName: dev
ProcessInput:
Type: 'AWS::ApiGateway::Resource'
Properties:
RestApiId: !Ref RecommenderApi
ParentId: !GetAtt
- RecommenderApi
- RootResourceId
PathPart: process-input
OfferLevel:
Type: 'AWS::ApiGateway::Resource'
Properties:
RestApiId: !Ref RecommenderApi
ParentId: !Ref ProcessInput
PathPart: offer-level
OrderLevel:
Type: 'AWS::ApiGateway::Resource'
Properties:
RestApiId: !Ref RecommenderApi
ParentId: !Ref ProcessInput
PathPart: order-level
OfferPostMethod:
DependsOn: RecommenderLambda
Type: 'AWS::ApiGateway::Method'
Properties:
RestApiId: !Ref RecommenderApi
ResourceId: !Ref OfferLevel
HttpMethod: POST
AuthorizationType: NONE
Integration:
Type: AWS_PROXY
IntegrationHttpMethod: POST
Uri: !Join
- ''
- - 'arn:aws:apigateway:'
- !Ref 'AWS::Region'
- ':lambda:path/2015-03-31/functions/'
- !GetAtt
- RecommenderLambda
- Arn
- /invocations
IntegrationResponses:
- StatusCode: 200
ResponseTemplates:
application/json: $input.json('$.body')
OrderPostMethod:
DependsOn: RecommenderLambda
Type: 'AWS::ApiGateway::Method'
Properties:
RestApiId: !Ref RecommenderApi
ResourceId: !Ref OrderLevel
HttpMethod: POST
AuthorizationType: NONE
Integration:
Type: AWS_PROXY
IntegrationHttpMethod: POST
Uri: !Join
- ''
- - 'arn:aws:apigateway:'
- !Ref 'AWS::Region'
- ':lambda:path/2015-03-31/functions/'
- !GetAtt
- RecommenderLambda
- Arn
- /invocations
IntegrationResponses:
- StatusCode: 200
ResponseTemplates:
application/json: $input.json('$.body')
Outputs:
RootUrl:
Description: Root URL of the API gateway
Value: !Join
- ''
- - 'https://'
- !Ref RecommenderApi
- .execute-api.
- !Ref 'AWS::Region'
- .amazonaws.com
OfferUrl:
Description: Root URL of the API gateway
Value: !Join
- ''
- - 'https://'
- !Ref RecommenderApi
- .execute-api.
- !Ref 'AWS::Region'
- .amazonaws.com
- /dev/process-input/offer-level
OrderUrl:
Description: Root URL of the API gateway
Value: !Join
- ''
- - 'https://'
- !Ref RecommenderApi
- .execute-api.
- !Ref 'AWS::Region'
- .amazonaws.com
- /dev/process-input/order-level

If you are using YAML for CloudFormation, the Policy can be in YAML. There is no need to use JSON for it. For example:
Parameters:
ApiAllowedIps:
Type: CommaDelimitedList
RestApi:
Type: AWS::ApiGateway::RestApi
Properties:
...
Policy:
Version: "2012-10-17"
Statement:
- Effect: Allow
Action: '*'
Principal: '*'
Resource: '*'
Condition:
IpAddress:
aws:SourceIp: !Ref ApiAllowedIps

Related

aws cloudformation template for spot request in ec2 instance

I need a sample cloud formation template to add spot requests while provisioning the ec2 instance in AWS.I have tried with console to provision spot instances but I couldn't find any exact template for add spot request in ec2
You Can create a SpotFleet resource, here's an example
SpotFleet:
Type: AWS::EC2::SpotFleet
Properties:
SpotFleetRequestConfigData:
IamFleetRole: !GetAtt [IAMFleetRole, Arn]
SpotPrice: '1000'
TargetCapacity:
Ref: TargetCapacity
LaunchSpecifications:
- EbsOptimized: 'false'
InstanceType:
Ref: InstanceType
ImageId:
Fn::FindInMap:
- AWSRegionArch2AMI
- Ref: AWS::Region
- Fn::FindInMap:
- AWSInstanceType2Arch
- Ref: InstanceType
- Arch
SubnetId:
Ref: Subnet1
WeightedCapacity: '8'
- EbsOptimized: 'true'
InstanceType:
Ref: InstanceType
ImageId:
Fn::FindInMap:
- AWSRegionArch2AMI
- Ref: AWS::Region
- Fn::FindInMap:
- AWSInstanceType2Arch
- Ref: InstanceType
- Arch
Monitoring:
Enabled: 'true'
SecurityGroups:
- GroupId:
Fn::GetAtt:
- SG0
- GroupId
SubnetId:
Ref: Subnet0
IamInstanceProfile:
Arn:
Fn::GetAtt:
- RootInstanceProfile
- Arn
WeightedCapacity: '8'
You need to create Spot-fleet resource.
Example :
"SpotFleet": {
"Type": "AWS::EC2::SpotFleet",
"Properties": {
"SpotFleetRequestConfigData": {
"IamFleetRole": { "Fn::GetAtt": [ "IAMFleetRole", "Arn"] },
"SpotPrice": "1000",
"TargetCapacity": { "Ref": "TargetCapacity" },
"LaunchSpecifications": [
{
"EbsOptimized": "false",
"InstanceType": { "Ref": "InstanceType" },
"ImageId": { "Fn::FindInMap": [ "AWSRegionArch2AMI", { "Ref": "AWS::Region" },
{ "Fn::FindInMap": [ "AWSInstanceType2Arch", { "Ref": "InstanceType" }, "Arch" ] }
]},
"SubnetId": { "Ref": "Subnet1" },
"WeightedCapacity": "8"
},
{
"EbsOptimized": "true",
"InstanceType": { "Ref": "InstanceType" },
"ImageId": { "Fn::FindInMap": [ "AWSRegionArch2AMI", { "Ref": "AWS::Region" },
{ "Fn::FindInMap": [ "AWSInstanceType2Arch", { "Ref": "InstanceType" }, "Arch" ] }
]},
"Monitoring": { "Enabled": "true" },
"SecurityGroups": [ { "GroupId": { "Fn::GetAtt": [ "SG0", "GroupId" ] } } ],
"SubnetId": { "Ref": "Subnet0" },
"IamInstanceProfile": { "Arn": { "Fn::GetAtt": [ "RootInstanceProfile", "Arn" ] } },
"WeightedCapacity": "8"
}
]
}
}
}
More details can be found in this link :
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ec2-spotfleet.html

Invalid Parameter Combination when creating a new Elasticache Redis Instance using Cloudformation

I am creating a new stack which includes a Redis instance, after starting the creation, I get this error for redis:
Cannot use the given parameters when creating new replication group
in an existing global replication group. (Service: AmazonElastiCache;
Status Code: 400; Error Code: InvalidParameterCombination;
I am new to using elasticache within cloudformation and the documentation doesn't provide a lot of helpful information for the valid/correct combination.
I am using the following snippet:
cachesubnet:
Type: AWS::ElastiCache::SubnetGroup
Properties:
CacheSubnetGroupName: !Join ["-" , ["rb", !Ref Environment, "redis-subnet-group"]]
Description: subnet group for redis
SubnetIds:
- !Ref private1a
- !Ref private1b
Tags:
- Key: environment
Value: !Ref Environment
redis2:
Type: AWS::ElastiCache::ReplicationGroup
Properties:
AtRestEncryptionEnabled: True
AutomaticFailoverEnabled: True
AutoMinorVersionUpgrade: True
CacheNodeType: cache.m5.large
CacheParameterGroupName: default.redis5.0
CacheSubnetGroupName: !Ref cachesubnet
Engine: redis
EngineVersion: 5.0.6
NumNodeGroups: 1
GlobalReplicationGroupId: !Join ["-" , ["rb-horizon", !Ref Environment]]
MultiAZEnabled: True
NodeGroupConfiguration:
- PrimaryAvailabilityZone: us-east-1a
- ReplicaAvailabilityZones:
- us-east-1b
- ReplicaCount: 1
PreferredCacheClusterAZs:
- us-east-1a
- us-east-1b
PreferredMaintenanceWindow: mon:06:30-mon:07:30
ReplicationGroupDescription: Horizon for env
ReplicationGroupId: !Join ["-" , ["rb-horizon", !Ref Environment, rgi]]
SecurityGroupIds:
- !GetAtt mainSecGroup.GroupId
Tags:
- Key: environment
Value: !Ref Environment
Ran into a similar issue today and was able to solve it via some information gleaned from this thread https://old.reddit.com/r/aws/comments/rvgv15/demystifying_redis_in_cloudformation_multiaz_and/
You do not need to specify parameters such as Engine, EngineVersion, CacheNodeType, etc... because ElastiCache will know to use the same values as the ones provided in the primary replication group of the global datastore.
While it says there that the params are not needed, they actually do raise the above error message. I added a condition around those fields and then my stack went through fine.
// edited to add example below
Example redis template. I removed the mappings and snipped the security group ingress rules which are not relevant for demonstration purposes.
The GlobalReplicationGroupId param should not be set for the PRIMARY redis group. Instead, the PRIMARY is joined to the global data store while creating the GlobalReplicationGroup. You should only add the primary member in that resource.
For creating the secondary groups, first launch redis with this param empty and then update it with the param pointing to the previously created GlobalReplicationGroup to join the global store.
{
"AWSTemplateFormatVersion": "2010-09-09",
"Parameters": {
"GlobalReplicationGroupId": {
"Description": "ID of the previously created Global Replication Group (optional)",
"Type": "String",
"Default": ""
}
},
"Conditions": {
"IsGlobalStore": {
"Fn::Not": [{
"Fn::Equals": [ { "Ref": "GlobalReplicationGroupId" }, "" ]
}]
}
},
"Resources": {
"RedisParameterGroup": {
"Type": "AWS::ElastiCache::ParameterGroup",
"Properties": {
"CacheParameterGroupFamily": "redis6.x",
"Description": {
"Fn::Join": [
"",
[
"Redis Parameter Group ",
{
"Ref": "EnvironmentName"
}
]
]
}
}
},
"RedisSubnetGroup": {
"Type": "AWS::ElastiCache::SubnetGroup",
"Properties": {
"CacheSubnetGroupName": {
"Fn::Join": [
"", [
"", { "Ref": "EnvironmentName" }, "-redis-subnet-group"
]
]
},
"SubnetIds": {
"Fn::FindInMap": [
"privateSubnets",
{
"Ref": "AWS::Region"
},
{
"Ref": "EnvironmentName"
}
]
},
"Description": {
"Fn::Join": [
"",
[
"Redis Subnet Group ",
{
"Ref": "EnvironmentName"
}
]
]
}
}
},
"RedisSecurityGroup": {
"Type": "AWS::EC2::SecurityGroup",
"Properties": {
"GroupDescription": {
"Fn::Join": [
"",
[
"Redis Security Group ",
{
"Ref": "EnvironmentName"
}
]
]
},
"VpcId": {
"Fn::FindInMap": [
"vpcId",
{
"Ref": "AWS::Region"
},
{
"Ref": "EnvironmentName"
}
]
},
"SecurityGroupIngress": [
],
"Tags": [
{
"Key": "Name",
"Value": {
"Fn::Join": [
"",
[
"Redis-",
{
"Ref": "EnvironmentName"
},
"-SecurityGroup"
]
]
}
},
{
"Key": "Product",
"Value": "Redis"
},
{
"Key": "Environment",
"Value": {
"Ref": "EnvironmentName"
}
}
]
}
},
"RedisReplicationGroup": {
"Type": "AWS::ElastiCache::ReplicationGroup",
"Properties": {
"GlobalReplicationGroupId": {
"Fn::If": [
"IsGlobalStore",
{"Ref": "GlobalReplicationGroupId"},
{"Ref": "AWS::NoValue"}
]
},
"AutomaticFailoverEnabled": "true",
"CacheNodeType": {
"Fn::If": [
"IsGlobalStore",
{
"Ref": "AWS::NoValue"
},
{
"Fn::FindInMap": [
"RedisNodeType",
{
"Ref": "AWS::Region"
},
{
"Ref": "EnvironmentName"
}
]
}
]
},
"CacheParameterGroupName": {
"Fn::If": [
"IsGlobalStore",
{
"Ref": "AWS::NoValue"
},
{
"Ref": "RedisParameterGroup"
}
]
},
"CacheSubnetGroupName": {
"Ref": "RedisSubnetGroup"
},
"Engine": {
"Fn::If": [
"IsGlobalStore",
{
"Ref": "AWS::NoValue"
},
"redis"
]
},
"EngineVersion": {
"Fn::If": [
"IsGlobalStore",
{
"Ref": "AWS::NoValue"
},
"6.2"
]
},
"NumCacheClusters": {
"Fn::FindInMap": [
"NumCacheClusters",
{
"Ref": "AWS::Region"
},
{
"Ref": "EnvironmentName"
}
]
},
"PreferredCacheClusterAZs": {
"Fn::FindInMap": [
"ZoneMap",
{
"Ref": "AWS::Region"
},
"AZ"
]
},
"ReplicationGroupDescription": {
"Fn::Join": [
"",
[
"Redis Replication Group ",
{
"Ref": "EnvironmentName"
}
]
]
},
"SecurityGroupIds": [
{
"Ref": "RedisSecurityGroup"
}
],
"Tags": [
{
"Key": "Name",
"Value": {
"Fn::Join": [
"",
[
"Redis-",
{
"Ref": "EnvironmentName"
}
]
]
}
},
{
"Key": "Product",
"Value": "Redis"
},
{
"Key": "Environment",
"Value": {
"Ref": "EnvironmentName"
}
}
]
}
}
},
"Outputs": {}
}

Why is AWS CloudFormation throwing "Encountered unsupported property InstanceGroups"?

When I deploy the below AWS CloudFormation script, I am getting the following error: "Encountered unsupported property InstanceGroups"
I have used InstanceGroups in the past without any issues. Here is an example of how others using it: https://noise.getoto.net/tag/amazon-emr/
I am using EMR 5.17.0, which I have used to set up before.
{
"Description": "Spark ETL EMR CloudFormation",
"Resources": {
"EMRCluster": {
"Type": "AWS::EMR::Cluster",
"Properties": {
"Applications": [
{
"Name": "Hadoop"
},
{
"Name": "Spark"
},
{
"Name": "Ganglia"
},
{
"Name": "Zeppelin"
}
],
"AutoScalingRole": "EMR_AutoScaling_DefaultRole",
"BootstrapActions": [
{
"Path": "s3://somepath/scripts/install_pip36_dependencies.sh",
"Args": [
"relay==0.0.1"
],
"Name": "install_pip36_dependencies"
}
],
"Configurations": [
{
"Classification": "yarn-site",
"Properties": {
"yarn.scheduler.fair.preemption": "False",
"yarn.resourcemanager.am.max-attempts": "1"
},
"Configurations": []
},
{
"Classification": "core-site",
"Properties": {
"fs.s3.canned.acl": "BucketOwnerFullControl"
},
"Configurations": []
}
],
"EbsRootVolumeSize": 10,
"InstanceGroups": [
{
"Name": "Master",
"Market": "ON_DEMAND",
"InstanceRole": "MASTER",
"InstanceType": "m5.2xlarge",
"InstanceCount": 1,
"EbsConfiguration": {
"EbsBlockDeviceConfigs": [
{
"VolumeSpecification": {
"SizeInGB": 100,
"VolumeType": "64"
},
"VolumesPerInstance": 1
}
],
"EbsOptimized": "True"
}
},
{
"Name": "Core",
"Market": "ON_DEMAND",
"InstanceGroupType": "CORE",
"InstanceType": "m5.2xlarge",
"InstanceCount": 5,
"EbsConfiguration": {
"EbsBlockDeviceConfigs": [
{
"VolumeSpecification": {
"SizeInGB": 100,
"VolumeType": "gp2"
},
"VolumesPerInstance": 1
}
],
"EbsOptimized": "True"
}
},
{
"Name": "Task - 3",
"Market": "ON_DEMAND",
"InstanceGroupType": "TASK",
"InstanceType": "m5.2xlarge",
"InstanceCount": 2,
"EbsConfiguration": {
"EbsBlockDeviceConfigs": [
{
"VolumeSpecification": {
"SizeInGB": 32,
"VolumeType": "gp2"
},
"VolumesPerInstance": 1
}
],
"EbsOptimized": "True"
}
}
],
"LogUri": "s3://somepath/emr-logs/",
"Name": "EMR CF",
"ReleaseLabel": "emr-5.17.0",
"ServiceRole": "EMR_DefaultRole",
"VisibleToAllUsers": "True"
}
}
}
}
When the CF script is loaded, it should create an AWS EMR cluster
Aws recommends that you set MasterInstanceGroup and CoreInstanceGroup under Instances
I give you an example of the Instances property of an EMR Cluster with Hadoop, Hbase, Spark, Ganglia and Zookeeper:
Instances:
Ec2KeyName: !Ref KeyName
Ec2SubnetId: !ImportValue MySubnetPrivateA
EmrManagedMasterSecurityGroup: !ImportValue EmrMasterSgId
AdditionalMasterSecurityGroups:
- !ImportValue EmrMasterAdditionalSgId
EmrManagedSlaveSecurityGroup: !ImportValue EmrSlaveSgId
AdditionalSlaveSecurityGroups:
- !ImportValue EmrSlaveAdditionalSgId
ServiceAccessSecurityGroup: !ImportValue EmrServiceSgId
MasterInstanceGroup:
InstanceCount: 1
InstanceType: !Ref MasterInstanceType
Market: ON_DEMAND
Name: Master
CoreInstanceGroup:
InstanceCount: !Ref NumberOfCoreInstances
InstanceType: !Ref CoreInstanceType
Market: ON_DEMAND
Name: Core
TerminationProtected: false
VisibleToAllUsers: true
JobFlowRole: !Ref EMRClusterinstanceProfile
ReleaseLabel: !Ref ReleaseLabel
LogUri: !Ref LogUri
Name: !Ref EMRClusterName
AutoScalingRole: EMR_AutoScaling_DefaultRole
ServiceRole: !Ref EMRClusterServiceRole
Tags:
-
Key: "cluster_name"
Value: "master.emr.my.com"
You can see the complete AWS template here.

AWS Deploying environment and create environments for dev and prod

Greeting all,
I'm looking for a way to deploy my application which contains:
API Gateway
DynamoDB
Lambda Functions
An S3 bucket
I looked at CloudFormation and CodeDeploy but I'm unsure how to proceed without EC2...
All the information I find is for EC2, I haven't found any information regarding deploying the app above...
The goal is to have a deployment script that deploys app to an environment automatically with technology from AWS. (Basically duplicating my environment)
Any help would greatly be appreciated.
EDIT: I need to be able to export from one AWS account then import onto another AWS account.
Cheers!
In order to deploy your CloudFormation stack into a "different" environment, you have to parameterize your CloudFormation stack name and resource names. (You don't have to parameterize the AWS::Serverless::Function function in this example because CloudFormation automatically creates a function name if no function name is specified, but for most other resources it's necessary)
Example CloudFormation template cfn.yml using the Serverless Application Model (SAM):
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: Deploys a simple AWS Lambda using different environments.
Parameters:
Env:
Type: String
Description: The environment you're deploying to.
Resources:
ServerlessFunction:
Type: AWS::Serverless::Function
Properties:
Handler: index.handler
Runtime: nodejs12.x
CodeUri: ./
Policies:
- AWSLambdaBasicExecutionRole
MyBucket:
Type: AWS::S3::Bucket
Properties:
BucketName: !Sub 'my-bucket-name-${Env}'
You can add further resources like a DynamoDB table. The API Gateway is automatically created if you're using SAM and provide an Events section in your AWS::Serverless::Function resource. See also this SAM example code from the serverless-app-examples repository.
Example deploy.sh script:
#!/usr/bin/env bash
LAMBDA_BUCKET="Your-S3-Bucket-Name"
# change this ENV variable depending on the environment you want to deploy
ENV="prd"
STACK_NAME="aws-lambda-cf-environments-${ENV}"
# now package the CloudFormation template which automatically uploads the Lambda function artifacts to S3 -> generated a "packaged" CloudFormation template cfn.packaged.yml
aws cloudformation package --template-file cfn.yml --s3-bucket ${LAMBDA_BUCKET} --output-template-file cfn.packaged.yml
# ... and deploy the packaged CloudFormation template
aws cloudformation deploy --template-file cfn.packaged.yml --stack-name ${STACK_NAME} --capabilities CAPABILITY_IAM --parameter-overrides Env=${ENV}
See the full example code here. Just deploy the script using ./deploy.sh and change the ENV variable.
Based JSON examples.
Lambda function AWS::Lambda::Function
This example creates a Lambda function and an IAM Role attached to it.
Language: NodeJs.
"LambdaRole": {
"Type": "AWS::IAM::Role",
"Properties": {
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": [
"lambda.amazonaws.com"
]
},
"Action": [
"sts:AssumeRole"
]
}
]
},
"Policies": [
{
"PolicyName": "LambdaSnsNotification",
"PolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowSnsActions",
"Effect": "Allow",
"Action": [
"sns:Publish",
"sns:Subscribe",
"sns:Unsubscribe",
"sns:DeleteTopic",
"sns:CreateTopic"
],
"Resource": "*"
}
]
}
}
]
}
},
"LambdaFunctionMessageSNSTopic": {
"Type": "AWS::Lambda::Function",
"Properties": {
"Description": "Send message to a specific topic that will deliver MSG to a receiver.",
"Handler": "index.handler",
"MemorySize": 128,
"Role": {
"Fn::GetAtt": [
"LambdaRole",
"Arn"
]
},
"Runtime": "nodejs6.10",
"Timeout": 60,
"Environment": {
"Variables": {
"sns_topic_arn": ""
}
},
"Code": {
"ZipFile": {
"Fn::Join": [
"\n",
[
"var AWS = require('aws-sdk');",
"};"
]
]
}
}
}
}
API Gateway AWS::ApiGateway::RestApi
This example creates Role, RestAPI, Usageplan, Keys and permission to execute lambda from a Request method.
"MSGGatewayRestApi": {
"Type": "AWS::ApiGateway::RestApi",
"Properties": {
"Name": "MSG RestApi",
"Description": "API used for sending MSG",
"FailOnWarnings": true
}
},
"MSGGatewayRestApiUsagePlan": {
"Type": "AWS::ApiGateway::UsagePlan",
"Properties": {
"ApiStages": [
{
"ApiId": {
"Ref": "MSGGatewayRestApi"
},
"Stage": {
"Ref": "MSGGatewayRestApiStage"
}
}
],
"Description": "Usage plan for stage v1",
"Quota": {
"Limit": 5000,
"Period": "MONTH"
},
"Throttle": {
"BurstLimit": 200,
"RateLimit": 100
},
"UsagePlanName": "Usage_plan_for_stage_v1"
}
},
"RestApiUsagePlanKey": {
"Type": "AWS::ApiGateway::UsagePlanKey",
"Properties": {
"KeyId": {
"Ref": "MSGApiKey"
},
"KeyType": "API_KEY",
"UsagePlanId": {
"Ref": "MSGGatewayRestApiUsagePlan"
}
}
},
"MSGApiKey": {
"Type": "AWS::ApiGateway::ApiKey",
"Properties": {
"Name": "MSGApiKey",
"Description": "CloudFormation API Key v1",
"Enabled": "true",
"StageKeys": [
{
"RestApiId": {
"Ref": "MSGGatewayRestApi"
},
"StageName": {
"Ref": "MSGGatewayRestApiStage"
}
}
]
}
},
"MSGGatewayRestApiStage": {
"DependsOn": [
"ApiGatewayAccount"
],
"Type": "AWS::ApiGateway::Stage",
"Properties": {
"DeploymentId": {
"Ref": "RestAPIDeployment"
},
"MethodSettings": [
{
"DataTraceEnabled": true,
"HttpMethod": "*",
"LoggingLevel": "INFO",
"ResourcePath": "/*"
}
],
"RestApiId": {
"Ref": "MSGGatewayRestApi"
},
"StageName": "v1"
}
},
"ApiGatewayCloudWatchLogsRole": {
"Type": "AWS::IAM::Role",
"Properties": {
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": [
"apigateway.amazonaws.com"
]
},
"Action": [
"sts:AssumeRole"
]
}
]
},
"Policies": [
{
"PolicyName": "ApiGatewayLogsPolicy",
"PolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:DescribeLogGroups",
"logs:DescribeLogStreams",
"logs:PutLogEvents",
"logs:GetLogEvents",
"logs:FilterLogEvents"
],
"Resource": "*"
}
]
}
}
]
}
},
"ApiGatewayAccount": {
"Type": "AWS::ApiGateway::Account",
"Properties": {
"CloudWatchRoleArn": {
"Fn::GetAtt": [
"ApiGatewayCloudWatchLogsRole",
"Arn"
]
}
}
},
"RestAPIDeployment": {
"Type": "AWS::ApiGateway::Deployment",
"DependsOn": [
"MSGGatewayRequest"
],
"Properties": {
"RestApiId": {
"Ref": "MSGGatewayRestApi"
},
"StageName": "DummyStage"
}
},
"ApiGatewayMSGResource": {
"Type": "AWS::ApiGateway::Resource",
"Properties": {
"RestApiId": {
"Ref": "MSGGatewayRestApi"
},
"ParentId": {
"Fn::GetAtt": [
"MSGGatewayRestApi",
"RootResourceId"
]
},
"PathPart": "delivermessage"
}
},
"MSGGatewayRequest": {
"DependsOn": "LambdaPermission",
"Type": "AWS::ApiGateway::Method",
"Properties": {
"ApiKeyRequired": true,
"AuthorizationType": "NONE",
"HttpMethod": "POST",
"Integration": {
"Type": "AWS",
"IntegrationHttpMethod": "POST",
"Uri": {
"Fn::Join": [
"",
[
"arn:aws:apigateway:",
{
"Ref": "AWS::Region"
},
":lambda:path/2015-03-31/functions/",
{
"Fn::GetAtt": [
"LambdaFunctionMessageSNSTopic",
"Arn"
]
},
"/invocations"
]
]
},
"IntegrationResponses": [
{
"StatusCode": 200
},
{
"SelectionPattern": "500.*",
"StatusCode": 500
},
{
"SelectionPattern": "412.*",
"StatusCode": 412
}
],
"RequestTemplates": {
"application/json": ""
}
},
"RequestParameters": {
},
"ResourceId": {
"Ref": "ApiGatewayMSGResource"
},
"RestApiId": {
"Ref": "MSGGatewayRestApi"
},
"MethodResponses": [
{
"StatusCode": 200
},
{
"StatusCode": 500
},
{
"StatusCode": 412
}
]
}
},
"LambdaPermission": {
"Type": "AWS::Lambda::Permission",
"Properties": {
"Action": "lambda:invokeFunction",
"FunctionName": {
"Fn::GetAtt": [
"LambdaFunctionMessageSNSTopic",
"Arn"
]
},
"Principal": "apigateway.amazonaws.com",
"SourceArn": {
"Fn::Join": [
"",
[
"arn:aws:execute-api:",
{
"Ref": "AWS::Region"
},
":",
{
"Ref": "AWS::AccountId"
},
":",
{
"Ref": "MSGGatewayRestApi"
},
"/*"
]
]
}
}
}
DynamoDB AWS::DynamoDB::Table
This example creates a DynamoDB table MyCrossConfig and an alarms for it.
"TableMyCrossConfig": {
"Type": "AWS::DynamoDB::Table",
"Properties": {
"TableName": "MyCrossConfig",
"AttributeDefinitions": [
{
"AttributeName": "id",
"AttributeType": "S"
}
],
"KeySchema": [
{
"AttributeName": "id",
"KeyType": "HASH"
}
],
"ProvisionedThroughput": {
"ReadCapacityUnits": "5",
"WriteCapacityUnits": "5"
}
}
},
"alarmTargetTrackingtableMyCrossConfigProvisionedCapacityLowdfcae8d90ee2487a8e59c7bc0f9f6bd9": {
"Type": "AWS::CloudWatch::Alarm",
"Properties": {
"ActionsEnabled": "true",
"AlarmDescription": {
"Fn::Join": [
"",
[
"DO NOT EDIT OR DELETE. For TargetTrackingScaling policy arn:aws:autoscaling:",
{
"Ref": "AWS::Region"
},
":",
{
"Ref": "AWS::AccountId"
},
":scalingPolicy:7558858e-b58c-455c-be34-6de387a0c6d1:resource/dynamodb/table/MyCrossConfig:policyName/DynamoDBReadCapacityUtilization:table/MyCrossConfig."
]
]
},
"ComparisonOperator": "LessThanThreshold",
"EvaluationPeriods": "3",
"MetricName": "ProvisionedReadCapacityUnits",
"Namespace": "AWS/DynamoDB",
"Period": "300",
"Statistic": "Average",
"Threshold": "5.0",
"AlarmActions": [
{
"Fn::Join": [
"",
[
"arn:aws:autoscaling:",
{
"Ref": "AWS::Region"
},
":",
{
"Ref": "AWS::AccountId"
},
":scalingPolicy:7558858e-b58c-455c-be34-6de387a0c6d1:resource/dynamodb/table/MyCrossConfig:policyName/DynamoDBReadCapacityUtilization:table/MyCrossConfig"
]
]
}
],
"Dimensions": [
{
"Name": "TableName",
"Value": "MyCrossConfig"
}
]
}
}
s3 bucket AWS::S3::Bucket
This example creates a Bucket with name configbucket- + AWS::AccountId
"ConfigBucket": {
"Type": "AWS::S3::Bucket",
"Properties": {
"BucketName": {
"Fn::Join": [
"",
[
"configbucket-",
{
"Ref": "AWS::AccountId"
}
]
]
}
},
"DeletionPolicy": "Delete"
}
Now you need to put altogether, make the reference in the template, Etc.
Hope it helps!
My guess would be that you could use CloudFormation for such an app, but I'm also unfamiliar.
What I have had success with is writing small scripts which leverage the awscli utility to accomplish this. Additionally, you'll need a strategy for how you setup a new environment.
Typically, what I have done is to use a different suffix on DynamoDB tables and S3 buckets to represent different environments. Lambda + API Gateway have the idea of different versions baked in, so you can support different environments there as well.
For really small projects, I have even setup my Dynamo schema to support many environments within a single table. This is nice for pet or small projects because it's cheaper.
Built my own SDK for deployments, it's in the making...
https://github.com/LucLaverdure/aws-sdk
You will need to use the following shell scripts within the containers:
export.sh
import.sh
Requirements:
AWS CLI
Python
pip
npm
jq

AWS Lambda scheduled event source via cloudformation

I already have my lambda / roles defined in cloudformation and would love to also use it to add a scheduled eventsources ... are there any docs or examples around ?
Use Aws::Event::Rule with a ScheduleExpression and a AWS::Lambda::Permission
// rule to periodically call the lambda
"TagWatcherRule": {
"Type": "AWS::Events::Rule",
"Properties": {
"ScheduleExpression": "rate(10 minutes)",
"Targets": [
{
"Id": "TagWatcherScheduler",
"Arn": {
"Fn::GetAtt": [
"TagWatcherFunction",
"Arn"
]
}
}
]
}
},
// role may call the lambda
"InvokeLambdaPermission": {
"Type": "AWS::Lambda::Permission",
"Properties": {
"FunctionName": {
"Fn::GetAtt": [
"TagWatcherFunction",
"Arn"
]
},
"Action": "lambda:InvokeFunction",
"Principal": "events.amazonaws.com",
"SourceArn": {
"Fn::GetAtt": [
"TagWatcherRule",
"Arn"
]
}
}
}
I solved same problem.
"RoleForLambdaStopEC2Instances" : {
"Type": "AWS::IAM::Role",
"Properties": {
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Service": "lambda.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
},
"Policies": [
{
"PolicyName": "LambdaStopEC2InstancesPolicy",
"PolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents",
"ec2:StopInstances"
],
"Resource": [
"arn:aws:logs:*:*:*",
"arn:aws:ec2:*"
]
}
]
}
}
],
"Path": "/"
}
},
"LambdaStopEC2Instances": {
"Type": "AWS::Lambda::Function",
"Properties": {
"Code": {
"S3Bucket": "XXXXXXXXXXXXXXXXX",
"S3Key": "XXXXXXXXXXXXXXXXXX"
},
"Handler": "stopEC2Instances.handler",
"Role": { "Fn::GetAtt" : ["RoleForLambdaStopEC2Instances", "Arn"] },
"Runtime": "nodejs4.3",
"Timeout": "5"
}
},
"StopEC2InstancesRule": {
"Type" : "AWS::Events::Rule",
"Properties" : {
"Name" : "StopEC2Instances",
"ScheduleExpression" : "cron(0 13 ? * MON-FRI *)",
"State": "ENABLED",
"Targets": [{
"Arn": { "Fn::GetAtt": ["LambdaStopEC2Instances", "Arn"] },
"Id": "stopEC2Instances"
}]
}
},
"LambdaInvokePermission": {
"Type": "AWS::Lambda::Permission",
"Properties": {
"FunctionName" : { "Fn::GetAtt" : ["LambdaStopEC2Instances", "Arn"] },
"Action": "lambda:InvokeFunction",
"Principal": "events.amazonaws.com",
"SourceAccount": { "Ref" : "AWS::AccountId" },
"SourceArn": { "Fn::GetAtt": ["StopEC2InstancesRule","Arn"] }
}
}
Unfortunately, configuring scheduled event sources for lambda functions is currently not supported by CloudFormation. You will need to deploy your lambda using CloudFormation and then manually configure your scheduled events.
CloudFormation does support an AWS::Lambda::EventSourceMapping resource type. However, this resource is limited configuring Kinesis or DynamoDB streams, so this is likely not helpful to you.
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-lambda-eventsourcemapping.html
**Update - as of April 2016, this is now supported using CloudWatch Events - https://aws.amazon.com/about-aws/whats-new/2016/04/amazon-cloudwatch-events-now-supported-in-aws-cloudformation-templates/
The YAML Version
ScheduledRule:
Type: AWS::Events::Rule
Properties:
Description: "ScheduledRule"
ScheduleExpression: "rate(10 minutes)"
State: "ENABLED"
Targets:
-
Arn:
Fn::GetAtt:
- "LambdaFunction"
- "Arn"
Id: "TargetFunctionV1"
PermissionForEventsToInvokeLambda:
Type: AWS::Lambda::Permission
Properties:
FunctionName:
Ref: "LambdaFunction"
Action: "lambda:InvokeFunction"
Principal: "events.amazonaws.com"
SourceArn:
Fn::GetAtt:
- "ScheduledRule"
- "Arn"
As of this week (18 April 2016) it is now possible to add a scheduled CloudWatch event rule that will trigger your Lambda function.
The AWS::Event::Rule has a ScheduleExpression field for the cron-style schedule and a Targets array which can accept a Lambda function ARN.
AWS supports periodic run through sourcedetails.
EventSource: "aws.config"
MaximumExecutionFrequency: Twelve_Hours
MessageType: "ScheduledNotification"
If you use function name as
"FunctionName": {
"Fn::GetAtt": [
"TagWatcherFunction",
"Arn"
]
}
and you not specify the function then it'll throw you
"Template is not valid: Template error: instance of Fn::GetAtt references undefined resource TagWatcherFunction"
So instead of function name you can directly specify the "lambda ARN".
you can see example below
"TagWatcherRule": {
"Type": "AWS::Events::Rule",
"Properties": {
"ScheduleExpression": "rate(10 minutes)",
"Targets": [
{
"Id": "TagWatcherScheduler",
"Arn": {
"Fn::GetAtt": [
"TagWatcherFunction",
"Arn"
]
}
}
]
}
},
// role may call the lambda
"InvokeLambdaPermission": {
"Type": "AWS::Lambda::Permission",
"Properties": {
"FunctionName": "arn:aws:lambda:<region>:<awsid>:function:<lambd name>",
"Action": "lambda:InvokeFunction",
"Principal": "events.amazonaws.com",
"SourceArn": {
"Fn::GetAtt": [
"TagWatcherRule",
"Arn"
]
}
}
}