I'm trying to create some resources using Cloudformation with serverless framework, In which I need to substitute resource name from another resource. Tried to use !Sub but still I couldn't get Arn of another resource created.
Tried all the approaches in this stackoverflow question How to use Sub and GetAtt functions at the same time in CloudFormation template? to no avail.
I appreciate any help.
Resources:
BasicParameter:
Type: AWS::SSM::Parameter
Properties:
Name: /data/config-name
Type: String
Value:
Fn::Base64:
!Sub |
{
"filter_configs": [{
"stream_name": !GetAtt tpRecordDeliveryStream.Arn,
"events": [
{
"name": "event_name1",
"stream": "streamname1"
},
{
"name": "event_name2"
}
]
}]
}
Description: Configuration for stream filters
Tags:
project: projectname
team: data
owner: owner_name
This was resolved by using serverless-pseudo-parameters serverless plugin. Serverless framework also uses ${} placeholder and it conflicts with Cloudformation placeholders. serverless-pseudo-parameters solves that by allowing us to replace those place holders with #{} which are replaced during sls deploy with cloud formation templates
Resources:
streamConfig:
Type: AWS::SSM::Parameter
Properties:
Name: config_name
Type: String
Value:
Fn::Base64: |
{
"filter_configs": [{
"firehose_stream_arn": "#{tpRecordDeliveryStream.Arn}",
"events": [
{
"name": "config0",
"filter1": "value1"
},
{
"name": "config1"
}
]
}]
}
Description: Configuration for stream filters
Since you have !Sub |, instead of
"stream_name": !GetAtt tpRecordDeliveryStream.Arn,
the following should be enough
"stream_name": "${tpRecordDeliveryStream.Arn}"
The alternative using !Sub in array notation:
Value:
Fn::Base64:
!Sub
- |
{
"filter_configs": [{
"stream_name": "${tpRecordDeliveryStreamArn}",
"events": [
{
"name": "event_name1",
"stream": "streamname1"
},
{
"name": "event_name2"
}
]
}]
}
- tpRecordDeliveryStreamArn: !GetAtt tpRecordDeliveryStream.Arn
Related
I've been all over the web searching for an answer to this.
Essentially, we're spinning up an API using Swagger, which is awesome and works great, but one thing doesn't work... When we make a call to an Endpoint, we get a 500 error (it's not a 500 error that we're providing either it's one from AWS). The error states "Execution failed due to configuration error: Invalid permissions on Lambda function" (https://youtu.be/H4LM_jw5zzs <- This is a video, from another user, of the error I'm getting).
I've gone down many ratholes, and have found an answer... It involves using the AWS CLI and looks a bit like this:
aws lambda add-permission \
--function-name FUNCTION_NAME \
--statement-id STATEMENT_ID \
--action lambda:InvokeFunction \
--principal apigateway.amazonaws.com \
--source-arn "arn:aws:execute-api:us-east-1:ACCOUNT_ID:API_ID/*/METHOD/ENDPOINT"
This is great and all, but we are using CloudFormation to spin up everything and we want this to be automated. Is there an easier way to go about this? Is there something in CloudFormation that will give us the resource policy that we need?
I'm hitting a bit of a wall with this, but I've been working on it for a few hours today and it's a bit of a blocker for our API release, so any help would be much appreciated. :)
There is a CloudFormation solution to this problem. See the following CloudFormation snippet:
"Permission": {
"Type": "AWS::Lambda::Permission",
"Properties": {
"FunctionName": { "Fn::GetAtt": [ "Lambda", "Arn" ] },
"Action": "lambda:InvokeFunction",
"Principal": "apigateway.amazonaws.com",
"SourceArn": { "Fn::Join": [ "", [
"arn:aws:execute-api:",
{ "Ref": "AWS::Region" }, ":",
{ "Ref": "AWS::AccountId" }, ":",
{ "Ref": "API" },
"/*/*/*"
] ] }
}
}
This grants API Gateway permissions to launch your Lambda function. Variables in this snippet you need to change are Lambda (line 4) and API (line 11).
For the invoke permissions:
"APIInvokePermission": {
"Type": "AWS::Lambda::Permission",
"Properties": {
"FunctionName": {
"Ref": "YOUR_LAMBDA_FUNCTION_RESOURCE_NAME"
},
"Action": "lambda:InvokeFunction",
"Principal": "apigateway.amazonaws.com",
"SourceArn": {
"Fn::Sub": "arn:aws:execute-api:${AWS::Region}:${AWS::AccountId}:${YOUR_REST_API_RESOURCE_NAME}/*/*/*"
}
}
},
Thanks https://twitter.com/edjgeek for helping me get this straight in my head.
This GIST shows how to use AWS::Serverless:Function with Events to automatically generate the needed AWS::Lambda::Permission to allow APIGateway (for a given route) to invoke your Lambda:
https://gist.github.com/rainabba/68df1567cbd0c4930d428c8953dc2316
Both of the following approaches assume an api such as (many fields omitted for readability):
MyApi:
Type: 'AWS::Serverless::Api'
Properties:
DefinitionBody:
Fn::Transform:
Name: AWS::Include
Parameters:
Location: openapi.yaml
Using "SAM Events"
The most relevant bit (I've omitted many required fields):
MyLambdaFunction:
Type: 'AWS::Serverless::Function'
Properties:
Events:
MyRouteEventToProxy:
Type: Api
Properties:
Method: POST
Path: '/some-route/{pathParm}'
RestApiId: !Ref MyApi # ResourceName of AWS::Serverless::Api
Auth:
Authorizer: NONE
Using "openapi binding"
If you'd rather declare the binding in the openapi.yaml, then see the following project (no Lambda/Events required). This approach requires an explict role to allow invoke.
template.yaml relevant bits:
MyLambdaFunction:
Type: 'AWS::Serverless::Function'
Properties:
# Events: # No need for Events when binding from openapi.yaml
MyHttpApiRole:
Type: "AWS::IAM::Role"
Properties:
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: "Allow"
Principal:
Service: "apigateway.amazonaws.com"
Action:
- "sts:AssumeRole"
openapi.yaml relevant bits:
paths:
post:
x-amazon-apigateway-integration:
httpMethod: POST
uri:
Fn::Sub: "arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${MyLambdaFunction.Arn}:live/invocations"
contentHandling: "CONVERT_TO_TEXT"
type: aws_proxy
credentials:
Fn::GetAtt: [MyHttpApiRole, Arn]
https://github.com/aws-samples/sessions-with-aws-sam/tree/master/http-api-direct-integration
I have an organization account with several managed accounts underneath it. Each managed account has multiple VPCs in them. One of the VPC in each managed account will have a tag "ServiceName":"True" while the others in that account will have a "ServiceName":"False" tag instead.
I'm trying to create a stackset with a stack dedicated to create a security group with ingress rules attached to it and I need to dynamically assign the "VpcId" property of that security group to be the "VpcId" of VPC with the "ServiceName":"True" tag in that account.
Obviously, if I don't specify a VPC ID in the VpcId field, it creates the security group but attach it to the default VPC of that account. I can't specify manually a VPC either since it's going to be ran in multiple accounts. Leaving me with the only option available to search and assign VPCs by running some sort of function to extract the "VpcId".
The stack itself works fine as I ran it in a test environment while specifying a VPC ID. So, it's just a matter getting that "VpcId" dynamically.
In the end, I'm looking to do something that would resemble this:
{
"Parameters": {
"MyValidVPCID": {
"Description": "My Valid VPC ID where ServiceName tag equals true. Do some Lambda Kung Fu to get the VPC ID using something that would let me parse the equivalent of aws ec2 describe-vpcs command.",
"Type": "String"
}
},
"Resources": {
"SG": {
"Type": "AWS::EC2::SecurityGroup",
"Properties": {
"GroupDescription": "Security Group Desc.",
"Tags": [
{
"Key": "Key1",
"Value": "ABC"
},
{
"Key": "Key2",
"Value": "DEF"
}
],
"VpcId" : { "Ref" : "MyValidVPCID" }
}
},
"SGIngressRule01":
{
"Type": "AWS::EC2::SecurityGroupIngress",
"DependsOn": "SG",
"Properties": {
"GroupId" : { "Fn::GetAtt": [ "SG", "GroupId" ] },
"Description": "Rule 1 description",
"IpProtocol": "tcp",
"FromPort": 123,
"ToPort": 456,
"CidrIp": "0.0.0.0/0"
}
}
}
I really don't know if it's a feasible approach or what would be the extra steps needed to recuperate that VpcId based on the tag. That's why if I could get some input from people used to work with CloudFormation, it would help me a lot.
getting that "VpcId" dynamically.
You have to use custom resource for that. You would have to create it as a lambda function which would take any input arguments you want, and using AWS SDK, would query or modify the VPC/Security groups in your stack.
Thanks Marcin for pointing me in the right direction with the custom resources. For those who are wondering what the basic code to make it work looks like, it looks something like this:
Resources:
FunctionNameLambdaFunctionRole:
Type: "AWS::IAM::Role"
Properties:
RoleName: FunctionNameLambdaFunctionRole
Path: "/"
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Principal:
Service: lambda.amazonaws.com
Action: sts:AssumeRole
FunctionNameLambdaFunctionRolePolicy:
Type: "AWS::IAM::Policy"
Properties:
PolicyName: admin3cx
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: "Allow"
Action: "*"
Resource: "*"
Roles:
- Ref: FunctionNameLambdaFunctionRole
FunctionNameLambdaFunctionCode:
Type: "AWS::Lambda::Function"
DeletionPolicy: Delete
DependsOn:
- FunctionNameLambdaFunctionRole
Properties:
FunctionName: FunctionNameLambdaFunctionCode
Role: !GetAtt FunctionNameLambdaFunctionRole.Arn
Runtime: python3.7
Handler: index.handler
MemorySize: 128
Timeout: 30
Code:
ZipFile: |
import boto3
import cfnresponse
ec2 = boto3.resource('ec2')
client = boto3.client('ec2')
def handler(event, context):
responseData = {}
filters =[{'Name':'tag:ServiceName', 'Values':['True']}]
vpcs = list(ec2.vpcs.filter(Filters=filters))
for vpc in vpcs:
responseVPC = vpc.id
responseData['ServiceName'] = responseVPC
cfnresponse.send(event, context, cfnresponse.SUCCESS, responseData, "CustomResourcePhysicalID")
FunctionNameLambdaFunctionInvocationCode:
Type: "Custom::FunctionNameLambdaFunctionInvocationCode"
Properties:
ServiceToken: !GetAtt FunctionNameLambdaFunctionCode.Arn
SGFunctionName:
Type: "AWS::EC2::SecurityGroup"
Properties:
GroupDescription: Description
VpcId: !GetAtt FunctionNameLambdaFunctionInvocationCode.ServiceName
...
Some stuff has been redacted and I made the switch to YAML. The code will be refined obviously. The point was just to make sure I was able to get a return value based on a filter in a Lambda function inside a CloudFormation stack.
I am trying to create external view using the data from the tables but receiving error Property validation failure: [Value of property {/TableInput/ViewOriginalText} does not match type {String}] while creating views in AWS glue using cloud formation.
Any idea about resolution would be appreciated. Below is the yaml file code snippet that I am using.
Thanks in Advance
---
AWSTemplateFormatVersion: 2010-09-09
Description: "Glue Athena database and table configuration"
Parameters:
MarketingAndSalesDatabaseName:
Type: String
MinLength: "4"
Default: "marketingandsales_qs"
Description: "Name of the AWS Glue database to contain this CloudFormation template's tables."
SalesPipelineTableName:
Type: String
MinLength: "4"
Default: "salespipeline_qs"
Description: "Name of the Sales Pipeline data table in AWS Glue."
Resources:
### AWS GLUE RESOURCES ###
MarketingAndSalesDatabase:
Type: "AWS::Glue::Database"
Properties:
DatabaseInput:
Description: "Marketing and Sales database (Amazon QuickSight Samples)."
Name: !Ref MarketingAndSalesDatabaseName
CatalogId: !Ref AWS::AccountId
SalesPipelineTable:
Type: "AWS::Glue::Table"
DependsOn: MarketingAndSalesDatabase
Properties:
TableInput:
Description: "Sales Pipeline table (Amazon QuickSight Sample)."
TableType: "VIRTUAL_VIEW"
Parameters: {
"CrawlerSchemaDeserializerVersion": "1.0",
"compressionType": "none",
"classification": "csv",
"recordCount": "16831",
"typeOfData": "file",
"CrawlerSchemaSerializerVersion": "1.0",
"columnsOrdered": "true",
"objectCount": "1",
"delimiter": ",",
"skip.header.line.count": "1",
"averageRecordSize": "119",
"sizeKey": "2002910",
"presto_view": "true"
}
StorageDescriptor:
StoredAsSubDirectories: False
Parameters: {
"CrawlerSchemaDeserializerVersion": "1.0",
"compressionType": "none",
"classification": "csv",
"recordCount": "16831",
"typeOfData": "file",
"CrawlerSchemaSerializerVersion": "1.0",
"columnsOrdered": "true",
"objectCount": "1",
"delimiter": ",",
"skip.header.line.count": "1",
"averageRecordSize": "119",
"sizeKey": "2002910"
}
InputFormat: "org.apache.hadoop.mapred.TextInputFormat"
OutputFormat: "org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat"
Columns:
- Type: string
Name: salesperson
- Type: string
Name: lead_name
ViewOriginalText:
'Fn::Sub':
- '/* Presto View: ${View} */'
- View:
'Fn::Base64': !sub '{"catalog": "awsdatacatalog",
"schema":"default",
"columns": [ { "name": "salesperson", "type": "varchar" }, { "name": "lead_name", "type": "varchar" }],
"originalSql": "SELECT salesperson AS salesperson, lead_name AS lead_name FROM marketing_qs_1"
}'
Retention: 0
Name: !Ref SalesPipelineTableName
DatabaseName: !Ref MarketingAndSalesDatabaseName
CatalogId: !Ref AWS::AccountId
These properties should be underneath StorageDescriptor instead of ViewOriginalText:
Compressed: False
Location: !Sub "s3://${DataBucketName}/sales/"
Recommend trying the CloudFormation Linter in VSCode to see some of these errors inline while authoring templates:
[cfn-lint] E3002: Property is an object instead of String at Resources/SalesPipelineTable/Properties/TableInput/ViewOriginalText
Sub also isn't required inside that Base64:
[cfn-lint] W1020: Fn::Sub isn't needed because there are no variables at Resources/SalesPipelineTable/Properties/TableInput/ViewOriginalText/Fn::Sub/1/View/Fn::Base64/Fn::Sub
I need to write a CFT for pipeline with Jenkins integration for build/test. I found this documentation to setup ActionTypeId for the jenkins stage. But this doc does not specify how to set the server url of the jenkins server. And also it is not clear to me where to give the Jenkins Provider name. Is it in the ActionTypeId or in configuration properties?
I could not find any example for this use case in the internet as well.
Please provide a proper example for setup Jenkins Action Provider for AWS Codepipeline using AWS Cloudformation template.
Following is a section from the sample cft I wrote from what I learnt from above doc.
"stages": [
{
"name": "Jenkins",
"actions": [
...
{
"name": "Jenkins Build",
"actionTypeId": {
"category": "Build",
"owner": "Custom",
"provider": "Jenkins",
"version": "1"
},
"runOrder": 2,
"configuration": {
???
},
...
}
]
},
...
]
The piece of information which was missing for me was that I need to create a Custom Action to use Jenkins as the Action provider for my codepipeline.
First I added the custom action as below:
JenkinsCustomActionType:
Type: AWS::CodePipeline::CustomActionType
Properties:
Category: Build
Provider: !Ref JenkinsProviderName
Version: 1
ConfigurationProperties:
-
Description: "The name of the build project must be provided when this action is added to the pipeline."
Key: true
Name: ProjectName
Queryable: false
Required: true
Secret: false
Type: String
InputArtifactDetails:
MaximumCount: 5
MinimumCount: 0
OutputArtifactDetails:
MaximumCount: 5
MinimumCount: 0
Settings:
EntityUrlTemplate: !Join ['', [!Ref JenkinsServerURL, "/job/{Config:ProjectName}/"]]
ExecutionUrlTemplate: !Join ['', [!Ref JenkinsServerURL, "/job/{Config:ProjectName}/{ExternalExecutionId}/"]]
Tags:
- Key: Name
Value: custom-jenkins-action-type
The jenkins server URL is given in the settings for Custom Action
and the Jenkins provider name is given for Provider. Which were the
issues I had initially.
Then configure the pipeline stage as following:
DevPipeline:
Type: AWS::CodePipeline::Pipeline
DependsOn: JenkinsCustomActionType
Properties:
Name: Dev-CodePipeline
RoleArn:
Fn::GetAtt: [ CodePipelineRole, Arn ]
Stages:
...
- Name: DevBuildVerificationTest
Actions:
- Name: JenkinsDevBVT
ActionTypeId:
Category: Build
Owner: Custom
Version: 1
Provider: !Ref JenkinsProviderName
Configuration:
# JenkinsDevBVTProjectName - Jenkins Job name defined as a parameter in the CFT
ProjectName: !Ref JenkinsDevBVTProjectName
RunOrder: 4
The Custom Action has to be created before the Pipeline. Hence DependsOn: JenkinsCustomActionType
I've been all over the web searching for an answer to this.
Essentially, we're spinning up an API using Swagger, which is awesome and works great, but one thing doesn't work... When we make a call to an Endpoint, we get a 500 error (it's not a 500 error that we're providing either it's one from AWS). The error states "Execution failed due to configuration error: Invalid permissions on Lambda function" (https://youtu.be/H4LM_jw5zzs <- This is a video, from another user, of the error I'm getting).
I've gone down many ratholes, and have found an answer... It involves using the AWS CLI and looks a bit like this:
aws lambda add-permission \
--function-name FUNCTION_NAME \
--statement-id STATEMENT_ID \
--action lambda:InvokeFunction \
--principal apigateway.amazonaws.com \
--source-arn "arn:aws:execute-api:us-east-1:ACCOUNT_ID:API_ID/*/METHOD/ENDPOINT"
This is great and all, but we are using CloudFormation to spin up everything and we want this to be automated. Is there an easier way to go about this? Is there something in CloudFormation that will give us the resource policy that we need?
I'm hitting a bit of a wall with this, but I've been working on it for a few hours today and it's a bit of a blocker for our API release, so any help would be much appreciated. :)
There is a CloudFormation solution to this problem. See the following CloudFormation snippet:
"Permission": {
"Type": "AWS::Lambda::Permission",
"Properties": {
"FunctionName": { "Fn::GetAtt": [ "Lambda", "Arn" ] },
"Action": "lambda:InvokeFunction",
"Principal": "apigateway.amazonaws.com",
"SourceArn": { "Fn::Join": [ "", [
"arn:aws:execute-api:",
{ "Ref": "AWS::Region" }, ":",
{ "Ref": "AWS::AccountId" }, ":",
{ "Ref": "API" },
"/*/*/*"
] ] }
}
}
This grants API Gateway permissions to launch your Lambda function. Variables in this snippet you need to change are Lambda (line 4) and API (line 11).
For the invoke permissions:
"APIInvokePermission": {
"Type": "AWS::Lambda::Permission",
"Properties": {
"FunctionName": {
"Ref": "YOUR_LAMBDA_FUNCTION_RESOURCE_NAME"
},
"Action": "lambda:InvokeFunction",
"Principal": "apigateway.amazonaws.com",
"SourceArn": {
"Fn::Sub": "arn:aws:execute-api:${AWS::Region}:${AWS::AccountId}:${YOUR_REST_API_RESOURCE_NAME}/*/*/*"
}
}
},
Thanks https://twitter.com/edjgeek for helping me get this straight in my head.
This GIST shows how to use AWS::Serverless:Function with Events to automatically generate the needed AWS::Lambda::Permission to allow APIGateway (for a given route) to invoke your Lambda:
https://gist.github.com/rainabba/68df1567cbd0c4930d428c8953dc2316
Both of the following approaches assume an api such as (many fields omitted for readability):
MyApi:
Type: 'AWS::Serverless::Api'
Properties:
DefinitionBody:
Fn::Transform:
Name: AWS::Include
Parameters:
Location: openapi.yaml
Using "SAM Events"
The most relevant bit (I've omitted many required fields):
MyLambdaFunction:
Type: 'AWS::Serverless::Function'
Properties:
Events:
MyRouteEventToProxy:
Type: Api
Properties:
Method: POST
Path: '/some-route/{pathParm}'
RestApiId: !Ref MyApi # ResourceName of AWS::Serverless::Api
Auth:
Authorizer: NONE
Using "openapi binding"
If you'd rather declare the binding in the openapi.yaml, then see the following project (no Lambda/Events required). This approach requires an explict role to allow invoke.
template.yaml relevant bits:
MyLambdaFunction:
Type: 'AWS::Serverless::Function'
Properties:
# Events: # No need for Events when binding from openapi.yaml
MyHttpApiRole:
Type: "AWS::IAM::Role"
Properties:
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: "Allow"
Principal:
Service: "apigateway.amazonaws.com"
Action:
- "sts:AssumeRole"
openapi.yaml relevant bits:
paths:
post:
x-amazon-apigateway-integration:
httpMethod: POST
uri:
Fn::Sub: "arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${MyLambdaFunction.Arn}:live/invocations"
contentHandling: "CONVERT_TO_TEXT"
type: aws_proxy
credentials:
Fn::GetAtt: [MyHttpApiRole, Arn]
https://github.com/aws-samples/sessions-with-aws-sam/tree/master/http-api-direct-integration