Publish S3 Bucket Notification to SQS - amazon-web-services

I am trying to set up my S3 to notify my SQS Queue for a "PUT" Object Creation Event.
I am able to achieve this using CLI by:
aws --profile QA s3api put-bucket-notification --bucket <BUCKET_NAME> --notification-configuration '{ "QueueConfiguration": { "Id": "<EVENT ID>", "Event": "s3:ObjectCreated:Put", "Queue": "<QUEUE ARN>" } }'
Also able to do the same using Java:
NotificationConfiguration notificationConfiguration = new QueueConfiguration(queueArn, EnumSet.of(S3Event.ObjectCreatedByPut));
BucketNotificationConfiguration bucketNotificationConfiguration = new BucketNotificationConfiguration("DropShipInboundQueueDelivery", notificationConfiguration);
client.setBucketNotificationConfiguration(bucketName, bucketNotificationConfiguration)
However when I tried to something similar using CloudFormation template, I cannot find any way to trigger a notification to SQS. The only option I see that works and is documented is to trigger notification to SNS.
I have referred the Cloud Formation Documentation:
I looked at the AWS::S3::Bucket docs to look at the outer syntax. I saw NotificationConfiguration which I need to set
However the Notification Configuration can only contain a list of TopicConfigurations with was the old constructor in JDK before QueueConfiguration was supported
I tried doing something like this:
"NotificationConfiguration" :{
"QueueConfiguration": {
"Id": "DropshipInboundEventNotification",
"Event": "s3:ObjectCreated:Put",
"Queue": "arn:aws:sqs:*:*:Dropship-Inbound-qa"
}
},
But this as expected threw an error: "Encountered unsupported property QueueConfiguration" from amazon.
Looked at this API documentation
I would like to know if someone has been able to do this using CloudFormation Templates as thats how I am maintaining all the other AWS resources and do not want to do anything special for this particular feature.
Any help is appreciated.

There is no need "Id" in Cloudformation Template ( You can check from QueueConfiguration Doc ) and your second mistake, that is not "QueueConfiguration", it's "QueueConfigurations". Because of that you get an error that says "Encountered unsupported property QueueConfiguration"
It must be something like that.
"S3Bucket":{
"Type" : "AWS::S3::Bucket",
"Properties" : {
"AccessControl" : String,
"BucketName" : String,
"CorsConfiguration" : CORS Configuration,
"LifecycleConfiguration" : Lifecycle Configuration,
"LoggingConfiguration" : Logging Configuration,
"NotificationConfiguration" :
{ "QueueConfigurations" : [ {
"Event" : "s3:ObjectCreated:Put",
"Queue" : "arn:YOURQUEUEARN"
} ] },
"Tags" : [ Resource Tag, ... ],
"VersioningConfiguration" : Versioning Configuration,
"WebsiteConfiguration" : Website Configuration Type
}
}
While you are reading cloudformation template documents, you must be careful about "Required:" sections. If it is not required, you don't need to fill it, just remove that line from your template if you don't use it( Like S3 Tags ).
Other Docs about it:
S3BucketDocs
NotificationConfigurationDocs

Related

Create Secrets Manager using CloudFormation but keeping values not visible

I have a CloudFormation template to create a Secret in Secrets Manager. My current template is similar to this (based on aws documentation https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-secretsmanager-secret.html):
{
"Resources": {
"MyCredentials": {
"Type": "AWS::SecretsManager::Secret",
"Properties": {
"Name": "prod/web/api",
"Description": "",
"SecretString": "{
\"Client_id\":\"my_client_id\",
\"Client_secret\":\"a_super_secret_value\"
}"
}
}
}
}
My problem is that I can not use the GenerateSecretString property because the password is defined from an external organization so I can not change or create the value on my own and in this way the secret value can be viewed from the template in CloudFormation.
Is possible to achieve this or I need to create the secrets manually?
You can use AWS SSM Parameter, where the external organization has given permissions to add/update password there or someone in the team do the same.
Once the password is there, you read in your cloudformation template either via dynamic references like below,
The following example uses an ssm-secure dynamic reference to set the password for an IAM user to a secure string stored in Systems Manager Parameter Store. As specified, CloudFormation will use version 10 of the IAMUserPassword parameter for stack and change set operations.
"MyIAMUser": {
"Type": "AWS::IAM::User",
"Properties": {
"UserName": "MyUserName",
"LoginProfile": {
"Password": "{{resolve:ssm-secure:IAMUserPassword:10}}"
}
}
}
Or static reference something like below :
here Accessing the AvailabilityZone param stored in SSM.
"AvailabilityZone": {
"Description": "Amazon EC2 instance Availablity Zone",
"Type": "AWS::SSM::Parameter::Value<String>",
"Default": "AvailabilityZone"
}
More examples in Using AWS Systems Manager Parameter Store Secure String parameters in AWS CloudFormation templates

Optional parameters when using AWS CLI to launch CloudFormation template

I'm trying to create a CloudFormation template that'll deploy a Lambda function, And I need the security options to be optional parameters.
I was able to partially accomplish this using the question here:
How to make a whole object in CloudFormation templates optional?
Interestingly, that method worked great to make the VpcConfig property optional in the AWS GUI Console, but it did NOT work to make it optional for the CLI. And unfortunately, I need it to work in the CLI, since I'll be using CodeBuild to call and deploy this template's resources.
Here are the relevant parameters:
"SecurityGroupIds" : {
"Type" : "CommaDelimitedList",
"Description" : "A list of one or more security groups IDs in the VPC that includes the resources to which your Lambda function requires access."
},
"SubnetIds" : {
"Type" : "CommaDelimitedList",
"Description" : "A list of one or more subnet IDs in the VPC that includes the resources to which your Lambda function requires access."
}
And conditions:
"HasVPC": {"Fn::And": [{"Fn::Not": [{"Fn::Equals": [{"Fn::Join": ["", {"Ref": "SubnetIds"}]}, ""]}]}, {"Fn::Not": [{"Fn::Equals": [{"Fn::Join": ["", {"Ref": "SecurityGroupIds"}]}, ""]}]}]}
And here's where that condition is used in the Lambda resource being defined in the Resources section of the template:
"VpcConfig": {
"Fn::If": [
"HasVPC",
{
"SecurityGroupIds" : {"Ref": "SecurityGroupIds"},
"SubnetIds" : {"Ref": "SubnetIds"}
},
{ "Ref":"AWS::NoValue" }
]
},
When I issue the command to deploy this stack in the CLI, I get the following error:
An error occurred (ValidationError) when calling the CreateChangeSet
operation: Parameters: [SecurityGroupIds, SubnetIds] must have values
Here's the AWS CLI command I'm issuing, from the same directory in which the template is located. Note: the ARN values have all been heavily modified to not be real values from my account, but I kept them in the right format so you can see the real format of the command:
aws cloudformation deploy --template-file lambda-template.json --stack-name "CLI-lambda-stack" --parameter-overrides S3BucketName="myBucket" S3FileLocation="lambda_function.zip" S3ObjectVersion="ZuB0iueEghOyh5q00.DiykLNudujdsc5" DeadLetterArn="arn:aws:sns:us-west-2:577898337216:CloudFormationTests" EnvironmentVariable="testing" KmsKeyArn="arn:aws:kms:us-west-2:504398934246:key/b24e7b72-a94d-6a3e-b848-165115c86212" HandlerFunctionName="lambda_function.lambda_handler" MemorySize="128" Role="arn:aws:iam::102893937243:role/serverless-test-default-us-east-1-lambdaRole" FuncName="myCLILambda"
You are not providing SecurityGroupIds neither SubnetIds default values and your are not providing them on your --parameter-overrides. Therefore, CloudFormation doesn't know how to process them if no values are provided.
Adding the Default statement should do the trick:
{
"Parameters" : {
"SecurityGroupIds" : {
"Type" : "CommaDelimitedList",
"Description" : "A list of one or more security groups IDs in the VPC that includes the resources to which your Lambda function requires access.",
"Default" : ""
},
"SubnetIds" : {
"Type" : "CommaDelimitedList",
"Description" : "A list of one or more subnet IDs in the VPC that includes the resources to which your Lambda function requires access.",
"Default" : ""
}
}

CloudFormation - Create SNS subscription in disabled state

Is there a way to create an SNS subscription in the disabled state? This is for a lambda if that makes a difference.
Example:
MySubscription:
Type: AWS::SNS::Subscription
Properties:
Endpoint: arn:aws:lambda:region:account-id:function:mylambda
Protocol: lambda
TopicArn: arn:aws:sns:region:account-id:topic
Enabled: false # like this
Couldn't find anything like this in the AWS CloudFormation documentation
You can create a condition on the subscription creation.
First add a parameter:
"CreateSubscription": {
"Type": "String",
"AllowedValues": [
"true",
"false"
],
"Description": "Create subscription to sns"
}
after the 'parameters' section, create a 'conditions' section:
"Conditions" : {
"CreateSubscription" : {"Fn::Equals" : [{"Ref" : "CreateSubscription"}, "true"]}
}
add the condition to your subscription
"Subscription": {
"Type": "AWS::SNS::Subscription",
"Condition" : "CreateSubscription",
[...]
}
when you want to "activate" you subscription, you just have to change the parameter value updating the stack using the same template
Conditions section reference
There isn't a way with Cloudformation which conforms to your example. According to the documentation, AWS::SNS::Subscription does not have 'Enabled' as a setting.
Although, the documentation does state that the owner of the endpoint must confirm the subscription before Amazon SNS creates the subscription. So, in a sense, it's already disabled because it doesn't exist until you confirm it on the SNS topic.

AWS put-bucket-notification-configuration won't let me use "Filter"

The command I use:
aws s3api put-bucket-notification-configuration --bucket bucket-name --notification-configuration file:///Users/chris/event_config.json
Works fine if I take out the "Filter" key. As soon as I add it in, I get:
Parameter validation failed:
Unknown parameter in NotificationConfiguration.LambdaFunctionConfigurations[0]: "Filter", must be one of: Id, LambdaFunctionArn, Events
Here's my JSON file:
{
"LambdaFunctionConfigurations": [
{
"LambdaFunctionArn": "arn:aws:lambda:us-east-1:000000000:function:name",
"Events": [
"s3:ObjectCreated:*"
],
"Filter": {
"Key": {
"FilterRules": [
{
"Name": "prefix",
"Value": "images/"
}
]
}
}
}
]
}
When I look at the command's docs (http://docs.aws.amazon.com/cli/latest/reference/s3api/put-bucket-notification-configuration.html), I don't see any mistake. I've tried copy/pasting, carefully looking over, etc... Any help would be greatly appreciated!
You need to be running at least version 1.7.46 of aws-cli, released 2015-08-20.
This release adds Amazon S3 support for event notification filters and fixes some issues.
https://aws.amazon.com/releasenotes/CLI/3585202016507998
The aws-cli utility contains a lot of built-in intelligence and validation logic. New features often require the code in aws-cli to be updated, and Filter on S3 event notifications is a relatively recent feature.
See also: https://aws.amazon.com/blogs/aws/amazon-s3-update-delete-notifications-better-filters-bucket-metrics/

AWS cloud formation Template- providing Tags for the stack in the template

We wanted to use company specific Tags to the resources that we create in AWS for billing purposes. I am using a cloud formation template to spin up our Elasticbeanstalk instance and other project dependent resources. When I use the CloudFormation console to create a stack it asks me for Tags in the page after parameters. I have to manually input the Tags for that stack. However is there a way to specify those Tags (Tags for the stack) with in the cloud formation template itself? That way the Tag gets propagated to the other resources? I know that the cloud formation automatically tags the resources with the stack name. But we need company specific tags to bill separate departments.
In the template anatomy, you can't set stack-level tags directly. However you can create a wrapper template, having a single resource of AWS::CloudFormation::Stack.
You can define stack-level tags on that resource:
{
"AWSTemplateFormatVersion": "2010-09-09",
"Description": "WrapperTemplate",
"Resources": {
"WrappedStackWithStackLevelTags": {
"Type" : "AWS::CloudFormation::Stack",
"Properties" : {
"Tags" : [ { "Key" : "Stage", "Value" : "QA" } ],
"TemplateURL" : "your-original-template-s3-url"
}
}
}
}
When launching AWS CloudFormation, the tags being requested will be applied to the CloudFormation Stack itself and (where possible) will also be propagated to the resources launched by the Stack.
These tags can be passed to the CreateStack API call, or from the CLI:
See: create-stack CLI documentation
These tags are applied to the whole Stack and aren't included in the CloudFormation template.
However, CloudFormation templates can include tags for specific resources that are being created. For example, when launching Amazon EC2 instances, tags can be included in the template:
"MyInstance" : {
"Type" : "AWS::EC2::Instance",
"Properties" : {
"SecurityGroups" : [{ "Ref" : "MySecurityGroup" }],
"AvailabilityZone" : "us-east-1a",
"ImageId" : "ami-20b65349",
"Volumes" : [{
"VolumeId" : { "Ref" : "MyEBS" },
"Device" : "/dev/sdk"
}],
"Tags" : [{
"Key" : "Stage",
"Value" : "QA"
}]
}
}
Contrary to what #lalyos says, you don't need to use nested stacks for this, just provide the tags that should apply to all resources as stack level tags.
These stack-level tags can be specified whether running the stack on the console or via CLI.
CLI example:
aws cloudformation create-stack --stack-name my-stack-name \
--template-body file://path-to-template-file.yaml \
--parameters ParameterKey=param1key,ParameterValue=param1value \
--tags Key=tag1key,Value=tag1value \
Key=tag2key,Value=tag2value \
Key=tag3key,Value=tag3value
... and generally add as many tags as you need, using the same format and allowing spaces between tag key-value pairs
See:
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-console-add-tags.html
You can build and deploy a CloudFormation template using aws-sam-cli. This command reads a samconfig.toml file where you can declare tags for all the resources of the stack (including CloudFormation stack itself)
Your samconfig.toml should look like:
[default.deploy.parameters]
stack_name = "your-application"
s3_bucket = "your-s3-for-cloudformation-stuff"
s3_prefix = "your-folder-name"
...
tags = "Stage=\"QA\""
and then run:
sam build --template <your-cloudformation-template.yml> && sam deploy
You don't need any wrapper..
You can add tags to the stack on when you create/update it:
In Console:
You can also use the aws cli:
aws cloudformation create-stack help
--tags (list)
Key-value pairs to associate with this stack. CloudFormation also
propagates these tags to supported resources in the stack. You can
specify a maximum number of 50 tags.
If you don't specify this parameter, CloudFormation doesn't modify
the stack's tags. If you specify an empty value, CloudFormation re-
moves all associated tags.
(structure)
The Tag type enables you to specify a key-value pair that can be
used to store information about an CloudFormation stack.
Key -> (string)
Required . A string used to identify this tag. You can spec-
ify a maximum of 128 characters for a tag key. Tags owned by
Amazon Web Services (Amazon Web Services) have the reserved
prefix: aws: .
Value -> (string)
Required . A string containing the value for this tag. You
can specify a maximum of 256 characters for a tag value.
Shorthand Syntax:
Key=string,Value=string ...
JSON Syntax:
[
{
"Key": "string",
"Value": "string"
}
...
]