I'm trying to do bulk provisioning from the AWS IOT console
i chose Parameter file from s3, selected a role and entered provision template
when i check the status,all the provisions are failing
my role has SystemAdministrator , AWSIoTFullAccess , AmazonS3FullAccess and this inline policy permissions
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": "sts:*",
"Resource": "*"
}
]
}
my parameter file is this
{"ThingName": "e-scooter_0", "ThingTypeName": "vehicle", "ThingId": "0"}
{"ThingName": "e-scooter_1", "ThingTypeName": "vehicle", "ThingId": "1"}
{"ThingName": "e-scooter_2", "ThingTypeName": "vehicle", "ThingId": "2"}
{"ThingName": "e-scooter_3", "ThingTypeName": "vehicle", "ThingId": "3"}
my Provisioning template is this
{
"Parameters": {
"ThingName": {
"Type": "String"
},
"ThingTypeName": {
"Type": "String"
},
"ThingId": {
"Type": "String"
}
},
"Resources": {
"thing": {
"Type": "AWS::IoT::Thing",
"Properties": {
"ThingName": {
"Ref": "ThingName"
},
"ThingTypeName": {
"Ref": "ThingTypeName"
},
"AttributePayload": {
"version": "v1",
"thingId": {
"Ref": "ThingId"
}
}
}
}
}
}
any help is appreciated
Related
I've been slamming my head against the wall for the past 2 days trying to get dynamic partitioning to work in the delivery stream I'm creating using a serverless application model template. I've tried multiple configurations, but it seems as soon as I fill in a prefix, enable dynamic partitioning and add a metadata extraction processor to get myself a field I can use as the prefix my stack stops working, I can generate alerts in aws and see the logs from the lambda processor I also added, but no files reach the bucket, I also can't seem to configure cloudwatch logs for the stream as well, since I've tried creating them through the script and manually and neither at least have logs pertaining to any error....
I'll leave my script here, hopefully I'm doing something wrong:
{
"AWSTemplateFormatVersion": "2010-09-09",
"Transform": "AWS::Serverless-2016-10-31",
"Description": "An AWS Serverless Application.",
"Parameters": {
"BucketArn": {
"Type": "String"
}
},
"Resources": {
"DeliveryPolicy": {
"Type": "AWS::IAM::Policy",
"Properties": {
"PolicyName": "K-POLICY",
"Roles": [
{
"Ref": "DeliveryRole"
}
],
"PolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:*"
],
"Resource": [
{
"Ref": "BucketArn"
}
]
},
{
"Effect": "Allow",
"Action": [
"lambda:InvokeFunction",
"lambda:GetFunctionConfiguration"
],
"Resource": [
{
"Fn::GetAtt": [
"Processor",
"Arn"
]
}
]
}
]
}
}
},
"DeliveryRole": {
"Type": "AWS::IAM::Role",
"Properties": {
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": [
"firehose.amazonaws.com"
]
},
"Action": [
"sts:AssumeRole"
]
}
]
}
}
},
"DeliveryStream": {
"Type": "AWS::KinesisFirehose::DeliveryStream",
"Properties": {
"DeliveryStreamName": "K-STREAM",
"DeliveryStreamType": "DirectPut",
"ExtendedS3DestinationConfiguration": {
"BucketARN": {
"Ref": "BucketArn"
},
"Prefix": "!{partitionKeyFromQuery:symb}",
"ErrorOutputPrefix": "errors/!{firehose:random-string}/!{firehose:error-output-type}/!{timestamp:yyyy/MM/dd}",
"RoleARN": {
"Fn::GetAtt": [
"DeliveryRole",
"Arn"
]
},
"DynamicPartitioningConfiguration": {
"Enabled": true
},
"ProcessingConfiguration": {
"Enabled": true,
"Processors": [
{
"Type": "MetadataExtraction",
"Parameters": [
{
"ParameterName": "MetadataExtractionQuery",
"ParameterValue": "{symb: .TICKER_SYMBOL}"
},
{
"ParameterName": "JsonParsingEngine",
"ParameterValue": "JQ-1.6"
}
]
},
{
"Type": "Lambda",
"Parameters": [
{
"ParameterName": "LambdaArn",
"ParameterValue": {
"Fn::GetAtt": [
"Processor",
"Arn"
]
}
}
]
}
]
}
}
}
},
"Processor": {
"Type": "AWS::Serverless::Function",
"Properties": {
"Handler": "K-PROCESSOR::KRATOS_PROCESSOR.Functions::FunctionHandler",
"FunctionName": "K-PROCESSOR",
"Runtime": "dotnetcore3.1",
"CodeUri": "./staging/app/3cs-int-k",
"MemorySize": 256,
"Timeout": 60,
"Role": null,
"Policies": [
"AWSLambdaBasicExecutionRole"
]
}
}
}
}
I am new to AWS CloudFormation, and I am trying to capture events from SQS Queue and place them in S3 bucket via AWS lambda. Flow of events is
SNS --> SQS <-- Lambda ---> S3 bucket.
I am trying to achieve above flow using cloudFormation template.I am getting below error message after deploying my CloudFormation template. Any help you can provide would be greatly appreciated. Thank you
11:51:56 2022-01-13 17:51:47,930 - INFO - ...
11:52:53 2022-01-13 17:52:48,511 - ERROR - Stack myDemoApp shows a rollback status ROLLBACK_IN_PROGRESS.
11:52:53 2022-01-13 17:52:48,674 - INFO - The following root cause failure event was found in the myDemoApp stack for resource EventStreamLambda:
11:52:53 2022-01-13 17:52:48,674 - INFO - Resource handler returned message: "Error occurred while GetObject. S3 Error Code: NoSuchKey.
S3 Error Message: The specified key does not exist. (Service: Lambda, Status Code: 400, Request
ID: 5f2f9882-a863-4a58-90bd-7e0d0dfdf4d5, Extended Request ID: null)" (RequestToken: 0a95acb4-a677-0a2d-d0bc-8b7487a858ad, HandlerErrorCode: InvalidRequest)
11:52:53 2022-01-13 17:52:48,674 - INFO - ..
MY lambda function is :
import json
import logging
import boto3
logger = logging.getLogger()
logger.setLevel(logging.INFO)
logging.basicConfig(level=logging.INFO,
format='%(asctime)s: %(levelname)s: %(message)s')
def lambda_handler(event, context):
logger.info(f"lambda_handler -- event: {json.dumps(event)}")
s3_bucket = boto3.resource("3")
event_message = json.loads(event["Records"][0]["body"])
s3_bucket.put_object(Bucket="S3DeployBucket", key="data.json", Body=json.dumps(event_message))
My complete CloudFormation template is :
{
"AWSTemplateFormatVersion": "2010-09-09",
"Description": "myDemoApp Resource Stack",
"Mappings": {
},
"Parameters": {
"S3DeployBucket": {
"Default": "myDemoApp-deploy-bucket",
"Description": "Bucket for deployment configs and artifacts for myDemoApp",
"Type": "String"
},
"EnvName": {
"Description": "Platform environment name for myDemoApp",
"Type": "String"
},
"AuditRecordKeyArn": {
"Description": "ARN for audit record key encryption for myDemoApp",
"Type": "String"
},
"ParentVPCStack": {
"Description": "The name of the stack containing the parent VPC for myDemoApp",
"Type": "String"
},
"StackVersion": {
"Description": "The version of this stack of myDemoApp",
"Type": "String"
},
"EventLogFolderName": {
"Type": "String",
"Description": "folder name for the logs for the event stream of myDemoApp",
"Default": "event_log_stream"
},
"EventLogPartitionKeys": {
"Type": "String",
"Description": "The partition keys that audit logs will write to S3. Use Hive-style naming conventions for automatic Athena/Glue comprehension.",
"Default": "year=!{timestamp:yyyy}/month=!{timestamp:MM}/day=!{timestamp:dd}/hour=!{timestamp:HH}"
},
"AppEventSNSTopicArn": {
"Description": "Events SNS Topic of myDemoApp",
"Type": "String"
},
"ReportingEventsRetentionDays": {
"Default": "2192",
"Description": "The number of days to retain a record used for reporting.",
"Type": "String"
}
},
"Resources": {
"AppEventSQSQueue": {
"Type": "AWS::SQS::Queue"
},
"AppEventSnsSubscription": {
"Type": "AWS::SNS::Subscription",
"Properties": {
"TopicArn": {
"Ref": "AppEventSNSTopicArn"
},
"Endpoint": {
"Fn::GetAtt": [
"AppEventSQSQueue",
"Arn"
]
},
"Protocol": "sqs"
}
},
"S3DeployBucket": {
"Type": "AWS::S3::Bucket",
"DeletionPolicy": "Retain",
"UpdateReplacePolicy": "Retain",
"Properties": {
"BucketEncryption": {
"ServerSideEncryptionConfiguration": [
{
"ServerSideEncryptionByDefault": {
"KMSMasterKeyID": {
"Ref": "AuditRecordKeyArn"
},
"SSEAlgorithm": "aws:kms"
}
}
]
},
"VersioningConfiguration": {
"Status": "Enabled"
},
"LifecycleConfiguration": {
"Rules": [
{
"ExpirationInDays": {
"Ref": "ReportingEventsRetentionDays"
},
"Status": "Enabled"
}
]
}
}
},
"EventStreamLogGroup": {
"Type": "AWS::Logs::LogGroup"
},
"EventLogStream": {
"Type": "AWS::Logs::LogStream",
"Properties": {
"LogGroupName": {
"Ref": "EventStreamLogGroup"
}
}
},
"EventStreamSubscriptionRole": {
"Type": "AWS::IAM::Role",
"Properties": {
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "sns.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
},
"Policies": [
{
"PolicyName": "SNSSQSAccessPolicy",
"PolicyDocument": {
"Version": "2012-10-17",
"Statement": {
"Action": [
"sqs:*"
],
"Effect": "Allow",
"Resource": "*"
}
}
}
]
}
},
"EventDeliveryRole": {
"Type": "AWS::IAM::Role",
"Properties": {
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "sqs.amazonaws.com"
},
"Action": "sts:AssumeRole",
"Condition": {
"StringEquals": {
"sts:ExternalId": {
"Ref": "AWS::AccountId"
}
}
}
}
]
}
}
},
"EventSqsQueuePolicy": {
"Type": "AWS::SQS::QueuePolicy",
"Properties": {
"PolicyDocument": {
"Version": "2012-10-17",
"Id": "SqsQueuePolicy",
"Statement": [
{
"Sid": "Allow-SNS-SendMessage",
"Effect": "Allow",
"Principal": "*",
"Action": [
"sqs:SendMessage",
"sqs:ReceiveMessage"
],
"Resource": {
"Fn::GetAtt": [
"EventStreamLambda",
"Arn"
]
},
"Condition": {
"ArnEquals": {
"aws:SourceArn": {
"Ref": "EventSNSTopicArn"
}
}
}
}
]
},
"Queues": [
{
"Ref": "EventSNSTopicArn"
}
]
}
},
"EventDeliveryPolicy": {
"Type": "AWS::IAM::Policy",
"Properties": {
"PolicyName": "sqs_delivery_policy",
"PolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:PutObject"
],
"Resource": [
{
"Fn::GetAtt": [
"S3DeployBucket",
"Arn"
]
},
{
"Fn::Join": [
"",
[
{
"Fn::GetAtt": [
"S3DeployBucket",
"Arn"
]
},
"/*"
]
]
}
]
},
{
"Effect": "Allow",
"Action": [
"logs:PutLogEvents"
],
"Resource": {
"Fn::Sub": "arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:log-group:${EventStreamLogGroup}:log-stream:${EventLogStreamLogStream}"
}
},
{
"Effect": "Allow",
"Action": [
"kms:Decrypt",
"kms:GenerateDataKey"
],
"Resource": [
{
"Ref": "AuditRecordKeyArn"
}
],
"Condition": {
"StringEquals": {
"kms:ViaService": {
"Fn::Join": [
"",
[
"s3.",
{
"Ref": "AWS::Region"
},
".amazonaws.com"
]
]
}
},
"StringLike": {
"kms:EncryptionContext:aws:s3:arn": {
"Fn::Join": [
"",
[
{
"Fn::GetAtt": [
"S3DeployBucket",
"Arn"
]
},
"/*"
]
]
}
}
}
}
]
},
"Roles": [
{
"Ref": "EventDeliveryRole"
}
]
}
},
"EventStreamLambda": {
"Type": "AWS::Lambda::Function",
"Properties": {
"Handler": "lambda_function.lambda_handler",
"MemorySize": 128,
"Runtime": "python3.8",
"Timeout": 30,
"FunctionName": "sqs_s3_pipeline_job",
"Role": {
"Fn::GetAtt": [
"SQSLambdaExecutionRole",
"Arn"
]
},
"Code": {
"S3Bucket": {
"Ref": "S3DeployBucket"
},
"S3Key": {
"Ref": "S3DeployBucket"
}
},
"TracingConfig": {
"Mode": "Active"
}
}
},
"SQSLambdaExecutionRole": {
"Type": "AWS::IAM::Role",
"Properties": {
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": [
"lambda.amazonaws.com"
]
},
"Action": [
"sts:AssumeRole"
]
}
]
},
"Policies": [
{
"PolicyName": "StreamLambdaLogs",
"PolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:*"
],
"Resource": "arn:aws:logs:*:*:*"
}
]
}
},
{
"PolicyName": "SQSLambdaPolicy",
"PolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"sqs:ReceiveMessage",
"sqs:DeleteMessage",
"sqs:GetQueueAttributes",
"sqs:ChangeMessageVisibility"
],
"Resource":"*"
}
]
}
}
]
}
}
},
"Outputs": {
"VpcSubnet3ExportKey": {
"Value": {
"Fn::Sub": "${ParentVPCStack}-privateSubnet3"
}
}
}
}
SubscriptionRoleArn is only for kinesis:
This property applies only to Amazon Kinesis Data Firehose delivery stream subscriptions.
I am setting up an API HTTP Gateway (V2) with lambda integrations via Cloudformation, and everything has been working so far. I have 2 working integrations, but my third integration is not working: Everything looks fine from the API Gateway side (it lists the correct route with a link to the Lambda), but the API endpoint in the lambda is listed as "https://c59boisn2k.execute-api.eu-central-1.amazonaws.com/productionnull". When I try to call the route, it says "Not Found". The odd thing is that I am using the same template for all three integrations.
I was thinking it could be a "dependsOn" issue, but I think I have all the correct dependencies. I tried re-creating the stack from scratch and now two of the three functions say "null" in their URL while the API Gateway still states the correct routes. Can this be a 'dependsOn' problem?
Here's my template for a single integration:
{
"Resources": {
"api": {
"Type": "AWS::ApiGatewayV2::Api",
"Properties": {
"Name": { "Ref": "AWS::StackName" },
"ProtocolType": "HTTP",
"CorsConfiguration": {
"AllowMethods": ["*"],
"AllowOrigins": ["*"]
}
}
},
"stage": {
"Type": "AWS::ApiGatewayV2::Stage",
"Properties": {
"Description": { "Ref": "AWS::StackName" },
"StageName": "production",
"AutoDeploy": true,
"ApiId": { "Ref": "api" },
"AccessLogSettings": {
"DestinationArn": {
"Fn::GetAtt": ["stageLogGroup", "Arn"]
}
}
}
},
"getSignedS3LambdaRole": {
"Type": "AWS::IAM::Role",
"Properties": {
"RoleName": {
"Fn::Sub": "${AWS::StackName}-getSignedS3"
},
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": ["lambda.amazonaws.com"]
},
"Action": ["sts:AssumeRole"]
}
]
},
"Policies": [
{
"PolicyName": "root",
"PolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Resource": "arn:aws:logs:*:*:*",
"Action": "logs:*"
},
{
"Effect": "Allow",
"Action": ["s3:*"],
"Resource": ["arn:aws:s3:::euromomo.eu/uploads/*"]
}
]
}
}
]
}
},
"getSignedS3Lambda": {
"Type": "AWS::Lambda::Function",
"DependsOn": ["getSignedS3LambdaRole"],
"Properties": {
"FunctionName": {
"Fn::Sub": "${AWS::StackName}-getSignedS3"
},
"Code": {
"S3Bucket": { "Ref": "operationsS3Bucket" },
"S3Key": { "Ref": "getSignedS3S3Key" }
},
"Runtime": "nodejs10.x",
"Handler": "index.handler",
"Role": { "Fn::GetAtt": ["getSignedS3LambdaRole", "Arn"] }
}
},
"getSignedS3Permission": {
"Type": "AWS::Lambda::Permission",
"DependsOn": ["api", "getSignedS3Lambda"],
"Properties": {
"Action": "lambda:InvokeFunction",
"FunctionName": { "Ref": "getSignedS3Lambda" },
"Principal": "apigateway.amazonaws.com",
"SourceArn": {
"Fn::Sub": "arn:aws:execute-api:${AWS::Region}:${AWS::AccountId}:${api}/*/*"
}
}
},
"getSignedS3Integration": {
"Type": "AWS::ApiGatewayV2::Integration",
"DependsOn": ["getSignedS3Permission"],
"Properties": {
"ApiId": { "Ref": "api" },
"IntegrationType": "AWS_PROXY",
"IntegrationUri": {
"Fn::Sub": "arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${getSignedS3Lambda.Arn}/invocations"
},
"PayloadFormatVersion": "2.0"
}
},
"getSignedS3Route": {
"Type": "AWS::ApiGatewayV2::Route",
"DependsOn": ["getSignedS3Integration"],
"Properties": {
"ApiId": { "Ref": "api" },
"RouteKey": "POST /getSignedS3",
"AuthorizationType": "NONE",
"Target": { "Fn::Sub": "integrations/${getSignedS3Integration}" }
}
}
}
}
After spending hours debugging this, I found that the problem was in my Lambda permission. I need to use the correct path in the permission.
This does not work:
arn:aws:execute-api:${AWS::Region}:${AWS::AccountId}:${api}/*/*
This does work:
arn:aws:execute-api:${AWS::Region}:${AWS::AccountId}:${api}/*/*/getSignedS3
I believe I could scope it even more to this:
arn:aws:execute-api:${AWS::Region}:${AWS::AccountId}:${api}/*/POST/getSignedS3
This fixed all my problems and shows the correct path in the lambda web console.
I am trying to create two user using CFT i am very new to cloudformation how do we define multiple users i have tried below but getting cft error.
{
"Resources": {
"AWSSCRIPTS": {
"Type": "AWS::IAM::User"
},
"AWSSCRIPTSPolicy": {
"Type": "AWS::IAM::ManagedPolicy",
"Properties": {
"Description" : "This policy allows to run scripts in new account.",
"PolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "*",
"Resource": "*"
}
]
},
"Users": [{
"Ref": "AWSSCRIPTS"
}]
}
},
"AWSSCRIPTSKeys": {
"Type": "AWS::IAM::AccessKey",
"Properties": {
"UserName": {
"Ref": "AWSSCRIPTS"
}
}
}
},
"ADDUSER": {
"Type": "AWS::IAM::User"
},
"ADDUSERPolicy": {
"Type": "AWS::IAM::ManagedPolicy",
"Properties": {
"Description" : "This policy allows to list IAM Roles for AAD User.",
"PolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "*",
"Resource": "*"
}
]
},
"Users": [{
"Ref": "ADDUSER"
}]
}
},
"ADDUSERKeys": {
"Type": "AWS::IAM::AccessKey",
"Properties": {
"UserName": {
"Ref": "ADDUSER"
}
}
},
"Outputs": {
"AccessKey": {
"Value": {
"Ref": "AWSSCRIPTS"
},
"Description": "Access Key ID of AWS Scripts"
},
"SecretKey": {
"Value": {
"Fn::GetAtt": [
"AWSSCRIPTSKeys",
"SecretAccessKey"
]
},
"Description": "Secret Key of AWS Scripts User"
},
"AccessKey2": {
"Value": {
"Ref": "ADDUSER"
},
"Description": "Access Key ID of ADD USER"
},
"SecretKey2": {
"Value": {
"Fn::GetAtt": [
"ADDUSERKeys",
"SecretAccessKey"
]
},
"Description": "Secret Key of ADD User"
}
}
}
I am getting below error
Invalid template property or properties [ADDUSERPolicy, ADDUSER, ADDUSERKeys]
Create credentials for the user, depending on the type of access the user requires:
Programmatic access: The IAM user might need to make API calls, use the AWS CLI, or use the Tools for Windows PowerShell. In that case, create an access key (access key ID and a secret access key) for that user.
AWS Management Console access: If the user needs to access the AWS Management Console, create a password for the user.
ADDUSER, ADDUSERPolicy and ADDUSERKeys should be in Resources, but they are on the same level:
I am trying to create my Encryption key with cloudformation. So just to test I have a very simple one as follow:
{
"AWSTemplateFormatVersion": "2010-09-09",
"Description": "Creates a KMS key and attaches a policy similar to the default policy. Also, creates two Roles which allow encryption and decryption under this key.",
"UserPrincipal": {
"Type": "String",
"Default": "user/datadog"
}
},
"Resources": {
"DemonstrationKey": {
"Type": "AWS::KMS::Key",
"Properties": {
"KeyPolicy": {
"Id": "DefaultKmsPolicy",
"Version": "2012-10-17",
"Statement": [{
"Sid": "Enable IAM User Permissions",
"Effect": "Allow",
"Principal": {
"AWS": [{
"Fn::Join": [
":", [
"arn:aws:iam:",
{
"Ref": "AWS::AccountId"
},
"root"
]
]
}]
},
"Action": "kms:*",
"Resource": "*"
}]
}
}
}
},
"Outputs": {
"KeyID": {
"Description": "Key ID",
"Value": {
"Ref": "DemonstrationKey"
}
}
}
}
And it works fine but this is not what I want. Instead I want to attach the already existing policy to it for example sth like this:
{
"AWSTemplateFormatVersion": "2010-09-09",
"Description": "Creates a KMS key and attaches a policy similar to the default policy. Also, creates two Roles which allow encryption and decryption under this key.",
"UserPrincipal": {
"Type": "String",
"Default": "user/datadog"
}
},
"Resources": {
"DemonstrationKey": {
"Type": "AWS::KMS::Key",
"Properties": {
"KeyPolicy": "arn:aws:iam::******:policy/testtestpol1"
}
}
},
"Outputs": {
"KeyID": {
"Description": "Key ID",
"Value": {
"Ref": "DemonstrationKey"
}
}
}
}
But this does not work and I get the following error:
MalformedPolicyDocumentException
Can anyone help me with that. Is it doable at all?