The below CF template is indented to create a stack set containing one role with specific parameter values depending on the OU where the account is located. So if the account is located / created in OU A the trust relationship of the debug role must point to account "111111111111".
AWSTemplateFormatVersion: "2010-09-09"
Description: Debug Role
Resources:
StackSet:
Type: AWS::CloudFormation::StackSet
Properties:
Description: Debug Role Stack Set
PermissionModel: SERVICE_MANAGED
Capabilities:
- CAPABILITY_NAMED_IAM
AutoDeployment:
Enabled: true
RetainStacksOnAccountRemoval: false
Parameters:
- ParameterKey: TrustAccountId
ParameterValue: "123456789123" # default value
OperationPreferences:
RegionConcurrencyType: PARALLEL
FailureToleranceCount: 0
MaxConcurrentPercentage: 100
StackInstancesGroup:
- DeploymentTargets:
OrganizationalUnitIds:
- ou-aaaa # A
ParameterOverrides:
- ParameterKey: "TrustAccountId"
ParameterValue: "111111111111"
Regions:
- 'eu-central-1'
- DeploymentTargets:
OrganizationalUnitIds:
- ou-bbbb # B
ParameterOverrides:
- ParameterKey: "TrustAccountId"
ParameterValue: "222222222222"
Regions:
- 'eu-central-1'
StackSetName: debug-role-stack-set
TemplateBody: |
AWSTemplateFormatVersion: 2010-09-09
Description: Debug Role
Parameters:
TrustAccountId:
Description: Trust to account
Type: String
AllowedPattern: \d{12}
Resources:
DebugRole:
Type: AWS::IAM::Role
Description: "Role for debugging accounts"
Properties:
RoleName: DebugRole
Path: /
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Principal:
AWS:
- Ref: TrustAccountId
Action:
- 'sts:AssumeRole'
ManagedPolicyArns:
- arn:aws:iam::aws:policy/AdministratorAccess
Now the issue is that all account within the stack instances group, regardless if existing or new account, get the stack applied with the default parameter value "123456789123". So all accounts get the role having the incorrect trust relationship to "123456789123".
The only way to apply the correct value based on the OU is to update the stack instances with the CLI.
aws cloudformation update-stack-instances --stack-set-name debug-role-stack-set --deployment-targets OrganizationalUnitIds=ou-aaaa --regions eu-central-1 --operation-preferences FailureToleranceCount=0,MaxConcurrentCount=5 --parameter-overrides Pa
rameterKey=TrustAccountId,ParameterValue=111111111111
Is there any way to apply the correct parameter values during the stack creation without updating them afterwards? Because for every new account all stack set instances needs to be updated.
Since the stack instances have not been executed with the correct parameter value, I ended up having one stack set for each OU. Of course, this solution is only feasible for static environments. So in my case, if new OUs pop up new stack sets must be created.
Related
I'm trying to create a cloudFormation stack with MSK Configuration and associating MSK Configuration with MSK Cluster. Creation of AWS::MSK::Configuration returns only ARN while I need ARN and Revision number to associate MSK Configuration to MSK Cluster. Is there any way to achieve this? Currently I'm hard-coding it to 1 which means it will work only for creating stack.
...
MSKConfiguration:
Type: AWS::MSK::Configuration
Properties:
Name: aws-msk-configuration
ServerProperties: |
auto.create.topics.enable = true
zookeeper.connection.timeout.ms = 1000
log.roll.ms = 604800000
MSKCluster:
Type: AWS::MSK::Cluster
Properties:
ClusterName: !Ref ClusterName
ClientAuthentication: !If
- UsingIamAuthentication
- Sasl:
Iam:
Enabled: true
- Sasl:
Scram:
Enabled: true
ConfigurationInfo:
Arn: !GetAtt MSKConfiguration.Arn
Revision: 1
...
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-msk-configuration.html
You can only get the latest version if you define a custom resource. Since you program the full logic of the resource, you can do what you want, including automatically setting up latest version for MKS.
I am creating a custom policy to attach it to the IAM role which has been autogenerated by AWS.
Below is the policy:-
rRotationLambdaDecryptPolicy:
Type: AWS::IAM::ManagedPolicy
DependsOn: rSecretRotationScheduleHostedRotationLambda
Properties:
Description: "Providing access to HostedLambda for decrypting KMS"
ManagedPolicyName: CustomedHostedLambdaKmsUserRolePolicy
PolicyDocument:
Version: '2012-10-17'
Statement:
- Sid: AllowLambdaDecryptKMS
Effect: Allow
Action:
- kms:Decrypt
- kms:CreateGrant
Resource:
- !Sub arn:aws:kms:*:${AWS::AccountId}:key/*
Condition:
ForAnyValue:StringLike:
kms:ResourceAliases: alias/SecretsManager_KMSKey
Roles: <friendly rolename>
In Roles parameter , as i am not fully aware of the Rolename, so have been trying to generate it from its arn.
Roles:
- !Select [!Split ["/", !Sub 'arn:aws:iam::${AWS::AccountId}:role/secret-rotat-SecretsManagerRDSPostgre-*']]
but once pushed to cloudformation, getting error as below:-
The specified value for roleName is invalid. It must contain only alphanumeric characters and/or the following: +=,.#_- (Service: AmazonIdentityManagement; Status Code: 400; Error Code: ValidationError; Request ID: fbd4b14e-8c0e-459f-867f-968052828620; Proxy: null)
Not sure what is wrong here, and how can i refer it!
You can't use a wildcard * in the role name.
You would need to save the role name somewhere you can retrieve it (e.g. in Systems Manager Parameter store), or pass it as a parameter in to the template and then !Ref it.
i looked out everywhere, and found that this will be part of next version release of SAM as it requires work to add more attributes to the HostedLambda.
Till the time i managed to retrieve it from jenkins using AWSCLI (by getting the Lambda attributes which has that autogenerated role attached) and then processing it.
aur_rolename=sh(script: """aws lambda get-function --function-name SecretsManager-research-creds-rotation-lambda --query Configuration.Role --output text""", returnStdout: true).trim().split('/')[1]
aur_policyarn="arn:aws:iam::${env.account}:policy/CustomedHostedLambdaKmsUserRolePolicy"
sh(script: """aws iam attach-role-policy --policy-arn ${aur_policyarn} --role-name ${aur_rolename}""", returnStdout: true)
I want to automate the process of bucket creation through CI/CD pipeline based on the data mentioned in one of the yaml file. So, I have got bucket.yaml file which contains the name of all the buckets. This file keeps changing as more buckets names will be added in future. Currently, this is how bucket.yaml looks
BucketName:
- test-bucket
- test-bucket2
- test-bucket3
I have got one template.yaml file which is a cloudformation template for s3 buckets creation. Here is how it looks:
Resources:
S3Bucket:
Type: 'AWS::S3::Bucket'
DeletionPolicy: Retain
Properties:
BucketName: This will come from bucket.yaml
Now, template.yaml will fetch the bucket names from bucket.yaml file and should create 3 buckets as mentioned in bucket.yaml. If someone adds 2 more buckets in bucket.yaml, then template.yaml should create those 2 new buckets as well. Also, if someone deletes any bucket name from bucket.yaml then those buckets should be deleted as well. I couldn't find out the process in my research, just found information in bits and pieces.So, here I have specific questions, if its possible to do:
How to fetch bucket names from bucket.yaml and template.yaml should create all the buckets.
If someone update/add/delete bucket name in bucket.yaml, template.yaml should update those accordingly.
Also, please explain how will I do it through CI/CD pipeline in Azure DevOps.
About your first question:
How to fetch bucket names from bucket.yaml and template.yaml should create all the buckets.
In bucket.yaml you can use Parameters to set up the BucketName.
For example:
parameters:
- name: BucketName
type: object
default:
- test-bucket
- test-bucket2
- test-bucket3
steps:
- ${{ each value in parameters.BucketName }}:
- script: echo ${{ value }}
The step in here can loop through the values of the parameter BucketName.
In the template.yaml you can call the bucket.yaml like as below.
trigger:
- main
extends:
template: bucket.yaml
For your second question:
If someone update/add/delete bucket name in bucket.yaml, template.yaml should update those accordingly.
There is no any easy way to do this. You can try to write a script to run in the pipeline to do the following things:
List all the buckets that have been created. This is the list of the the existing buckets.
Compare the list of the existing buckets with the values list of the parameter BucketName to check which buckets need to be added and which need to be deleted.
If a bucket is listed in the parameter but not in the existing buckets, this bucket should be created as a new bucket.
If a bucket is listed in the existing buckets but not in the parameter, this bucket should be deleted.
BucketName:
- test-bucket
- test-bucket2
- test-bucket3
The requirements imply that all S3 buckets will be created in the same way and that no deviation from the given Cloudformation template (AWS::S3::Bucket) is required.
The requirements require us to track what S3 buckets need to be deleted. Cloudformation will not delete the S3 buckets as the Cloudformation template snippet contains a DeletionPolicy of Retain.
Solution:
The S3 buckets can be tagged in a specific way to identify them as being owned by the current CI/CD pipeline. S3 buckets can be listed and all the S3 buckets that are tagged in the correct way, and yet, does not exist in bucket.yaml can then be deleted.
I would personally just create S3 buckets required by the CI/CD pipeline using the AWS SDK and manually manage the S3 bucket deletion. If an application requires a S3 bucket then they should create it themselves in their application's Cloudformation stack so that they can !Ref it and customize it the way they want (eg encryption at rest, versioning, lifecycle rules, etc).
Technical note:
For a S3 bucket to be deleted its contents will also need to be deleted. This will require us to list all the objects in the S3 bucket and then delete them. Some documentation for the Java SDK [here].
Only subsequently will the API call to delete the S3 bucket succeed.
You can get Cloudformation to delete your S3 objects using a custom resource. That said, I don't find the custom resources that fun to work with - so if you can use the AWS SDK inside your CI/CD pipeline I would probably just use that.
The custom resource to delete a bucket's contents might look something like this in Cloudformation: (Its a custom resource that kicks of a Lambda. The Lambda will delete the S3 bucket contents if the custom resource gets deprovisioned)
# https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-cfn-customresource.html
# https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/walkthrough-custom-resources-lambda-lookup-amiids.html
ExampleBucketOperationCustomResource:
Type: AWS::CloudFormation::CustomResource
DependsOn: [Bucket, ExampleBucketOperationLambdaFunction]
Properties:
ServiceToken: !GetAtt ExampleBucketOperationLambdaFunction.Arn
# Custom properties
BucketToUse: !Ref S3BucketName
ExampleBucketOperationLambdaFunctionExecutionRole:
Type: AWS::IAM::Role
Properties:
RoleName: "ExampleBucketOperationLambda-ExecutionRole"
Path: "/"
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- sts:AssumeRole
Principal:
Service:
- lambda.amazonaws.com
Policies:
- PolicyName: "ExampleBucketOperationLambda-CanAccessCloudwatchLogs"
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- logs:CreateLogGroup
- logs:CreateLogStream
- logs:PutLogEvents
Resource: arn:aws:logs:*:*:*
- PolicyName: "ExampleBucketOperationLambda-S3BucketLevelPermissions"
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- s3:ListBucket
Resource:
- !Sub "arn:aws:s3:::${S3BucketName}"
- PolicyName: "ExampleBucketOperationLambda-S3ObjectLevelPermissions"
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- s3:DeleteObject
- s3:PutObject
Resource:
- !Sub "arn:aws:s3:::${S3BucketName}/*"
# Test payload:
# {"RequestType":"Create","ResourceProperties":{"BucketToUse":"your-bucket-name"}}
ExampleBucketOperationLambdaFunction:
Type: AWS::Lambda::Function
DependsOn: ExampleBucketOperationLambdaFunctionExecutionRole
# DeletionPolicy: Retain
Properties:
FunctionName: "ExampleBucketOperationLambda"
Role: !GetAtt ExampleBucketOperationLambdaFunctionExecutionRole.Arn
Runtime: python3.8
Handler: index.handler
Timeout: 30
Code:
ZipFile: |
import boto3
import cfnresponse
def handler(event, context):
eventType = event["RequestType"]
print("The event type is: " + str(eventType));
bucketToUse = event["ResourceProperties"]["BucketToUse"]
print("The bucket to use: " + str(bucketToUse));
try:
# Requires s3:ListBucket permission
if (eventType in ["Delete"]):
print("Deleting everyting in bucket: " + str(bucketToUse));
s3Client = boto3.client("s3")
s3Bucket = boto3.resource("s3").Bucket(bucketToUse)
for currFile in s3Bucket.objects.all():
print("Deleting file: " + currFile.key);
s3Client.delete_object(Bucket=bucketToUse, Key=currFile.key)
print("All done")
responseData = {}
cfnresponse.send(event, context, cfnresponse.SUCCESS, responseData)
except Exception as e:
responseData = {}
errorDetail = "Exception: " + str(e)
errorDetail = errorDetail + "\n\t More detail can be found in CloudWatch Log Stream: " + context.log_stream_name
print(errorDetail)
cfnresponse.send(event=event, context=context, responseStatus=cfnresponse.FAILED, responseData=responseData, reason=errorDetail)
Thanks for the above answers. I took a different path to solve this issue. I used AWS CDK to implement what I exactly wanted. I personally used AWS CDK for Python and created infrastructure using that.
I currently have a Cognito user pool configured to trigger a pre sign up lambda. Right now I am setting up the staging environment, and I have the exact same setup on dev (which works). I know it is the same because I am creating both envs out of the same terraform files.
I have already associated the invoke permissions with the lambda function, which is very often the cause for this error message. Everything looks the same in both environments, except that I get "PreSignUp invocation failed due to configuration" when I try to sign up a new user from my new staging environment.
I have tried to remove and re-associate the trigger manually, from the console, still, it doesn't work
I have compared every possible setting I can think of, including "App client" configs. They are really the same
I tried editing the lambda code in order to "force" it to update
Can it be AWS taking too long to invalidate the permissions cache? So far I can only believe this is a bug from AWS...
Any ideas!?
There appears to be a race condition with permissions not being attached on the first deployment.
I was able to reproduce this with cloudformation.
Deploying a stack with the same config twice appears to "fix" the permissions issue.
I actually added a 10-second delay on the permissions attachment and it solved my first deployment issue...
I hope this helps others who run into this issue. 😃
# Hack to fix Cloudformation bug
# AWS::Lambda::Permission will not attach correctly on first deployment unless "delay" is used
# DependsOn & every other thing did not work... ¯\_(ツ)_/¯
CustomResourceDelay:
Type: Custom::Delay
DependsOn:
- PostConfirmationLambdaFunction
- CustomMessageLambdaFunction
- CognitoUserPool
Properties:
ServiceToken: !GetAtt CustomResourceDelayFunction.Arn
SecondsToWait: 10
CustomResourceDelayFunctionRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement: [{ "Effect":"Allow","Principal":{"Service":["lambda.amazonaws.com"]},"Action":["sts:AssumeRole"] }]
Policies:
- PolicyName: !Sub "${AWS::StackName}-delay-lambda-logs"
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action: [ logs:CreateLogGroup, logs:CreateLogStream, logs:PutLogEvents ]
Resource: !Sub arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/${AWS::StackName}*:*
CustomResourceDelayFunction:
Type: AWS::Lambda::Function
Properties:
Handler: index.handler
Description: Wait for N seconds custom resource for stack debounce
Timeout: 120
Role: !GetAtt CustomResourceDelayFunctionRole.Arn
Runtime: nodejs12.x
Code:
ZipFile: |
const { send, SUCCESS } = require('cfn-response')
exports.handler = (event, context, callback) => {
if (event.RequestType !== 'Create') {
return send(event, context, SUCCESS)
}
const timeout = (event.ResourceProperties.SecondsToWait || 10) * 1000
setTimeout(() => send(event, context, SUCCESS), timeout)
}
# ------------------------- Roles & Permissions for cognito resources ---------------------------
CognitoTriggerPostConfirmationInvokePermission:
Type: AWS::Lambda::Permission
## CustomResourceDelay needed to property attach permission
DependsOn: [ CustomResourceDelay ]
Properties:
Action: lambda:InvokeFunction
FunctionName: !GetAtt PostConfirmationLambdaFunction.Arn
Principal: cognito-idp.amazonaws.com
SourceArn: !GetAtt CognitoUserPool.Arn
In my situation the problem was caused by the execution permissions of the Lambda function: While there was a role configured, that role was empty due to some unrelated changes.
Making sure the role actually had permissions to do the logging and all the other things that the function was trying to do made things work again for me.
I have a CloudFormation template that creates an AWS::Events::Rule and an AWS::SSM::Document. I need to provide a list of Targets for the SSM::Rule, but each target expects an ARN:
mySSMDocument:
Type: AWS::SSM::Document
Properties:
DocumentType: 'Command'
Content:
schemaVersion: '2.2'
description: "Code that will be run on EC2"
mainSteps:
- action: "aws:runShellScript"
name: runShellScript
inputs:
runCommand:
- 'Some command to execute'
myEventRule:
Type: AWS::Events::Rule
Properties:
Description: "A description for the Rule."
EventPattern:
source:
- "aws.autoscaling"
detail-type:
- "EC2 Instance-terminate Lifecycle Action"
detail:
AutoScalingGroupName:
- !Ref 'someAutoScalingGroupInThisTemplate'
RoleArn: 'some role ARN'
State: "ENABLED"
Targets:
- Id: "some-unique-id"
Arn: <-- This is the value that I need to fill in.
RunCommandParameters:
RunCommandTargets:
- Key: "tag: Name"
Values:
- 'The name of the EC2 machine'
I think that I need to replace the <-- This is the value that I need to fill in. with the ARN of mySSMDocument, but I don't see any way to retrieve this value from within the template itself. The documentation does not specify any GetAtt functionality on SSM::Document that allows to get the ARN. Anyone know how to solve this issue?
This is ARN pattern of Document
arn:${Partition}:ssm:${Region}:${Account}:document/${DocumentName}
example:
arn:aws:ssm:us-east-2:12345678912:document/demoooo
You can use Ref function to get name of document, then Sub to create final ARN
refer: https://docs.aws.amazon.com/IAM/latest/UserGuide/list_awssystemsmanager.html#awssystemsmanager-resources-for-iam-policies
!Sub arn:${AWS::Partition}:ssm:${AWS::Region}:${AWS::AccountId}:document/${mySSMDocument}
You can produce the ARN format for AWS::SSM::Document using the return Value for AWS::SSM::Document, the Pseudo Parameters for Partition, Region, and AccountId, and the Sub intrinsic function