I have followed those instructions step by step: https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/backups-exporting.html
However I have the following error:
An error occurred (InvalidParameterValue) when calling the CopySnapshot operation: Elasticache was unable to validate the authenticated user has access on the S3 bucket ...
The bucket is in the same region of the backup
This is my bucket configuration:
{
"LocationConstraint": "eu-central-1"
}
{
"Version": "2012-10-17",
"Id": "xxxxxxxx",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "eu-central-1.elasticache-snapshot.amazonaws.com"
},
"Action": [
"s3:PutObject",
"s3:GetObject",
"s3:ListBucket",
"s3:GetBucketAcl",
"s3:ListMultipartUploadParts",
"s3:ListBucketMultipartUploads"
],
"Resource": [
"arn:aws:s3:::my-bucket-name/*",
"arn:aws:s3:::my-bucket-name"
]
}
]
}
This is the snapshot
{
"Snapshots": [
{
"SnapshotName": "my-snapshot-name",
"CacheClusterId": "xxxxxxxx-xxx",
"SnapshotStatus": "available",
"SnapshotSource": "manual",
"CacheNodeType": "cache.t2.micro",
"Engine": "redis",
"EngineVersion": "5.0.3",
"NumCacheNodes": 1,
"PreferredAvailabilityZone": "eu-central-1c",
"CacheClusterCreateTime": "xxxxxxx",
"PreferredMaintenanceWindow": "mon:02:30-mon:03:30",
"Port": 6379,
"CacheParameterGroupName": "default.redis5.0",
"CacheSubnetGroupName": "internal",
"VpcId": "xxxxx",
"AutoMinorVersionUpgrade": true,
"SnapshotRetentionLimit": 7,
"SnapshotWindow": "00:00-02:00",
"NodeSnapshots": [
{
"CacheNodeId": "0001",
"CacheSize": "33 MB",
"CacheNodeCreateTime": "xxxxxx",
"SnapshotCreateTime": "xxxxxx"
}
],
"ARN": "arn:aws:elasticache:eu-central-1:000000000:snapshot:my-snapshot-name",
"DataTiering": "disabled"
}
]
}
UPDATE
Apparently AWS updated their docs by adding a crucial piece of information regarding the ACL, look the accepted answer for more info.
Here very important step is to add ACL as mentioned in doc:
Add grantee Canonical Id 540804c33a284a299d2547575ce1010f2312ef3da9b3a053c8bc45bf233e4353 with the following options:
Objects: List, Write
Bucket ACL: Read, Write
I added this ACL permission and it started working like a charm.
After adding this ACL my configuration looks below.
Full Ref document link: https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/backups-exporting.html#backups-exporting-grant-access
Based on the article you linked, you also need additional S3 permission:
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": [
"s3:GetBucketLocation",
"s3:ListAllMyBuckets",
"s3:PutObject",
"s3:GetObject",
"s3:DeleteObject",
"s3:ListBucket"
],
"Resource": "arn:aws:s3:::*"
}]
}
Have you verified that your IAM user has these access?
When you create the S3 bucket, enable ACLs. Then you can continue with the Permissions section of the instructions.
Related
I have two buckets:
"source-bucket" (in AWS account 88888888).
"destination-bucket" (in AWS account 99999999)
Both buckets have versioning enabled and are located in the same region (eu-west-1).
In the source bucket, I've created a Replication-rule with the following settings:
I opted for automatic role creation, which created a role with the following policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"s3:ListBucket",
"s3:GetReplicationConfiguration",
"s3:GetObjectVersionForReplication",
"s3:GetObjectVersionAcl",
"s3:GetObjectVersionTagging",
"s3:GetObjectRetention",
"s3:GetObjectLegalHold"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::source-bucket",
"arn:aws:s3:::source-bucket/*",
"arn:aws:s3:::destination-bucket",
"arn:aws:s3:::destination-bucket/*"
]
},
{
"Action": [
"s3:ReplicateObject",
"s3:ReplicateDelete",
"s3:ReplicateTags",
"s3:ObjectOwnerOverrideToBucketOwner"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::source-bucket/*",
"arn:aws:s3:::destination-bucket/*"
]
}
]
}
According to the documentation found here https://docs.aws.amazon.com/AmazonS3/latest/userguide/replication-walkthrough-2.html
, I've added a bucket policy to "destination-bucket", which looks as follows:
{
"Version": "2012-10-17",
"Id": "",
"Statement": [
{
"Sid": "Set permissions for objects",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::88888888:role/service-role/auto-created-role"
},
"Action": [
"s3:ReplicateObject",
"s3:ReplicateDelete"
],
"Resource": "arn:aws:s3:::destination-bucket/*"
},
{
"Sid": "Set permissions on bucket",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::88888888:role/service-role/auto-created-role"
},
"Action": [
"s3:GetBucketVersioning",
"s3:PutBucketVersioning"
],
"Resource": "arn:aws:s3:::destination-bucket"
}
]
}
But, when I add a file to the source bucket, nothing seems happens.
Does anyone have any idea what could be wrong here?
The AWS docs aren't the best here. From your pictures I see you have enabled the setting "change object ownership to destination bucket owner" (as most people would).
However, this requires an extra permission on the destination side give them ownership. s3:ObjectOwnerOverrideToBucketOwner
The following policy should work for you
{
"Version": "2012-10-17",
"Id": "",
"Statement": [
{
"Sid": "Set permissions for objects",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::88888888:role/service-role/auto-created-role"
},
"Action": [
"s3:ReplicateObject",
"s3:ReplicateDelete",
"s3:ObjectOwnerOverrideToBucketOwner"
],
"Resource": "arn:aws:s3:::destination-bucket/*"
},
{
"Sid": "Set permissions on bucket",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::88888888:role/service-role/auto-created-role"
},
"Action": [
"s3:GetBucketVersioning",
"s3:PutBucketVersioning"
],
"Resource": "arn:aws:s3:::destination-bucket"
}
]
}
To debug this issue I used aws s3api head-object --bucket <bucket> --key <prefix> --query ReplicationStatus to see the replication failed and then I added s3:* permission on the destination side to see if it was a permission issue. Which in this case it was.
Check if this helps.
By default, Amazon S3 doesn't replicate objects that are stored at rest using server-side encryption with AWS Key Management Service (AWS KMS) customer master keys (CMKs). To replicate encrypted objects, you modify the bucket replication configuration to tell Amazon S3 to replicate these objects.
https://docs.aws.amazon.com/AmazonS3/latest/userguide/replication-walkthrough-4.html
I am building a CI/CD pipeline using AWS Codepipeline, the repository source is on bitbucket and I used the AWS-Codestarconnections to create a connection between the bitbucket repository and the pipeline.
The pipeline details are below:
{
"pipeline": {
"name": "test_pipeline",
"roleArn": "arn:aws:iam::<AccountId>:role/PipelineServiceRole",
"artifactStore": {
"type": "S3",
"location": "tadadadada-artifact"
},
"stages": [
{
"name": "Source",
"actions": [
{
"name": "Source",
"actionTypeId": {
"category": "Source",
"owner": "AWS",
"provider": "CodeStarSourceConnection",
"version": "1"
},
"runOrder": 1,
"configuration": {
"BranchName": "dev",
"ConnectionArn": "arn:aws:codestar-connections:us-east-2:<AccountId>:connection/4ca7b1cf-2917-4fda-b681-c5239944eb33",
"FullRepositoryId": "<username>/repository_name",
"OutputArtifactFormat": "CODE_ZIP"
},
"outputArtifacts": [
{
"name": "SourceArtifact"
}
],
"inputArtifacts": [],
"region": "us-east-2",
"namespace": "SourceVariables"
}
]
},
{
"name": "Build",
"actions": [
{
....
}
]
}
],
"version": 1
},
"metadata": {
"pipelineArn": "arn:aws:codepipeline:us-east-2:<AccountId>:test_pipeline",
"created": 1611669087.267,
"updated": 1611669087.267
}
}
The PipelineServiceRole + that policy attached to it are:
Service Role
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "codepipeline.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
Policy
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "IamPassRolePolicy",
"Effect": "Allow",
"Action": [
"iam:PassRole"
],
"Resource": "*",
"Condition": {
"StringEqualsIfExists": {
"iam:PassedToService": [
"cloudformation.amazonaws.com",
"ec2.amazonaws.com",
"ecs-tasks.amazonaws.com"
]
}
}
},
{
"Sid": "CodeBuildPolicy",
"Effect": "Allow",
"Action": [
"codebuild:BatchGetBuilds",
"codebuild:StartBuild"
],
"Resource": "*"
},
{
"Sid": "S3AccessPolicy",
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:GetObjectVersion",
"s3:GetBucketAcl",
"s3:GetBucketLocation"
],
"Resource": "*"
},
{
"Sid": "ECRAccessPolicy",
"Effect": "Allow",
"Action": [
"ecr:DescribeImages"
],
"Resource": "*"
},
{
"Sid": "CodeStarConnectionsAccessPolicy",
"Effect": "Allow",
"Action": [
"codestar-connections:UseConnection"
],
"Resource": "*"
}
]
}
The source stage fails with an error :
[Bitbucket] Upload to S3 failed with the following error: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 085999D90C19E650; S3 Extended Request ID: gJ6l08+cX3U6i2Vj0+fW7PiqA/UzM6ZGCfyECmWb+Jit4Knu+gi/L4y3F24uqkFWUfGy9tZo0VE=; Proxy: null) (Service: null; Status Code: 0; Error Code: null; Request ID: null; S3 Extended Request ID: null; Proxy: null) (Service: null; Status Code: 0; Error Code: null; Request ID: null; S3 Extended Request ID: null; Proxy: null)
The error message lacks details, I am not sure which service is trying to access s3, shouldn't it be code-pipeline (which in this case has PutObject permission)?
Resolved this by changing the OutputArtifactFormat from "OutputArtifactFormat": "CODE_ZIP" to "OutputArtifactFormat": "CODEBUILD_CLONE_REF".
CODEBUILD_CLONE_REF - from the console description is a Full clone, in which case
AWS CodePipeline passes metadata about the repository that allows subsequent actions to do a full git clone. Only supported for AWS CodeBuild actions.
The "CODE_ZIP" option does not include the git metadata about the repository
This issue appears to be related to a recent change in the CDK's default IAM Role for the BitBucketSourceAction.
I found that by adding the "s3:PutObjectAcl" action to the list I was able to successfully integrate the BitBucketSourecAction (for GitHub version 2 connection). Note: this did not require:
Changing the OutputArtifactFormat from "OutputArtifactFormat": "CODE_ZIP" to "OutputArtifactFormat": "CODEBUILD_CLONE_REF", or,
S3-full-access "s3:*"
As detailed in this CDK issue, I was using the BitBucketSourceAction to integrate with a GitHub repository. I got the following error when the CodePipeline first attempted the GitHub (Version2) action:
[GitHub] Upload to S3 failed with the following error: Access Denied
On a previous pipeline I released with the BitBucketSourceAction the "s3:PutObject*" wildcarded action was included in the synthesized template. On reviewing the IAM role generated during my latest cdk deployment (using version 1.91.0) the BitBucketSourceAction only had the "s3:PutObject" action (i.e. not wildcarded). This excludes the "s3:PutObjectAcl" action which seems to be required to upload the source repository from GitHub to S3 and free it up for use further along in the pipeline.
Adding the s3:PutObjectAcl action permission to the role policy associated with the Pipeline Bucket Store worked for me.
I had to add the following permissions:
s3:GetObject
s3:GetObjectVersion
s3:PutObject
s3:GetBucketVersioning
s3:PutObjectAcl
I had the same problem using GitHub.
[GitHub] Upload to S3 failed with the following error: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: foo; S3 Extended Request ID: bar; Proxy: null)
But in the artifact store S3 bucket, object was updated.
So I changed s3 service policy to full access.
"s3:PutObject",
"s3:GetObject",
"s3:GetObjectVersion",
"s3:GetBucketVersioning",
↓
"s3:*",
Had this exact problem today and idk why this fixed it but the policy attached to the PipelineGithubRole had 2 s3 statements, one contained just List* action and the other contained all the Read & Put actions, so I just moved them into a single statement and it started working.
I just tried adding tags to some buckets and then I created an inline IAM role policy that'd give that role access to the S3 buckets however that didn't work. I tried both iam:ResourceTag/tagName and s3:ResourceTag/tagName as conditionals but neither worked.
As everything looked just fine I started thinking that AWS might not have implemented this yet for S3. Is that the case? I tried reviewing documentation and indeed I didn't find anything about this use of tags working with S3.
For example the role HumanResources should have to all buckets tagged with HR, Recruitment etc. but no other buckets.
In looking at Actions, Resources, and Condition Keys for Amazon S3 - AWS Identity and Access Management, there does not appear to be the ability to specify a Bucket Tag in an IAM Policy.
One alternative is to use a wildcard in a bucket name. For example, you could grant permission to access:
acme-hr-1
You could grant permissions based on a bucket name of acme-hr-*.
Yes you can but you will need do on each S3 Resource Policy.
Here is an S3 Policy to grant access to the bucket for only IAM users and roles with a Tag department set to "hr".
To ensure HR employee only have access to these buckets you will need to remove all S3 access from their IAM user/role access polices.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "DenyObjectOthers",
"Effect": "Deny",
"Principal": "*",
"Action": [
"s3:AbortMultipartUpload",
"s3:ListMultipartUploadParts",
"s3:DeleteObject*",
"s3:PutObject*",
"s3:GetObject*",
"s3:RestoreObject*"
],
"Resource": [
"arn:aws:s3:::BUCKET_NAME/*"
],
"Condition": {
"StringNotLike": {
"aws:PrincipalTag/department": [
"hr"
]
}
}
},
{
"Sid": "DenyListOthers",
"Effect": "Deny",
"Principal": "*",
"Action": [
"s3:ListBucket*"
],
"Resource": [
"arn:aws:s3:::BUCKET_NAME"
],
"Condition": {
"StringNotLike": {
"aws:PrincipalTag/department": [
"hr"
]
}
}
},
{
"Sid": "AllowObject",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::AWS_ACCOUNT_NUMBER:root"
},
"Action": [
"s3:AbortMultipartUpload",
"s3:ListMultipartUploadParts",
"s3:DeleteObject*",
"s3:PutObject*",
"s3:GetObject*",
"s3:RestoreObject*"
],
"Resource": [
"arn:aws:s3:::BUCKET_NAME/*"
],
"Condition": {
"StringLike": {
"aws:PrincipalTag/department": [
"hr"
]
}
}
},
{
"Sid": "AllowList",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::AWS_ACCOUNT_NUMBER:root"
},
"Action": [
"s3:ListBucket*"
],
"Resource": [
"arn:aws:s3:::BUCKET_NAME"
],
"Condition": {
"StringLike": {
"aws:PrincipalTag/department": [
"hr"
]
}
}
}
]
}
Previous Wrong Answer
From: IAM Policy Elements: Variables and Tags - AWS Identity and Access Management
"Resource": ["arn:aws:s3:::bucket/${aws:PrincipalTag/department}"]
Also make sure to include the version at 2012-10-17.
I want to add a cloudwatch subscription to a AWS lambda logs thereby making my AWS lambda triggered by cloudwatch logs. What permissions should I add to the role which lambda is using to enable this?
Your Lambda will by default have access to CloudWatch to write logs (with the default AWSLambdaBasicExecutionRole), however if you want to manually add it this is the policy with the required permissions:
{
"document": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": "*"
}
]
},
"name": "AWSLambdaBasicExecutionRole",
"id": "xxxxx",
"type": "managed",
"arn": "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
}
Lambda function policy for CloudWatch event trigger on Lambda:
{
"Version": "2012-10-17",
"Id": "default",
"Statement": [
{
"Sid": "uuid",
"Effect": "Allow",
"Principal": {
"Service": "events.amazonaws.com"
},
"Action": "lambda:invokeFunction",
"Resource": "arn:aws:lambda:us-east-x:xxxxxxxxxxxx:function:LambdaFunction",
"Condition": {
"ArnLike": {
"AWS:SourceArn": "arn:aws:events:us-east-x:xxxxxxxxxxxx:rule/CloudWatchRule"
}
}
}
]
}
I have an AWS managed Elasticsearch Service (say smallES) which has an properly working S3 bucket attached to containing day wise rolling indices of last 1 year. I've created another AWS managed ES cluster (say bigES) for some business reason. I want to restore last 1 year's data from bucket into bigES. It's guaranteed that smallES bigES and bucket all are in the same region and same VPC.
So, I created a policy :
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:GetBucketLocation",
"s3:ListBucketMultipartUploads",
"s3:ListBucketVersions"
],
"Resource": [
"arn:aws:s3:::bucket"
]
},
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObject",
"s3:AbortMultipartUpload",
"s3:ListMultipartUploadParts"
],
"Resource": [
"arn:aws:s3:::bucket/*"
]
}
]
}
And attached the policy with a role. Trust relationship of that role is
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "s3.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
Now, when I create a snapshot by http request within the same VPC, it can create a snapshot-repo for the bigES and I can query for that too
curl -XGET 'http://bigESid.region.es.amazonaws.com:80/_snapshot'
Output
{
"snapshot-repo": {
"type": "s3",
"settings": {
"bucket": "bucket",
"region": "region",
"role_arn": "role_arn"
}
}
}
But when I try to see the snapshots in this snapshot repo I get error (described below)
curl -XGET 'http://bigESid.region.es.amazonaws.com:80/_cat/snapshots/snapshot-repo'
I get the following error:
{
"error": {
"root_cause": [
{
"type": "a_w_s_security_token_service_exception",
"reason": "User: arn:aws:sts::acountid:assumed-role/cp-sts-grant-role/swift-region-prod-365021432299 is not authorized to perform: sts:AssumeRole on resource: role-arn (Service: AWSSecurityTokenService; Status Code: 403; Error Code: AccessDenied; Request ID: some-id)"
}
],
"type": "a_w_s_security_token_service_exception",
"reason": "User: arn:aws:sts::acountid:assumed-role/cp-sts-grant-role/swift-region-prod-365021432299 is not authorized to perform: sts:AssumeRole on resource: role-arn (Service: AWSSecurityTokenService; Status Code: 403; Error Code: AccessDenied; Request ID: some-id)"
},
"status": 500
}
I've given all access of s3 to my role, but no luck. I've posted all the http requests from a ec2 machine inside the VPC.
Also to mention, if I query like following, I see expected result
curl -XGET 'http://smallESid.region.es.amazonaws.com:80/_cat/snapshots/snapshot-repo'
IDK why I tried making a role which has trust relationship like following. Still no luck.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "ec2.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
Thanks in advance for any kind of help/suggestions.
I had the same issue, and it was because I'd not allowed the Elasticsearch service to assume the role. I had to update my trust relationship policy document to include es.amazonaws.com.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Service": [
"es.amazonaws.com",
"ec2.amazonaws.com"
]
},
"Action": "sts:AssumeRole"
}
]
}
I solved this problem using the following policy
{
"Statement": [
{
"Action": [
"s3:ListBucket",
"s3:GetBucketLocation",
"s3:ListBucketMultipartUploads",
"s3:ListBucketVersions"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::bucket-name"
]
},
{
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObject",
"s3:AbortMultipartUpload",
"s3:ListMultipartUploadParts",
"iam:PassRole"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::bucket-name/*"
]
}
],
"Version": "2012-10-17"
}
Then I attached the policy to the role. I think "iam:PassRole"has done the work.