I'm trying to narrow down the minimal policy to run a predefined machine image. The image is based on two snapshots and I only want "m1.medium" instance types to be launched.
Based on that and with the help of this page and this article, I worked out the following policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt1385026304010",
"Effect": "Allow",
"Action": [
"ec2:RunInstances"
],
"Condition": {
"StringEquals": {
"ec2:InstanceType": "m1.medium"
}
},
"Resource": [
"arn:aws:ec2:us-east-1::instance/*",
"arn:aws:ec2:us-east-1::image/ami-f1c3e498",
"arn:aws:ec2:us-east-1::snapshot/snap-e2f51ffa",
"arn:aws:ec2:us-east-1::snapshot/snap-18ca2000",
"arn:aws:ec2:us-east-1::key-pair/shenton",
"arn:aws:ec2:us-east-1::security-group/sg-6af56d02",
"arn:aws:ec2:us-east-1::volume/*"
]
}
]
}
The policy narrows down the exact image, snapshots, security group and key-pair while leaving the specific instance and volume open.
I'm using the CLI tools as follows, as described here:
aws ec2 run-instances --dry-run \
--image-id ami-f1c3e498 \
--key-name shenton \
--security-group-ids sg-6af56d02 \
--instance-type m1.medium
The ~/.aws/config is as follows:
[default]
output = json
region = us-east-1
aws_access_key_id = ...
aws_secret_access_key = ...
The command results in a generic You are not authorized to perform this operation message and the encoded authorization failure message indicates that none of my statements were matched and therefore it rejects the action.
Changing to "Resource": "*" resolves the issue obviously, but I want to gain more understanding as to why the above doesn't work. I fully realize that this involves some degree of guess work, so I welcome any ideas.
I've been contacted by Jeff Barr from Amazon Web Services and he kindly helped me find out what the issue was.
First you need to decode the authorization failure message using the following statement:
$ aws sts decode-authorization-message --encoded-message 6gO3mM3p....IkgLj8ekf
Make sure the IAM user / role has permission for the sts:DecodeAuthorizationMessage action.
The response contains a DecodedMessage key comprising another JSON encoded body:
{
"allowed": false,
"explicitDeny": false,
"matchedStatements": {
"items": []
},
"failures": {
"items": []
},
"context": {
"principal": {
"id": "accesskey",
"name": "testuser",
"arn": "arn:aws:iam::account:user/testuser"
},
"action": "ec2:RunInstances",
"resource": "arn:aws:ec2:us-east-1:account:instance/*",
"conditions": { ... }
}
}
Under context => resource it will show what resource it was attempting to match against the policy; as you can see, it expects an account number. The arn documentation should therefore be read as:
Unless otherwise specified, the region and account are required.
Adding the account number or * in the affected ARN's fixed the problem:
"Resource": [
"arn:aws:ec2:us-east-1:*:instance/*",
"arn:aws:ec2:us-east-1:*:image/ami-f1c3e498",
"arn:aws:ec2:us-east-1:*:snapshot/snap-e2f51ffa",
"arn:aws:ec2:us-east-1:*:snapshot/snap-18ca2000",
"arn:aws:ec2:us-east-1:*:key-pair/shenton",
"arn:aws:ec2:us-east-1:*:security-group/sg-6af56d02",
"arn:aws:ec2:us-east-1:*:volume/*"
]
Related
I am trying to register a snapshot for my elasticsearch on AWS. My goal is to create a snapshot of elasticsearch domain on a s3 bucket. Below is the command I am using:
curl -XPUT https://vpc-xxxxxxx.ap-southeast-2.es.amazonaws.com/_snapshot/es-snapshot -d '
{
"type": "s3",
"settings": {
"bucket": "$MY_BUCKET",
"region": "ap-southeast-2",
"role_arn": "arn:aws:iam::xxxx:role/es-snapshot-role"
}
}'
But I got this error:
{"Message":"User: anonymous is not authorized to perform: iam:PassRole on resource: arn:aws:iam::xxxx:role/es-snapshot-role"}
It seems like a role permission issue. I have configured the role policy as:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Action": [
"es:*",
"s3:*",
"iam:PassRole",
"es:ESHttpPut"
],
"Resource": [
"*"
]
}
]
}
And its trust relationship is:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "es.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
I wonder what else I missed here.
This post AccessDenied for EC2 Instance with attached IAM Role doesn't seem to relate to my issue.
Registering a Manual Snapshot Repository
You must register a snapshot repository with Amazon Elasticsearch Service before you can take manual index snapshots. This one-time operation requires that you sign your AWS request with credentials that are allowed to access TheSnapshotRole, as described in Manual Snapshot Prerequisites.
You can't use curl to perform this operation, because it doesn't support AWS request signing. Instead, use the sample Python client, Postman, or some other method to send a signed request to register the snapshot repository. The request takes the following form:
PUT elasticsearch-domain-endpoint/_snapshot/my-snapshot-repo
{
"type": "s3",
"settings": {
"bucket": "s3-bucket-name",
"region": "region",
"role_arn": "arn:aws:iam::123456789012:role/TheSnapshotRole"
}
}
Reference from AWS Documentation: Working with Amazon Elasticsearch Service Index Snapshots
Add iam:PassRole permissions to your IAM user and try the command,
I have an AWS account with Organizations enabled. I want to ensure that certain logs from my child accounts go to my Kinesis stream in a logging account. The idea is that in future if I create a new child account in Organizations, the logs should go to Kinesis.
For this, I have created a Kinesis log destination in my logging account using aws logs put-destination command. I added a destination policy to it. The policy I used was:
{
"Version": "2012-10-17",
"Statement": {
"Sid": "PutSubscriptionFilter",
"Effect": "Allow",
"Principal": {
"AWS": ["*"]
},
"Action": "logs:PutSubscriptionFilter",
"Resource": "arn:aws:logs:us-east-1:123456789012:destination:mytestLogDestination",
"Condition": {
"StringEquals": {
"aws:PrincipalOrgID": "o-abcde12345"
}
}
}
}
The command I used to add the destination policy was:
aws logs put-destination-policy \
--destination-name mytestLogDestination \
--access-policy file://destination_policy.json
This added the destination policy successfully. I can confirm this by running the command: aws logs describe-destinations --destination-name-prefix mytestLogDestination. When I try to create a new subscription filter in one of my member accounts using the following command, it errors out. The command I tried is:
aws logs put-subscription-filter \
--log-group-name "/aws/lambda/GetOrgIdFunction" \
--filter-name randomsubscriptionfilter --filter-pattern "" \
--destination-arn arn:aws:logs:us-east-1:123456789012:destination:mytestLogDestination
Error message is:
An error occurred (AccessDeniedException) when calling the PutSubscriptionFilter operation: User with accountId: 210987654321 is not authorized to perform: logs:PutSubscriptionFilter on resource: arn:aws:logs:us-east-1:123456789012:destination:mytestLogDestination
When I remove the condition and restrict the Principal to just my account (210987654321), it works fine. Is it possible to get this setup working or does AWS currently not support it?
As of August 02, 2019
After talking to AWS Support, this is CloudWatch Logs limitation as they don't yet support PrincipalOrgID. We would have to add each account separately when creating the log destination policy.
Marking this as an answer for now.
Update: January 06, 2021
According to a new AWS release, this is now supported. AWS documentation for reference: https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CreateDestination.html
How annoying, I wasted so much time testing different methods to try and get this running. Glad I eventually found your answer!
I don't suppose they gave you any further information regarding dates when it might be supported? I'm assuming no conditions work with these policies as I tried PrincipleArn and I was having the same issue.
I was only able to get it to work with the aws:SourceArn condition which is fairly frustrating.
{
"Version": "2008-10-17",
"Id": "__default_policy_ID",
"Statement": [
{
"Sid": "__default_statement_ID",
"Effect": "Allow",
"Principal": {
"Service": "cloudwatch.amazonaws.com",
"AWS": "*"
},
"Action": [
"SNS:GetTopicAttributes",
"SNS:SetTopicAttributes",
"SNS:AddPermission",
"SNS:RemovePermission",
"SNS:DeleteTopic",
"SNS:Subscribe",
"SNS:ListSubscriptionsByTopic",
"SNS:Publish",
"SNS:Receive"
],
"Resource": "<topic arn>",
"Condition": {
"ArnLike": {
"aws:SourceArn": [
"arn:aws:cloudwatch:<region>:<account a>:alarm:*",
"arn:aws:cloudwatch:<region>:<account b>:alarm:*",
"arn:aws:cloudwatch:<region>:<account c>:alarm:*"
]
}
}
}
]
}
I created a cloudformation stack with redshift cluster and a masteruser: testuser
"RedshiftCluster" : {
"IamRoles" : [
{
"Fn::GetAtt": [
"IAMInstanceRole",
"Arn"
]
}
]
... other configurations
It uses the below IAM role (IAMInstanceRole) which is in in-sync status and the redshift cluster is up and running:
"IAMInstanceRole": {
"Properties": {
"RoleName": "test-iam-role",
"AssumeRolePolicyDocument": {
"Statement": [
{
"Action": [
"sts:AssumeRole"
],
"Effect": "Allow",
"Principal": {
"Service": [
"ec2.amazonaws.com",
"redshift.amazonaws.com",
"s3.amazonaws.com"
]
}
}
]
},
"Path": "/",
"Policies": [
{
"PolicyName": "root",
"PolicyDocument": {
"Version" : "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "*",
"Resource": "*"
}
]
}
}
]
}
I'm trying to load csv file from s3 to redshift using copy command and iam_role as credential. The iam_role has the arn of IAMInstanceRole (declared above).
Whenever I execute the below command:
copy test_table from 's3://test-bucket/test.csv' CREDENTIALS 'aws_iam_role=arn:aws:iam::<account-id>:role/test-iam-role' MAXERROR 100000 removequotes TRIMBLANKS emptyasnull blanksasnull delimiter '|';
I get the error:
ERROR: User arn:aws:redshift:us-west-2:189675173661:dbuser:automated-data-sanity-redshiftcluster-fbp9fgls6lri/sanityuser is not authorized to assume IAM Role arn:aws:iam::189675173661:role/sanity-test-iam-instance-role
DETAIL:
-----------------------------------------------
error: User arn:aws:redshift:us-west-2:<account-id>:dbuser:test-redshiftcluster-fbp9fgls6lri/testuser is not authorized to assume IAM Role arn:aws:iam::<account-id>:role/test-iam-role
code: 8001
context: IAM Role=arn:aws:iam::<account-id>:role/test-iam-role
query: 1139
location: xen_aws_credentials_mgr.cpp:236
process: padbmaster [pid=29280]
-----------------------------------------------
Please suggest some resolution.
I ran into the same problem but after a good 1 hour of troubleshooting, I realised I had failed to add the Redshift role to the cluster while I was creating it. If you select the cluster from Redshift, choose the drop-down on 'Actions' and select 'Manage IAM roles' from there you will be able to attach the Redshift role you may have created for this cluster.
That solved the problem for me, anyways.
Hope this helps.
I resolved this issue !!
By default, IAM roles that are available to an Amazon Redshift cluster are available to all users on that cluster. You can choose to restrict IAM roles to specific Amazon Redshift database users on specific clusters or to specific regions.
To permit only specific database users to use an IAM role, take the following steps.
To identify specific database users with access to an IAM role
Identify the Amazon Resource Name (ARN) for the database users in your Amazon Redshift cluster. The ARN for a database user is in the format: arn:aws:redshift:region:account-id:dbuser:cluster-name/user-name.
Open the IAM Console at url="https://console.aws.amazon.com/.
In the navigation pane, choose Roles.
Choose the IAM role that you want to restrict to specific Amazon Redshift database users.
Choose the Trust Relationships tab, and then choose Edit Trust Relationship. A new IAM role that allows Amazon Redshift to access other AWS services on your behalf has a trust relationship as follows:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "redshift.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
Add a condition to the sts:AssumeRole action section of the trust relationship that limits the sts:ExternalId field to values that you specify. Include an ARN for each database user that you want to grant access to the role.
For example, the following trust relationship specifies that only database users user1 and user2 on cluster my-cluster in region us-west-2 have permission to use this IAM role.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "redshift.amazonaws.com"
},
"Action": "sts:AssumeRole",
"Condition": {
"StringEquals": {
"sts:ExternalId": [
"arn:aws:redshift:us-west-2:123456789012:dbuser:my-cluster/user1",
"arn:aws:redshift:us-west-2:123456789012:dbuser:my-cluster/user2"
]
}
}
}]
}
7.Choose Update Trust Policy.
Here's a template that works fine:
{
"AWSTemplateFormatVersion": "2010-09-09",
"Resources": {
"RedshiftRole": {
"Type": "AWS::IAM::Role",
"Properties": {
"RoleName": "Redshift-Role",
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": [
"redshift.amazonaws.com"
]
},
"Action": [
"sts:AssumeRole"
]
}
]
},
"Path": "/",
"Policies": [
{
"PolicyName": "root",
"PolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "s3:*",
"Resource": "*"
}
]
}
}
]
}
},
"RedshiftSG": {
"Type": "AWS::EC2::SecurityGroup",
"Properties": {
"GroupName": "Redshift Security Group",
"GroupDescription": "Enable access to redshift",
"VpcId": "vpc-11223344",
"SecurityGroupIngress": [
{
"IpProtocol": "tcp",
"FromPort": 5439,
"ToPort": 5439,
"CidrIp": "0.0.0.0/0"
}
],
"Tags": [
{
"Key": "Name",
"Value": "Redshift Security Group"
}
]
}
},
"RedshiftCluster": {
"Type": "AWS::Redshift::Cluster",
"Properties": {
"ClusterType": "single-node",
"NodeType": "dc2.large",
"MasterUsername": "master",
"MasterUserPassword": "YourPassword",
"IamRoles": [
{
"Fn::GetAtt": [
"RedshiftRole",
"Arn"
]
}
],
"VpcSecurityGroupIds": [
{
"Ref": "RedshiftSG"
}
],
"PubliclyAccessible": true,
"Port": 5439,
"DBName": "foo"
}
}
}
}
Be sure to insert your own VpcId in the security group.
The Role can be assumed by Redshift and grants access to s3:* (which you should reduce in scope).
I was trying to access Glue data catalog from Redshift. I created the role with the necessary policies attached (AWSGlueServiceRole, AmazonS3FullAccess), and added it to the cluster. However, I had set the AWS service as Glue but it should've been Redshift since Redshift is the service needing the access. Attaching these policies the Redshift role I have (and adding the role to the cluster, if necessary) solved the problem for me.
Resolved it
Complete Steps followed :
Create s3 bucket in the same region as redshift ( move-redshift-data)
create a folder inside it. ( move-redshift-data)
create an IAM role (move-redshift-data-role) ,attach S3Fullaccesss and add the following to trust relationship
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::126111577039:root”
},
"Action": "sts:AssumeRole"
}
]
}
where 126111577039 is the account ID of redshift cluster
Find you already attached role to your cluster
Open your redshift cluster
Click on actions -->Manage IAM roles
You could see the role (mine is RedshiftDynamoDBAccess)
Open the role in IAM console and attach the following inline policy to it .
Add below policy to the role already associated to redshift cluster (See in manage cluster)
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"sts:AssumeRole"
],
"Resource": "arn:aws:iam::888850378087:role/move-redshift-data-role"
}
]
}
where 888850378087 : account which has s3 bucket in it and move-redshift-data role
Finally Run command
unload ('select * from sellercompliancestate')
to 's3://unload-swarnimg/unload-swarnimg/'
iam_role 'arn:aws:iam::126111577039:role/RedshiftDynamoDBAccess,arn:aws:iam::888850378087:role/move-redshift-data-role'
allowoverwrite
format as csv;
Got the solution after searching for a while. I created separate IAM role for redshift as John suggested, which is a correct advice but was not the issue in my case.
Then followed the thread to resolve the issue: Copy from remote S3 using IAM Role - not authorized to assume IAM Role
I'd to activate the region where my cluster was in Account Settings
I solved by editing function like this
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": [
"sagemaker.amazonaws.com",
"redshift.amazonaws.com"
]
},
"Action": "sts:AssumeRole"
}
]
}
I add "sagemaker.amazonaws.com" to my function AmazonRedshiftML
Function
I figured it out.
There is no use of deleting a cluster, rebooting or managing IAM Roles in redshift Cluster.
Though I did all above many times, still I was getting the error. Then I tried below steps.
Give Access Key Id and secret Key in COPY command, instead of IAM role. Example as below.
copy users from 's3://awssampledbuswest2/tickit/allusers_pipe.txt'
credentials 'aws_access_key_id=;SKDFHSJKD;aws_secret_access_key=SDJHFJHajhsdjh'
delimiter '|' region 'us-west-2';
I am trying to build a Kibana dashboard fed with twitter data collected via AWS Kinesis firehose where data passes into an S3 bucket which triggers a Lambda function which passes the data to AWS Elastic Search and then to Kibana. I am following this blog https://aws.amazon.com/blogs/big-data/building-a-near-real-time-discovery-platform-with-aws/
The data is loading into the S3 bucket correctly but it never arrives in Kibana, I believe this is because the Lambda function is not being triggered by events in S3 as I would have hoped (there are no invocations or logs). I think this is because I have not set permissions properly. The Lambda function can be invoked manually by the test event.
On the Lambda function page I chose an existing role which I called lambda_s3_exec_role which has the AWSLambdaExecute policy attached to it but I feel I'm missing something else more specific to S3. I have been unable to follow this line in the blog in the create lambda function section because I do not recognise those options:
"10. Choose lambda_s3_exec_role (if this value does not exist, choose Create new role S3 execution role)."
Can anyone help me create the appropriate role/policy for the Lambda function, or spot what the problem may be?
From view permissions on the Lambda function I currently have:
FUNCTION POLICY
{
"Version": "2012-10-17",
"Id": "default",
"Statement": [
{
"Sid": "****",
"Effect": "Allow",
"Principal": {
"Service": "s3.amazonaws.com"
},
"Action": "lambda:InvokeFunction",
"Resource": "****",
"Condition": {
"ArnLike": {
"AWS:SourceArn": "arn:aws:s3:::****"
}
}
}
]
}
EXECUTION ROLE
{
"roleName": "lambda_s3_exec_role",
"policies": [
{
"document": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:*"
],
"Resource": "arn:aws:logs:*:*:*"
},
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject"
],
"Resource": "arn:aws:s3:::*"
}
]
},
"name": "AWSLambdaExecute",
"id": "****",
"type": "managed",
"arn": "arn:aws:iam::aws:policy/AWSLambdaExecute"
}
]
}
The permissions you have listed look OK so I am going to try provide some steps that might help find the issue as it is difficult to understand specifically where your issue might be.
Does the execution role have the trust relationship with a trusted entity of lambda.amazonaws.com
Does your event prefix match the prefix in firehose. In the tutorial they are both twitter/raw-data/. If firehose is writing to a path that isn't the event prefix then the event won't be invoked.
Does the lambda trigger any errors when you manually invoke it
Does the lambda write to the logs when you manually invoke it
Test the lambda using dummy data (example data below)
CLI
aws lambda invoke \
--invocation-type RequestResponse \
--function-name helloworld \
--region region \
--log-type Tail \
--payload file://dummy_event.json \
--profile adminuser \
outputfile.txt
Example data
source
dummy_event.json
{
"Records":[
{
"eventVersion":"2.0",
"eventSource":"aws:s3",
"awsRegion":"us-west-2",
"eventTime":"1970-01-01T00:00:00.000Z",
"eventName":"ObjectCreated:Put",
"userIdentity":{
"principalId":"AIDAJDPLRKLG7UEXAMPLE"
},
"requestParameters":{
"sourceIPAddress":"127.0.0.1"
},
"responseElements":{
"x-amz-request-id":"C3D13FE58DE4C810",
"x-amz-id-2":"FMyUVURIY8/IgAtTv8xRjskZQpcIZ9KG4V5Wp6S7S/JRWeUWerMUE5JgHvANOjpD"
},
"s3":{
"s3SchemaVersion":"1.0",
"configurationId":"testConfigRule",
"bucket":{
"name":"sourcebucket",
"ownerIdentity":{
"principalId":"A3NL1KOZZKExample"
},
"arn":"arn:aws:s3:::sourcebucket"
},
"object":{
"key":"HappyFace.jpg",
"size":1024,
"eTag":"d41d8cd98f00b204e9800998ecf8427e",
"versionId":"096fKKXTRTtl3on89fVO.nfljtsv6qko"
}
}
}
]
}
Struggled with this for a long time and eventually realized that your rule that triggers the lambda cannot have exactly the same name as the lambda itself or it won't work.
I'm attempting to update an IAM policy to restrict it to a single region but I have a syntax error that I can't seem to work out.
Here's the JSON for the policy:
{
"Version":"2012-10-17",
"Statement":[
{
"Action":[
"ec2:*"
],
"Resource":[
"*"
],
"Effect":"Allow",
"Condition":{
"condition":{
"StringEquals":{
"ec2:Region":"us-east-1"
}
}
}
}
]
}
It's the condition that's causing the issue but I can't quite seem to figure out where I've gone wrong. I've tried using the AWS IAM policy builder to generate it but it still gives me a syntax error when I try creating it via that so I'm guessing it's a syntax issue.
All I'm trying to do is restrict the policy to EC2 operations in us-east-1 only.
I'm using this command to upload it
aws iam put-group-policy --group-name eastern-contractors --policy-document file://ec2.json --policy-name ec2
And I get the following response
A client error (MalformedPolicyDocument) occurred when calling the
PutGroupPolicy operation: There are invalid conditions in this policy.
I've tried using similar answer on this site such as AWS IAM Permissions for EC2 – Controlling Access on Specific Instances with particular region but it doesn't seem to work.
Here is the working policy
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt1410456206000",
"Effect": "Allow",
"Action": [
"ec2:*"
],
"Condition": {
"StringEquals": {
"ec2:Region": "us-east-1"
}
},
"Resource": [
"*"
]
}
]
}
The nested "condition" was the issue.