Redshift Scheduler unable to create schedule - amazon-web-services

My AWS has 2 different Users: admin, s3_readonly
I am the main admin and have 1 cluster in Redshift(cluster1).
Now I am trying to schedule a query that just calls those procedures every hour (CALL <procedure_name>)
For this task, I have followed the official documentation from AWS (Scheduling a query on the Amazon Redshift console - Amazon Redshift) and to be exact this document steps (Scheduling SQL queries on your Amazon Redshift data warehouse | AWS Big Data Blog).
So I created new IAM role RedshiftScheduler, which has Redshift Customizable option and have attached AmazonRedshiftDataFullAccess to it. Then I edited the Trust relationship and added:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "redshift.amazonaws.com"
},
"Action": "sts:AssumeRole"
},
{
"Sid": "S2",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::<ACCOUNT_ID>:user/admin"
},
"Action": "sts:AssumeRole"
},
{
"Sid": "S1",
"Effect": "Allow",
"Principal": {
"Service": "events.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
I then went back to my AWS user (admin) and attached a new policy granted with Assume Role permissions:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "S3",
"Effect": "Allow",
"Action": "sts:AssumeRole",
"Resource": "arn:aws:iam::<ACCOUNT_ID>:role/RedshiftScheduler"
}
]
}
Now, I logged in to the Redshift cluster via AWS service. Used Temporary credentials to connect to cluster1 and user as dbuser. However, when I try to schedule the query it throws an error
To view the schedule history of this schedule, add sts:AssumeRole for IAM role arn:aws:iam::<ACCOUNT_ID>:role/RedshiftScheduler to your IAM role. You also need to add your IAM user ARN to the role’s trust policy.

You need to add your IAM user ARN to the role’s trust policy like this
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::<account #>:user/<admin username"
},
"Action": "sts:AssumeRole",
"Condition": {}
}
after
{
"Sid": "S1",
"Effect": "Allow",
"Principal": {
"Service": "events.amazonaws.com"
},
"Action": "sts:AssumeRole"
}

Related

Correct way to write multiple Principal AWS in AWS IAM Role TrustEntity Relation

I am new to AWS IAM Roles.
Here scenario is, I have an IAM Role (DDBReadRole) for DynamoDB read access (in Account P lets say).
And we have 2 lambda execution roles L1,L2 in Account B, Account C respectively.
Now these 2 lambda executions roles need to be added to DDBReadRole access Trust Entities relation
For this I am writing
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": ["arn:aws:iam::<AccountBId>:role/<AccountBRole>", "arn:aws:iam::
<AccountCId>:role/<AccountCRole>"]
},
"Action": "sts:AssumeRole"
}
]
}
I got an other option
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::<AccountBId>:role/<AccountBRole>
},
"Action": "sts:AssumeRole"
},
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::<AccountCId>:role/<AccountCRole>"
},
"Action": "sts:AssumeRole"
}
]
}
Can some one please help me which is the correct way to add 2 AWS Principals in trust entities!!
Both are correct, and you can use any of them. But the first form is usually used, because its shorter.

Access Denied when querying in Athena for data in S3 bucket in another AWS account

I want to use Glue Crawler to crawl data from an S3 bucket. This S3 bucket is in another AWS account. Let's call is Account A. My Glue Crawler is in Account B.
I have created a Role in Account B and called it AWSGlueServiceRole-Reporting
I have attached the following policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "BucketAccess",
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:GetBucketLocation"
],
"Resource": [
"arn:aws:s3:::AccountAbucketname"
]
},
{
"Sid": "ObjectAccess",
"Effect": "Allow",
"Action": "s3:GetObject",
"Resource": [
"arn:aws:s3:::AccountABucketName/Foldername/*"
]
}
]
}
And also AWSGlueServiceRole policy.
In Account A that has the S3 bucket, I've attached the following bucket policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::AccountB:role/AWSGlueServiceRoleReporting”
},
"Action": [
"s3:ListBucket",
"s3:GetBucketLocation"
],
"Resource": "arn:aws:s3:::AccountABucketName"
},
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::AccountB:role/AWSGlueServiceRoleReporting”
},
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::AccountABucketName/FolderName/*"
}
]
}
I'm able to run a Glue Crawler in Account B on this S3 bucket and it created Glue Tables. But when I try to query them in Athena, I get Access Denied.
Can anybody help me how to query it in Athena??
When Amazon Athena queries run, they use the permissions of the user that is running the query.
Therefore, you will need to modify the Bucket Policy on the bucket in Account A to permit access by whoever is running the query in Amazon Athena:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam::AccountB:role/AWSGlueServiceRoleReporting",
"arn:aws:iam::AccountB:user/username"
]
},
"Action": [
"s3:ListBucket",
"s3:GetBucketLocation"
],
"Resource": "arn:aws:s3:::AccountABucketName"
},
{
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam::AccountB:role/AWSGlueServiceRoleReporting",
"arn:aws:iam::AccountB:user/username"
]
},
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::AccountABucketName/FolderName/*"
}
]
}
The user will also need sufficient S3 permissions (on their IAM User) to access that S3 bucket. (For example, having s3:ListBucket and s3:GetObject on S3 buckets. They likely already have this, but it is worth mentioning.)
This is different to AWS Glue, which uses an IAM Role. Athena does not accept an IAM Role for running queries.

Why is this bucket policy being shown as public

I am relatively new to AWS.
So recently, we creates an AWS S3 bucket through the AWS console. Then we set the following policy on the bucket:
{
"Version": "2008-10-17",
"Statement": [
{
"Sid": "AddJPGImageFilter",
"Effect": "Allow",
"Principal": {
"AWS": "IAM user ARN"
},
"Action": "s3:PutObject",
"Resource": "bucket_name/*.jpg"
},
{
"Sid": "AddJPEGImageFilter",
"Effect": "Allow",
"Principal": {
"AWS": ""IAM user ARN""
},
"Action": "s3:PutObject",
"Resource": "bucket_name/*.jpeg"
},
{
"Sid": "AddPNGImageFilter",
"Effect": "Allow",
"Principal": {
"AWS": ""IAM user ARN""
},
"Action": "s3:PutObject",
"Resource": "bucket_name/*.png"
}
]
}
Now from what I understand the above policy means the resource would be accessible (with the S3:PutObject action allowed) to only the IAM user ARN mentioned in the Principal.
For it to be public, the Principal would need to be *.
Is my understanding correct ?
If yes, why is the AWS console still showing the above bucket as Public ?
AWS web console has page caching issues. Try logout, clear your cache and re-login again.
Thanks
Sanjog

AWS Firehose cross region/account policy

I am trying to create Firehose streams that can receive data from different regions in Account A, through AWS Lambda, and output into a redshift table in Account B. To do this I created an IAM role on Account A:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Service": "firehose.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
I gave it the following permissions:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Action": [
"s3:AbortMultipartUpload",
"s3:GetBucketLocation",
"s3:GetObject",
"s3:ListBucket",
"s3:ListBucketMultipartUploads",
"s3:PutObject"
],
"Resource": [
"arn:aws:s3:::b-bucket/*",
"arn:aws:s3:::b-bucket"
]
},
{
"Effect": "Allow",
"Action": [
"firehose:*"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"redshift:*"
],
"Resource": "*"
}
]
}
On Account B I created a role with this trust policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Service": "firehose.amazonaws.com"
},
"Action": "sts:AssumeRole",
"Condition": {
"StringEquals": {
"sts:ExternalId": "11111111111"
}
}
}
]
}
I gave that role the following access:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:*"
],
"Resource": [
"arn:aws:s3:::b-bucket",
"arn:aws:s3:::b-bucket/*",
"arn:aws:s3:::b-account-logs",
"arn:aws:s3:::b-account-logs/*"
]
},
{
"Effect": "Allow",
"Action": [
"firehose:*"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": "redshift:*",
"Resource": "arn:aws:redshift:us-east-1:cluster:account-b-cluster*"
}
]
}
I also edited the access policy on the S3 buckets to give access to my Account A role:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::11111111111:role/AccountAXAccountBPolicy"
},
"Action": "s3:*",
"Resource": ["arn:aws:s3:::b-bucket","arn:aws:s3:::b-bucket/*"]
}
]
}
However, none of this works. When I try to create the the stream in Account A it does not list the buckets in Account B nor the redshift cluster. Is there any way to make this work?
John's answer is semi correct. I would recommend that the account owner of the Redshift Cluster creates the FireHose Stream. Creating through CLI requires you to supply the user name and password. Having the cluster owner create the stream and sharing IAM Role permissions on the stream is safer for security and in case of credential change. Additionally, you cannot create a stream that accesses a database outside of the region, so have the delivery application access the correct stream and region.
Read on to below to see how to create the cross account stream.
In my case both accounts are accessible to me and to lower the amount of changes and ease of monitoring I created the stream on Account A side.
The above permissions are right however, you cannot create a Firehose Stream from Account A to Account B through AWS Console. You need to do it through AWS Cli:
aws firehose create-delivery-stream --delivery-stream-name testFirehoseStreamToRedshift
--redshift-destination-configuration 'RoleARN="arn:aws:iam::11111111111:role/AccountAXAccountBRole", ClusterJDBCURL="jdbc:redshift://<cluster-url>:<cluster-port>/<>",
CopyCommand={DataTableName="<schema_name>.x_test",DataTableColumns="ID1,STRING_DATA1",CopyOptions="csv"},Username="<Cluster_User_name>",Password="<Cluster_Password>",S3Configuration={RoleARN="arn:aws:iam::11111111111:role/AccountAXAccountBRole",
BucketARN="arn:aws:s3:::b-bucket",Prefix="test/",CompressionFormat="UNCOMPRESSED"}'
You can test this by creating a test table on the other AWS Account:
create table test_schema.x_test
(
ID1 INT8 NOT NULL,
STRING_DATA1 VARCHAR(10) NOT NULL
)
distkey(ID1)
sortkey(ID1,STRING_DATA1);
You can send test data like this:
aws firehose put-record --delivery-stream-name testFirehoseStreamToRedshift --record '{"DATA":"1,\"ABCDEFGHIJ\""}'
This with the permissions configuration above should create the cross account access for you.
Documentation:
Create Stream - http://docs.aws.amazon.com/cli/latest/reference/firehose/create-delivery-stream.html
Put Record - http://docs.aws.amazon.com/cli/latest/reference/firehose/put-record.html
No.
Amazon Kinesis Firehose will only output to Amazon S3 buckets and Amazon Redshift clusters in the same region.
However, anything can send information to Kinesis Firehose by simply calling the appropriate endpoint. So, you could have applications in any AWS Account and in any Region (or anywhere on the Internet) send data to the Firehose and then have it stored in a bucket or cluster in the same region as the Firehose.

AWS IAM Access Management

I know that you can set up an IAM policy to restrict access to services. However, is it possible to set up a policy to allow access to a part of a service.
E.g. I am two EC2 instances. I need to create two users such that they have an access to the AWS console, but only to one EC2 instance each.
Yes you can do this with Resource-Level Permissions for EC2
The structure of the resource is stated in the documentation as follows:
arn:aws:[service]:[region]:[account]:resourceType/resourcePath
Here is how you would structure the IAM policies for each user:
User 1
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": "ec2:*",
"Resource": "arn:aws:ec2:us-east-1:123456789012:instance/InstanceIdOne"
}
]
}
User 2
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": "ec2:*",
"Resource": "arn:aws:ec2:us-east-1:123456789012:instance/InstanceIdTwo"
}
]
}
Policy without access to EC2:DescribeInstance will not work. You need to allow DescribeInstances access on all resources and manage additional access like modify, delete to specific instances depending on what the need is.
In short, allow all basic operations like Describe Tags, Instances, NetworkACLs, Images etc to all users and allow specific destructive actions like Modify and Delete to select user.
List of EC2 actions for your reference here
http://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_Operations.html
So you have 2 options-
Create one policy like below and attach the same policy to both users
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": "ec2:*Describe*",
"Resource":"*",
},
{
"Effect": "Allow",
"Action": [
"ec2:*Modify*",
"ec2:*Delete*"
],
"Principal": { "AWS": "arn:aws:iam::AWS-account-ID:user/**user-name-1**" },
"Resource": "arn:aws:ec2:us-east-1:AWS-account-ID:instance/**InstanceIdOne**"
},
{
"Effect": "Allow",
"Action": [
"ec2:*Modify*",
"ec2:*Delete*"
],
"Principal": { "AWS": "arn:aws:iam::AWS-account-ID:user/**user-name-2**" },
"Resource": "arn:aws:ec2:us-east-1:AWS-account-ID:instance/**InstanceIdTwo**"
}
]}
Create 2 different policies. Example for one below
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": "ec2:*Describe*",
"Resource":"*",
},
{
"Effect": "Allow",
"Action": [
"ec2:*Modify*",
"ec2:*Delete*"
],
"Principal": { "AWS": "arn:aws:iam::AWS-account-ID:user/**user-name-1**" },
"Resource": "arn:aws:ec2:us-east-1:AWS-account-ID:instance/**InstanceIdOne**"
}
]}