I am using Athena using lambda, and just noticed that result of query execution is success but there is nothing in S3 bucket. S3 bucket is public and has permission to put objects, so I can't find any problem with S3 bucket. I wonder if there is solution for this.
Providing S3 write permission to lambda execution role may help.
Related
My use-case is to push data from one AWS account S3 bucket to another AWS account S3 bucket continuously. A cross account push.
I’m using lambda to do this job.
Assume in AWS account A, data is frequently landed from some source into S3 bucket. I need to create an S3 trigger which will invoke Lambda function in AWS account A and push account A S3 bucket data to another S3 bucket in AWS account B.
Is this possible?
Yes!
Firstly, if the buckets are in different regions, you could use Cross-Region Replication and Amazon S3 will do it all for you automatically.
If not, then your proposed plan looks fine. It would involve:
An Amazon S3 Event to trigger the Lambda function whenever a new object is created
The Lambda function receives the Bucket Name and Key of the new object
The Lambda function should then call CopyObject() to copy the object to the other bucket (in the other account)
The most important element is to give permissions to the Lambda function running in Account-A to write to the bucket in Account-B. This can be done by:
Creating an IAM Role (Role-A) in Account-A that is used by the Lambda function
Adding a Bucket Policy to the bucket in Account-B that permits PutObject from Role-A (by specifying the ARN of Role-A)
Can I create a database and table in Athena service within my account to access S3 data in another account?
I went over the below link and I assume as per this documentation both Amazon Athena and S3 bucket have to be in the same account and access is provided to the user in another account.
https://console.aws.amazon.com/athena/home?force®ion=us-east-1#query
From Access Control Policies - Amazon Athena:
To run queries in Athena, you must have the appropriate permissions for:
The Athena actions.
The Amazon S3 locations where the underlying data is stored that you are going to query in Athena.
...
So, it seems that the IAM User who is executing the Athena query requires access to the Amazon S3 location.
This could be done by adding a Bucket Policy to the S3 bucket in the other account that permits the IAM User access to the bucket.
To explain better:
Account-A with IAM-User-A and AWS Athena
Account-B with Bucket-B that has a Bucket Policy granting access to IAM-User-A
This answer deals with the additional information that:
A Lambda function in Account-A must be able to create a table in Amazon Athena in Account-B
I haven't tested it, but I think you will require:
Role-A in Account-A for the Lambda function that:
Permits AssumeRole on Role-B
Role-B in Account-B that:
Permits access to Amazon Athena and the source bucket in Amazon S3
Trusts Role-A
The Lambda function will run with Role-A. It will then use credentials from Role-A to call AssumeRole on Role-B. This will return a new set of credentials that can be used to call Amazon Athena in Account-B.
I have manually created a Glue table with S3 bucker as the source.
The S3 bucket has a bucket policy defined to allow access only from
root
my user_id
or a role defined for Glue
Now when a different user who has AWSGlueConsoleFullAccess tries to access the table from Glue console he gets access denied although Glue has service access to the S3 bucket.
Request help in understanding this behavior.
Thanks
Can you please look into the policy details of role "AWSGlueConsoleFullAccess"? Most probably its expecting the S3 bucket will have certain prefix e.g. "aws-glue-*". In that case either update your policy or rename your bucket to have aws-glue- prefix.
"Resource": [
"arn:aws:s3:::aws-glue-*"
Is it possible to get the ARN of an S3 bucket via the AWS command line?
I have looked through the documentation for aws s3api ... and aws s3 ... and have not found a way to do this.
It's always arn:PARTITION:s3:::NAME-OF-YOUR-BUCKET. If you know the name of the bucket and in which partition it's located, you know the ARN. No need to 'get' it from anywhere.
The PARTITION will be aws, aws-us-gov, or aws-cndepending on whether you're in general AWS, GovCloud, or China resepectively.
You can also select your S3 bucket ARN by selecting it using the tick mark at the s3 management console, which will pop up a Side bar. where there is a provision to copy your S3 bucket ARN.S3 management console with bucket ARN
aws articles spell out the arn format but never say go here to see it. Highlighting my s3 bucket and seeing that Copy Bucket ARN kept me sane for a few more hours.
Is it possible to configure S3 bucket to run a Lambda function created in a different account? Basically what I'm trying to accomplish is that when new items are added to S3 bucket I want to run a lambda function in another account
You can do this by providing the full Lambda Function ARN to your S3 bucket. For example inside your bucket settings in the AWS Console:
This article will help you configure the correct IAM for cross account invocation. Also take a look at the AWS Lambda Permissions Model. Note that as far as I know the bucket and the Lambda function have to be in the same region!