I have 2 AWS accounts: Account1, and Account2. I have some data stored in S3 in Account1, and I registered that data into an Athena table in Account1. Now, I would like to access the same Athena table from Account2. I realize that I could create an Athena table in Account2 to query data in Account1, but ideally I would like to keep all the tables under Account1.
Since May 2021 it is now possible to register a data catalog from a different account in Amazon Athena, see the User Guide.
Athena Query Engine v2 is required though and there are some other limitations.
As of today, it seems possible only by deploying in Account2 (the account you want to query from) a Lambda with the proper cross-account permissions to access the data catalog in Account1.
See this other answer and the associated post on AWS blog.
Related
I'm grasping at straws at this point. I have a Database in LakeFormation and I've given the quicksight service role access to it as well as underlying tables (and their S3 buckets). I've also verified that the quicksight location (us-east-1 N. Virginia) is the same as the s3 bucket locations for the underlying tables. After all of this, when I click on "new dataset" and select Athena, I still can't see my database under the AwsDataCatalog. If I go to athena directly, I can see my DBs + tables there. What else do I need to do?
Thanks!
I had the same issue and it seems QuickSight does not consider the Lake Formation permissions granted to an IAM role or user but rather looks for Lake Formation permissions granted to a specific QuickSight user (or to the QuickSight group to which the QuickSight user belongs).
For QuickSight, Lake Formation permissions thus need to be granted to QuickSight ARNs, e.g. arn:aws:quicksight:us-east-1:111122223333:user/default/YourUserName
I've just started to use AWS, I have a question.
I'm trying to send data from DynamoDB (Account A) to Athena (Account B).
What I would like to do is,
Transfer and update everyday the tables which are stored in Account A's DynamoDB TO ANOTHER ACCOUNT which is Account B.
Account B is going to execute queries in Athena with the tables received from Account A.
Do you have any solution ?
Thanks a lot !
Note that we can not transfer data to Athena, which does not hold data itself.
If the query data source is in the same account, Athena Federated Query would be useful, but it does not seem to be supported cross-account yet.
So for your case, export DynamoDB table to S3 like this aws blog.
Then query S3 data using Athena.
Amazon EventBridge can be used for everyday scheduled actions.
Can I use Informatica EDC instead of Glue catalog in AWS.
does AWS Athena tightly coupled with Glue catalog?
Did you check here: [https://docs.aws.amazon.com/athena/latest/ug/glue-upgrade.html?
Looks like you need to perform some AWS Glue upgrade, and also add policies so that Athena can pull catalog information. Also, FAQ is available here https://docs.aws.amazon.com/athena/latest/ug/glue-faq.html. I have not worked on this scenario yet, but working on Glue - Redshift.
In the FAQ, its mentioned as follows:
Why do I need to add AWS Glue policies to Athena users?
Before you upgrade, Athena manages the data catalog, so Athena actions must be allowed for your users to perform queries. After you upgrade to the AWS Glue Data Catalog, Athena actions no longer apply to accessing the AWS Glue Data Catalog, so AWS Glue actions must be allowed for your users. Remember, the managed policy for Athena has already been updated to allow the required AWS Glue actions, so no action is required if you use the managed policy.
What happens if I don’t allow AWS Glue policies for Athena users?
If you upgrade to the AWS Glue Data Catalog and don't update a user's customer-managed or inline IAM policies, Athena queries fail because the user won't be allowed to perform actions in AWS Glue. For the specific actions to allow, see Step 2 - Update Customer-Managed/Inline Policies Associated with Athena Users.
I have set up a reporting stack using data stored in S3, schema mapped by AWS Glue, queried by Amazon Athena, and visualized in Amazon QuickSight.
I gave QuickSight permissions to access the three aws-athena-query-results buckets I have (see below)
However, when I try to build reports based on my Athena table, it throws an error. I went back in and explicitly gave it access to the S3 bucket that holds my raw data, and now I have visualizations.
My question is whether or not this is how it should need to be set up. My assumption was that Athena has access to S3, and QuickSight has access to Athena and it's results, so it shouldn't need direct access to each S3 bucket storing raw data. It seems it would generate a lot of overhead each time there is a new S3 bucket to be reported on that you need to go grant Athena and QuickSight access.
From reading this page: Troubleshoot Athena Insufficient Permissions, it's unclear which buckets are required.
Yes, at the moment, QuickSight needs to be granted explicit access to both Athena and the underlying buckets that Athena accesses. I got this answer from discussion with Amazon so, unfortunately, I don't have source to link.
Is it possible to directly access AWS Glue Data Catalog of Account B via the Athena interface of Account A?
I was just trying to resolve this same issue in my own setup, but then stumbled across this bummer (the last bullet under Cross-Account Access Limitations on this page):
Cross-account access to the Data Catalog is not supported when using an AWS Glue crawler, Amazon Athena, or Amazon Redshift.
So it sounds like even with the cross-account access that is possible today, they won't naturally replicate through those services (including the asked about Athena).
That said, I was able to set up cross-account access to the AWS Glue Data Catalog in a way that allowed me to use Account A to pull all relevant info about Data Catalog objects from Account B. I can update my answer to incorporate how far I got, if you want, but a hacky method that might solve this question would be to set up the cross-account access that is possible today then run a recurring Lambda function that replicates over all the relevant metadata in the Data Catalog from Account B to Account A so users in Account A can view that within Account A's AWS Glue Data Catalog. I'm not sure whether Athena specifically would work in that setup, as I know it requires PutObject access when it queries data in S3 (which could be solved via the appropriate S3 bucket policies, but that'd be another cross-account permissions thing to manage).
Let me know whether you'd like to see those details on what cross-account stuff I was able to get working.
AWS has started supporting this using Lambda, please follow below link
https://aws.amazon.com/blogs/big-data/cross-account-aws-glue-data-catalog-access-with-amazon-athena/
Since May 2021 it is now possible to register a data catalog from a different account in Amazon Athena, see the User Guide.
Athena Query Engine v2 is required though and there are some other limitations.