I have set up a reporting stack using data stored in S3, schema mapped by AWS Glue, queried by Amazon Athena, and visualized in Amazon QuickSight.
I gave QuickSight permissions to access the three aws-athena-query-results buckets I have (see below)
However, when I try to build reports based on my Athena table, it throws an error. I went back in and explicitly gave it access to the S3 bucket that holds my raw data, and now I have visualizations.
My question is whether or not this is how it should need to be set up. My assumption was that Athena has access to S3, and QuickSight has access to Athena and it's results, so it shouldn't need direct access to each S3 bucket storing raw data. It seems it would generate a lot of overhead each time there is a new S3 bucket to be reported on that you need to go grant Athena and QuickSight access.
From reading this page: Troubleshoot Athena Insufficient Permissions, it's unclear which buckets are required.
Yes, at the moment, QuickSight needs to be granted explicit access to both Athena and the underlying buckets that Athena accesses. I got this answer from discussion with Amazon so, unfortunately, I don't have source to link.
Related
No data is getting printed when I run the table in redshift which has huge data and it is working fine in Athena but not in redshift, can anybody help me with ways to access the data.
There could be multiple causes of this issue:
Role you have created external table does not have access to S3 bucket.
If the role has access to S3 bucket that also needs to be associated with the redshift cluster.
There could be a S3-based policy restricting the getObject access with bucket name or bucket_name_* in the resources.
Check if there is endpoint created in VPC for S3 bucket and check any policy restricting the S3 bucket access..
For testing, create separate bucket load data into it, and try creating an external table using data. catalog and check if it is accessible.
From reading the AWS manuals,
it is not clear to me whether one can enforce access to S3 data for an IAM user via a AWS View only.
I am going to try it next week, but some up-front research is what I am doing first so as to save time.
No, you cannot, via AWS Glue Data Catalog. Need Athena or Redshift Spectrum for such an approach.
There is a S3 bucket that has "Bucket and objects not public" access. Within Athena, there is table that is pulling data from the S3 bucket successfully. However, I cannot pull the data from Athena to Quicksight. My conclusion is that it is because the S3 bucket has "Bucket and objects not public" access. Is this correct?
Is it the case that Athena has some kind of special access to the S3 bucket, but Quicksight doesn't?
Here is a crude illustration of the issue:
I'm a total beginner when it comes to AWS so I apologise for missing any information.
Thanks in advance.
To verify that you can connect Amazon QuickSight to Athena, check the following settings:
AWS resource permissions inside of Amazon QuickSight
AWS IAM policies
S3 bucket location
Query results location
If S3 bucket location and Query results location are correct, you might have issues with Amazon QuickSight resource permissions. You have to make sure that Amazon QuickSight can access the S3 buckets used by Athena:
Choose your profile name. Choose Manage QuickSight, then choose Security & permissions.
Choose Add or remove.
Locate Athena, select it to enable Athena. (choose Connect both)
Choose the buckets that you want to access and click Select.
Choose Update.
I am trying to unload my organizations redshift data to a vendors s3 bucket. They have provided me with Access Keys and Secret Access Keys but Im getting a 403 Access Denied Error. I want to make sure its not an issue with the credentials they sent me and am reaching out. Is this even possible?
Yes, That's possible. Here are the steps.
unload the redshift data to bucket associated to the current account.
unload ('select * from table')
to 's3://mybucket/table'
Mybucket should have replication configured to the vendor s3 bucket. use the below s3 replication to configure the replication
Is it possible to directly access AWS Glue Data Catalog of Account B via the Athena interface of Account A?
I was just trying to resolve this same issue in my own setup, but then stumbled across this bummer (the last bullet under Cross-Account Access Limitations on this page):
Cross-account access to the Data Catalog is not supported when using an AWS Glue crawler, Amazon Athena, or Amazon Redshift.
So it sounds like even with the cross-account access that is possible today, they won't naturally replicate through those services (including the asked about Athena).
That said, I was able to set up cross-account access to the AWS Glue Data Catalog in a way that allowed me to use Account A to pull all relevant info about Data Catalog objects from Account B. I can update my answer to incorporate how far I got, if you want, but a hacky method that might solve this question would be to set up the cross-account access that is possible today then run a recurring Lambda function that replicates over all the relevant metadata in the Data Catalog from Account B to Account A so users in Account A can view that within Account A's AWS Glue Data Catalog. I'm not sure whether Athena specifically would work in that setup, as I know it requires PutObject access when it queries data in S3 (which could be solved via the appropriate S3 bucket policies, but that'd be another cross-account permissions thing to manage).
Let me know whether you'd like to see those details on what cross-account stuff I was able to get working.
AWS has started supporting this using Lambda, please follow below link
https://aws.amazon.com/blogs/big-data/cross-account-aws-glue-data-catalog-access-with-amazon-athena/
Since May 2021 it is now possible to register a data catalog from a different account in Amazon Athena, see the User Guide.
Athena Query Engine v2 is required though and there are some other limitations.