I am trying to create a crawler in AWS Glue, but it gives error: {"service":"AWSGlue","statusCode":400,"errorCode":"AccessDeniedException","requestId":"<requestId>","errorMessage":"Account <accountId> is denied access.","type":"AwsServiceError"}.
This is what I've done so far:
Create a database in AWS Glue
Add tables in the database using a crawler
Name the crawler
Choose Amazon S3 as the data store and specified a path to a csv file inside a bucket in my account
Choose an existing IAM role I've created before
Choose a database I've created before
Press finish.
When I press finish, the above error is occurred.
I have grant AdministratorAccess both to IAM user and role used to create the crawler, so I assume there is no lack of permission issues. The bucket used is not encrypted and located in the same region as the AWS Glue.
I also have tried to create another database and specified a path to a different csv file but it is not solved the problem.
Any help would be very appreciated. Thanks.
I have contacted the owner (the root user) of this account and the owner asked for help to AWS Premium Support. The AWS Premium Support told us that all the required permissions to create AWS Glue Crawler are already provided and there is no SCPs attached to the account. After waiting around 7-working-day, finally I can create AWS Glue Crawler without any errors.
Unfortunately, I don't have any further information on how the AWS Premium Support solve the issue. For those of you who encounter similar errors like me, just try to contact the owner of the account, because most likely the issue is out of your control. Hope this helps in the future. Thanks.
Related
I am getting this message when I try to create a crawler on AWS Glue:
{"service":"AWSGlue","statusCode":400,"errorCode":"AccessDeniedException","requestId":"RequestIDNumber","errorMessage":"Account <AccountID> is denied access.","type":"AwsServiceError"}
already attached all those policies below to the IAM
All my Policies here
Already setup permissions to the AWS lake formation for the role too
Already created a custom policy kms to it too
And I am stucked, I cannot create a crawler!
I am in the root account, actually there`s no other account just the root, It's a super new account I created in AWS so I don't know what to do to be able to create this simple crawler
My ideia its to use it with dynamodb as data source
The message says there is something wrong with my account permission not with a role
Someone has an idea?
Thank you so much
Not sure the reason, but loads of people have been having this issue. You can submit a ticket through AWS account support. I actually submitted two, including one through the unpaid support version and one through my paid account. They answered both tickets.
Basically, just tell them you need access to Glue and they should alter whatever it is to give you access. Sorry it's not a better answer, but I found no other useful information anywhere.
I propose to try using a separate user and attaching the policy you created to it.
How to get the usage metrics of S3.
Currently, IAM users are uploading/downloading files from the S3 bucket. Each IAM user has a separate folder. How to access, how many GB of data were transferred/downloaded from S3?.
You can't. There is no such metric provided by AWS. You have to develop a custom solution for that. If you, for example, have CloudTrial trial enabled for S3 operations, you can parse the past logs and based on them build up a report on who downloaded/upload what. Once you know what objects were uploaded/downloaded by a give IAM user, you can add up their sizes.
I was trying to edit schema from AWS Glue console.
I am getting following error while trying to save my changes.
{"service":"AWSGlue","statusCode":400,"errorCode":"AccessDeniedException","requestId":"644bfcb2-75a8-456b-b17a-e22e829345d2","errorMessage":"Insufficient Lake Formation permission(s): Required Alter on recidivism_clustering_model_output_csv","type":"AwsServiceError"}
I have provided necessary access to the s3 buckets and tables in glue using Lakeformation.
Can anyone tell me what the best way to edit schema via glue?
And wny am i getting that error?
I had to manually add my user as a database owner to the database resource. Not sure why this sometimes happens. My role is a datalake administrator but for some reason that did not work if I added my role as a Database admin then I could alter the database.
There is probably something else going on but I could not find it.
My IAM users can't see the Athena tables I've created a long time ago using the root account.
Their group has the following permissions:
AmazonS3FullAccess
AmazonAthenaFullAccess
They only see the sampledb databases, which is unfortunate, because they need the one we actually use. The documentation is not clear on how to make the databases accessible to everyone. How do I achieve that?
Your permissions are correct.
Athena's context is not currently shared across regions. Ensure that the users are viewing Athena from the same region as the root account. When they login to AWS, they may be initially placed in another region.
You need Glue permissions, Glue is the service in charge of manage Databases and Tables in AWS
I am new to Amazon EMR and Hadoop in general. I am currently trying to set up a Pig job on an EMR cluster and to import and export data from S3. I have set up a bucket in s3 with my data named "datastackexchange". In an attempt to begin to copy the data to Pig, I have used the following command:
ls s3://datastackexchange
And I am met with the following error message:
AWS Access Key ID and Secret Access Key must be specified as the username or password (respectively) of a s3 URL, or by setting the fs.s3.awsAccessKeyId or fs.s3.awsSecretAccessKey properties (respectively).
I presume I am missing some critical steps (presumably involving setting up the access keys). As I am very new to EMR, could someone please explain what I need to do to get rid of this error and allow me to use my S3 data in EMR?
Any help is greatly appreciated - thank you.
As you correctly observed, your EMR instances do not have the privileges to access the S3 data. There are many ways to specify the AWS credentials to access your S3 data, but the correct way is to create IAM role(s) for accessing your S3 data.
Configure IAM Roles for Amazon EMR explains the steps involved.