I'm working through some of the example Sagemaker notebooks, and I receive the following Access Denied error when trying to run the linear_time_series_forecast example:
ValueError: Error training linear-learner-2017-12-21-15-29-34-676: Failed Reason: ClientError: Data download failed:AccessDenied (403): Access Denied
I can manually download and upload from my S3 bucket using the AWS command line interface, but the Jupyter notebook fails.
Note that I am running the notebook through Sagemaker's notebook instance.
Looks like this question was also answered on the AWS Forums.
The IAM Role referenced by
role = get_execution_role()
needs to have a policy attached to it that grants S3:GetObject permission on the S3 bucket holding your training data.
Note that as of at least October 28, 2022, the linked forum post now re-directs to a page which states (among other things):
The thread you are trying to access has outdated guidance, hence we have archived it.
Please keep this in mind as it is possible that this answer no longer works, or that it at some point in the future will no longer work.
Related
I am trying to register a respository on AWS S3 to store ElasticSearch snapshots.
I am following guide and ran the very first command listed in the doc.
But I am getting the error Access Denied while executing that command.
The role that is being used to perform operations on S3 is the AmazonEKSNodeRole.
I have assigned the appropriate permissions to the role to perform operations on the S3 bucket.
Also, here is another doc which suggests to use kibana for ElasticSearch version > 7.2 but I am doing the same via cURL requests.
Below is trust Policy of the role through which I am making the request to register repository in the S3 bucket.
Also, below are the screenshots of the permissions of the trusting and trusted accounts respectively -
I need to run the command aws s3 cp <filename> <bucketname> from an EC2 RHEL instance to copy a file from the instance to an S3 bucket.
When I run this command, I receive this error: An error occurred (ExpiredToken) when calling the PutObject operation: The provided token has expired
I also found that this same error occurs when trying to run many other CLI commands from the instance.
I do not want to change my IAM role because the command was previously working perfectly fine and IAM policy changes must go through an approval process. I have double checked the IAM role the instance is assuming and it still contains the correct configuration for allowing PutObject on the correct resources.
What can I do to allow AWS CLI commands to work again in my instance?
AWS API tokens are time-sensitive, and VMs in the cloud tend to suffer from clock drift.
Check that time is accurate on the RHEL instance, and use ntp servers to make sure any drift is regularly corrected.
I am using the AWS Glue connector for BigQuery. My glue jobs were running fine in multiple AWS accounts but suddenly it started failing with the below response in all the accounts together:
Access Denied: Table common-infra-services:detailedcost.gcp_billing_export_resource_v1_01E8AD_3E792E_BB0E5D: Permission bigquery.tables.get denied on table common-infra-services:detailedcost.gcp_billing_export_resource_v1_01E8AD_3E792E_BB0E5D (or it may not exist).", "reason": "accessDenied"
Please review and let me know what could be the issue of this problem.
I am using the GCP IAM service account role to run queries using Glue to BigQuery with the following set of permissions:
bigquery.jobs.create
bigquery.tables.getData
bigquery.tables.list
And with these permissions, all jobs were running fine till yesterday.
Based on that error message I'd check if table common-infra-services:detailedcost.gcp_billing_export_resource_v1_01E8AD_3E792E_BB0E5D exists. If it does you might need to add permission bigquery.tables.get to your service account.
I am trying to get AWS Sagemaker to run locally. I found this jupyter notebook
https://gitlab.com/juliensimon/aim410/-/blob/master/local_training.ipynb
I logged into AWS via saml2aws and hence have valid credentials, entered my specific region as well as the Sagemaker Execution Role ARN and specify below the specific image I want to pull.
However when starting the .fit() i getthe following ClientError:
ClientError: An error occurred (InvalidClientTokenId) when calling the GetCallerIdentity operation: The security token included in the request is invalid.
Can someone give my a hint or suggestion how to solve this issue?
Thanks!
Try to verify your AWS credentials are setup properly, bypassing Boto3, by running a cell with something like:
!aws sagemaker list-endpoints
If this fails, then your AWS CLI credentials aren't setup correctly, or your saml2aws process, or your role has no SageMaker permissions.
I'm running Spark 2.4 on an EC2 instance. I am assuming an IAM role and setting the key/secret key/token in the sparkSession.sparkContext.hadoopConfiguration, along with the credentials provider as "org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider".
When I try to read a dataset from s3 (using s3a, which is also set in the hadoop config), I get an error that says
com.amazonaws.services.s3.model.AmazonS3Exception: Status Code: 403, AWS Service: Amazon S3, AWS Request ID: 7376FE009AD36330, AWS Error Code: null, AWS Error Message: Forbidden
read command:
val myData = sparkSession.read.parquet("s3a://myBucket/myKey")
I've repeatedly checked the S3 path and it's correct. My assumed IAM role has the right privileges on the S3 bucket. The only thing I can figure at this point is that spark has some sort of hidden credential chain ordering and even though I have set the credentials in the hadoop config, it is still grabbing credentials from somewhere else (my instance profile???). But I have no way to diagnose that.
Any help is appreciated. Happy to provide any more details.
spark-submit will pick up your env vars and set them as the fs.s3a access +secret + session key, overwriting any you've already set.
If you only want to use the IAM credentials, just set fs.s3a.aws.credentials.provider to com.amazonaws.auth.InstanceProfileCredentialsProvider; it'll be the only one used
Further Reading: Troubleshooting S3A