I'm new to AWS and I'm trying to download a bunch of files from my S3 bucket to my local machine using aws s3 sync as described in http://docs.aws.amazon.com/cli/latest/reference/s3/sync.html.
I used the following command:
aws s3 sync s3://outputbucket/files/ .
I got the following error:
A client error (AccessDenied) occurred when calling the ListObjects operation: Access Denied
Completed 1 part(s) with ... file(s) remaining
Even though I have configured my access key ID & secret access key as described in http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-set-up.html
Where might the problem be?
Assuming that you are an Administrator and/or you have set your credentials properly, it is possible that you are using an old AWS CLI.
I encountered this while using the packaged AWS CLI with Ubuntu 14.04.
The solution that worked for me is to remove the AWS CLI prepackaged with Ubuntu, and download it from python-pip instead:
sudo apt-get remove awscli
sudo apt-get install python-pip
sudo pip install awscli
Many thanks to this link:
https://forums.aws.amazon.com/thread.jspa?threadID=173124
To perform a file sync, two sets of permissions are required:
ListObjects to obtain a list of files to copy
GetObjects to access the objects
If you are using your "root" user that comes with your AWS account, you will automatically have these permissions.
If you are using a user created within Identity and Access Management (IAM), you will need to assign these permissions to the User. The easiest way is to assign the AmazonS3FullAccess policy, which gives access to all S3 functions.
In my case the credentials stored in ~/.aws/config were being clobbered by a competing profile sourced in ~/.zshrc. Run env | grep AWS to check.
Related
I am trying to copy all the data from an AWS S3 bucket to a GCS bucket. Acc. to this answer rsync command should have been able to do that. But I am receiving the following error when trying to do that
Caught non-retryable exception while listing s3://my-s3-source/: AccessDeniedException: 403 InvalidAccessKeyId
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>InvalidAccessKeyId</Code><Message>The AWS Access Key Id you provided does not exist in our records.</Message><AWSAccessKeyId>{REDACTED}</AWSAccessKeyId><RequestId>{REDACTED}</RequestId><HostId>{REDACTED}</HostId></Error>
CommandException: Caught non-retryable exception - aborting rsync
This is the command I am trying to run
gsutil -m rsync -r s3://my-s3-source gs://my-gcs-destination
I have the AWS CLI installed which is working fine with the same AccessKeyId and listing buckets as well as objects in the bucket.
Any idea what am I doing wrong here?
gsutil can work with both Google Storage and S3.
gsutil rsync -d -r s3://my-aws-bucket gs://example-bucket
You just need to configure it with both - Google and your AWS S3 credentials. For GCP you need to add the Amazon S3 credentials to ~/.aws/credentials or you can also store your AWS credentials in the .boto configuration file for gsutil. However, when you're accessing an Amazon S3 bucket with gsutil, the Boto library uses your ~/.aws/credentials file to override other credentials, such as any that are stored in ~/.boto.
=== 1st update ===
Also make sure you have to make sure you have the correct IAM permissions on the GCP side and the correct AWS IAM credentials. Also depending if you have a prior version of Migrate for Compute Engine (formerly Velostrata) use this documentation and make sure you set up the VPN, IAM credentials and AWS network. If you are using the current version (5.0), use the following documentation to check everything is configured correctly.
I have installed the latest versions of the aws-cli-2 and docker, as well as ran "aws configure" and entered my access key and secret key. I have also verified the aws.config is correct and showing the right region and output format. My credentials in AWS are admin. I keep getting the following error:
'''Unable to locate credentials. You can configure credentials by running "aws configure".
Error: Cannot perform an interactive login from a non TTY device'''
Even though I have already ran 'aws configure.' I am running the commands prefixed with 'sudo' as well. Any thoughts?! Thank you for your time!
The aws configure command was being run as the local user, whereas the ecr command was being run as sudo.
If you run commands as sudo it will not have access to your local users config, it will instead default to the root users.
Instead ensure all commands are run as the same user.
If you want to use the aws credentials file from the default location you can also specify the location via the AWS_CONFIG_FILE environment variable.
Was able to set up a pull from an S3 bucket on a Mac seamlessly, but have been struggling with an identical process on a PC (Windows). Here is what I have done -- any help along the way would be much appreciated.
Installed awscli using pip
Ran aws configure in the command prompt and inputed the proper access key id and secret access key.
Ran the s3 code: G:\>aws s3 cp --recursive s3://url-index-given/ . (where the url was replaced with url-index-given for example purposes).
And got this error:
fatal error: Could not connect to the endpoint URL: "https://url-index-given.s3.None.amazonaws.com/?list-type=2&prefix=&encoding-type=url"
I have tried uninstalling the awscli package and followed this process recommended by Amazon without any errors.
The error indicates have you have given an invalid value for Region when using aws configure. (See the None in the URL? That is where the Region normally goes.)
You should run aws configure again and give it a valid region (eg us-west-2).
I'm currently trying to automate the Cognito User Pool creation process via bash scripts on AWS-CLI. However, following the steps from the AWS console, I'm trying to reproduce the same steps via the CLI. I like to know which commands I should be looking at and in what sequence? The AWS docs don't really say much and the commands sometimes tend to be confusing.
Any ideas will be greatly appreciated.
Cheers!
Nyah
First install aws cli using following command
sudo pip install awscli
Configure AWS credentials, Run below commonond, system will ask following input AWS Access Key ID, AWS Secret Access Key, Default region name, Default output format
sudo aws configure
Create user pool
sudo aws cognito-idp create-user-pool --pool-name MyUserPool
You can install first awscli
sudo pip install awscli
configure aws cli with your private key and access key.
to run configure aws-cli run the command:
aws configure
For all the details of Cognito, you can find available command for it over here: https://docs.aws.amazon.com/cli/latest/reference/cognito-idp/index.html
Create the user pool in cognito:
aws Cognito-idp create-user-pool --pool-name <Whatever name you want to add>
Update the user pool incognito
aws cognito-idp update-user-pool --user-pool-id <value>
You can also refer this bash script :
https://github.com/awslabs/aws-cognito-angular-quickstart/blob/master/aws/createResources.sh
I'm attempting a simple rsync (or cp) from AWS S3 to GCP Storage.
For e.g.
gsutil rsync -d -r -n s3://mycustomer-src gs://mycustomer-target
I get an error message as below when attempting this on a VM on GCP.
Note that if I install aws cli on the VM, then I can access / browse AWS S3 contents just fine. The AWS credentials are stored in ~/.aws/credentials file.
Building synchronization state...
Caught non-retryable exception while listing s3://musiclab-etl-dev/: AccessDeniedException: 403 InvalidAccessKeyId
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>InvalidAccessKeyId</Code><Message>The AWS Access Key Id you provided does not exist in our records.</Message><AWSAccessKeyId>ASIAJ3XGCQ7RGZYPD5UA</AWSAccessKeyId><RequestId>CE8919045C68DEC4</RequestId><HostId>i7oMBM61US3FyePJka8O+rjoHSo1rIZbRGnVZvIGkjEVPh6lXdbp03pZOtJ68F3pPdAAW1UvF5s=</HostId></Error>
CommandException: Caught non-retryable exception - aborting rsync
Is this a bug in gsutil ? Any workarounds or tips appreciated.
NOTE - The client's AWS account is setup for federated access and requires using AWS keys as obtained using a script similar to this-
https://aws.amazon.com/blogs/security/how-to-implement-a-general-solution-for-federated-apicli-access-using-saml-2-0/
The AWS keys are set to expire when the session token expires.
If I use a different AWS account (no federation) with typical AWS keys (non-expiring), the rsync (or cp) works fine.
It appears that gsutil still uses the legacy AWS_SECURITY_TOKEN instead of AWS_SESSION_TOKEN. If your script doesn't set it up automatically, you can do it manually like this:
export AWS_SECURITY_TOKEN=$AWS_SESSION_TOKEN
After this you should be able to use gsutil normally.