I am attempting to use wasabi but the aws s3 cli seams to ignore --enpoint-url when ever I specify s3://my-wasabi-bucket
For Example
aws s3 ls --endpoint-url=https://s3.wasabisys.com --profile wasabi
Will list my wasabi buckets, but when I do
aws s3 ls --endpoint-url=https://s3.wasabisys.com --profile wasabi s3://my-bucket
Where my-bucket is a bucket that was in the list from above, I get
Could not connect to the endpoint URL: "https://s3.us-central-1.amazonaws.com/my-bucket?list-type=2&prefix=&delimiter=%2F&encoding-type=url"
Looks like I was using the wrong endpoint url, my bucket is hosted in wasabi's central-1 so I needed to use --endpoint-url=https://s3.us-central-1.wasabisys.com
Related
So I have created a VPC Interface endpoint for S3.
I have tried running the below command:
aws s3api getobject --bucket <bucketname> --key <file_key> --endpoint-url https://<bucketname>.bucket.s3.
<region>.amazonaws.com
and it works fine.
Then I tried to update my /etc/hosts file with the 2 subnet IPs of my VPC like below:
<subnet-ip-1> s3.<region>.amazonaws.com
<subnet-ip-2> s3.<region>.amazonaws.com
And then I do a aws s3 ls. Works fine- I can see all my buckets
Then I run the below command (1st command excluding endpoint-url):
aws s3api getobject --bucket <bucketname> --key <file_key> <local_file_name>
Timeout error.
I try -
aws s3 cp s3://<bucketname>/<filekey> <local_file_name>
Timeout again..
Then I updated my /etc/hosts file like below..
<subnet-ip-1> <bucketname>.s3.<region>.amazonaws.com
<subnet-ip-2> <bucketname>.s3.<region>.amazonaws.com
And I ran the command..
aws s3 cp s3://<bucketname>/<filekey> <local_file_name>
It works!!
But I cant add bucket names like this because I have almost 100 such buckets. And I cannot use endpoint url in my s3 commands because Im setting this up for greengrass. And greengrass wont allow me to configure a s3 endpoint. So it has to be like greengrass will call its bucket commands at s3..amazonaws.com but they should be redirected to my VPC endpoint by default.
How do I set that up?
Note: I do not and cannot hv internet access.
I have the ff s3 bucket, see image below:
I know how to delete them on the page but how can i delete them using aws cli?
I tried using
aws s3api delete-objects --bucket
elasticbeanstalk-ap-southeast-1-613285248276
but it wont work.
You can use 'rb' option
E.g. aws s3 rb s3://elasticbeanstalk-ap-southeast-1-613285248276
For more details click here
If the bucket is not empty then use the --force flag
The following commands deletes the Bucket from S3
aws s3api delete-bucket --bucket "your-bucket-name" --region "your-region".
I am deleting S3 bucket from following command without any extra hustle of emptying it first
aws s3 rb s3://<BUCKATE-NAME> --force
for example
aws s3 rb s3://alok.guha.myversioningbucket --force
PS : Make sure you have programmatic access with policy "AmazonS3FullAccess"
I am trying to copy a file from S3 to an Ec2 instance, here is the strange behavior
Following command runs perfectly fine and show me the contents of s3, that I want to access
$aws s3 ls
2016-05-05 07:40:57 folder1
2016-05-07 15:04:42 my-folder
then I issue following command (also successful)
$ aws s3 ls s3://my-folder
2016-05-07 16:44:50 6007 myfile.txt
but when I try to copy this file, I recive an error as follows
$aws s3 cp s3://my-folder/myfile.txt ./
A region must be specified --region or specifying the region in a
configuration file or as an environment variable. Alternately, an
endpoint can be specified with --endpoint-url
I simply want to copy txt file from s3 to ec2 instance.
At least modify the above command to copy the contents. I am not sure about region as If I visit S3 from web it says
"S3 does not require region selection"
What is happening on the earth?
Most likely something is not working right, you should not be able to list the bucket if your regions is not setup as default in the aws configure.
Therefore from my experience with S3 if this works:
aws s3 ls s3://my-folder
then this should work as well:
aws s3 cp s3://my-folder/myfile.txt ./
However if it's asking you for region, then you need to provide it.
Try this to get the bucket region:
aws s3api get-bucket-location --bucket BUCKET
And then this to copy the file:
aws s3 cp --region <your_buckets_region> s3://my-folder/myfile.txt ./
If I visit S3 from web it says
"S3 does not require region selection"
S3 and bucket regions can be very confusing especially with that message. As it is the most misleading information ever IMO when it comes to s3 regions. Every bucket has got specific region (default is us-east-1) unless you have enabled cross-region replication.
You can choose a region to optimize latency, minimize costs, or
address regulatory requirements. Objects stored in a region never
leave that region unless you explicitly transfer them to another
region. For more information about regions, see Accessing a Bucket: in
the Amazon Simple Storage Service Developer Guide.
How about
aws s3 cp s3://my-folder/myfile.txt .
# or
aws s3 cp s3://my-folder/myfile.txt myfile.txt
I suspect the problem is something to do with the local path parser.
aws cli s3 fileformat parser
It is kinda strange because aws cli read the credential and region config.
The fix is specifying the region, below explains how to get the bucket region if you cant get it from the cli.
aws s3 cp s3://xxxxyyyyy/2008-Nissan-Sentra.pdf myfile.pdf --region us-west-2
I need to get the contents of one S3 bucket into another S3 bucket.
The buckets are in two different accounts.
I was told not to create a policy to allow access to the destination bucket from the origin bucket.
Using the AWS CLI how can I download all the contents of the origin bucket, and then upload the contents to the destination bucket?
To copy locally
aws s3 sync s3://origin /local/path
To copy to destination bucket:
aws s3 sync /local/path s3://destination
The aws cli allows you to configure named profiles which lets you use a different set of credentials for each individual cli command. This will be helpful because your buckets are in different accounts.
To create your named profiles you'll need to make sure you already have IAM users in each of your accounts and each user will need a set of access keys. Create your two named profiles like this.
aws configure --profile profile1
aws configure --profile profile2
Each of those commands will ask you for your access keys and a default region to use. Once you have your two profiles, use the aws cli like this.
aws s3 cp s3://origin /local/path --recursive --profile profile1
aws s3 cp /local/path s3://destination --recursive --profile profile2
Notice that you can use the --profile parameter to tell the cli which set of credentials to use for each command.
I need to download files recursively from a s3 bucket. The s3 bucket lets anonymous access.
How to list files and download them without providing AWS Access Key using an anonymous user?
My command is:
aws s3 cp s3://anonymous#big-data-benchmark/pavlo/text/tiny/rankings/uservisits uservisit --region us-east --recursive
The aws compains that:
Unable to locate credentials. You can configure credentials by running "aws configure"
You can use no-sign-request option
aws s3 cp s3://anonymous#big-data-benchmark/pavlo/text/tiny/rankings/uservisits uservisit --region us-east --recursive --no-sign-request
you probably have to provide an access keys and secret key, even if you're doing anonymous access. don't see an option for anonymous for the AWS cli.
another way to do this, it to hit the http endpoint and grab the files that way.
In your case: http://big-data-benchmark.s3.amazonaws.com
You will get and XML listing all the keys in the bucket. You can extract the keys and issues requests for each. Not the fastest thing out there but it will get the job done.
For example: http://big-data-benchmark.s3.amazonaws.com/pavlo/sequence-snappy/5nodes/crawl/000741_0
for getting the files curl should be enough. for parsing the xml depending on what you like you can go as lo-level as sed and as high-level as a proper language.
hope this helps.