I am copying multiple files to s3 using one command
[ec2-user#ip-172-31-38-250 ~]$ ls *.rpm
1.rpm
2.rpm
3.rpm
how do I copy these 3 rpms in 1 aws cli command?
I tried
aws s3 cp *.rpm s3://mybucket1/rpms/
I got an error
Unknown options: 1.rpm, 2.rpm, 3.rpm,s3://mybucket1/rpms/
You can use filters:
aws s3 cp . s3://mybucket1/rpms/ --recursive --exclude "*" --include "*.rpm"
or with sync:
aws s3 sync . s3://mybucket1/rpms/ --exclude "*" --include "*.rpm"
Related
I would like to copy files matching a file name pattern from my machine to an AWS S3 bucket using AWS CLI. Using the standard unix file name wildcards does not work:
$ aws s3 cp *.csv s3://wesam-data/
Unknown options: file1.csv,file2.csv,file3.csv,s3://wesam-data/
I followed this SO answer addressing a similar problem that advises using the --exclude and --include filters as explained here as shown below without success.
$ aws s3 cp . s3://wesam-data/ --exclude "*" --include "*.csv"
Solution
$ aws s3 cp . s3://wesam-data/ --exclude "*" --include "*.csv" --recursive
Explanation
It turns out that I have to use the --recursive flag with the --include & --exclude flags since this is a multi-file operation.
The following commands are single file/object operations if no --recursive flag is provided.
cp
mv
rm
I want to delete all S3 keys starting with some prefix using AWS data Pipeline.
I am using AWS Shell Activity for this.
These are the argument
"scriptUri": "https://s3.amazonaws.com/my_s3_bucket/hive/removeExitingS3.sh",
"scriptArgument": "s3://my_s3_bucket/output/2017-03-19",
I want to delete all S3 keys starting with 2017-03-19 in output folder. What should be command to do this?
I have tried this command in .sh file
sudo yum -y upgrade aws-cli
aws s3 rm $1 --recursive
This is not working.
Sample files are
s3://my_s3_bucket/output/2017-03-19/1.txt
s3://my_s3_bucket/output/2017-03-19/2.txt
s3://my_s3_bucket/output/2017-03-19_3.txt
EDIT:
The date(2017-03-19) is dynamic and this is output of #{format(#scheduledStartTime,"YYYY-MM-dd")}. So effectively
"scriptArgument": "s3://my_s3_bucket/output/{format(#scheduledStartTime,"YYYY-MM-dd")}"
Try
aws s3 rm $1 --recursive --exclude "*" --include "2017-03-19*" --include "2017-03-19/*"
with
"scriptArgument": "s3://my_s3_bucket/output/"
EDIT:
As the date is a dynamic param, pass it as the second scriptArgument to the Shell command activity,
aws s3 rm $1 --recursive --exclude "*" --include "$2*" --include "$2/*"
I have 2 buckets, Bucket A and Bucket B. Bucket A contains Javascript files and bucket B contains a mix of javascript and other file types. I am trying to do a sync of only JS files from bucket A to B.
I am using the following:
aws s3 sync s3://bucket-a s3://bucket-b --delete --exclude "*"
--include "*.js"
I was assuming that this will leave bucket B with an exact copy of all of the js files. However the above command will start deleting all of the non js files in Bucket B.
When I run the following command:
aws s3 sync . s3://bucket-b --delete --exclude "*"
--include "*.js"
With the current directory containing a copy of bucket A, bucket B will have the same js files as bucket A and non js files will not be affected.
Why is this command functioning differently when syncing local to bucket compared to bucket to bucket?
EDIT: Added CLI input/output to reproduce problem.
Darrrens-MBP:testFolder darren$ aws --version
aws-cli/1.7.4 Python/2.7.9 Darwin/14.4.0
Darrrens-MBP:testFolder darren$ aws s3 ls s3://tmp-test-bucket-a
2015-09-01 11:40:44 2 test1.js
2015-09-01 11:40:43 2 test2.js
Darrrens-MBP:testFolder darren$ aws s3 ls s3://tmp-test-bucket-b
2015-09-01 11:39:32 2 test1.js
2015-09-01 11:39:34 2 test2.js
2015-09-01 11:39:34 3 test3.php
Darrrens-MBP:testFolder darren$ ls
test1.js test2.js
Darrrens-MBP:testFolder darren$ aws s3 sync . s3://tmp-test-bucket-b --delete --exclude "*" --include "*.js"
Darrrens-MBP:testFolder darren$ aws s3 sync s3://tmp-test-bucket-a s3://tmp-test-bucket-b --delete --exclude "*" --include "*.js"
delete: s3://tmp-test-bucket-b/test3.php
copy: s3://tmp-test-bucket-a/test2.js to s3://tmp-test-bucket-b/test2.js
copy: s3://tmp-test-bucket-a/test1.js to s3://tmp-test-bucket-b/test1.js
If I understand the problem correctly,--delete deletes files that exist in the destination but not in the source during sync, which are the ones you exclude.better use aws s3 cp instead of aws s3 sync.
Check AWS CLI reference
I'm trying to upload all files of type .flv to an S3 bucket using the AWS CLI from a Windows server 2008 command line.
I do this:
aws s3 sync . s3://MyBucket --exclude '*.png'
And it begins uploading .png files instead.
I'm trying to follow the documentation and it gives an example that reads:
Local directory contains 3 files:
MyFile1.txt
MyFile2.rtf
MyFile88.txt
'''
aws s3 sync . s3://MyBucket/MyFolder --exclude '*.txt'
upload: MyFile2.rtf to s3://MyBucket/MyFolder/MyFile2.rtf
So what am I doing wrong?
Use:
aws s3 sync . s3://MyBucket/ --exclude "*" --include "*.flv"
It excludes all files, then includes .flv files. The order of parameters is important.
You can also use:
aws s3 cp . s3://MyBucket/ --recursive --exclude "*" --include "*.flv"
The difference is that sync will not re-copy a file that already exists in the destination.
I have zipped files in an S3 bucket that I need to bring back to my EC2 instance. In the past, I moved the documents to S3 with the following command:
aws s3 cp /my/ec2/path/ s3://my/s3/path/ --exclude '*' --include '2014-01*’ —-recursive
To move files from January 2014 back to EC2, I have tried the following command:
aws s3 cp s3://my/s3/path/ //my/ec2/path/ --exclude '*' --include '2014-01*' --recursive
My understanding is that this command excludes all files but then includes all files with the prefix '2014-01'. I have confirmed that this is how the files I want start. I have also tried only one forward slash before mainstorage and including fewer files.
I have followed these two links from Amazon:
http://docs.aws.amazon.com/cli/latest/reference/s3/index.html
http://docs.aws.amazon.com/cli/latest/userguide/using-s3-commands.html
Figured it out. The key was to define the filepath in --include , i.e. --include '2014-1'. Correct command:
aws s3 cp s3://my/s3/path //my/ec2/path/ --exclude '*' --include '*2014-01*' --recursive