I have 2 buckets, Bucket A and Bucket B. Bucket A contains Javascript files and bucket B contains a mix of javascript and other file types. I am trying to do a sync of only JS files from bucket A to B.
I am using the following:
aws s3 sync s3://bucket-a s3://bucket-b --delete --exclude "*"
--include "*.js"
I was assuming that this will leave bucket B with an exact copy of all of the js files. However the above command will start deleting all of the non js files in Bucket B.
When I run the following command:
aws s3 sync . s3://bucket-b --delete --exclude "*"
--include "*.js"
With the current directory containing a copy of bucket A, bucket B will have the same js files as bucket A and non js files will not be affected.
Why is this command functioning differently when syncing local to bucket compared to bucket to bucket?
EDIT: Added CLI input/output to reproduce problem.
Darrrens-MBP:testFolder darren$ aws --version
aws-cli/1.7.4 Python/2.7.9 Darwin/14.4.0
Darrrens-MBP:testFolder darren$ aws s3 ls s3://tmp-test-bucket-a
2015-09-01 11:40:44 2 test1.js
2015-09-01 11:40:43 2 test2.js
Darrrens-MBP:testFolder darren$ aws s3 ls s3://tmp-test-bucket-b
2015-09-01 11:39:32 2 test1.js
2015-09-01 11:39:34 2 test2.js
2015-09-01 11:39:34 3 test3.php
Darrrens-MBP:testFolder darren$ ls
test1.js test2.js
Darrrens-MBP:testFolder darren$ aws s3 sync . s3://tmp-test-bucket-b --delete --exclude "*" --include "*.js"
Darrrens-MBP:testFolder darren$ aws s3 sync s3://tmp-test-bucket-a s3://tmp-test-bucket-b --delete --exclude "*" --include "*.js"
delete: s3://tmp-test-bucket-b/test3.php
copy: s3://tmp-test-bucket-a/test2.js to s3://tmp-test-bucket-b/test2.js
copy: s3://tmp-test-bucket-a/test1.js to s3://tmp-test-bucket-b/test1.js
If I understand the problem correctly,--delete deletes files that exist in the destination but not in the source during sync, which are the ones you exclude.better use aws s3 cp instead of aws s3 sync.
Check AWS CLI reference
Related
I setup an SFTP server on Debian EC2 instance. I setup a cron job using aws s3 sync <source dir on EC2> <destination S3 bucket>. I issue is that my EC2 will get full as uploads come in.
Once a file is uploaded to EC2 instance, I want the file to moved to S3 bucket. sync command just copies it and doesn't delete from source. How can I accomplish this?
The aws s3 mv command actually performs a CopyObject() and a Delete.
To move a whole directory, use:
aws s3 mv --recursive localdir s3://bucket-name/
If you want to move the 'contents' of the directory, try:
aws s3 mv --recursive localdir s3://bucket-name/ --exclude "*" --include "localdir/*"
See: mv — AWS Command Reference
I am copying multiple files to s3 using one command
[ec2-user#ip-172-31-38-250 ~]$ ls *.rpm
1.rpm
2.rpm
3.rpm
how do I copy these 3 rpms in 1 aws cli command?
I tried
aws s3 cp *.rpm s3://mybucket1/rpms/
I got an error
Unknown options: 1.rpm, 2.rpm, 3.rpm,s3://mybucket1/rpms/
You can use filters:
aws s3 cp . s3://mybucket1/rpms/ --recursive --exclude "*" --include "*.rpm"
or with sync:
aws s3 sync . s3://mybucket1/rpms/ --exclude "*" --include "*.rpm"
Have over 2000+ folders reside in S3 bucket. I do not want to copy all folders to my local server.
Is there a way or a script to loop through to copy 200 folders out of 2000+ folders from that particular bucket. for eg.
Need to copy over 200-400 folders out of 2000+ from s3 bucket, is there a regex group capture or script to automate to copy certain list of folders
input.....
faob/
halb/
mcgb/
mgvb/
nxhb/
ouqb/
pdyb/
qwdb/
output...
ouqb/
pdyb/
qwdb/
aws s3 cp s3://s3-bucket/* /tmp/
Yes, you can use multiple --include parameters to specify multiple input locations.
aws s3 cp s3://bucket-name /local/folder --recursive --exclude "*" --include "faob/*" --include "halb/*" --include "mcgb/*"
But you can't have multiple destination folders.
hope this helps.
This seems to work:
aws s3 cp --recursive s3://my-bucket /tmp/ --exclude "*" --include "*b/*"
For information about using wildcards in aws s3 cp, see: Use of Exclude and Include Filters
I'm trying to upload all files of type .flv to an S3 bucket using the AWS CLI from a Windows server 2008 command line.
I do this:
aws s3 sync . s3://MyBucket --exclude '*.png'
And it begins uploading .png files instead.
I'm trying to follow the documentation and it gives an example that reads:
Local directory contains 3 files:
MyFile1.txt
MyFile2.rtf
MyFile88.txt
'''
aws s3 sync . s3://MyBucket/MyFolder --exclude '*.txt'
upload: MyFile2.rtf to s3://MyBucket/MyFolder/MyFile2.rtf
So what am I doing wrong?
Use:
aws s3 sync . s3://MyBucket/ --exclude "*" --include "*.flv"
It excludes all files, then includes .flv files. The order of parameters is important.
You can also use:
aws s3 cp . s3://MyBucket/ --recursive --exclude "*" --include "*.flv"
The difference is that sync will not re-copy a file that already exists in the destination.
I have zipped files in an S3 bucket that I need to bring back to my EC2 instance. In the past, I moved the documents to S3 with the following command:
aws s3 cp /my/ec2/path/ s3://my/s3/path/ --exclude '*' --include '2014-01*’ —-recursive
To move files from January 2014 back to EC2, I have tried the following command:
aws s3 cp s3://my/s3/path/ //my/ec2/path/ --exclude '*' --include '2014-01*' --recursive
My understanding is that this command excludes all files but then includes all files with the prefix '2014-01'. I have confirmed that this is how the files I want start. I have also tried only one forward slash before mainstorage and including fewer files.
I have followed these two links from Amazon:
http://docs.aws.amazon.com/cli/latest/reference/s3/index.html
http://docs.aws.amazon.com/cli/latest/userguide/using-s3-commands.html
Figured it out. The key was to define the filepath in --include , i.e. --include '2014-1'. Correct command:
aws s3 cp s3://my/s3/path //my/ec2/path/ --exclude '*' --include '*2014-01*' --recursive