The below aws sync command does execute, but I can not seem to exclude the xxxx files as that have the --include pattern in them.
It will always be xxxx but I am trying to exclude them from the sync. Thank you :).
files in directory
xxxx.id.1.bam
xxxx.id.1.bai
aaa.id.1.bam
aaa.id.1.bai
bbb.bam
bbb.bai
desired
aaa.id.1.bam
aaa.id.1.bai
command
aws s3 sync . s3://bucket/ --exclude "*" --exclude "*xxxx" --include "*.id.1.bam" --include "*.id.1.bai" --dryrun
The order of --exclude and --include metters. It should be:
aws s3 sync . s3://bucket/ --exclude "*" --include "*.id.1.bam" --include "*.id.1.bai" --exclude "xxxx.*" --dryrun
Related
I'm syncing the entire contents of an external hard drive, used with macOS, to an S3 bucket. I'd like to exclude all macOS hidden files.
I've tried:
aws s3 sync --dryrun --exclude "^\." --exclude "\/\." ./ s3://bucketname
However, the result when I run that is exactly the same as just:
aws s3 sync --dryrun . s3://bucketname
So, I must be doing something wrong.
Any suggestions?
Thanks.
aws s3 sync --dryrun . s3://bucketname --exclude ".*" --exclude "*/.*"
Adding two exclusion arguments will hide both the specified files in the current directory as well as any in subfolders.
This seems to work:
aws s3 sync --dryrun . s3://bucketname --exclude ".*"
However, I don't think it will exclude such files in sub-directories.
Try this:
aws s3 sync --dryrun --exclude '*/.*'
This should remove any hidden files, including in subfolders.
aws s3 sync --recursive --dryrun --exclude '/.'
It says in docs for aws cli that wild cards are not supported. You can use --include and --exclude options. But that could take a while when files structure is wide.
aws s3 rm s3://your-bucket/your-folder/year=2020/month=05/ --exclude "*" --include "*/provider=400/qk=0001" --include "*/provider=400/qk=0002" --include "*/provider=400/qk=0003" --include "*/provider=400/qk=0010" ...
So what are other options?
In shell terminal you can do next trick:
for i in `s3://your-bucket/your-folder/year=2020/month=05/day={01,02,03,04,05,06,07,08,09,10...}/provider=400/qk={0001,0002,0003,0010,...}; do aws s3 rm $i --recursive; done
I'd like to upload a file.txt on aws s3 that is located in something like main/part1/part2/file.txt, where part1 and part2 are unknown (those folders always change).
I can do that with the command aws s3 cp ./main s3://mybucket --exclude "*" --include "*.txt" --recursive, but then in my bucket I have the file located in part1/part2/file.txt. I'd like file.txt to be at the base of the bucket, not inside part1/part2
Is that possible given that part1 and part2 are constantly changing?
for dir1 in $(ls main); do
for dir2 in $(ls main/$dir1); do
aws s3 cp ./main/$dir1/$dir2/ s3://my-bucket --exclude "*" --include "*.txt" --recursive
done
done
upload: main/part1/part2/file.txt to s3://my-bucket/file.txt
upload: main/part11/part22/file2.txt to s3://my-bucket/file2.txt
The following will work if main will never contain more than 1 subdirectory at a time (part1) & that subdirectory in-turn will never contain more than 1 subdirectory at a time (part2):
aws s3 cp ./main/*/*/ s3://my-bucket --exclude "*" --include "*.txt" --recursive
upload: main/part1/part2/file.txt to s3://my-bucket/file.txt
I would like to copy files matching a file name pattern from my machine to an AWS S3 bucket using AWS CLI. Using the standard unix file name wildcards does not work:
$ aws s3 cp *.csv s3://wesam-data/
Unknown options: file1.csv,file2.csv,file3.csv,s3://wesam-data/
I followed this SO answer addressing a similar problem that advises using the --exclude and --include filters as explained here as shown below without success.
$ aws s3 cp . s3://wesam-data/ --exclude "*" --include "*.csv"
Solution
$ aws s3 cp . s3://wesam-data/ --exclude "*" --include "*.csv" --recursive
Explanation
It turns out that I have to use the --recursive flag with the --include & --exclude flags since this is a multi-file operation.
The following commands are single file/object operations if no --recursive flag is provided.
cp
mv
rm
I have zipped files in an S3 bucket that I need to bring back to my EC2 instance. In the past, I moved the documents to S3 with the following command:
aws s3 cp /my/ec2/path/ s3://my/s3/path/ --exclude '*' --include '2014-01*’ —-recursive
To move files from January 2014 back to EC2, I have tried the following command:
aws s3 cp s3://my/s3/path/ //my/ec2/path/ --exclude '*' --include '2014-01*' --recursive
My understanding is that this command excludes all files but then includes all files with the prefix '2014-01'. I have confirmed that this is how the files I want start. I have also tried only one forward slash before mainstorage and including fewer files.
I have followed these two links from Amazon:
http://docs.aws.amazon.com/cli/latest/reference/s3/index.html
http://docs.aws.amazon.com/cli/latest/userguide/using-s3-commands.html
Figured it out. The key was to define the filepath in --include , i.e. --include '2014-1'. Correct command:
aws s3 cp s3://my/s3/path //my/ec2/path/ --exclude '*' --include '*2014-01*' --recursive