I used aws s3 sync myfolder my-bucket to copy files from EC2 to a bucket in s3. Now, they're loose (not in a folder). The usual mkdir and mv commands don't seem to be available - how can I make a folder and put my files into it?
One would be to to sync again, but this time with the folder name which you want (it will be created automatically):
aws s3 sync myfolder s3://my-bucket/my-folder-on-s3
The other way would be mv with --recursive option:
aws s3 mv s3://my-bucket s3://my-bucket/my-folder-on-s3 --recursive
The syntax to copy them to S3 would be:
aws s3 sync localdir s3://my-bucket
If you wish to move files around an Amazon S3 bucket, use:
aws s3 mv s3://my-bucket/object1.txt s3://my-bucket/folder1/
Related
I setup an SFTP server on Debian EC2 instance. I setup a cron job using aws s3 sync <source dir on EC2> <destination S3 bucket>. I issue is that my EC2 will get full as uploads come in.
Once a file is uploaded to EC2 instance, I want the file to moved to S3 bucket. sync command just copies it and doesn't delete from source. How can I accomplish this?
The aws s3 mv command actually performs a CopyObject() and a Delete.
To move a whole directory, use:
aws s3 mv --recursive localdir s3://bucket-name/
If you want to move the 'contents' of the directory, try:
aws s3 mv --recursive localdir s3://bucket-name/ --exclude "*" --include "localdir/*"
See: mv — AWS Command Reference
I'm trying to download one file from my s3 bucket
I'm trying this command:
aws s3 sync %inputS3path% %inputDatapath% --include "20211201-1500-euirluclprd01-olX8yf.1.gz"
and I habve also tried_
aws s3 sync %inputS3path% %inputDatapath% --include "*20211201-1500-euirluclprd01-olX8yf.1*.gz"
but when command is executing, I'm get all file that's include folder
Folder looks like :
/2021/12/05
20211201-1500-euirluclprd01-olX8yf.1.gz
20211201-1505-euirluclprd01-olX8yf.1.gz
You can use aws s3 cp to copy a specific file. For example:
aws s3 cp s3://bucketname/path/file.gz .
Looking at your variables, you could probably use:
aws s3 cp %inputS3path%/20211201-1500-euirluclprd01-olX8yf.1.gz %inputDatapath%
I want to sync a directory in my S3 bucket and delete files that are in the destination, that are not in the source.
I try this command but, the files that are only in destination was keep.
aws s3 sync my-directory/ s3://my-bucket
I found the solution here, I just add --delete.
aws s3 sync my-directory/ s3://my-bucket --delete
I'm trying to copy/move/sync the files from local directory to S3 using the AWS Command-Line Interface (CLI).
I was able to successfully upload files for the very first time to the S3 bucket but when I try to run the same command again for uploading the second time it fails to upload. The command doesn't throw any error.
Here is the command which I ran for moving the files.
aws s3 mv --recursive my-directory s3://my-files/
For instance, I had files file1.pdf, file2.pdf and file3.pdf.
If I delete file2.pdf from the s3 bucket and try to copy the file again using cp or sync or mv. It won't be uploading the file back again to s3 bucket.
AWS CLI Version: aws-cli/1.15.10 Python/2.6.6 Linux/2.6.32-642.6.2.el6.x86_64 botocore/1.10.10
Any thoughts?
Initially I ran the aws s3 mv --recursive my-directory s3://my-files/ which transfers the files and deletes them from the local directory. Only the files were deleted, folders still exist. Files didn't exist in those folders so the subsequent cp & sync commands didn't work.
I'm looking through the documentation of aws cli and I cannot find the way to copy the only files in some directory structure to other bucket with "flattened" structure(I want one directory and all files inside of it).
for example
/a/b/c/1.pg
/a/2.jpg
/a/b/3.jpg
i would want to have in different bucket:
/x/1.jpg
/x/2.jpg
/x/3.jpg
Am I missing something or is it impossible?
Do you have an idea how could I do that?
Assuming that you have aws cli configured on the system and assuming that both the buckets are in the same region.
What you can do is first dowload the s3 bucket to your local machine using:
aws s3 sync s3://originbucket /localdir/
Post this, use a find command to get all the files into one dir
find /localdir/ -type f -exec mv {} /anotherlocaldir/
Finally, you can upload the files to s3 again!
aws s3 sync /anotherlocaldir/ s3://destinationbucket
You don't need to download files locally, as suggested in another answer. Instead, you could write a shell script or something that does the following:
Run ls on s3://bucket1 to get fully-qualified names of all files in it.
For each file, run cp to copy it from current location to s3://bucket2/x/
Here are some examples for your reference:
aws s3 sync /a/b/c/1.pg s3://bucketname/
aws s3 sync /a/2.jpg s3://bucketname/
aws s3 sync /a/b/3.jpg s3://bucketname/
To sync all contents of a dir to S3 bucket:
aws s3 sync /directoryPath/ s3://bucketname/
AWS reference url: http://docs.aws.amazon.com/cli/latest/reference/s3/sync.html