Trying to copy one file with Amazon S3 CLI - amazon-web-services

I made a folder with 3 .jpg files in it to test. This folder is called c:\Work\jpg.
I am trying to upload it to a bucket with this command:
aws s3 cp . s3://{bucket}/Test
I get the following every time:
[Errno 2] No such file or directory: "C:\Work\jpg\".
Obviously, it correctly translated the current folder "." into the correct folder, but then it says it doesn't exist?!?
Any help out there to simply copy 3 files?

Are you confusing aws s3 sync with aws s3 cp. For copy, you need to specify the source file. The destination file can be current directory.
aws s3 cp test.txt s3://mybucket/test2.txt

Ensure that your path is correctly written.
Remember add --recursive option, because is folder
aws s3 cp ./ s3://{bucket}/Test --recursive

Related

AWS CLI to download file with its entire folder structure from S3 to local and/or one S3 to another S3

AWS CLI to download file with its entire folder structure from S3 to local and/or one S3 to another S3
I am looking to download the file from S3 bucket to local with its entire folder structure. For example,
s3://test-s3-dev/apps/test-prd/test/data/sets/frs/bblr/type/level=low/type=data/bd=2022-08-25/region=a/entity=c/ss=tt/dev=mtp/datasetV=1/File123.txt
Above is the S3 path which i need to download on local with it's entire folder structure from S3.
However, by
cp --recursive and synch both are only downloading the File123.txt in current local folder and not downloading the FIle123.txt file with its entire folder structure.
**Please advice how to achieve the File gets downloaded from S3 with its entire folder structure from S3 for ->
To download on local system and/or
Copy from one s3 connection to another S3 connection.**
aws --endpoint-url http://abc.xyz.pqr:9020 s3 cp --recursive s3://test-s3-dev/apps/test-prd/test/data/sets/frs/bblr/type/level=low/type=data/bd=2022-08-25/region=a/entity=c/ss=tt/dev=mtp/datasetV=1/File123.txt ./
OR
aws --endpoint-url http://abc.xyz.pqr:9020 s3 cp --recursive s3://test-s3-dev/apps/test-prd/test/data/sets/frs/bblr/type/level=low/type=data/bd=2022-08-25/region=a/entity=c/ss=tt/dev=mtp/datasetV=1/ ./
OR
aws --endpoint-url http://abc.xyz.pqr:9020 s3 sync s3://test-s3-dev/apps/test-prd/test/data/sets/frs/bblr/type/level=low/type=data/bd=2022-08-25/region=a/entity=c/ss=tt/dev=mtp/datasetV=1/ ./
Above Three aws commands are downloading the file directly in current local folder without copying/sync the file entire directory structure from S3.

How can i download specified file from s3 bucket

I'm trying to download one file from my s3 bucket
I'm trying this command:
aws s3 sync %inputS3path% %inputDatapath% --include "20211201-1500-euirluclprd01-olX8yf.1.gz"
and I habve also tried_
aws s3 sync %inputS3path% %inputDatapath% --include "*20211201-1500-euirluclprd01-olX8yf.1*.gz"
but when command is executing, I'm get all file that's include folder
Folder looks like :
/2021/12/05
20211201-1500-euirluclprd01-olX8yf.1.gz
20211201-1505-euirluclprd01-olX8yf.1.gz
You can use aws s3 cp to copy a specific file. For example:
aws s3 cp s3://bucketname/path/file.gz .
Looking at your variables, you could probably use:
aws s3 cp %inputS3path%/20211201-1500-euirluclprd01-olX8yf.1.gz %inputDatapath%

How to download larg file from aws s3

im trying to download the functional map of the world dataset. It is stored in aws s3. The github page of the dataset only describes how to download a .json file from the bucket, but not the record itself. I tried to download the whole bucket with aws s3 cp s3://spacenet-dataset/Hosted-Datasets/fmow/fmow-rgb/, which returns the error that this path doesn't exist.
Does anyone know which commands I need to download the data? Thanks in advance.
What you should be running if you're copying a specific prefix is the following command
aws s3 cp s3://spacenet-dataset/Hosted-Datasets/fmow/fmow-rgb . --recursive
Or if you want the whole bucket then the following
aws s3 cp s3://spacenet-dataset . --recursive
In this case the . will copy to your current directory so if you want it to go to another directory specify the path by replacing the . argument.
You can also use aws s3 sync:
aws s3 sync s3://spacenet-dataset/Hosted-Datasets/fmow/fmow-rgb/ .
where . is the current working folder.
Or
aws s3 sync s3://spacenet-dataset/Hosted-Datasets/fmow/fmow-rgb/ <path-to-dest-folder>
where <path-to-dest-folder> is an existing folder.

AWS CLI cp doesn't copy the files second time

I'm trying to copy/move/sync the files from local directory to S3 using the AWS Command-Line Interface (CLI).
I was able to successfully upload files for the very first time to the S3 bucket but when I try to run the same command again for uploading the second time it fails to upload. The command doesn't throw any error.
Here is the command which I ran for moving the files.
aws s3 mv --recursive my-directory s3://my-files/
For instance, I had files file1.pdf, file2.pdf and file3.pdf.
If I delete file2.pdf from the s3 bucket and try to copy the file again using cp or sync or mv. It won't be uploading the file back again to s3 bucket.
AWS CLI Version: aws-cli/1.15.10 Python/2.6.6 Linux/2.6.32-642.6.2.el6.x86_64 botocore/1.10.10
Any thoughts?
Initially I ran the aws s3 mv --recursive my-directory s3://my-files/ which transfers the files and deletes them from the local directory. Only the files were deleted, folders still exist. Files didn't exist in those folders so the subsequent cp & sync commands didn't work.

AWS s3 mv command deletes the source directory

The objective is to move all the files from s3 "dir2" directory to EMR directory "mydir".
I am using the command:
aws s3 mv s3:///dir1/dir2/ /mnt/mydir/ --recursive
This command gets executed but the dir2 directory from s3 gets deleted.
The files within dir2 although moves to mydir of EMR.
How can I only move the files from source dir of s3 without removing the source directory?
When dealing with multiple objects, you want to use sync not cp or mv:
aws s3 sync s3:///dir1/dir2/ /mnt/mydir/
There are ways to load data into EMR directly from S3, so you may want to look into those.
Update: I have confirmed that:
aws s3 mv s3://bucket/f1/f2 . --recursive
Will move all of the files inside f2/ while leaving f2 in the bucket.
Directories or folders do not actually exist in S3. What you are calling a directory is simply a common file name prefix in S3. When there are no files with that prefix anymore then the "directory" does not exist anymore in S3.