aws s3 sync <> <> --delete works fine but I have a scenario wherein someone created directories using AWS console and put in some files inside those directories using manual upload. So now when I run the sync command, those files get removed but the manually created directories still persist.
Is this an expected behavior of the command?
This issue is tracked here - https://github.com/aws/aws-cli/issues/2685 . This is a known bug and no direct solution is available yet so we need to create wayarounds according to our condition.
Related
I have a folder "Execution" folder in s3 bucket.
It has folders and files like
Execution
Exec_06-06-2022/
file1.json
file2.json
Exec_07-06-2022/
file3.json
file4.json
I need to configure to delete the Exec_datestamp folders and the inside files after X days.
I tried this using AWS lifecycle config for the prefix "Execution/"
But it deletes the folder Execution/ after X days ( set this to 1 day to test)
Is there any other way to achieve this?
There are no folders in S3. Execution is part of objects name, specifically its key prefix. The S3 console only makes Execution to appear as a folder, but there is no such thing in S3. So your lifecycle deletes Execution/ object (not folder), because it matches your query.
You can try with Execution/Exec* filter.
Got it, https://stackoverflow.com/a/41459761/5010582. The prefix has to be "Execution/Exec" without any wildcard.
I have a powershell script, that downloads all files form an S3 bucket, and then removes the files from the bucket. All the files I'm removing are stored in a subfolder in the S3 bucket, and I just want to delete the files but maintain the subfolders.
I'm currently using the following command to delete the files in S3 once the file has been downloaded from S3.
Remove-S3Object -BucketName $S3Bucket -Key $key -Force
My problem is that if it removes all the files in the subfolder, the subfolder is removed as well. Is there a way to remove the file, but keep the subfolder present using powerhsell. I believe I can do something like this,
aws s3 rm s3://<key_to_be_removed> --exclude "<subfolder_key>"
but not quite sure if that'll work.
I'm looking for the best way to accomplish this, and at the moment, my only option is to recreate the subfolder via the script if the subfolder not longer exists.
The only way to accomplish having an empty folder is to create a zero-length object which has the same name as the folder you want to keep. This is actually how the S3 console enables you to create an empty folder.
You can check this by running $ aws s3 ls s3://your-bucket/folderfoo/ and observing an output object having length of zero bytes.
See more on this topic here.
As already commented, S3 does not really have folders the way file systems do. The folders as presented by most S3 browsers are just generated based on the paths of the files/objects. If you upload an object/file named folder/file, the browsers will present folder as folder with file as a file in the folder. But technically, all that there is is the file/object folder/file. The folder does not exist on its own.
You can explicitly create a folder by creating an empty empty-named object with "in the folder": folder/. If you do that, it will appear the the folder exists even if there are no files in it. But if you do not do that, the virtual folder disappears once you remove all objects in the folder.
Now the question is whether your command removes even the empty named object representing the folder or not. I cannot tell that.
I want to synchronize a directory on my machine with a bucket in s3. The problem is that I find that the aws cli option of sync does not seem to do what I expected.
The behavior that I'm looking for is such that when I run the command it evaluates the content of the local directory and the content of the s3 bucket and updates the one with the old content with the changes in the other.
Can't guess the best approach. Thanks in advance.
I think it's a one way sync. If anything is the source is new or updated then it will sync it to the destination. Even if you may have a new file in the destination, it won't sync that down to your source. You would need to do a sync again from destination back to source. It's a one way sync, from source to destination.
You can't do that with awscli. aws sync s3://bucket . will synchronize the data in both sides
If you want a tool that acts according to your needs you have to develop a python script using https://github.com/boto/boto3
Npm package (and cli command) s3-cli can sync a local folder and an S3 bucket, both ways.
s3-cli sync [--delete-removed] /path/to/folder/ s3://bucket/key/on/s3/
s3-cli sync [--delete-removed] s3://bucket/key/on/s3/ /path/to/folder/
I am trying to run a demo project for uploading to S3 with Grails 3.
The project in question is this, more specifically the S3 upload is only for the 'Hotel' example at the end.
When I run the project and go to upload the image, I get an updated message but nothing actually happens - there's no inserted url in the dbconsole table.
I think the issue lies with how I am running the project, I am using the command:
grails -Daws.accessKeyId=XXXXX -Daws.secretKey=XXXXX run-app
(where I am supplementing the X's for my keys obviously).
This method of running the project appears to be slightly different to the method shown in the example. I run my project from the command line and I do not use GGTS, just Sublime.
I have tried inserting my AWS keys into the application.yml but I receive an internal server error then.
Can anyone help me out here?
Check your bucket policy in s3. You need to grant permissions to the API user to allow uploads.
Using the S3 CLI, I can sync a local directory with an S3 bucket using the following command:
aws s3 sync s3://mybucket/ ./local_dir/
This command is a complete sync. It uploads new files, updates changed files, and deletes removed files. I am trying to figure out how to do something equivalent using the OpenStack Object Storage CLI:
http://docs.openstack.org/cli-reference/content/swiftclient_commands.html
The upload command has a --changed option. But I need a complete sync that is also capable of deleting local files that were removed.
Does anyone know if I can do something equivalent to s3 sync?
The link you mentioned has this :
`
objects –
A list of file/directory names (strings) or SwiftUploadObject instances containing a source for the created object, an object name, and an options dict (can be None) to override the options for that individual upload operation`
I'm thinking, if you pass the directory and the --changed option it should work.
I don't have a swift to test with. Can you try again?