Long time listener, first time caller...
If I have an S3 bucket which is versioned (as per the documentation for replication) and bucket replication is enabled; what happens if the source bucket object is deleted before replication has taken place?
I'm assuming (as the source bucket is versioned) the replication will still take place. The replication logic surely sees a new object creation event and replicates this in the destination bucket. Because I've not specified any other configuration, the delete marker subsequently placed on the source bucket object will not be replicated onto the destination object. Therefore it should just appear as normal in the destination bucket and not be visible in the source bucket.
I can't find anything concrete in the documentation that clarifies the position on this situation.
It depends on which option you choose in your replication configuration:
V1: the delete marker is replicated - a subsequent GET request to the deleted object does not return the object in either the source or the destination bucket.
V2: the delete marker is not replicated - a subsequent GET request to the deleted object returns the object only in the destination bucket.
For more, see Managing delete marker replication in Amazon S3.
Related
With S3 replication, if a previously replicated file is deleted in the destination bucket, is the default behaviour that the file will be re-copied? I assume this is the case and if so, is there any way to change this behaviour so files are only ever replicated once?
if a previously replicated file is deleted in the destination bucket, is the default behaviour that the file will be re-copied
NO, it wont be recopied because according to docs "By default, replication only supports copying new Amazon S3 objects after it is enabled."
So literally s3 sees it as an existing object which is already replicated to destination no matter whether it is deleted in destination or not, it won't replicate AGAIN!
I'm trying to understand the delete operation of an object in aws S3.
In cross region replication, if I delete an object from the source, this delete is not propagated to the destination.
The official text - "If you specify an object version ID to delete in
a DELETE request, Amazon S3 deletes that object version in the source
bucket, but it doesn't replicate the deletion in the destination
bucket. In other words, it doesn't delete the same object version from
the destination bucket. This protects data from malicious deletions. "
In other case, I read that
The official text - Amazon S3 offers eventual consistency for
overwrite PUTS and DELETES in all Regions
When I made a test, the delete is not propagated. Then, there is a divergence between the replica !
Is it normal ? how about the eventual consistency of the delete ?
This is not about replication, it's about simple buckets from Introduction to AWS S3.
Amazon S3 offers eventual consistency for overwrite PUTS and DELETES in all Regions
The right answer - "it doesn't delete the same object version from the destination bucket. This protects data from malicious deletions".
If you need "consistency of the delete" - you can try to automate it with aws s3 sync with --delete flag.
I recently deleted by error a complete folder in an S3 bucket without versioning activated. This folder used to have a lifecycle policy, a lot of objects were in a glacier state. Now I can't find them, event in a vault.
Can I retrieve the deleted folder or is it a lost cause?
Unfortunately, I believe if you didn't have versioning running on the bucket to the best of my knowledge you cannot recover the items.
As stated on https://docs.aws.amazon.com/AmazonS3/latest/user-guide/undelete-objects.html
"To be able to undelete a deleted object, you must have had versioning
enabled on the bucket that contains the object before the object was
deleted."
We’ve been using Google Cloud Storage Transfer service and in our data source (AWS) we had a directory accidentally deleted, so we figured it would be in the data sink however upon taking a looking it wasn’t there despite versioning being on.
This leads us to believe in Storage Transfer the option deleteObjectsUniqueInSink hard deletes objects in the sink and removes them from the archive.
We'e been unable to confirm this in the documentation.
Is GCS Transfer Service's deleteObjectsUniqueInSink parameter in the TransferSpec mutually exclusive with GCS's object versioning soft-delete?
When the deleteObjectsUniqueInSink option is enabled, Google Cloud Storage Transfer will
List only the live versions of objects in source and destination buckets.
Copy any objects unique in the source to the destination bucket.
Issue a versioned delete for any unique objects in the destination bucket.
If the unique object is still live at the time that Google Cloud Storage Transfer issues the deletion, it will be archived. If another process, such as Object Lifecycle Management, archived the object before the deletion occurs, the object could be permanently deleted at this point rather than archived.
Edit: Specifying the version in the delete results in a hard delete (Objects Delete Documentation), so transfer service is currently performing hard deletes for unique objects. We will update the service to instead perform soft deletions.
Edit: The behavior has been changed. From now on deletions in versioned buckets will be soft deletes rather than hard deletes.
I have a few JSON files in my S3 bucket that I would like replicated to another bucket. Not just a one time copy but will mirror the changes of the original when an update occurs, similar to a multicast to two different buckets.
If I just right click and hit copy and paste it into a another bucket, will that do the trick?
There are 2 solutions to this question
Create a Lambda file which maps to the Object Created (All) event type. So whenever a new object is created or modified or copied, the Lambda will get triggered; where the event will contain reference to the object. Copy the object from bucket A to bucket B inside the Lambda which will achieve your requirement.
However, this wouldn't work for delete functionality. For Delete, you might have to create another Lambda for Object Removed (All) event, and delete the S3 object from bucket B when the object gets deleted in Bucket A. This is near realtime
Write a scheduler which runs every N minutes where it copies all the contents of the bucket from A to B- but not a good scalable solution. This is more of a batch solution
My 2 paisas, hope it helps