Two days after having manually deleted all the objects in a multi-region Cloud Storage bucket (e.g. us.artifacts.XXX.com) without Object Versioning I noticed that the bucket size hadn't decreased at all. Only when trying to delete the bucket I discovered that it actually stills containing the objects that I had presumably deleted.
Why aren't those objects displayed in the bucket list view, even when enabling Show deleted data?
When deploying a Function for the first time, two buckets are created automatically:
gcf-sources-XXXXXX-us-central1
us.artifacts.project-ID.appspot.com
You can observe these two buckets from the GCP Console by clicking on Cloud Storage from the left panel.
The files you're seeing in bucket us.artifacts.project-ID.appspot.com are related to a recent change in how the runtime (for Node 10 and up) is built as this post explains.
I also found out that this bucket doesn't have object versioning, retention policy or any lifecycle rule. Although you delete this bucket, it will be created again when you deploy the related function, so, if you are seeing unexpected amounts of Cloud Storage used, this is likely caused by a known issue with the cleanup of artifacts created in the function deployment process as indicated here.
Until the issue is resolved, you can avoid hitting storage limits by creating an auto-deletion rule in the Cloud Console:
In the Cloud Console, select your project > Storage > Browser to open the storage browser.
Select the "artifacts" bucket from the list.
Under the Lifecycle tab, add a rule to auto-delete old images. Choose a deletion interval that works within your normal rate of deployments.
If possible, try to reproduce this scenario with a new function. In the meantime, take into account that if you delete many objects at once, you can track deletion progress by clicking the Notifications icon in the Cloud Console.
In addition, the Google Cloud Status Dashboard provides information about regional or global incidents affecting Google Cloud services such as Cloud Storage.
Nevermind! Eventually (at some point between 2-7 days after the deletion) the bucket size decreased and the objects are no longer displayed in the "Delete bucket" dialog.
Related
I was inspecting the infrastructure points I have on my Google Cloud to remove any lose points...
Then i noticed that google cloud storage have 5 buckets [even that i just created 2 of them]
these 5 buckets are:
1 - bucket i created
2 - bucket i created
3 - PROJECT.backups
4 - gcf-sources-CODE-us-central1
5 - us.artifacts.PROJECT.appspot.com
I understand that the backups bucket come from firebase realtime database backups and the sources bucket come from the firebase cloud functions code. BUT where does the artifacts bucket comes from? this bucket alone has TWICE the size of all other buckets together.
Its contents are just binary files named like "sha256:HASH" some of which are larger than 200MB
I deleted this bucket and it was re-created [without my interaction] again next day.
Does anyone know what might be using it? how can i track it down? what is it for?
The us.artifacts.<project id>.appspot.com bucket is created and used by Cloud Build to store container images generated by the Cloud Build service. One of the processes that generates objects in this bucket is Cloud Function, and you can realize this because the first time that you create a function, GCP asks you to enable the Cloud Build API and this bucket appears in the Cloud Storage section. App Engine also stores objects in this bucket each time you deploy a new version of an app.
As it is mentioned in the documentation, in the case of App Engine, once the deployment has been completed, the images in the us.artifacts.<project id>.appspot.com bucket are no longer needed, so it is safe to delete them. However, in the case that you are only using Cloud Functions, it is not recommended to delete the objects in this bucket. Although you are not experiencing issues now, there is a possibility that you can experience them in the future, so instead of delete all of the objects manually, you can use the Lifecycle Object Management to delete the objects in this bucket every certain period of time, for instance, every 7 days. You can do it by navigating to the Lifecycle tab of the us.artifacts.<project id>.appspot.com bucket and adding a new lifecycle rule which deletes objects that have the age greater than X days.
This is your docker registry. Each time you push (either via docker push or by using the Cloud Build service) GCP stores image layers in those buckets.
My Gcp project name is Mobisium. I found out that there are 2 bucket auto created in the storage browser named mobisum-bucket and mobisium-daisy-bkt-asia.I have never used bucket in the project. mobisium-bucket bucket is empty and the mobisium-daisy-bkt-asia contains one file called daisy.log. Both buckets are Location Type: Multi-region. I read in a stack overflow question's comments that If bucket are created automatically multi-region, you will be charged.
My questions is:
Am I being charged for this buckets.
Are these buckets required, If not should I delete them.
According to documentation you are charged for:
data storage
network
operations
So you will be charged for them if they contain data. You can also view all charges assosciated with your billing account
This buckets names suggests that some services created them - the buckets name is hard to figure out which services. Sometimes when you turn on the services, they create buckets for themselves.
Creating new project, there shouldn't be any buckets, so if this is really new project (created from scratch) you could try to delete them.
If this will be repeated for another project (nor only for this one) it will be good idea to contact support, because this is not normal action.
Is there a way to make a Google Cloud Storage bucket "append-only"?
To clarify, I want to make it so that trying to overwrite/modify an existing object returns an error.
Right now the only way I see to do this is client-side, by checking if the object exists before trying to write to it, but that doubles the number of calls I need to make.
There are several Google Cloud Storage features that you can enable:
Object Versioning
Bucket Lock
Retention Policies
The simplest method is to implement Object Versioning. This prevents objects from being overwritten or deleted. This does require changes to client code to know how to request a specific version of an object if multiple versions have been created due to object overwrites and deletes.
Cloud Storge Object Versioning
For more complicated scenarios implement bucket lock and retention policies. These features allow you to configure a data retention policy for a Cloud Storage bucket that governs how long objects in the bucket must be retained
Retention policies and Bucket Lock
I have a certain image in my Google Cloud Storage Bucket but it was gone w/out us knowing how it was deleted. When I checked the activity logs.. All I can see is "updated". By the way I see it, updated will be the status whether the files is deleted or created. So how can I check this? I want to know at least when it was deleted
In order to see audit logs for Cloud Storage you have to first enable them.
To enable them in the console go to the IAM & Admin -> Audit Logs and than by selecting Google Cloud Storage you will see on the right side of the screen under LOG TYPE different services which you can enable or disable audit logging for. Please refer to this documentation where the procedure is described.
After the Cloud Storage logs are enabled you can see all the activity by going in the console to ACTIVITY and under Activity types select Data access. After this you will be able to see the operations on the bucket (considering you have the right permissions to see those).
We’ve been using Google Cloud Storage Transfer service and in our data source (AWS) we had a directory accidentally deleted, so we figured it would be in the data sink however upon taking a looking it wasn’t there despite versioning being on.
This leads us to believe in Storage Transfer the option deleteObjectsUniqueInSink hard deletes objects in the sink and removes them from the archive.
We'e been unable to confirm this in the documentation.
Is GCS Transfer Service's deleteObjectsUniqueInSink parameter in the TransferSpec mutually exclusive with GCS's object versioning soft-delete?
When the deleteObjectsUniqueInSink option is enabled, Google Cloud Storage Transfer will
List only the live versions of objects in source and destination buckets.
Copy any objects unique in the source to the destination bucket.
Issue a versioned delete for any unique objects in the destination bucket.
If the unique object is still live at the time that Google Cloud Storage Transfer issues the deletion, it will be archived. If another process, such as Object Lifecycle Management, archived the object before the deletion occurs, the object could be permanently deleted at this point rather than archived.
Edit: Specifying the version in the delete results in a hard delete (Objects Delete Documentation), so transfer service is currently performing hard deletes for unique objects. We will update the service to instead perform soft deletions.
Edit: The behavior has been changed. From now on deletions in versioned buckets will be soft deletes rather than hard deletes.