My GCP project automatically created 2 storage buckets - google-cloud-platform

My Gcp project name is Mobisium. I found out that there are 2 bucket auto created in the storage browser named mobisum-bucket and mobisium-daisy-bkt-asia.I have never used bucket in the project. mobisium-bucket bucket is empty and the mobisium-daisy-bkt-asia contains one file called daisy.log. Both buckets are Location Type: Multi-region. I read in a stack overflow question's comments that If bucket are created automatically multi-region, you will be charged.
My questions is:
Am I being charged for this buckets.
Are these buckets required, If not should I delete them.

According to documentation you are charged for:
data storage
network
operations
So you will be charged for them if they contain data. You can also view all charges assosciated with your billing account
This buckets names suggests that some services created them - the buckets name is hard to figure out which services. Sometimes when you turn on the services, they create buckets for themselves.
Creating new project, there shouldn't be any buckets, so if this is really new project (created from scratch) you could try to delete them.
If this will be repeated for another project (nor only for this one) it will be good idea to contact support, because this is not normal action.

Related

How do I make an AWS Budget Action stop serving public files from an S3 bucket?

I have an Amazon S3 bucket hosting some images for a static website. To keep things simple, I made folders in the bucket publicly readable with "make public using ACL" in the web console (so I can just link to the images from elsewhere).
I want to set up a budget limit on this bucket so I don't wake up to a huge S3 charge if it gets Reddit-hugged etc. Can I make a Budget Action that disables public read access for an S3 bucket (folder)?
None of the built-in budget actions in the web console seem relevant. The bewildering AWS docs mention something about SCPs but I don't see those could apply here. Someone else asked a similar question here but got no answers.
So far I've
Made my bucket and the files in it publicly readable (that's the point)
Attached a user-defined tag (eg "budget_limit_foo") to the bucket
Activated this tag as a cost allocation tag
Created a budget in the Billing Console set to Fixed Budget, Blended Costs, and using tag "budget_limit_foo" as a filter dimension.
Created a budget alert with threshold at 90% (that's the one that'll trip the action)
Gotten to "attach actions" in the "Create Budgets" template and hit a wall.
And to clarify, I only want to disable public read — or file transfer out — on this one bucket when it hits its limit. I'd prefer it didn't delete the files!

Google Cloud storage bucket not listing deleted objects

Two days after having manually deleted all the objects in a multi-region Cloud Storage bucket (e.g. us.artifacts.XXX.com) without Object Versioning I noticed that the bucket size hadn't decreased at all. Only when trying to delete the bucket I discovered that it actually stills containing the objects that I had presumably deleted.
Why aren't those objects displayed in the bucket list view, even when enabling Show deleted data?
When deploying a Function for the first time, two buckets are created automatically:
gcf-sources-XXXXXX-us-central1
us.artifacts.project-ID.appspot.com
You can observe these two buckets from the GCP Console by clicking on Cloud Storage from the left panel.
The files you're seeing in bucket us.artifacts.project-ID.appspot.com are related to a recent change in how the runtime (for Node 10 and up) is built as this post explains.
I also found out that this bucket doesn't have object versioning, retention policy or any lifecycle rule. Although you delete this bucket, it will be created again when you deploy the related function, so, if you are seeing unexpected amounts of Cloud Storage used, this is likely caused by a known issue with the cleanup of artifacts created in the function deployment process as indicated here.
Until the issue is resolved, you can avoid hitting storage limits by creating an auto-deletion rule in the Cloud Console:
In the Cloud Console, select your project > Storage > Browser to open the storage browser.
Select the "artifacts" bucket from the list.
Under the Lifecycle tab, add a rule to auto-delete old images. Choose a deletion interval that works within your normal rate of deployments.
If possible, try to reproduce this scenario with a new function. In the meantime, take into account that if you delete many objects at once, you can track deletion progress by clicking the Notifications icon in the Cloud Console.
In addition, the Google Cloud Status Dashboard provides information about regional or global incidents affecting Google Cloud services such as Cloud Storage.
Nevermind! Eventually (at some point between 2-7 days after the deletion) the bucket size decreased and the objects are no longer displayed in the "Delete bucket" dialog.

Google creates a storage bucket without my interaction

I was inspecting the infrastructure points I have on my Google Cloud to remove any lose points...
Then i noticed that google cloud storage have 5 buckets [even that i just created 2 of them]
these 5 buckets are:
1 - bucket i created
2 - bucket i created
3 - PROJECT.backups
4 - gcf-sources-CODE-us-central1
5 - us.artifacts.PROJECT.appspot.com
I understand that the backups bucket come from firebase realtime database backups and the sources bucket come from the firebase cloud functions code. BUT where does the artifacts bucket comes from? this bucket alone has TWICE the size of all other buckets together.
Its contents are just binary files named like "sha256:HASH" some of which are larger than 200MB
I deleted this bucket and it was re-created [without my interaction] again next day.
Does anyone know what might be using it? how can i track it down? what is it for?
The us.artifacts.<project id>.appspot.com bucket is created and used by Cloud Build to store container images generated by the Cloud Build service. One of the processes that generates objects in this bucket is Cloud Function, and you can realize this because the first time that you create a function, GCP asks you to enable the Cloud Build API and this bucket appears in the Cloud Storage section. App Engine also stores objects in this bucket each time you deploy a new version of an app.
As it is mentioned in the documentation, in the case of App Engine, once the deployment has been completed, the images in the us.artifacts.<project id>.appspot.com bucket are no longer needed, so it is safe to delete them. However, in the case that you are only using Cloud Functions, it is not recommended to delete the objects in this bucket. Although you are not experiencing issues now, there is a possibility that you can experience them in the future, so instead of delete all of the objects manually, you can use the Lifecycle Object Management to delete the objects in this bucket every certain period of time, for instance, every 7 days. You can do it by navigating to the Lifecycle tab of the us.artifacts.<project id>.appspot.com bucket and adding a new lifecycle rule which deletes objects that have the age greater than X days.
This is your docker registry. Each time you push (either via docker push or by using the Cloud Build service) GCP stores image layers in those buckets.

Backup S3 bucket with identical links

I have a Question Bank application with +10k questions.
The questions are a combination of text (stored within a database) and images (hosted on Amazon S3). The images are embedded as links through a admin panel.
For security purposes, I want to figure out if I can back-up the entire S3 bucket and later restore it with the same identical links.
Any experience with this?
I presume that your users are accessing the images via an Amazon S3 URL.
The URL consists of the bucket name and the Key (filename) of the object. To provide a file with the "same identical link" simply means putting it in the same bucket and giving it the same Key.
Please note that data stored in Amazon S3 is replicated between multiple Availability Zones. Therefore, it is typically not necessary to backup data for resilience purposes. However, it can be wise to backup in case somebody accidentally or maliciously deletes the objects. (Turning on Versioning could also be a way to safeguard against such cases.)
If you do wish to "backup an entire bucket", one option is to use Same-Region replication, which will automatically replicate objects between buckets. It requires that Versioning is activated.

Is it better to have multiple s3 buckets or one bucket with sub folders?

Is it better to have multiple s3 buckets per category of uploads or one bucket with sub folders OR a linked s3 bucket? I know for sure there will be more user-images than there will be profille-pics and that there is a 5TB limit per bucket and 100 buckets per account. I'm doing this using aws boto library and https://github.com/amol-/depot
Which is the structure my folders in which of the following manner?
/app_bucket
/profile-pic-folder
/user-images-folder
OR
profile-pic-bucket
user-images-bucket
OR
/app_bucket_1
/app_bucket_2
The last one implies that its really a 10TB bucket where a new bucket is created when the files within bucket_1 exceeds 5TB. But all uploads will be read as if in one bucket. Or is there a better way of doing what I'm trying to do? Many thanks!
I'm not sure if this is correct... 100 buckets per account?
https://www.reddit.com/r/aws/comments/28vbjs/requesting_increase_in_number_of_s3_buckets/
Yes, there is actually a 100 bucket limit per account. I asked the reason for that to an architect in an AWS event. He said this is to avoid people hosting unlimited static websites on S3 as they think this may be abused. But you can apply for an increase.
By default, you can create up to 100 buckets in each of your AWS
accounts. If you need additional buckets, you can increase your bucket
limit by submitting a service limit increase.
Source: http://docs.aws.amazon.com/AmazonS3/latest/dev/BucketRestrictions.html
Also, please note that there are actually no folders in S3, just a flat file structure:
Amazon S3 has a flat structure with no hierarchy like you would see in
a typical file system. However, for the sake of organizational
simplicity, the Amazon S3 console supports the folder concept as a
means of grouping objects. Amazon S3 does this by using key name
prefixes for objects.
Source: http://docs.aws.amazon.com/AmazonS3/latest/UG/FolderOperations.html
Finally, the 5TB limit only applies to a single object. There is no limit on the number of objects or total size of the bucket.
Q: How much data can I store?
The total volume of data and number of objects you can store are
unlimited.
Source: https://aws.amazon.com/s3/faqs/
Also the documentation states there is no performance difference between using a single bucket or multiple buckets so I guess both option 1 and 2 would be suitable for you.
Hope this helps.
Simpler Permission with Multiple Buckets
If the images are used in different use cases, using multiple buckets will simplify the permissions model, since you can give clients/users bucket level permissions instead of directory level permissions.
2-way doors and migrations
On a similar note, using 2 buckets is more flexible down the road.
1 to 2:
If you switch from 1 bucket to 2, you now have to move all clients to the new set-up. You will need to update permissions for all clients, which can require IAM policy changes for both you and the client. Then you can move your clients over by releasing a new client library during the transition period.
2 to 1:
If you switch from 2 buckets to 1 bucket, your clients will already have access to the 1 bucket. All you need to do is update the client library and move your clients onto it during the transition period.
*If you don't have a client library than code changes are required in both cases for the clients.