I am looking to increase the disk size for the 5GB persistent disk that Google provides for Cloud Shell.
I have read all the docs but, they only mention EC2 machines. Though, the Cloud Shell is also an EC2 machine when i loom at the VM's the Cloud Shell machi e is not listed.
PLEASE HELP.
You cannot change the size of the Cloud Shell disk. Cloud Shell runs on a Compute Engine instance that Google controls and is not a resource in your project. Cloud Shell is a container and you can deploy your own container on Compute Engine COS or via Docker.
Related
Does GCP assign a global location (gs://...something...) for the home directory in Cloud Shell? Like buckets are gs://$BUCKET_NAME/, where $BUCKET_NAME is globally unique.
At the shell prompt, all I can see is:
$USERNAME#cloudshell:~ ($PROJECT_ID)
Cloud Shell is globally distributed across multiple Google Cloud Platform regions. When you first connect to Cloud Shell, you will be automatically assigned to the closest available geographical region. You cannot pick your own region and in the event that Cloud Shell does not pick the most optimal region, it will try to migrate your Cloud Shell VM to a closer region when it is not in use.
To view your current region, run the following command from a Cloud Shell session:
$ curl metadata/computeMetadata/v1/instance/zone
Please note that Cloud Shell provisions 5 GB of free persistent disk storage mounted as your $HOME directory on the virtual machine instance. This storage is on a per-user basis and is available across projects. Unlike the instance itself, this storage does not time out on inactivity. All files you store in your home directory, including installed software, scripts and user configuration files like .bashrc and .vimrc, persist between sessions. Your $HOME directory is private to you and cannot be accessed by other users.
No, the cloud shell VM storage is not available as a storage bucket.
But if your intention was to use copy data in and out of the cloud-shell VM storage, you can easily do that
From doc ,
gcloud cloud-shell scp cloudshell:~/data.txt localhost:~data.txt
I used Deployment Manager to create a LAMP Stack for phpMyAdmin. Is it possible to access files on the VM from the Google Cloud Shell? If so, how would I navigate to the files in Google Cloud Shell?
When you start Cloud Shell, it provisions a Google Compute Engine virtual machine running a Debian-based Linux operating system. Cloud Shell instances are provisioned on a per-user, per-session basis. The instance persists while your Cloud Shell session is active; after an hour of inactivity, your session terminates and its VM, discarded. For more on usage quotas, refer to the limitations guide 1.
Yes you can access your LAMP VM-Instance using the cloud shell command as shown below:
gcloud beta compute ssh --zone "us-central1-a" "vm-name" --project "project-id".
Note: (Please replace the zone , vm-name and project-id as per your naming conventions).
Please follow the link 2 to get more information on Cloud Shell How to Guides.
I was trying to run a Docker image with Cloud run and realised that there is no option for adding a persistent storage. I found a list of services in https://cloud.google.com/run/docs/using-gcp-services#connecting_to_services_in_code but all of them are access from code. I was looking to share volume with persistent storage. Is there a way around it ? Is it because persistent storage might not work shared between multiple instances at the same time ? Is there are alternative solution ?
Cloud Run is serverless: it abstracts away all infrastructure management.
Also is a managed compute platform that automatically scales your stateless containers.
Filesystem access The filesystem of your container is writable and is
subject to the following behavior:
This is an in-memory filesystem, so writing to it uses the container
instance's memory. Data written to the filesystem does not persist
when the container instance is stopped.
You can use Google Cloud Storage, Firestore or Cloud SQL if your application is stateful.
3 Great Options for Persistent Storage with Cloud Run
What's the default storage for Google Cloud Run?
Cloud Run (fully managed) has known services that's not yet supported including Filestore which is also a persistent storage. However, you can consider running your Docker image on Cloud Run Anthos which runs on GKE and there you can use persistent volumes which are typically backed by Compute Engine persistent disks.
Having persistent storage in (fully managed) Cloud Run should be possible now.
Cloud Run's second generation execution environment (gen2) supports network mounted file systems.
Here are some alternatives:
Cloud Run + GCS: Using Cloud Storage FUSE with Cloud Run tutorial
Cloud Run + Filestore: Using Filestore with Cloud Run tutorial
If you need help deciding between those, check this:
Design an optimal storage strategy for your cloud workload
NOTE: At the time of this answer, Cloud Run gen2 is in Preview.
From a Google Cloud Run docker registry associated container, when I try to mount a Google Storage Bucket, the following is what I receive. Obviously without having a privileged docker execution this is expected, and as far as I have investigated, "Google Cloud Run" instances are not meant to support privileged container execution like Google Compute Engine.
Yet I am still asking if anyone has any other knowledge about this, is there any other way to mount a bucket via Google Run container ?
Opening GCS connection... Opening bucket... Mounting file system...
daemonize.Run: readFromProcess: sub-process: mountWithArgs:
mountWithConn: Mount: mount: running fusermount: exit status 1
stderr:
fusermount: fuse device not found, try 'modprobe fuse' first
Posting this as a Community Wiki as it's based on the comments of #JohnHanley and #SuperEye:
Based on what you mentioned:
My docker images are not web services.
If this is the case, you cannot use Cloud Run for what you are trying to do. Cloud Run is an HTTP Request/Response system. Your container must respond to HTTP requests, otherwise it will be terminated.
Also, for your other comment:
Google Compute Engine cannot run docker images from Container Registry
That is an incorrect assumption. Compute Engine supports Container Registry.
In conclusion, for your final goal of mounting a bucket as a file system, Cloud Run does not support that ability. An alternative is to use App Engine Flex.
According to the docs:
Cloud Shell provisions 5 GB of free persistent disk storage mounted as your $HOME directory on the virtual machine instance.
I would need more (paid) storage though that I can access from the Cloud Shell environment and that is persistent across my sessions. It's mostly used to store local clones of git repositories and images. I would be the only one to access these files.
It seems that the 5 GB storage is a hard limit, so it won't expand dynamically and bill me for the exceeding amount. It is possible to use Boost Mode, but that does not affect the storage size. And I also can't provision more storage with a custom Cloud Shell environment. I couldn't figure out if I can mount another GCE persistent disk to my $HOME. I was considering gcs-fuse as suggested in this answer but I'm not sure if it is suitable for git repos.
Is there any way to have more storage available in Cloud Shell?
Google Cloud Shell is a container that runs on a hidden Compute Engine instance managed by Google. You can download, modify and redeploy this container to Cloud Shell or to your own container running in the cloud or on your desktop.
The base image of the container is available at gcr.io/cloudshell-images/cloudshell:latest, per this page.
For your use case, I would use Compute Engine with Container OS and run the Cloud Shell container within COS. You can scale the CPUs, memory, and storage to fit your requirements.
You can also set up a Compute Engine instance, install the CLIs, SDKs, and tools and have a more powerful system.
Notes for future readers based upon the first answer:
Filestore is a great product, but pay attention to costs. The minimum deployment is 1 TB at $200+ per month. You will need to mount the NFS share each time Cloud Shell restarts - this can be put into login scripts. Note: I am not sure if you can actually mount an NFS share from Filestore in Cloud Shell. I have never tested this.
You will have the same remount problem with FUSE, plus you will have bandwidth costs to access Cloud Storage.
Cloud Shell is a great product that is well implemented, but when you need to exceed its capabilities it is better to deploy a small/medium size GCE instance. this enables persistent, snapshots, etc.
There is another way to have more disk space in the cloud shell. It's to create a cloud storage bucket and map the cloud storage bucket as a folder. This way you can store larger files in the cloud storage bucket and it doesn't require any compute instance.
Go to cloud storage and create a new storage bucket
Copy the storage bucket's name, eg. my_storage_bucket
Go to cloud shell and create a folder in your home folder
mkdir ~/my_bucket_folder
Mount the storage bucket to this folder
gcsfuse my_storage_bucket ~/my_bucket_folder
Change directory to your my_bucket_folder
cd ~/my_bucket_folder
Voila! you have unlimited space!
To unmount please run the following
fusermount -u ~/my_bucket_folder
I'm using gcsfuse and works fine. You don't have to remount every time if you put the mount command in .customize_environment (run on boot up).
#!/bin/sh
#.customize_environmnet run in background as root, wait for your user to initialize
sleep 20
sudo -u [USER] gcsfuse -o nonempty -file-mode=777 -dir-mode=777 --uid=1000 --debug_gcs [BUCKET_NAME] /home/[USER]/[FOLDER_NAME]
You can read more at Unlimited persistent disk in google cloud shell
There is no way of adding more storage to the Cloud Shell. You can create a VM and install the Cloud SDK and have as much storage as you'd like but it is not currently possible to add storage space to the Cloud Shell.
Depending on how you plan on using the saved repos, Cloud Storage may be ideal as it has a storage category just perfect archiving.
Filestore will be your best option as it is great for file systems and it is scalable. It fits your needs as you have described.
You can use Cloud Storage with FUSE. Keep in mind that this method, although great, depends on how it will be used as costs are based on storage category.
You can see a brief comparison of the Storage solutions the Cloud Platform has to offer here.