I am using Google Compute Engine VM for a web server. I currently have 3TB as disk space but want to bring it down to 1TB. Can someone tell me where I can do this from? Thanks.
As #John Hanley only increasing disk size is supported by Google Cloud:
gcloud compute disks resize resizes a Compute Engine disk(s).
Only increasing disk size is supported. Disks can be resized regardless of whether hey are attached.
I also found an answer in Serverfault that could help you with this topic here.
It uses fsarchiver tool in order to manage the boot disk resizing:
If the disk is not a boot disk you can do the following:
add a new disk with the required size and format it.
mount the new disk
cp -r --preserve=all /mnt/disk1/. /mnt/disk2/
edit the /etc/fstab to mount the new disk instead the old one
If you have standard disk and you want to shorten the cp time. You can
first create new ssd disk from snapshot and copy it to 2T ssd disk.
Then make a snapshot from the 2T disk and create a new 2T standard
disk.
If your disk is a boot disk, you can use a tool like fsarchiver:
Create an archive from the boot disk. fsarchiver savefs /mnt/backup/boot_disk.fsa /dev/sda
Restore the archive on the new disk fsarchiver restfs /mnt/backup/boot_disk.fsa id=0,dest=/dev/sdb
Related
I am using the AI platform of GCP. I need to choose the size of a boot disk and a data disk during the creation of the VM instance please see this attached image.
I have 500GB of data to be analysed in the AI platform. Should I choose more than 500GB disk size for both the boot disk and the data disk, OR is it enough to increase only one of the disk sizes (e.g., data disk size to be 500GB)? I am also not sure about the different roles between boot disk and data disk. Thank you.
When you create a new Notebook instance using Notebooks API, the API will create a VM with 2 disks:
Boot disk where the OS/libraries/initialization scripts live
Data disk which is mapped to /home/jupyter folder.
The reason to do this is to support Upgrade, where you can upgrade your VM by replacing the Boot disk with a new OS while preserving your data.
To answer your question, I would allocate space in data disk.
It depends!!
You have the boot disk dedicated to the boot system. 100Gb is generally enough for this task
Data disk can be used to store your data. And it's depends on the volume of your data.
However, your data disk can store your data, it's not mandatory, it depends how you store your data. You can store your data on the boot disk. it's not a best practice because, most of the time, you can consider that the boot disk is dependent to the VM, the data not.
Imagine this process: you create a small VM, store your data in the data disk and create your model and training loop. When done, you delete your VM, create a bigger one (with GPUs for example) and attach your data disk. No need to reload the data, your model and so on, to train your model with this big VM
I am running samtools on a google VM with 8CPUs. It seems that when the process is finished, the program crashes giving the below error. At the same time, there is a problem with the bucket, showing this. Any ideas? Problems with saving the file?
Error:
username#instance-1:~/my_bucket$ /usr/local/bin/bin/samtools view -#20 -O sam -f 4 file_dedup.realign
ed.cram > file.unmapped.sam
samtools view: error closing standard output: -1
Also this comes up when tying ls in the bucket directory:
ls: cannot open directory '.': Transport endpoint is not connected
As we discovered at the comment section this issue is related to the difference between a FUSE and a POSIX file systems.
You can solve this issue in two ways:
Increase disk space on your VM instance (by following the documentation Resize the disk and Resize the file system and partitions) and stop using Google Cloud Storage Bucket mounted via FUSE.
Save data received from samtools to the VM's disk at first and then move them to the Google Cloud Storage Bucket mounted via FUSE.
You can estimate cost for each scenario with Google Cloud Pricing Calculator.
Keep in mind that persistent disks have restrictions, among them:
Each persistent disk can be up to 64 TB in size, so there is no need to manage arrays of disks to create large logical volumes.
Most instances can have up to 128 persistent disks and up to 257 TB of total persistent disk space attached. Total persistent disk space
for an instance includes the size of the boot persistent disk.
In addition, please have a look Quotas & limits for Google Cloud Storage.
I am working in Google cloud platform and I have a situation where I need to attach local SSD disk to a existing VM machine but I cannot find a way to do it, Basically this is the process I need to do.
1. Attach a new local SSD disk
2. Copy the existing data to new disk
4. Unmount the old disk
5. Mount new disk to old data path
Thanks.
As John said, you can't if you initially created your VM without local SSD. But you can do something like this
Go to the console, select your current VM
Go the the boot disk section and select your boot disk
Create an image of the boot disk
Go back to the current VM page
Click on "create similar" button, on the top
Select the boot disk image that you just created
Add additional disk, with type local scratch SSD
Now you have a similar VM, with the same boot disk image but with local SSD.
Then, you have to detach the existing persistent disk on the old VM and attach it to the new one, make your copy, and delete it.
This is the situation:
I hooked a disk to a VM. Reboot the VM (for any reasons). I have to remount the disk otherwise it is not available (unmounted) after the restart. So I remount the disk with the following command: sudo mount -o discard, defaults /dev/[DEVICE_ID] /mnt/disks/[MNT_DIR]
Does the fact that I have to remount the disk also mean that I have lost all the data inside?
Thanks in advance
The document that you shared with us says:
"If you detach this zonal persistent disk or create a snapshot from the boot disk for this instance, edit the /etc/fstab file and remove the entry for this zonal persistent disk"
Therefore, if you are not creating snapshots from the BOOTDISK you can reboot your instance without having any issue with your data.
However, if you are using snapshots or schedule snapshots of your SSD disk, I would recommend you to follow these best practises to create it :
https://cloud.google.com/compute/docs/disks/snapshot-best-practices
But also you can do persistent disk snapshots at any time without unmounting your disk. These recommendations are only in order to have a greater reliability and create the snapshots more quickly (this is also explained in the documentation: https://cloud.google.com/compute/docs/disks/snapshot-best-practices#prepare_for_consistency)
In the document that you linked, there is a description of how to add your mount point to /etc/fstab. Using the command line sudo mount -o ... mounts the disk temporarily, but the mount will be lost across reboots. Editing the /etc/fstab will cause the mount point to persist across reboots because that file is read during startup.
Thanks a lot for your answers. I'm sorry for my question that's not complete. I'm developer and I new in sysadmin.
As you can see here, I added "zonal persistent disk" (type permanent ssd disk) to my VM on Compute Engine (https://cloud.google.com/compute/docs/disks/add-persistent-disk).
Here says if I have a schedule snaphots it's no possible set automatically mount the disk to my VM after restart (that I may need for any reasons). So the question: how I sure that the restart, in addition to unmount the disc, it won't lose the data?
Having disk snapshots I would still be able to restore the data, but I would still understand what happens in this case. Meantime I read your suggestions on linux mount and I have understand that I will not lose the data on the disk with restart of the machine.
Thanks
My Google compute engine instance disk size is dramatically different from my snapshot size created from this instance. My used disk space is under 40GB, while snapshot size is 123 GB. Below is illustration. My instance is Ubunt 16.04
Here is output of df command
Here is size of snapshot created from this instance:
I expect them to be of approximately the same size. Do you need clear trash bin before creating snapshot or do something else?
Google cloud snapshot size will change depending on the changes made to datas as only the first snapshot will be a full disk snapshot and then all the other will be differential.
This mean that depending on the actions made on the disk you could have a snapshot containing only 1GB of change and the next one could be 200GB for the same 1TB disk.
Google will maintain consistency between snapshot, you can find more information here: https://cloud.google.com/compute/docs/disks/restore-and-delete-snapshots#deleting_snapshot