Extend GlusterFS on top of LVM - centos7

I need to add more space to one of our gluster volumes. The volumes are replica 2 and sit on top of an LVM. The file system is XFS. The current size is 4TB and I want to resize to 6TB. The LVM has enough Free PEs on both replica servers.
--- Physical volume ---
PV Name /dev/sdb
VG Name gluster
PV Size 10,91 TiB / not usable 4,00 MiB
Allocatable yes
PE Size 4,00 MiB
Total PE 2861183
Free PE 1633407
Allocated PE 1227776
PV UUID F3CwNm-dceK-ezPY-7w12-OYT5-FLAH-U0a239
-
--- Physical volume ---
PV Name /dev/sdb
VG Name gluster
PV Size 10,91 TiB / not usable 4,00 MiB
Allocatable yes
PE Size 4,00 MiB
Total PE 2861183
Free PE 1618047
Allocated PE 1243136
PV UUID dWDEgF-0brq-9e6r-eqpO-jTeK-GJfb-c3MGbE
I've read somewhere, that it's enough to extend the LVM and to resize the FS on both hosts.
# lvextend -L +2T <lvm>
# xfs_growfs <lvm mountpoint>
I know that XFS has to be reseize while it's mounted. The LVM can also be resized during operations (although not recommended). And I've read somewhere that GlusterFS will automatically adapt to the new volume size as soon as both/all volumes have the new size.
Since the storage is used in an productive environment it's important to do this on the fly.
Has anyone any experience with this combination or can confirm that my approach is correct?
Thanks in advance.

To answer my own question:
Eventually we had to increase Gluster storage without downtime.
After extending the logical volume of all Gluster bricks to the same size and growing the xfs filesystem to the maximum size of the volume, the Gluster volume automatically adjusted the new size.
I'm not sure if it's the recommended way, but it worked for us like a charm (both replica 2 and replica 3 volumes).

According to the https://access.redhat.com/solutions/1517993 if you want, you can grow a glusterfs volume by growing its bricks rather than adding new ones.
You can do it following a usual file system resizing method (the method will depend on the underlying block device).
Example : the brick is set on an XFS file system on the LV VGgluster/brick1. We want to add another 500MB to this brick.
Check that you have free space on the VG containing the brick :
~~~
# vgs VGgluster
~~~
Grow the file system :
~~~
# lvextend --resizefs -L+500M VGgluster/brick1
~~~
The size has been updated on the client :
~~~
# df /glusterfs/mountpoint
From my experience: if it's a replica volume, you have to resize all volume's bricks. Until all bricks have the same size, the actual size of particular gluster volume will be equal the size of the smallest brick at the moment.

Related

Why df -h and lsblk show different sizes of my one and only xvda1?

So when i do lsblk, it shows that my xvda1 size is 10 GB. but when i do df -h, it shows that my xvda1 size is 7.7 GB
because the commands work differently.
lsblk: Use lsblk command to view your available disk devices and their mount points (if applicable) to help you determine the correct device name to use. The output of lsblk removes the /dev/ prefix from full device paths. It tells you the size of volume and partition(which in your case both 10 GB).
df-h:Use df -h command to verify the size of the file system for each volume. Sometimes the size of the filesystem might be default, so you need to extend that using resize2fs or xfs_growfs commands depending on the type of file system.
For more details please check: recognize-expanded-volume-linux

Reading directory contents in FAT32

I am trying to write bare metal FAT32 file system driver on RPI 3B.
I am able to read FAT sectors and root directory sectors using emmc driver.
I have doubt on how to follow FAT entries linked list when next entry pointer (next cluster number) doesn't fit in the current FAT sector?
Should I read the FAT sector each time I get to the new cluster number?
My current understanding is as follows:
Get first cluster number (cluster_number) of directory/file
Read the FAT sector which contains first_cluster_number entry.
Say I read FAT sector as
uint8_t fat_sector[512] = { 0 };
uint32_t this_fat_sector_num, this_fat_entry_offset;
this_fat_sector_num = unusedSectors + reservedSectorCount + ((cluster_number * 4)/ bytesPerSector);
this_fat_entry_offset = (cluster_number * 4)/ bytesPerSector;
read_fat_sector(this_fat_sector_num, & fat_sector[0]);
// Calculate next cluster in chain
uint32_t next_cluster_number = ((uint32_t * fat_sector[this_fat_entry_offset])) & 0x0fffffff;
// Calculate next cluster in chain 1 more time, is below code correct ?
uint32_t next_next_cluster_number = ((uint32_t * fat_sector[next_cluster_number])) & 0x0fffffff;
What happens when the next cluster number is not present in already read fat_sector buffer (512 bytes)?
If cluster number = index of next entry in fat_sector, Do I need to multiply it by 4 given that fat 32 entries span 4 bytes.
If anyone could give some clarity, that will be helpful. Thanks in advance.
Implement a cache (in RAM) of the FAT. Let's say that the cache has enough RAM for 20 sectors and starts out empty.
Next write a "getFATentry" function that checks if the sector is in the cache and finds the right entry in the cache if it is; or (if necessary) evicts something from the cache to make room, fetches the right sector from disk into the cache, then finds the right entry in the cache.
Once that's done you can next_cluster = getFATentry(previous_cluster); without worrying about the cache or any disk IO (but will want to do something when the FAT is modified - e.g. adopt a "write-through" or "write-back" policy).
Note: By adjusting the size of the "FAT cache" you can improve performance or reduce RAM consumption. It'd be nice to allow the cache to grow/shrink dynamically (e.g. grow to be as large as the whole FAT if nothing else needs RAM, but shrink to bare minimum if all the RAM is needed for something else).
I have found solution.
First read the initial fat sector for given cluster number.
Find out thisFatEntryOffset and read the next fat entry.
New fat entry will be new cluster number. Find out the thisFatNumber and thisFatEntryOffset for new cluster number.
If new fat sector != old fat sector then read new fat sector and read entry using thisFatEntryOffset.

Add unallocated free space to LVM

I added 5 GB to primary disk under VMware. I can see 5GB free :
Number Start End Size Type File system Flags
32.3kB 1049kB 1016kB Free Space
1 1049kB 256MB 255MB primary ext2 boot
256MB 257MB 1048kB Free Space
2 257MB 215GB 214GB extended
5 257MB 215GB 214GB logical lvm
215GB 220GB 5370MB Free Space
How can I add this free space to my group volume ? I need create a new partition first?
Yes, you need to create a partition first and change partition's system id for LVM (code 8e).
Example:
fdisk /dev/sdb #Assuming this is the disk you added
press "n" #to create a new partition and following along.
press "t" #change the partition's system id to "8e"
press "w" #to write the changes
#Create PV.
pvcreate /dev/sdb1
#Add PV to the Volume Group
vgextend <VolumeGroup> /dev/sdb1
#Extend the logical volume
lvextend -l +100%FREE /dev/VolumeGroup/yourLV
Make sure the LV is not mounted when you extend it. Run "lvdisplay" to ensure the Logical Volume extended.
NOTE: If the filesystem doesn't expand you may try rebooting the machine or do fsck and then resize.

Correcting VirtualBox raw disk VMDK file after resizing a partition

I have a Windows 8.1 installation on second partition of my second HDD (/dev/sdb2 in Ubuntu) created using the command
VBoxManage internalcommands createrawvmdk -filename sdb2.vmdk -rawdisk /dev/sdb -partitions 2
Everything worked just fine - Windows installation was runable from VirtualBox and even bootable normally from GRUB. Last time when installing some software in Windows (PC booted up directly into Windows), I discovered there's not enough space on the system partition (/dev/sdb2) and enlarged it by 15 GB that were left spare on the HDD.
These changes made the Windows installation unusable in VirtualBox of course - it fails to boot offering some repair options. The first thing which I realized that is needed to do was enlarging the partition in the VMDK file, so I backed up the old sdb2.vmdk and sdb2-pt.vmdk files and recreated them with the same command as before.
This, however, made no change, because sdb2-pt.vmdk seems to be storing the boot record (MBR in my case, currently with GRUB) and some more stuff needed for Windows to work properly. My next attempt was replacing the new sdb2-pt.vmdk with the old one (with Windows bootloader and perhaps the old partition table) - this didn't work either.
How to update the VMDK files with the new partition size to make the enlarged Windows 8.1 installation bootable from VirtualBox again?
I have finally found the solution myself. Since the VBoxManage internalcommands createrawvmdk -filename sdb2.vmdk -rawdisk /dev/sdb -partitions 2 command produces two valid files based on the current disk structure, the only change needed was to recover the Windows boot loader from the old sdb2-pt.vmdk file which is a rather straightforward process. If you only wish to learn the recovery steps, you can skip the following theoretical part.
Some background information on the VMDK file format
VMWare Disk Format (VMDK) consists of two files - a descriptor file (sdb2.vmdk in the original question) and an extent file (sdb2-pt.vmdk). Their internal structure is well defined in the specification from VMWare. I'll sum up the most important parts:
The descriptor file (sdb2.vmdk) contains a section annotated # Extent description which can look something like this:
# Extent description
RW 63 FLAT "sdb2-pt.vmdk" 0
RW 41943040 ZERO
RW 83886080 FLAT "/dev/sdb" 58722304
RW 2 FLAT "sdb2-pt.vmdk" 63
RW 1191843568 ZERO
One extent description (a row from those above) has the following structure:
Access Size in sectors Type of extent Filename (Offset)
The offset parameter (specified only for FLAT type extents) specifies the offset (in sectors) of the given extent within the file Filename. Notice that file sdb2-pt.vmdk consists of two extents, the first 63 sectors long and the second only 2 sectors long.
The FLAT extent file sdb2-pt.vmdk is a raw data binary file identical to one you would obtain e.g. using the dd command on Unix-like systems. Since the sector size was 512 bytes in my case (I don't know if this is a general rule), the sdb2-pt.vmdk file (based on the new disk partitioning described in the extent description above) was (63+2)*512 bytes long.
Now to the second extent (the one with only 2 sectors in size). This is a padding extent which arose in my new partition table after enlarging the Windows partition (third extent in the description table). Since my previous partition table did not contain any such padding, the old sdb2-pt.vmdk file only contained the first 63 sectors long extent and thus was 1 024 bytes smaller than the new one generated by the VBoxManage internalcommands createrawvmdk -filename sdb2.vmdk -rawdisk /dev/sdb -partitions 2 command. This obviously rendered the old extent file and the new one incompatible.
The recovery process
Please be aware that the following steps apply to the old MBR disk structure only!
You surely want to keep the new partition structure and to propagate any changes made in the partition table to the VMDK file. Proceed with these steps:
Backup your old description file (sdb2.vmdk) and extent file (sdb2-pt.vmdk). In the following steps, you will only need the second one but you never know what else could happen.
Generate new descriptor and extent files issuing the command:
VBoxManage internalcommands createrawvmdk -filename sdb2.vmdk -rawdisk /dev/sdb -partitions 2
Now, the first extent entry in your new description file (sdb2.vmdk) should look like this:
RW ## FLAT "sdb2-pt.vmdk" 0
With the knowledge that you want to keep the new partition table (with everything what follows) and only restore the Windows boot loader stored in the backed up extent file (old sdb2-pt.vmdk), you have to copy the first 440 bytes (boot loader) from the old extent file to the new one. This can either be done with a hex editor (copy all values starting from address 0x0 up to 0x1B8 exclusive) or on a Unix-like system using the command:
dd if=old-sdb2-pt.vmdk of=sdb2-pt.vmdk bs=1 count=440
Violà.
On github there is a tool that will do that automatically (and re-running with same options will update its vmdk and auxiliary files, so you can change partitions too later) https://github.com/vasi/vmdk-raw-parts

LVM: Extending Physical Volume (under VMWare)

I have a virtual server under VMware, where I got 10 more GB harddisk from the sysadm.
Inside Linux (running Ubuntu 11.04 server), things are set up with LVM. In [c]fdisk, I do find these extra 10 gigs - they are present directly after the primary LVM partition.
I understand that I could make a new LVM partition out of this extra space, and then add this to the volume group, and then to the logical volume, and then resize the filesystem.
However, is it possible to assimilate these extra gigs into the existing LVM partition? So that the LVM's view of the physical volume increased instead of adding another physical volume into the volume group (preferably automatically percolating up to the volume group)?
I just built a test virtual machine with a 15GB disk and LVM on /dev/sda2. Then I grew the disk to 20GB and made LVM see the extra 5GB without adding another physical volume.
Here are the steps I followed:
apt-get install gnu-fdisk (or yum install gnu-fdisk)
I had to use gfdisk to make the whole thing work. No luck with the "standard" fdisk.
gfdisk /dev/sda
Switch to "sectors" as the unit (this is critical!!!) and print the partition table:
Command (m for help): u
Changing display/entry units to sectors
Command (m for help): p
Disk /dev/sda: 21 GB, 21089617920 bytes
255 heads, 63 sectors/track, 2564 cylinders, total 41190660 sectors
Units = sectors of 1 * 512 = 512 bytes
Device Boot Start End Blocks Id System
/dev/sda1 * 2048 499711 257008 83 Linux
Warning: Partition 1 does not end on cylinder boundary.
/dev/sda2 501758 29798632 14643247 8e Linux LVM
Warning: Partition 2 does not end on cylinder boundary.
Command (m for help):
Write down the "start" sector of the "Linux LVM" partition (/dev/vda2).
Delete the partition and recreate it using the same "start" sector (501758) and the same partition type (8e):
Command (m for help): d
Partition number (1-2): 2
Command (m for help): n
Partition type
e extended
p primary partition (1-4)
p
First sector (default 63s): 501758
Last sector or +size or +sizeMB or +sizeKB (default 41190659s):
Command (m for help): p
Disk /dev/sda: 21 GB, 21089617920 bytes
255 heads, 63 sectors/track, 2564 cylinders, total 41190660 sectors
Units = sectors of 1 * 512 = 512 bytes
Device Boot Start End Blocks Id System
/dev/sda1 * 2048 499711 257008 83 Linux
Warning: Partition 1 does not end on cylinder boundary.
/dev/sda2 501758 41190659 20338290 83 Linux
Command (m for help): t
Partition number (1-2): 2
Hex code (type L to list codes): 8e
Changed type of partition 2 to 8e (Linux LVM)
Command (m for help):
WARNING: please note that I didn't accept the default start sector for the partition, I entered it manually so that it matches the original value! I did accept the default value for the "last sector" though, because I want this partition to be as large as the disk.
Verify that you did everything correctly with "p" and write the new partition table to disk:
Command (m for help): w
Reboot the virtual machine.
Now log-in the virtual machine and run:
root#git:~# pvresize /dev/sda2
Physical volume "/dev/sda2" changed
1 physical volume(s) resized / 0 physical volume(s) not resized
Done! Now run vgdisplay and you will see the extra 5GB available as free extents.
CAVEATS:
If LVM is not the last (or only) partition on the virtual machine disk, you might have an hard time extending the size of the partition. It might even be impossible.
If LVM is on a logical partition (which is where Debian puts it by default at install time) or in other words if LVM is on /dev/sda5 instead of /dev/sda2, you must write down the starting sector of both the extended partition (let's call it /dev/sda2) and the logical partition (let's call it /dev/sda5), then delete both partitions, then recreate them with the same starting sector. The last sector of both /dev/sda2 and /dev/sda5 should be the last sector of the disk.
Since this is a risky procedure I recommend doing a backup of the virtual machine before attempting it.
Here are the instructions I always use when I'm doing the same thing...
Do “fdisk –l /dev/sda” I’m assuming /dev/sda2 is your LVM partition
and you don’t have a /dev/sda3. Adjust the instructions to match your config.
1) Make sure you have expanded the VM disk.
2) do “fdisk /dev/sda”
p (to see existing partitions)
n (new partition)
p (primary partition) -- give it the next available partition number, probably 3
accept the defaults. It will use the rest of the disk
w (write changes)
3) REBOOT
4) run ‘pvcreate /dev/sda3’
5) run ‘vgdisplay’ to get the name of the volume group. In this example, it’s “VolGroup”, as in:
6) run vgextend VolGroup /dev/sda3
7) run “df” to see the name of the root volume group.
8) run lvextend /dev/mapper/VolGroup-lv_root -r -l+100%FREE
( the (-r) option causes it to format it, too)
9 run “df –h” to see your new disk space
This is one of the reasons I prefer to have one big physical volume on the VMs that is separate from the /boot and other initial parts. This allows you to create a whole disk as an LVM physical disk (pvcreate /dev/sdb). Then if you need to do a resize, none of the file system changes need to happen because it is directly on disk.
In that scenario, you simply run pvresize /dev/sdb and move on with your vgextend, lvextend and resize2fs. Much simpler to maintain.
Personally I followed the instruction in this blog.
To summarize the steps:
Increase the virtual disk size using the vm-ware tools vmware-vdiskmanager
Boot on another system (GParted live CD or mount an Iso CD-ROM to boot on)
Use the Gparted tools (easy and graphic interface) to increase the recently expended drive
remove the CD-ROM (or the ISO) and boot on the original partition.
For expanding my Linux VM (RedHat EL 6) I did boot on the GParted ISO image (from here) and I was able to extend the partition I wanted. Note I had to move the partition I wanted unchanged at the end.
It was safe with one of my colleague as well, but I would recommand to backup the VM before running Gparted Linux.
extend disk without reboot
echo 1 > /sys/block/sda/device/rescan
echo 1 > /sys/block/sdb/device/rescan
partprobe
gdisk fix warnging
parted change partion size
## parted can executed as command line. but this is very dangerous
parted -s /dev/sdb "resizepart 2 -1" quit
parted -s /dev/sdb "resizepart 3 100%" quit
resizepart 3 100%
pvresize /dev/sda3
lvextend -l +100%FREE cs/root
xfs_growfs /dev/cs/root