I had an issue recently with logs on an external Platform Services Controller using up over 90% of the available free space on the volume. In the past, when I’ve needed to extend disk space on a VCSA I have followed this VMware KB which instructs how to extend disk space using ‘vpxd_servicecfg storage lvm autogrow’, so I thought I’d use the same process for a PSC. However, when I attempted to extend the volume, I found that the ‘vpxd_servicecfg’ script isn’t available on the PSC. Seeing as I’d already increased the size of the VMDK on the PSC, I decided to have a go at extending the volume using native linux tools.
I had started by increasing the VMDK for the log volume (VMDK 5 – see KB mentioned above), from 10GB to 15GB. Running ‘df -h’ in a shell session on the PSC shows it’s current size:
v6-psc:~ # df -h Filesystem Size Used Avail Use% Mounted on /dev/sda3 11G 2.3G 7.9G 23% / udev 1004M 164K 1004M 1% /dev tmpfs 1004M 32K 1004M 1% /dev/shm /dev/sda1 128M 38M 84M 31% /boot /dev/mapper/core_vg-core 5.0G 139M 4.6G 3% /storage/core /dev/mapper/log_vg-log 9.9G 381M 9.0G 4% /storage/log /dev/mapper/db_vg-db 9.9G 159M 9.2G 2% /storage/db /dev/mapper/dblog_vg-dblog 7.8M 1.1M 6.3M 15% /storage/dblog /dev/mapper/seat_vg-seat 7.8M 1.1M 6.3M 15% /storage/seat /dev/mapper/netdump_vg-netdump 7.8M 1.1M 6.3M 15% /storage/netdump /dev/mapper/autodeploy_vg-autodeploy 7.8M 1.1M 6.3M 15% /storage/autodeploy /dev/mapper/invsvc_vg-invsvc 7.8M 1.1M 6.3M 15% /storage/invsvc
We can check the disk the /dev/mapper/log_vg-log volume is on using fdisk. We already know it is the 5th disk, so the identifier is ‘sde’:
v6-psc:/ # fdisk -l /dev/sde Disk /dev/sde: 10.7 GB, 10737418240 bytes 255 heads, 63 sectors/track, 1305 cylinders, total 20971520 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000
The output shows that this matches the expected size of the disk. We can double check this by using the ‘pvdisplay’ command:
v6-psc:/ # pvdisplay /dev/sde --- Physical volume --- PV Name /dev/sde VG Name log_vg PV Size 10.00 GiB / not usable 7.00 MiB Allocatable yes (but full) PE Size 8.00 MiB Total PE 1279 Free PE 0 Allocated PE 1279 PV UUID FI2Dyl-UwXS-Thro-kDaz-h5P0-cHA0-HJoV25
This shows we have a volume group called ‘log_vg’, created on /dev/sde. The system is currently showing the size of this physical volume to be 10GB. I’ve already changed the size of the disk to 15GB, so we need to issue a command to have the system rescan it’s disks:
v6-psc:/ # rescan-scsi-bus.sh -w --forcerescan
Syncing file systems
Scanning SCSI subsystem for new devices and remove devices that have disappeared
Scanning host 0 for SCSI target IDs 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15, all LUNs
Scanning for device 0 0 0 0 ...
OLD: Host: scsi0 Channel: 00 Id: 00 Lun: 00
Vendor: VMware Model: Virtual disk Rev: 1.0
..........
‘rescan-scsi-bus.sh’ is a script found on linux distributions which allows you to scan for SCSI devices (or re-scan), without having to reboot the OS. With that done, running the fdisk command again, we can see that the disk is now 15GB:
v6-psc:/ # fdisk -l /dev/sde Disk /dev/sde: 16.1 GB, 16106127360 bytes 255 heads, 63 sectors/track, 1958 cylinders, total 31457280 sectors Units = sectors of 1 * 512 = 512 bytes
Next we need to extend the physical volume to make use of the extra capacity:
v6-psc:/ # pvresize /dev/sde Physical volume "/dev/sde" changed 1 physical volume(s) resized / 0 physical volume(s) not resized
Checking the properties of the physical volume now, it is 15 GB:
v6-psc:/ # pvdisplay /dev/sde --- Physical volume --- PV Name /dev/sde VG Name log_vg PV Size 15.00 GiB / not usable 7.00 MiB Allocatable yes
Finally, we need to extend the logical volume. We can check the logical volumes available using the lvdisplay command:
v6-psc:/ # lvdisplay --- Logical volume --- LV Name /dev/log_vg/log VG Name log_vg LV UUID UfBZR7-KuZP-nuHj-wc1M-hGfg-K7SZ-WowreB LV Write Access read/write LV Creation host, time localhost.localdom, 2016-05-05 10:03:54 +0000 LV Status available # open 1 LV Size 9.99 GiB Current LE 1279 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 1024 Block device 253:6
To resize the logical volume, run:
v6-psc:/ # lvresize --resizefs -l +100%FREE /dev/mapper/log_vg-log Extending logical volume log to 14.99 GiB Logical volume log successfully resized resize2fs 1.41.9 (22-Aug-2009) Filesystem at /dev/mapper/log_vg-log is mounted on /storage/log; on-line resizing required old desc_blocks = 1, new_desc_blocks = 1 Performing an on-line resize of /dev/mapper/log_vg-log to 3930112 (4k) blocks. The filesystem on /dev/mapper/log_vg-log is now 3930112 blocks long.
This command resizes the volume to use 100% of the available free space. Once done, the additional space should be available:
v6-psc:/ # df -h Filesystem Size Used Avail Use% Mounted on /dev/sda3 11G 2.3G 7.9G 23% / udev 1004M 164K 1004M 1% /dev tmpfs 1004M 32K 1004M 1% /dev/shm /dev/sda1 128M 38M 84M 31% /boot /dev/mapper/core_vg-core 5.0G 139M 4.6G 3% /storage/core /dev/mapper/log_vg-log 15G 384M 14G 3% /storage/log /dev/mapper/db_vg-db 9.9G 159M 9.2G 2% /storage/db /dev/mapper/dblog_vg-dblog 7.8M 1.1M 6.3M 15% /storage/dblog /dev/mapper/seat_vg-seat 7.8M 1.1M 6.3M 15% /storage/seat /dev/mapper/netdump_vg-netdump 7.8M 1.1M 6.3M 15% /storage/netdump /dev/mapper/autodeploy_vg-autodeploy 7.8M 1.1M 6.3M 15% /storage/autodeploy /dev/mapper/invsvc_vg-invsvc 7.8M 1.1M 6.3M 15% /storage/invsvc
Note that whilst this process appears to work, I’m not sure that it’s a supported method – I haven’t seen a KB article that details this process and I haven’t used this in a production environment, just in my test lab. There is a KB article that discusses space issues for the /storage/log volume on an external PSC however, with details on how to alter some log rotation settings – so worth a look if you’re experiencing free space issues.
For more info on working with LVM in general, check out this tutorial.