top of page
  • viquarmca

VCF 4.2.1 SDDC Going down due to root partition issue

In this blog we will take about my experience with VCF 4.2.1 GUI down issue, it was very wired that the GUI was not loading . After a lot of troubleshooting found the issue . This issue has been fixed in VCF 4.4



  • When check the SDDC VM , we found the root /var/log has Audit.log 12GB and Auth.log 12GB

  • df -h

Filesystem Size Used Avail Use% Mounted on

devtmpfs 7.9G 0 7.9G 0% /dev

tmpfs 7.9G 4.0K 7.9G 1% /dev/shm

tmpfs 7.9G 298M 7.6G 4% /run

tmpfs 7.9G 0 7.9G 0% /sys/fs/cgroup

/dev/sda4 28G 28G 0 100% /

tmpfs 7.9G 64K 7.9G 1% /tmp

/dev/sda2 120M 27M 85M 24% /boot

/dev/mapper/data_vg-data 7.9G 185M 7.3G 3% /data

/dev/mapper/lcmmount_vg-nfs--mount 503G 81G 397G 17% /nfs/vmware/vcf/nfs-mount

/dev/mapper/data_vg-vmware 63G 3.6G 56G 7% /var/log/vmware

/dev/mapper/data_vg-vcf 181G 1.7G 170G 1% /opt/vmware/vcf

tmpfs1.6G01.6G0% /run/user/1000


  • cd /var/log/

  • df -h .

Filesystem Size Used Avail Use% Mounted on

/dev/sda4 28G 28G 0 100% /


root@vcfsddc[ /var/log ]# ls -l

total 12610320

drwx------ 2 root root 4096 Feb 3 2021 audit

-rw------- 1 root root 12884303872 Aug 12 12:23 auth.log

-rw-r--r-- 1 root root 833 May 7 12:19 boot.log

-rw-rw---- 1 root utmp 11520 Aug 10 15:58 btmp

-rw------- 1 root root 4624384 Aug 12 12:20 cron

drwx------ 2 root root 4096 Aug 9 04:01 devicelist

lrwxrwxrwx 1 root root 30 Jan 21 2021 dracut.log -> /var/opt/dracut/log/dracut.log

-rw-r--r-- 1 root root 0 Feb 3 2021 installer-kickstart.log

-rw-r--r-- 1 root root 0 Feb 3 2021 installer.log

drwxr-sr-x 3 root systemd-journal 4096 Jun 16 08:20 journal

-rw-rw-r-- 1 root utmp 292292 Aug 12 12:23 lastlog

drwxr-x--- 2 root root 4096 Aug 11 18:10 loginsight-agent

-rw-r--r-- 1 root root 2519040 Aug 12 12:20 maillog

-rw-r----- 1 root root 1478656 Aug 12 12:26 messages

drwxr-xr-x 2 root root 4096 Jun 24 03:15 nginx

drwx------ 2 postgres users 12288 Aug 10 00:00 postgres

drwx------ 2 root root 4096 Feb 3 2021 private

drwx------ 2 root root 4096 Aug 9 04:01 rpmcheck

-rw-r--r-- 1 root root 19582976 Aug 12 12:20 sa

drwx------ 2 root root 4096 Aug 9 04:01 sgidlist

-rw------- 1 root root 11297 Jun 16 08:31 stigreport.log

drwx------ 2 root root 4096 Aug 9 04:02 suidlist

-rw------- 1 root root 64128 Aug 12 12:26 tallylog

-rw-r--r-- 1 root root 0 Feb 3 2021 tdnf.log

drwxr-xr-x 6 root root 4096 Jul 22 12:35 vmware

-rw------- 1 root root 719 Aug 10 16:00 vmware-network.1.log

-rw------- 1 root root 741 Aug 10 15:53 vmware-network.2.log

-rw------- 1 root root 741 Jul 20 08:20 vmware-network.3.log

-rw------- 1 root root 719 Jul 19 12:41 vmware-network.4.log

-rw------- 1 root root 741 Jul 19 12:29 vmware-network.5.log

-rw------- 1 root root 719 Jul 19 08:36 vmware-network.6.log

-rw------- 1 root root 741 Jul 19 08:20 vmware-network.7.log

-rw------- 1 root root 719 Jul 15 11:36 vmware-network.8.log

-rw------- 1 root root 719 Jun 11 15:09 vmware-network.9.log

-rw------- 1 root root 719 Aug 10 19:45 vmware-network.log

-rw------- 1 root root 31554 Aug 10 19:40 vmware-vgauthsvc.log.0

-rw------- 1 root root 45668 Aug 10 19:45 vmware-vmsvc-root.log

-rw------- 1 root root 3480 Aug 10 19:45 vmware-vmtoolsd-root.log

-rw-rw-r-- 1 root utmp 184320 Aug 12 12:23 wtmp


Cause : It occurs due to log rotation policy for auth.log and audit.log.


Fix :

  • Take the snapshots of the SDDC Manager.

  • Deleted the auth and audit logs both

root@vcfsddc[ /var/log ]# uptime

12:29:27 up 1 day, 16:49, 2 users, load average: 0.00, 0.02, 0.00

root@vcfsddc[ /var/log ]# rm auth.log

root@vcfsddc[ /var/log ]# cd audit

root@vcfsddc[ /var/log/audit ]# ls

audit.log

root@vcfsddc[ /var/log/audit ]# ls -l

total 12425896

-rw------- 1 root root 12724088832 Aug 12 12:31 audit.log

  • Reboot the sddc manager , then the space was back to normal and the GUI was able to login.

  • After few weeks the partition got again full , later vmware gss gave an workaround which is mention in the KB 86401

83 views0 comments

Recent Posts

See All

VCF 5.1.1 Release with VMware Private AI Foundation NVIDIA

March 26th 2024 marks an exciting milestone for VMware as it unveils the VMware Cloud Foundation 5.1.1 release, introducing groundbreaking support for VMware Private AI Foundation in collaboration wit

VCF 4.5 Precheck Is Failing On NSX-T Audit

In this blog, we'll delve into resolving VCF 4.5 precheck failures specifically related to NSX-T audit, providing you with actionable insights to overcome these hurdles. Precheck Failure Error : Healt

Comments


bottom of page