VCF 4.5 Precheck Is Failing On NSX-T Audit

In this blog, we'll delve into resolving VCF 4.5 precheck failures specifically related to NSX-T audit, providing you with actionable insights to overcome these hurdles.

Precheck Failure Error :

Health Status Red Error Description NSX-T Manager Audit for http://nsxtnsxlab3m1.avc.vmw.com failed with unknown exception Impact High: Do not perform upgrade without addressing this issue unless the available upgrade is for NSX-T. Remediation Audit check failed for NSX-T. Check if the SDDC Manager is able to communicate with NSX-T Manager. If not, login to NSX-T and check if upgrade is running and wait for the completion. Also please ensure that credential of type API for NSX-T manager is not expired.

We have tried below steps to fix the issue : 1. https://kb.vmware.com/s/article/85764 2. https://kb.vmware.com/s/article/86047 3. Password expiry on NSX-T manager

4. Certificate we are all good 5. Edge nodes curl -k -u 'admin' -H 'Accept:application/json' -H 'Content-Type:application/json' -X GET https://nsxtnsxlab3m1-01.avc.vmw.com/api/v1/configs/inventory

Findings from Log LCM-debug :

2023-01-17T18:47:28.582+0000 DEBUG [vcf_lcm,d3cfaeb310f8bd07,d083,auditId=ae740b5a-36b0-45e7-9758-41c97036e40f,resourceType=NSX_T_MANAGER,resourceId=nsxtnsxlab3m1.avc.vmw.com,name=nsxtnsxlab3m1.avc.vmw.com] [c.v.e.s.l.p.i.n.NsxtInventoryLoader,Scheduled-8] Transport Node nsxlab3m1esx01 is an Edge Node or does not exist in VCF inventory
2023-01-17T18:47:28.582+0000 DEBUG [vcf_lcm,d3cfaeb310f8bd07,d083,auditId=ae740b5a-36b0-45e7-9758-41c97036e40f,resourceType=NSX_T_MANAGER,resourceId=nsxtnsxlab3m1.avc.vmw.com,name=nsxtnsxlab3m1.avc.vmw.com] [c.v.e.s.l.p.i.n.NsxtInventoryLoader,Scheduled-8] Transport Node nsxlab3m1esx02 is an Edge Node or does not exist in VCF inventory
2023-01-17T18:47:28.582+0000 DEBUG [vcf_lcm,d3cfaeb310f8bd07,d083,auditId=ae740b5a-36b0-45e7-9758-41c97036e40f,resourceType=NSX_T_MANAGER,resourceId=nsxtnsxlab3m1.avc.vmw.com,name=nsxtnsxlab3m1.avc.vmw.com] [c.v.e.s.l.p.i.n.NsxtInventoryLoader,Scheduled-8] Transport Node nsxlab3m1esx03 is an Edge Node or does not exist in VCF inventory
2023-01-17T18:47:28.582+0000 DEBUG [vcf_lcm,d3cfaeb310f8bd07,d083,auditId=ae740b5a-36b0-45e7-9758-41c97036e40f,resourceType=NSX_T_MANAGER,resourceId=nsxtnsxlab3m1.avc.vmw.com,name=nsxtnsxlab3m1.avc.vmw.com] [c.v.e.s.l.p.i.n.NsxtInventoryLoader,Scheduled-8] Transport Node nsxlab3m1esx04 is an Edge Node or does not exist in VCF inventory
2023-01-17T18:47:28.582+0000 DEBUG [vcf_lcm,d3cfaeb310f8bd07,d083,auditId=ae740b5a-36b0-45e7-9758-41c97036e40f,resourceType=NSX_T_MANAGER,resourceId=nsxtnsxlab3m1avc.vmw.com,name=nsxtnsxlab3m1.avc.vmw.com] [c.v.e.s.l.p.i.n.NsxtInventoryLoader,Scheduled-8] Transport Node nsxlab3m1esx05 is an Edge Node or does not exist in VCF inventory
2023-01-17T18:47:28.582+0000 DEBUG [vcf_lcm,d3cfaeb310f8bd07,d083,auditId=ae740b5a-36b0-45e7-9758-41c97036e40f,resourceType=NSX_T_MANAGER,resourceId=nsxtnsxlab3m1.fsc.fedex.com,name=nsxnsxlab3m1.avc.vmw.com] [c.v.e.s.l.p.i.n.NsxtInventoryLoader,Scheduled-8] Transport Node nsxlab3m1esx06 is an Edge Node or does not exist in VCF inventory
2023-01-17T18:47:28.582+0000 DEBUG [vcf_lcm,d3cfaeb310f8bd07,d083,auditId=ae740b5a-36b0-45e7-9758-41c97036e40f,resourceType=NSX_T_MANAGER,resourceId=nsxtnsxlab3m1.avc.vmw.com,name=nsxtnsxlab3m1.avc.vmw.com] [c.v.e.s.l.p.i.n.NsxtInventoryLoader,Scheduled-8] Transport Node nsxlab3m1esx07 is an Edge Node or does not exist in VCF inventory

We are seeing this across multiple hosts - which is causing a failure to load NSX-T Cluster Inventory.

Specific hosts with the issue:

cat lcm-debug.log | grep -i 'is an Edge Node or does not exist in VCF inventory' | awk '{print $7}' | sort -d | uniq -c
    108 ennsxlab3m1-01
    108 ennsxlab3m1-02
    106 ennsxlab3w1-01.avc.vmw.com
    106 ennsxlab3w1-02.avc.vmw.com
    106 ennsxlab3w1-03.avc.vmw.com
    106 ennsxlab3w1-04.avc.vmw.com
    108 nsxlabesx01
    108 nsxlabsx02
    108 nsxlabesx03
    108 nsxlabesx04
    108 nsxlabesx05
    108 nsxlabesx06
    108 nsxlabesx07

Checking their names in the SDDC DB:

ennsxlab3m1-01 -> ennsxlab3m1-01.avc.vmw.com (Edge Transport Node)
ennsxlab3m1-02 -> ennsxlab3m1-02.avc.vmw.com (Edge Transport Node)
ennsxlab3w1-01.avc.vmw.com (Edge nodes that do not exist in SDDC Manager)
ennsxlab3w1-02.avc.vmw.com (Edge nodes that do not exist in SDDC Manager)
ennsxlab3w1-03.avc.vmw.com (Edge nodes that do not exist in SDDC Manager)
ennsxlab3w1-04.avc.vmw.com (Edge nodes that do not exist in SDDC Manager)
nsxlabesx01 -> nsxlab3m1esx01.avc.vmw.com
nsxlabesx02 -> nsxlab3m1esx02.avc.vmw.com
nsxlabesx03 -> nsxlab3m1esx03.avc.vmw.com
nsxlabesx04 -> nsxlab3m1esx04.avc.vmw.com
nsxlabesx05 -> nsxlab3m1esx05.avc.vmw.com
nsxlabesx06 -> nsxlab3m1esx06.avc.vmw.com
nsxlabesx07 -> nsxlab3m1esx07.avc.vmw.com

Resolution :

1. For all the components that are showing up as short name, go into the NSX-T UI > System > Fabric

2. Edit the disply names of all of them to be the FQDN.

3. Once the renaming is done , perform the rooling reboot of all 3 nsxt managers. one at a time to force sync inventory.

4. After rolling reboot run precheck again all was green.

Note : During the issue we had help from VMware GSS , who was able to give us the resolution .