top of page
  • viquarmca

VCF 4.4 to VCF 4.5 vCenter Upgrade got Stuck

Updated: Apr 17, 2023

Todays blog we will talk about yet another upgrade issue with VCF 4.5 this time it was on vCenter the upgrade was actually completed but it didn't fail rather it was stuck , when I look at the vCenter VAMI it shows the upgrade got completed , but the version of the vCenter didn't upgrade. let me share how the issue was fixed .


On the SDDC GUI the upgrade was failed for VC with below error msg , it says Install Failed. But if we do ssh to the VC and check the version of VC its reflecting the right version (7.0 Update 3h 20395099) but the VAMI pay of VC shows the older version of vc . To make sure actually the upgrade got completed or not we did ssh to the VC found the version is latest . Since the upgrade i now stuck and failed from sddc side we have checked below listed logs .





If u run the command on VC : cat /storage/core/software-update/install_operation


"state": "INSTALL_FAILED",

    "version": "7.0.3.01000",

    "latest_query_time": "2023-03-08T14:19:12Z",

    "operation_id": "/storage/core/software-update/install_operation"


Did SSH to VC and check the log /var/log/vmware/lookupsvc/lookupserver-default.log:


localhost-startStop-1 ERROR com.vmware.vim.lookup.impl.LdapStorage] LDAP


action failed; host=vcvcf-vmware.com, port=389

com.vmware.sso.interop.ldap.LdapException: Can't contact LDAP server

at com.vmware.sso.interop.ldap.LdapErrorChecker.CheckError(LdapErrorChecker.java:869) ~[ldap-lib-0.0.1-SNAPSHOT.jar:?]

at com.vmware.sso.interop.ldap.OpenLdapClientLibrary.CheckError(OpenLdapClientLibrary.java:1219) ~[ldap-lib-0.0.1-SNAPSHOT.jar:?]

at com.vmware.sso.interop.ldap.OpenLdapClientLibrary.ldap_sasl_srp_bind_s(OpenLdapClientLibrary.java:765) ~[ldap-lib-0.0.1-SNAPSHOT.jar:?]

at com.vmware.sso.interop.ldap.LdapConnection.bindSaslSrpConnection(LdapConnection.java:670) ~[ldap-lib-0.0.1-SNAPSHOT.jar:?]

at com.vmware.vim.lookup.impl.LdapStorage$LdapQuery.execute(LdapStorage.java:2283) ~[lookupservice-impl-0.0.1-SNAPSHOT.jar:?]

at com.vmware.vim.lookup.impl.LdapStorage.getSite(LdapStorage.java:2538) ~[lookupservice-impl-0.0.1-SNAPSHOT.jar:?]

at com.vmware.vim.lookup.impl.LdapStorage.<init>(LdapStorage.java:208) ~[lookupservice-impl-0.0.1-SNAPSHOT.jar:?]

at com.vmware.vim.lookup.impl.LdapStorage.<init>(LdapStorage.java:200) ~[lookupservice-impl-0.0.1-SNAPSHOT.jar:?]

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_291]

at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_291]

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_291]

at ava.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_291]

at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:204) ~[spring-beans-5.2.13.RELEASE.jar:5.2.13.RELEASE]

at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:117) ~[spring-beans-5.2.13.RELEASE.jar:5.2.13.RELEASE]


logs: 

2023-03-09T10:15:23 AM UTC [48536]DEBUG:vmware.appliance.update.update_state:Operation in progress is finished

2023-03-09T10:15:23 AM UTC [48536]DEBUG:vmware.vherd.transport.vapi:State 'INSTALL_FAILED'

2023-03-09T10:15:23 AM UTC [48536]DEBUG:vmware.vherd.transport.vapi:Only ['INSTALL_IN_PROGRESS'] block []

2023-03-09T10:15:23 AM UTC [48536]DEBUG:vmware.appliance.update.update_state:In State._get using state file /etc/applmgmt/appliance/software_update_state.conf

2023-03-09T10:15:23 AM UTC [48536]INFO:vmware.appliance.update.update_state:Found operation in progress /storage/core/software-update/install_operation

2023-03-09T10:15:23 AM UTC [48536]DEBUG:vmware.appliance.update.update_state:Operation in progress is finished

2023-03-09T10:15:23 AM UTC [48536]DEBUG:vmware.appliance.update.update_state:Writing to state file from State._get

2023-03-09T10:15:23 AM UTC [48536]DEBUG:vmware.vherd.base.authorization_local:Verify privileges user (root) privilege ['ViewConfiguration']

2023-03-09T10:15:23 AM UTC [48536]DEBUG:root:Validated user privileges in localstore or SSO

2023-03-09T10:15:23 AM UTC [48536]DEBUG:vmware.appliance.update.update_state:In State._writeInfo writing to state file /etc/applmgmt/appliance/software_update_state.conf

2023-03-09T10:15:23 AM UTC [48536]DEBUG:vmware.vapi.data.value:Invalid Type, expected String, instead got NoneType

2023-03-09T10:15:23 AM UTC [48536]ERROR:vmware.vapi.bindings.skeleton:Error in invoking MethodIdentifier(InterfaceIdentifier('com.vmware.appliance.update'), 'get') - Invali

d Type, expected String, instead got NoneType

Traceback (most recent call last):

  File "/usr/lib/applmgmt/vapi/lib/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/bindings/skeleton.py", line 378, in invoke

    output_type)

  File "/usr/lib/applmgmt/vapi/lib/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/bindings/converter.py", line 1117, in convert_to_vapi

    binding_type.accept(visitor)

  File "/usr/lib/applmgmt/vapi/lib/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/bindings/type.py", line 40, in accept

    visitor.visit(self)

  File "/usr/lib/applmgmt/vapi/lib/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/lib/visitor.py", line 43, in visit


To fix the issue followed the below steps .


FIX :

  1. Take snapshot for vcenter.

  2. Take backup of install_operation

  3. cp -fp /storage/core/software-update/install_operation /storage/core/ 

  4. Remove install_operation

  5. rm /storage/core/software-update/install_operation 

  6. Copy the below files from upgraded vCenter to affected vCenter.

  7. /etc/applmgmt/appliance/software_update_state.conf

  8. /etc/applmgmt/appliance/update.conf 

  9. restart applmgmt service

  10. service-control --stop applmgmt

  11. service-control --start applmgmt

  12. To verify that the workaround fixed the problem, reboot your vCenter Server Appliance once.

After the worked the SDDC and VC VAMI shows the vCenter upgrade got completed with right version .

The reference kb talks about similar issue

Recent Posts

See All

VCF 5.1.1 Release with VMware Private AI Foundation NVIDIA

March 26th 2024 marks an exciting milestone for VMware as it unveils the VMware Cloud Foundation 5.1.1 release, introducing groundbreaking support for VMware Private AI Foundation in collaboration wit

VCF 4.5 Precheck Is Failing On NSX-T Audit

In this blog, we'll delve into resolving VCF 4.5 precheck failures specifically related to NSX-T audit, providing you with actionable insights to overcome these hurdles. Precheck Failure Error : Healt

Comments


bottom of page