Todays blog we will talk about yet another upgrade issue with VCF 4.5 this time it was on vCenter the upgrade was actually completed but it didn't fail rather it was stuck , when I look at the vCenter VAMI it shows the upgrade got completed , but the version of the vCenter didn't upgrade. let me share how the issue was fixed .
On the SDDC GUI the upgrade was failed for VC with below error msg , it says Install Failed. But if we do ssh to the VC and check the version of VC its reflecting the right version (7.0 Update 3h 20395099) but the VAMI pay of VC shows the older version of vc . To make sure actually the upgrade got completed or not we did ssh to the VC found the version is latest . Since the upgrade i now stuck and failed from sddc side we have checked below listed logs .
If u run the command on VC : cat /storage/core/software-update/install_operation
"state": "INSTALL_FAILED",
"version": "7.0.3.01000",
"latest_query_time": "2023-03-08T14:19:12Z",
"operation_id": "/storage/core/software-update/install_operation"
Did SSH to VC and check the log /var/log/vmware/lookupsvc/lookupserver-default.log:
localhost-startStop-1 ERROR com.vmware.vim.lookup.impl.LdapStorage] LDAP
action failed; host=vcvcf-vmware.com, port=389
com.vmware.sso.interop.ldap.LdapException: Can't contact LDAP server
at com.vmware.sso.interop.ldap.LdapErrorChecker.CheckError(LdapErrorChecker.java:869) ~[ldap-lib-0.0.1-SNAPSHOT.jar:?]
at com.vmware.sso.interop.ldap.OpenLdapClientLibrary.CheckError(OpenLdapClientLibrary.java:1219) ~[ldap-lib-0.0.1-SNAPSHOT.jar:?]
at com.vmware.sso.interop.ldap.OpenLdapClientLibrary.ldap_sasl_srp_bind_s(OpenLdapClientLibrary.java:765) ~[ldap-lib-0.0.1-SNAPSHOT.jar:?]
at com.vmware.sso.interop.ldap.LdapConnection.bindSaslSrpConnection(LdapConnection.java:670) ~[ldap-lib-0.0.1-SNAPSHOT.jar:?]
at com.vmware.vim.lookup.impl.LdapStorage$LdapQuery.execute(LdapStorage.java:2283) ~[lookupservice-impl-0.0.1-SNAPSHOT.jar:?]
at com.vmware.vim.lookup.impl.LdapStorage.getSite(LdapStorage.java:2538) ~[lookupservice-impl-0.0.1-SNAPSHOT.jar:?]
at com.vmware.vim.lookup.impl.LdapStorage.<init>(LdapStorage.java:208) ~[lookupservice-impl-0.0.1-SNAPSHOT.jar:?]
at com.vmware.vim.lookup.impl.LdapStorage.<init>(LdapStorage.java:200) ~[lookupservice-impl-0.0.1-SNAPSHOT.jar:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_291]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_291]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_291]
at ava.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_291]
at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:204) ~[spring-beans-5.2.13.RELEASE.jar:5.2.13.RELEASE]
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:117) ~[spring-beans-5.2.13.RELEASE.jar:5.2.13.RELEASE]
logs:
2023-03-09T10:15:23 AM UTC [48536]DEBUG:vmware.appliance.update.update_state:Operation in progress is finished
2023-03-09T10:15:23 AM UTC [48536]DEBUG:vmware.vherd.transport.vapi:State 'INSTALL_FAILED'
2023-03-09T10:15:23 AM UTC [48536]DEBUG:vmware.vherd.transport.vapi:Only ['INSTALL_IN_PROGRESS'] block []
2023-03-09T10:15:23 AM UTC [48536]DEBUG:vmware.appliance.update.update_state:In State._get using state file /etc/applmgmt/appliance/software_update_state.conf
2023-03-09T10:15:23 AM UTC [48536]INFO:vmware.appliance.update.update_state:Found operation in progress /storage/core/software-update/install_operation
2023-03-09T10:15:23 AM UTC [48536]DEBUG:vmware.appliance.update.update_state:Operation in progress is finished
2023-03-09T10:15:23 AM UTC [48536]DEBUG:vmware.appliance.update.update_state:Writing to state file from State._get
2023-03-09T10:15:23 AM UTC [48536]DEBUG:vmware.vherd.base.authorization_local:Verify privileges user (root) privilege ['ViewConfiguration']
2023-03-09T10:15:23 AM UTC [48536]DEBUG:root:Validated user privileges in localstore or SSO
2023-03-09T10:15:23 AM UTC [48536]DEBUG:vmware.appliance.update.update_state:In State._writeInfo writing to state file /etc/applmgmt/appliance/software_update_state.conf
2023-03-09T10:15:23 AM UTC [48536]DEBUG:vmware.vapi.data.value:Invalid Type, expected String, instead got NoneType
2023-03-09T10:15:23 AM UTC [48536]ERROR:vmware.vapi.bindings.skeleton:Error in invoking MethodIdentifier(InterfaceIdentifier('com.vmware.appliance.update'), 'get') - Invali
d Type, expected String, instead got NoneType
Traceback (most recent call last):
File "/usr/lib/applmgmt/vapi/lib/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/bindings/skeleton.py", line 378, in invoke
output_type)
File "/usr/lib/applmgmt/vapi/lib/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/bindings/converter.py", line 1117, in convert_to_vapi
binding_type.accept(visitor)
File "/usr/lib/applmgmt/vapi/lib/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/bindings/type.py", line 40, in accept
visitor.visit(self)
File "/usr/lib/applmgmt/vapi/lib/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/lib/visitor.py", line 43, in visit
To fix the issue followed the below steps .
FIX :
Take snapshot for vcenter.
Take backup of install_operation
cp -fp /storage/core/software-update/install_operation /storage/core/
Remove install_operation
rm /storage/core/software-update/install_operation
Copy the below files from upgraded vCenter to affected vCenter.
/etc/applmgmt/appliance/software_update_state.conf
/etc/applmgmt/appliance/update.conf
restart applmgmt service
service-control --stop applmgmt
service-control --start applmgmt
To verify that the workaround fixed the problem, reboot your vCenter Server Appliance once.
After the worked the SDDC and VC VAMI shows the vCenter upgrade got completed with right version .
The reference kb talks about similar issue
留言