VCF 4.3.1 to 4.4.1 Upgrade Failed At Configuration Drift Stage

viquarmca

Sep 20, 2022

2 min read

Today Blog we will talk about VCF 4.3.1 to 4.4.1 SDDC Drift Configuration failure with error Failed to Run Clean VUM DB . The configuration drift bundle applies configuration changes required for 2nd party software components in the VMware Cloud Foundation Bill of Materials for the target release i.e. VCF 4.4.1 . This issue I have seen it for the 1st time , the Error msg says about Failed to run clean vum db , however that was not the issue . Lets check out the below steps how we find the issue and fix it.

Check the logs on /var/log/vmware/vcf/lcm/thirdparty/upgrades/ , found the error msg as

2022-09-14T03:02:12.886+0000 INFO [vcf_migration,0000000000000000,0000] [c.v.e.s.c.v.ExecutionContextToValidationConverter,main] Collecting processing task errors: FAILED_TO_RUN_CLEAN_VUM_DB for validation aggregation

2022-09-14T03:02:12.886+0000 INFO [vcf_migration,0000000000000000,0000] [c.v.e.s.c.v.ExecutionContextToValidationConverter,main] Collecting processing task errors: FAILED_TO_RUN_CLEAN_VUM_DB for validation aggregation

2022-09-14T03:02:12.889+0000 DEBUG [vcf_migration,0000000000000000,0000] [c.v.v.m.ContractRecipeExecutor,main] Validation output Optional[com.vmware.vcf.migration.model.WorkflowOutput@621500b3]

2022-09-14T03:02:12.898+0000 INFO [vcf_migration,0000000000000000,0000] [c.v.v.m.ContractRecipeExecutor,main] Execution ID: cb156fe7-fa79-463f-bfde-689b4eba1482, Status: COMPLETED_WITH_FAILURE

2022-09-14T03:02:12.905+0000 INFO [vcf_migration,0000000000000000,0000] [c.v.v.migration.SddcManagerMigration,main] Output Values: {}

2022-09-14T03:02:12.905+0000 INFO [vcf_migration,0000000000000000,0000] [com.vmware.vcf.migration.Application,main] SDDC Manager migration completed

2022-09-14T03:01:24.713+0000 ERROR [vcf_migration,0000000000000000,0000]

[c.v.e.s.o.model.error.ErrorFactory,pool-5-thread-15] [6469SC] FAILED_TO_RUN_CLEAN_VUM_DB Failed to run clean VUM DB.

com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Failed to run clean VUM DB.

at com.vmware.vcf.migration.actions.workarounds.CleanVumDBAction.execute(CleanVumDBAction.java:315)

at com.vmware.vcf.migration.actions.workarounds.CleanVumDBAction.execute(CleanVumDBAction.java:40)

at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionState.invoke(FsmActionState.java:62)

at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionPlugin.invoke(FsmActionPlugin.java:159)

at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionPlugin.invoke(FsmActionPlugin.java:144)

at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.invokeMethod(ProcessingTaskSubscriber.java:400)

at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.processTask(ProcessingTaskSubscriber.java:520)

at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.accept(ProcessingTaskSubscriber.java:124)

at sun.reflect.GeneratedMethodAccessor598.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at com.google.common.eventbus.Subscriber.invokeSubscriberMethod(Subscriber.java:87)

at com.google.common.eventbus.Subscriber$1.run(Subscriber.java:72)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

Caused by: com.vmware.vapi.std.errors.ServiceUnavailable: ServiceUnavailable (com.vmware.vapi.std.errors.service_unavailable) => {

messages = [LocalizableMessage (com.vmware.vapi.std.localizable_message) => {

id = com.vmware.vapi.endpoint.cis.ServiceUnavailable,

defaultMessage = Service unavailable.,

args = [],

params = <null>,

localized = <null>

}],

data = <null>,

errorType = SERVICE_UNAVAILABLE

Tried to follow the VUM DB Reset KB 2147284 , but didnt work. the upgrade failed again .

On vCenter Appliance checked the /var/log/vmware/applmgmt: Failed to get issuers certificates

2022-09-14T05:31:16.0 [2661]DEBUG:vmware.appliance.extensions.authorization.authorization_sso:User=vsphere.local\vpxd-extension-d93b7f11-f22a-4436-a03e-eb9e72c68489, groups=

set()

2022-09-14T05:31:16.0 [2661]DEBUG:root:Validated user privileges in localstore or SSO

2022-09-14T05:31:16.0 [2661]DEBUG:vmware.appliance.extensions.authorization.authorization_sso:Required privileges = ['ModifyLocalConf']

2022-09-14T05:31:16.4 [2661]ERROR:vmware.appliance.extensions.authorization.authorization_sso:FindAllParentGroups Failed {(vmodl.fault.SystemError) {

dynamicType = <unset>,

dynamicProperty = (vmodl.DynamicProperty) [],

msg = 'Internal server error',

faultCause = <unset>,

faultMessage = (vmodl.LocalizableMessage) [],

reason = "Failed to get issuers certificates: [Can't contact LDAP server]"

}}

Traceback (most recent call last):

File "/usr/lib/applmgmt/lib/extensions/py/vmware/appliance/extensions/authorization/authorization_sso.py", line 285, in get_groups

user_id)

File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 583, in <lambda>

self.f(*(self.args + (obj,) + args), **kwargs)

File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 373, in _InvokeMethod

return self._stub.InvokeMethod(self, info, args)

File "/usr/lib/vmware/site-packages/pyVmomi/SoapAdapter.py", line 1802, in InvokeMethod

raise obj

pyVmomi.VmomiSupport.vmodl.fault.SystemError: (vmodl.fault.SystemError) {

dynamicType = <unset>,

dynamicProperty = (vmodl.DynamicProperty) [],

msg = 'Internal server error',

faultCause = <unset>,

faultMessage = (vmodl.LocalizableMessage) [],

reason = "Failed to get issuers certificates: [Can't contact LDAP server]"

}

signing_chain = self.validate_certificate()

File "/usr/lib/applmgmt/lib/extensions/py/vmware/appliance/extensions/authentication/authentication_sso.py", line 685, in validate_certificate

'One or more certificates cannot be verified.')

vmware.appliance.extensions.authentication.authentication_sso.AuthenticationError: One or more certificates cannot be verified.

Fix :

1. Reset the STS certificate on the Management vCenter (https://kb.vmware.com/s/article/76719) 2. Reset the solution user certificates on the Management vCenter using Option 6 of the certificate manager (https://kb.vmware.com/s/article/2112283) That should restart all services on the vCenter as well. 3. Restart services on all remaining vCenter . 4. Attempt the Configuration Drift Bundle update again from the SDDC Manager UI.

VCF 4.3.1 to 4.4.1 Upgrade Failed At Configuration Drift Stage

Recent Posts

Comments