VCF 4.3.1 to 4.4.1 Upgrade Failed At Configuration Drift Stage

Today Blog we will talk about VCF 4.3.1 to 4.4.1 SDDC Drift Configuration failure with error Failed to Run Clean VUM DB . The configuration drift bundle applies configuration changes required for 2nd party software components in the VMware Cloud Foundation Bill of Materials for the target release i.e. VCF 4.4.1 . This issue I have seen it for the 1st time , the Error msg says about Failed to run clean vum db , however that was not the issue . Lets check out the below steps how we find the issue and fix it.

Check the logs on /var/log/vmware/vcf/lcm/thirdparty/upgrades/ , found the error msg as

2022-09-14T03:02:12.886+0000 INFO [vcf_migration,0000000000000000,0000] [c.v.e.s.c.v.ExecutionContextToValidationConverter,main] Collecting processing task errors: FAILED_TO_RUN_CLEAN_VUM_DB for validation aggregation

2022-09-14T03:02:12.886+0000 INFO [vcf_migration,0000000000000000,0000] [c.v.e.s.c.v.ExecutionContextToValidationConverter,main] Collecting processing task errors: FAILED_TO_RUN_CLEAN_VUM_DB for validation aggregation

2022-09-14T03:02:12.889+0000 DEBUG [vcf_migration,0000000000000000,0000] [c.v.v.m.ContractRecipeExecutor,main] Validation output Optional[com.vmware.vcf.migration.model.WorkflowOutput@621500b3]

2022-09-14T03:02:12.889+0000 DEBUG [vcf_migration,0000000000000000,0000] [c.v.v.m.ContractRecipeExecutor,main] Validation output Optional[com.vmware.vcf.migration.model.WorkflowOutput@621500b3]

2022-09-14T03:02:12.898+0000 INFO [vcf_migration,0000000000000000,0000] [c.v.v.m.ContractRecipeExecutor,main] Execution ID: cb156fe7-fa79-463f-bfde-689b4eba1482, Status: COMPLETED_WITH_FAILURE

2022-09-14T03:02:12.898+0000 INFO [vcf_migration,0000000000000000,0000] [c.v.v.m.ContractRecipeExecutor,main] Execution ID: cb156fe7-fa79-463f-bfde-689b4eba1482, Status: COMPLETED_WITH_FAILURE

2022-09-14T03:02:12.905+0000 INFO [vcf_migration,0000000000000000,0000] [c.v.v.migration.SddcManagerMigration,main] Output Values: {}

2022-09-14T03:02:12.905+0000 INFO [vcf_migration,0000000000000000,0000] [c.v.v.migration.SddcManagerMigration,main] Output Values: {}

2022-09-14T03:02:12.905+0000 INFO [vcf_migration,0000000000000000,0000] [com.vmware.vcf.migration.Application,main] SDDC Manager migration completed

2022-09-14T03:02:12.905+0000 INFO [vcf_migration,0000000000000000,0000] [com.vmware.vcf.migration.Application,main] SDDC Manager migration completed

2022-09-14T03:01:24.713+0000 ERROR [vcf_migration,0000000000000000,0000]

[c.v.e.s.o.model.error.ErrorFactory,pool-5-thread-15] [6469SC] FAILED_TO_RUN_CLEAN_VUM_DB Failed to run clean VUM DB.

com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Failed to run clean VUM DB.

at com.vmware.vcf.migration.actions.workarounds.CleanVumDBAction.execute(CleanVumDBAction.java:315)

at com.vmware.vcf.migration.actions.workarounds.CleanVumDBAction.execute(CleanVumDBAction.java:40)

at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionState.invoke(FsmActionState.java:62)

at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionPlugin.invoke(FsmActionPlugin.java:159)

at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionPlugin.invoke(FsmActionPlugin.java:144)

at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.invokeMethod(ProcessingTaskSubscriber.java:400)

at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.processTask(ProcessingTaskSubscriber.java:520)

at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.accept(ProcessingTaskSubscriber.java:124)

at sun.reflect.GeneratedMethodAccessor598.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at com.google.common.eventbus.Subscriber.invokeSubscriberMethod(Subscriber.java:87)

at com.google.common.eventbus.Subscriber$1.run(Subscriber.java:72)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

Caused by: com.vmware.vapi.std.errors.ServiceUnavailable: ServiceUnavailable (com.vmware.vapi.std.errors.service_unavailable) => {

messages = [LocalizableMessage (com.vmware.vapi.std.localizable_message) => {

id = com.vmware.vapi.endpoint.cis.ServiceUnavailable,

defaultMessage = Service unavailable.,

args = [],

params = <null>,

localized = <null>

}],

data = <null>,

errorType = SERVICE_UNAVAILABLE

Tried to follow the VUM DB Reset KB 2147284 , but didnt work. the upgrade failed again .

On vCenter Appliance checked the /var/log/vmware/applmgmt: Failed to get issuers certificates

2022-09-14T05:31:16.0 [2661]DEBUG:vmware.appliance.extensions.authorization.authorization_sso:User=vsphere.local\vpxd-extension-d93b7f11-f22a-4436-a03e-eb9e72c68489, groups=

set()

2022-09-14T05:31:16.0 [2661]DEBUG:root:Validated user privileges in localstore or SSO

2022-09-14T05:31:16.0 [2661]DEBUG:vmware.appliance.extensions.authorization.authorization_sso:Required privileges = ['ModifyLocalConf']

2022-09-14T05:31:16.4 [2661]ERROR:vmware.appliance.extensions.authorization.authorization_sso:FindAllParentGroups Failed {(vmodl.fault.SystemError) {

dynamicType = <unset>,

dynamicProperty = (vmodl.DynamicProperty) [],

msg = 'Internal server error',

faultCause = <unset>,

faultMessage = (vmodl.LocalizableMessage) [],

reason = "Failed to get issuers certificates: [Can't contact LDAP server]"

}}

Traceback (most recent call last):

File "/usr/lib/applmgmt/lib/extensions/py/vmware/appliance/extensions/authorization/authorization_sso.py", line 285, in get_groups

user_id)

File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 583, in <lambda>

self.f(*(self.args + (obj,) + args), **kwargs)

File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 373, in _InvokeMethod

return self._stub.InvokeMethod(self, info, args)

File "/usr/lib/vmware/site-packages/pyVmomi/SoapAdapter.py", line 1802, in InvokeMethod

raise obj

pyVmomi.VmomiSupport.vmodl.fault.SystemError: (vmodl.fault.SystemError) {

dynamicType = <unset>,

dynamicProperty = (vmodl.DynamicProperty) [],

msg = 'Internal server error',

faultCause = <unset>,

faultMessage = (vmodl.LocalizableMessage) [],

reason = "Failed to get issuers certificates: [Can't contact LDAP server]"

}

signing_chain = self.validate_certificate()

File "/usr/lib/applmgmt/lib/extensions/py/vmware/appliance/extensions/authentication/authentication_sso.py", line 685, in validate_certificate

'One or more certificates cannot be verified.')

vmware.appliance.extensions.authentication.authentication_sso.AuthenticationError: One or more certificates cannot be verified.

Fix :

1. Reset the STS certificate on the Management vCenter (https://kb.vmware.com/s/article/76719) 2. Reset the solution user certificates on the Management vCenter using Option 6 of the certificate manager (https://kb.vmware.com/s/article/2112283) That should restart all services on the vCenter as well. 3. Restart services on all remaining vCenter . 4. Attempt the Configuration Drift Bundle update again from the SDDC Manager UI.