vCenter event collectors start and run for a couple of weeks then will not start

[tony_phillips]

We are on San Diego patch9 and have several vCenter Event Collectors. These start and stay up for a couple of weeks, but then go down and can't be restarted. Even recreating the event collectors doesn't help. We get this error back when they fail to start, "Remote Exception Permission to perform this operation was denied.".

Searching the MID server logs, we found stack traces like these:

ECCQueueMonitor.1 SEVERE *** ERROR *** Error in executing Extension ECCMessage: Remote Exception Permission to perform this operation was denied.
com.snc.automation_common.mid.extension.api.InvalidExtensionParameterException: Remote Exception Permission to perform this operation was denied.
    at com.service_now.mid.extension.vcenter.VCenterExtension.validateParameters(VCenterExtension.java:348)
    at com.service_now.mid.extension.vcenter.VCenterExtension.connect(VCenterExtension.java:258)
    at com.service_now.mid.extension.vcenter.VCenterExtension.start(VCenterExtension.java:304)
    at com.service_now.mid.extension.container.ExtensionContainer.start(ExtensionContainer.java:392)
    at com.service_now.mid.extension.container.ExtensionMessageRouter.routeMessage(ExtensionMessageRouter.java:54)
    at com.service_now.mid.message_executors.ExtensionExecutor.execute(ExtensionExecutor.java:41)
    at com.service_now.monitor.ECCQueueMonitor.executeMessage(ECCQueueMonitor.java:310)
    at com.service_now.monitor.ECCQueueMonitor.processMessage(ECCQueueMonitor.java:243)
    at com.service_now.monitor.ECCQueueMonitor.processMessages(ECCQueueMonitor.java:264)
    at com.service_now.monitor.ECCQueueMonitor.run(ECCQueueMonitor.java:201)
    at com.service_now.monitor.AMonitor.runit(AMonitor.java:155)
    at com.service_now.monitor.AMonitor.access$200(AMonitor.java:44)
    at com.service_now.monitor.AMonitor$MonitorTask.execute(AMonitor.java:148)
    at com.service_now.monitor.base.AMonitorTask.run(AMonitorTask.java:29)
    at java.base/java.util.TimerThread.mainLoop(Timer.java:556)
    at java.base/java.util.TimerThread.run(Timer.java:506)
Caused by: com.vmware.vim25.NoPermission: Permission to perform this operation was denied.
    at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
    at java.base/java.lang.Class.newInstance(Class.java:584)
    at com.vmware.vim25.ws.XmlGenDom.fromXml(XmlGenDom.java:253)
    at com.vmware.vim25.ws.XmlGenDom.parseSoapFault(XmlGenDom.java:151)
    at com.vmware.vim25.ws.XmlGenDom.fromXML(XmlGenDom.java:109)
    at com.vmware.vim25.ws.SoapClient.unMarshall(SoapClient.java:253)
    at com.vmware.vim25.ws.WSClient.invoke(WSClient.java:96)
    at com.vmware.vim25.ws.VimStub.login(VimStub.java:1732)
    at com.vmware.vim25.mo.SessionManager.login(SessionManager.java:153)
    at com.vmware.vim25.mo.ServiceInstance.login(ServiceInstance.java:238)
    at com.vmware.vim25.mo.ServiceInstance.constructServiceInstance(ServiceInstance.java:131)
    at com.vmware.vim25.mo.ServiceInstance.<init>(ServiceInstance.java:79)
    at com.vmware.vim25.mo.ServiceInstance.<init>(ServiceInstance.java:69)
    at com.service_now.mid.extension.vcenter.VCenterExtension.getServiceInstance(VCenterExtension.java:224)
    at com.service_now.mid.extension.vcenter.VCenterExtension.validateParameters(VCenterExtension.java:344)
    ... 15 more

We've tested credentials against the ones that will not start and the credentials are valid.

Anyone else seen this type of behavior?

[Aron Campbell]

I now am getting the same error.  I am not sure when it started but several of the event collectors are stopped with the same status and cannot start again.

[Potter69]

The identical mistake is now occurring for me.

[Ceasar]

Getting same error too for 2 vCenter after Tokyo upgrade, but some of the collectors are working. Seems something not right even with valid credentials.

[Ceasar]

@tony_phillips , @Potter69 , @Aron Campbell :

I just found a fix on this issues. Delete the credential affinities of the affected vCenter and recreate it fixed my issue.