vCenter Connection Issues and Debugging

srikanthnelapud · ‎07-22-2020

Overview

During vCenter discovery, various connection issues can be encountered so the goal of this article is what to do when any of the below issues occurs thus providing the relief to the user.

Related KB: KB0831417

PROBLEM STATEMENT #1: - vCenter Probe (E.g. VMWarevCenterDatacentersProbe) is taking too much time for processing or Stuck in the "Processing" state.

Root Cause

The processing time of the probes depends on the data we retrieve from the Cloud so more the data then more the time taken for processing but if it takes longer time, say more than an hour or so and ECC Queue stuck in the "Processing" state then it can be due to the issue with the MID server i.e., either the MID is Down/Restored to Invalidated state/went for a toss.

Solution

Any of the below 3 ways can work:

Restarting the MID Server
Validating/Re-Validating the MID Server
Re-Installing the MID Server.

Please note if none of the above solutions worked then the problem is mostly with the target system, not the MID. If the target isn’t a vCenter and accepts a connection then it never responds so nothing can be done with MID instead that specific IP needs to be excluded from the schedule.

PROBLEM STATEMENT #2: Discovery log showing - "No credential would validate for <<vCenter IP>>/sdk"

Root Cause

It's a credential issue, due to which the authentication cannot be made with the vCenter by the MID.

Solution

If credentials are not added in the System then add the vCenter proper valid credentials and run the Discovery again.

If credential added already but still Discovery is failing with the above error message then probably the credentials might have changed so please confirm the same by doing below:

Go to "Discovery" -> "Credentials"
Open the record that we wanted to test.
Please click on the "Test credential" action available in the "Related Links" of the page
Provide the IP, leave the port number as 443, select the mid sever and click "Ok"
If credentials are proper then it returns "Credential validated" else "Authentication failed"

Please note, the most common credential problem is confusion over the credential type so please ensure to select credentials whose type is "VMware".

PROBLEM STATEMENT #3: Discovery log showing - "<<vCenter IP>> is not a reachable host (no response to target ports scanned by MID)"

Root Cause

vCenter IP is not reachable by the MID server that has been used for the Discovery initiated or the MID configured for the Discovery schedule.

Also, another root cause can be the ports - 5480 and 9443 aren't configured for the host to listen due to which the "vmapp" port probe won't receive any response to trigger vCenter probes.

Solution

Try ping and ensure if the IP is reachable by the below steps:

Log in to the host where the MID server is deployed and configured
Open the Terminal(Linux)/Command prompt(Windows) and provide the command - ping <<vCenter IP>>.
If IP isn't reachable then that leaves the user to configure the MID properly.
Also, ensure that the host has ports, 5480 and 9443, configured.
Having a firewall block 5480 or 9443 is also a common use-case.

Please note, blocking those ports means that we won’t classify the system as a vCenter (via port probe – the process classifier can still work) but the host can be classified as something else based on the response we get from other port probes.

PROBLEM STATEMENT #4: "VMWare - vCenter VM Tags" reports the error - "Session cannot be established"

Root Cause

To get the vCenter VMs tags information, we rely on the REST API provided by the vCenter itself and the very first URL which is used is to establish a session and query for the tags information further. So, if the vCenter version which the user has doesn't support the API i.e., no availability of such API then the probe will result in the error that "Session cannot be established".

Solution

One quick check is the discovery of the tags for vCenter VMs is supported for the vCenter version 6 and above due to the availability of the API.

Ensure if the REST API is available by redirecting to vCenter console and following the below steps:

Open any browser, just provide the vCenter IP alone and Hit enter
On the right side, see if you can find - "Browse vSphere REST APIs"
In the Select API, see if "cis" is available. If so then please select it
Under "session" try the first API (/com/vmware/cis/session) by providing "ServiceNow" as the "vmware-use-header-authn"
This way we can confirm is API is supportable or not.

PROBLEM STATEMENT #5: VMWarevCenterVMTagsProbe cannot discover tags in vCenter

Root Cause

To get the vCenter VMs tags information, we rely on the REST API provided by the vCenter itself and the very first URL which is used is to establish a session and query for the tags information further. The pre-requisite for this one is vCenter resources should be provided with possible READ access to get the tags information associated with the VM. If no proper access is provided then the empty information is returned in the ECC inputs for VMWarevCenterVMTagsProbe.

Solution

Instead of providing or having the read-only access from the resource level i.e., for example, cluster down, provide at the highest vCenter level. Post checking it then run the discovery again and the tag values will be discovered.

PROBLEM STATEMENT #6: vCenter Discovery reporting in the error log - "This issue may be exploited to execute arbitrary code when deserializing an untrusted Java object.- CVE-2017-5641"

Root Cause

VMware vCenter is the centralized management tool for the vSphere suite. The target is missing Update 3b, which corrects the following security issue: A remote code execution vulnerability via BlazeDS due to processing AMF3 messages. This issue may be exploited to execute arbitrary code when deserializing an untrusted Java object.- CVE-2017-5641

Solution

Upgrade the vCenter Server to Build 5318200 or apply the latest VMware vCenter Server build. Similarly, upgrade the vCenter Server Appliance to Build 5318203 or apply the latest VMware vCenter Server Appliance build. Refer to VMware KB2149816 for further details and following are links for downloading patches to fix the vulnerabilities:

VMSA-2017-0007: VMware vCenter Server 6.0 VMSA-2017-0007: VMware vCenter Server Appliance 6.0

PROBLEM STATEMENT #7: vCenter Discovery reporting in the error - "Unable to establish connection to https://<<vCenter IP>>/sdk: java.lang.NullPointerException: None of url, username can be null."

Root Cause

When the user name is null, com.vmware.vim25.mo.ServiceInstance will throw a NullPointerException, and in the mid server script includes, in the AVMWareProbe._getServiceInstance, when looping through all VMWare credentials, we don't catch the NullPointerException, so the mid server won't even try the credentials after the one with an empty user name.

Solution

The problem is fixed as a part of PRB1337097 [Fixed in PARIS. Backported to ORLANDO, NEW YORK].

PROBLEM STATEMENT #8: vCenter Discovery reporting in the error - " vCenter Exception - com.vmware.vim25.NoPermission handling"

Root Cause

In the AVMWareProbe MID Server script includes tryCredential() function, we attempt to connect to vCenter for each credential. If vCenter throws exceptions such as com.vmware.vim25.NoPermission, the catch block does not handle this very well and ended up rethrowing the exception and stops the credential retry cycle.

Another root cause can be the if the vCenter doesn't have proper READ permission then we get this error.

Solution

The problem is fixed as a part of PRB1337097 [Fixed in PARIS. Backported to ORLANDO, NEW YORK]. Also please ensure that vCenter is enabled with proper READ permissions.

PROBLEM STATEMENT #9: vCenterExtension reporting the exception - "java.lang.NullPointerException; at com.service_now.mid.extension.vcenter.VCenterExtension.getServiceInstance"

Root Cause

The NPE occurs in the catch block of VCenterExtension.getServiceInstance() method in line 228. When looping through the VMWare creds, VMWare throws RemoteException for one of the credentials without exception message, ie, ex.getMessage() is null due to this we get NPE: ex.getMessage().contains(VC_5_5_LOGIN_ERROR)

Once NPE is hit, the extension stops trying the rest of the credentials, and the Collector Context can never be started.

Please note the line number depends on the version of the code we dealing with./the release version which the customer is in.

Solution

The problem is fixed as a part of PRB1329996 [Fixed in NEW YORK].

PROBLEM STATEMENT #10: vCenter Discovery reporting the Discovery log - "No sensors defined"

Root Cause

The error can be due to 2 reasons:

Payload attachment exceeds com.glide.attachment.max_get_size
The Sensors mappings to the vCenter probes are missing

Solution

For Root Cause-1, the solution is addressed via PRB633324 [Fixed in GENEVA].

For Root Cause-2, if multiple sensors mappings are missing then Instance has some issues so it's better if the instance can be zBooted to restore the OOTB content.

PROBLEM STATEMENT #11: Timeout Errors cloning VMware VMs

Root Cause

Timeout error while executing the Workflow - VMware Clone IP0 which is part of the "Linux Virtual Machine" catalog item. When running a build in Madrid, the "VMware Clone IP0" workflow fails on the "Clone VM" activity with a timeout message.

SEVERE *** ERROR *** java.rmi.RemoteException: An error occurred getting a response from the connection at URL https://<<vCenter IP>>/sdk; nested exception is:
java.net.SocketTimeoutException: Read timed out

On looking at the error, we can confirm that it's read times out. So, we can just update the read timeout & connection timeout values.

Solution

Update the AVMwareprobe script include as below:

serviceInstance = new vim25.mo.ServiceInstance(url, credential.getUsername(), credential.getPassword(), _this.ignoreCert, 1000000, 2000000);

PROBLEM STATEMENT #12: vCenter Discovery breaks when login credentials contain predefined XML entities ", &, ', <, or >

Root Cause

Escaping HTML content for the username and password is not being done due to which the vCenter discovery fails because of that reason.

Solution

Any of the below 2 ways can work:

One quick fix is changing the password and avoid customizing the script.
In the vCenter MID Scripts, Please replace a couple of code lines in the scripts. PRB1400358 [Fixed in PARIS] is taken care of to address the problem.

FROM

var username = this.probe.getParameter("username") || "";
var password = this.probe.getParameter("password") || "";

TO
var StringUtil = Packages.com.glide.util.StringUtil;
var username = StringUtil.escapeHTML(this.probe.getParameter("username") || "");
var password = StringUtil.escapeHTML(this.probe.getParameter("password") || "");