Issue:
Discovery is getting stuck on the classification phase currently for workstations.
Troubleshooting steps:
Run a discovery with a specific mid server that i'm testing with
At this time, the powershellprobe worker starts.
01/10/20 10:49:17 (818) Worker-Interactive:Shazzam Worker starting: Shazzam source: See Payload
01/10/20 10:49:17 (818) Worker-Interactive:Shazzam DEBUG: probe()
01/10/20 10:49:17 (819) Worker-Interactive:Shazzam DEBUG: source: See Payload
01/10/20 10:49:17 (819) Worker-Interactive:Shazzam DEBUG: rangeXML: <?xml version="1.0" encoding="UTF-8"?><discovery_ranges><meta_coll><ip_list><ip>10.146.47.189</ip></ip_list></meta_coll></discovery_ranges>
01/10/20 10:49:17 (819) Worker-Interactive:Shazzam Shazzam will scan 1 IP addresses
01/10/20 10:49:17 (819) Worker-Interactive:Shazzam DEBUG: Chunk size: 1
01/10/20 10:49:17 (819) Worker-Interactive:Shazzam DEBUG: processChunk()
01/10/20 10:49:18 (346) LogStatusMonitor.60 stats threads: 293, memory max: 5462.0mb, allocated: 299.0mb, used: 90.0mb, standard.queued: 112 probes, standard.processing: 25 probes, expedited.queued: 0 probes, expedited.processing: 0 probes, interactive.queued: 0 probes, interactive.processing: 1 probes
01/10/20 10:49:20 (825) Worker-Interactive:Shazzam DEBUG: source: See Payload
01/10/20 10:49:20 (825) Worker-Interactive:Shazzam DEBUG: rangeXML: <?xml version="1.0" encoding="UTF-8"?><discovery_ranges><meta_coll><ip_list><ip>10.146.47.189</ip></ip_list></meta_coll></discovery_ranges>
01/10/20 10:49:20 (826) Worker-Interactive:Shazzam DEBUG: Queuing Shazzam wmi,snmp,ssh,http,wins,dns,slp,wbem,vmapp,winrm
01/10/20 10:49:20 (826) Worker-Interactive:Shazzam Sending to queue: Shazzam,wmi,snmp,ssh,http,wins,dns,slp,wbem,vmapp,winrm,See Payload
01/10/20 10:49:20 (826) Worker-Interactive:Shazzam Document: <?xml version="1.0" encoding="UTF-8"?><results active="1" alive="1" full_range="<?xml version="1.0" encoding="UTF-8"?><discovery_ranges><meta_coll><ip_list><ip>10.146.47.189</ip></ip_list></meta_coll></discovery_ranges>" probe_time="3008" range="<?xml version="1.0" encoding="UTF-8"?><discovery_ranges><meta_coll><ip_list><ip>10.146.47.189</ip></ip_list></meta_coll></discovery_ranges>" result_code="0" total="1"><result active="true" alive="true" ip_address="10.146.47.189"><scanner name="GenericTCP" port="135" portprobe="wmi" protocol="tcp" result="open" service="epmap"/><scanner name="NBT" port="137" portprobe="wins" protocol="udp" result="resolved" service="ms-nb-ns"><domain_name>PMCORP</domain_name><host_name>SUD-062662</host_name></scanner><scanner name="SLP" port="427" portprobe="slp" protocol="udp" result="timed_out" service="slp"/><scanner name="DNS" port="53" portprobe="dns" protocol="udp" result="unresolved" service="dns"/></result><parameters><parameter name="mid_selector_details" value="{"mode":"specific_mid"}"/><parameter name="agent" value="mid.server.wpgsnmid09"/><parameter name="used_by_runbook" value="true"/><parameter name="range" value="<?xml version="1.0" encoding="UTF-8"?><discovery_ranges><meta_coll><ip_list><ip>10.146.47.189</ip></ip_list></meta_coll></discovery_ranges>"/><parameter name="range_index" value="0"/><parameter name="shazzam_probes" value="1"/><parameter name="source" value="See Payload"/><parameter name="sys_id" value="8a1f1d49db9a8490509dfcea0c96195f"/><parameter name="shazzam_chunk_size" value="50"/><parameter name="from_host" value=""/><parameter name="sys_created_on" value="2020-01-10 16:49:08"/><parameter name="sys_domain" value="global"/><parameter name="used_by_discovery" value="true"/><parameter name="state" value="ready"/><parameter name="probe_name" value="Shazzam"/><parameter name="port_probe_spec" value="<portprobes><portprobe><name>wmi</name><scanner>GenericTCP</scanner><description>Windows WMI</description><conditional>false</conditional><service><name>epmap</name><port>135</port><type>TCP</type></service></portprobe><portprobe><name>snmp</name><scanner>SNMP</scanner><description>Simple Network Mangement Protocol</description><conditional>false</conditional><service><name>snmp</name><port>161</port><type>UDP</type></service></portprobe><portprobe><name>ssh</name><scanner>BannerTCP</scanner><description>Secure Shell Login</description><conditional>false</conditional><service><name>ssh</name><port>22</port><type>TCP</type></service></portprobe><portprobe><name>http</name><scanner>HTTP</scanner><description>Web Servers</description><conditional>false</conditional><service><name>http</name><port>80</port><type>TCP</type></service><service><name>https</name><port>443</port><type>TCP</type></service></portprobe><portprobe><name>wins</name><scanner>NBT</scanner><description>WINS Name Resolver</description><conditional>true</conditional><service><name>ms-nb-ns</name><port>137</port><type>UDP</type></service></portprobe><portprobe><name>dns</name><scanner>DNS</scanner><description>Domain Name Resolver</description><conditional>true</conditional><service><name>dns</name><port>53</port><type>TCP/UDP</type></service></portprobe><portprobe><name>slp</name><scanner>SLP</scanner><description>Service Location Protocol</description><conditional>false</conditional><service><name>slp</name><port>427</port><type>TCP/UDP</type></service></portprobe><portprobe><name>wbem</name><scanner>GenericTCP</scanner><description>WBEM (CIM)</description><conditional>false</conditional><serviceRegistryQuery querierClassname='SLPQuery'></serviceRegistryQuery><service><name>wbem_https</name><port>5989</port><type>TCP</type></service></portprobe><portprobe><name>vmapp</name><scanner>BannerTCP</scanner><description>vCenter Server Appliance Web user interface HTTPS</description><conditional>false</conditional><service><name>vmapp_https</name><port>5480</port><type>TCP</type></service><service><name>vmapp6_https</name><port>9443</port><type>TCP</type></service></portprobe><portprobe><name>winrm</name><scanner>HTTP</scanner><description>Windows Remote Management/HTTP</description><conditional>false</conditional><service><name>winrm</name><port>5985</port><type>TCP</type></service></portprobe></portprobes>"/><parameter name="debug" value="true"/><parameter name="response_to" value=""/><parameter name="from_sys_id" value=""/><parameter name="priority" value="0"/><parameter name="agent_correlator" value="8e1f1d49db9a8490509dfcea0c96195b"/><parameter name="HTTP_waitForResponseMS" value="500"/><parameter name="probe" value="eb95df760ab301550015543188996362"/><parameter name="processed" value=""/><parameter name="error_string" value=""/><parameter name="sequence" value="16f905c08b00000001"/><parameter name="mid.discovery.max_payload_size" value="-1"/><parameter name="GenericTCP_waitForConnectMS" value="1000"/><parameter name="full_range" value="<?xml version="1.0" encoding="UTF-8"?><discovery_ranges><meta_coll><ip_list><ip>10.146.47.189</ip></ip_list></meta_coll></discovery_ranges>"/><parameter name="name" value="wmi,snmp,ssh,http,wins,dns,slp,wbem,vmapp,winrm"/><parameter name="mid_range" value="<?xml version="1.0" encoding="UTF-8"?><discovery_ranges><meta_coll><ip_list><ip>10.146.47.189</ip></ip_list></meta_coll></discovery_ranges>"/><parameter name="topic" value="Shazzam"/><parameter name="queue" value="output"/><parameter name="ecc_queue" value="8a1f1d49db9a8490509dfcea0c96195f"/></parameters></results>
01/10/20 10:49:20 (829) Worker-Interactive:Shazzam Enqueuing: C:\ServiceNow\agent\work\monitors\ECCSender\output_0\ecc_queue.8a1f1d49db9a8490509dfcea0c96195f.xml
01/10/20 10:49:20 (832) Worker-Interactive:Shazzam Worker completed: Shazzam source: See Payload time: 0:00:03.008
01/10/20 10:49:21 (704) ECCSender.1 Sending ecc_queue.8a1f1d49db9a8490509dfcea0c96195f.xml
01/10/20 10:49:27 (626) Worker-Interactive:PowershellProbe Worker starting: WMIRunner source: 10.146.47.189
01/10/20 10:49:56 (741) Worker-Interactive:PowershellProbe Enqueuing: C:\ServiceNow\agent\work\monitors\ECCSender\output_0\ecc_queue.392f59c9131608506d463ff18144b0ea.xml
01/10/20 10:49:56 (744) Worker-Interactive:PowershellProbe Worker completed: WMIRunner source: 10.146.47.189 time: 0:00:29.111
As you can see from the mid server logs above, the worker completed fine without any errors and did capture all values from the source device correctly. The ECCSender sent the queue .xml file back to service now without returning any errors.
01/10/20 10:49:21 (704) ECCSender.1 Sending ecc_queue.8a1f1d49db9a8490509dfcea0c96195f.xml
Now when we check the discovery, we see that the device is active and is in the Classifying stage (according to discovery log)
If we check the Shazzam ECC queue input, it shows the discovered data from the device (below)
The first line below tells us the device is active and alive, which is good.
<results active="1" alive="1" full_range="<?xml version="1.0" encoding="UTF-8"?><discovery_ranges><meta_coll><ip_list><ip>10.146.47.189</ip></ip_list></meta_coll>
We can then go to the ECC queue and see WMIRunner is stuck on the Classify stage.
The payloads show no errors for either the Input or output of the WMIRunner. All items are processed correctly according to visible logs.
What now happens is the discovery hangs/gets stuck on this stage and if a max run time is specified (in this discovery above I did it without one) it will then time out and give the error show below
I’m unable to identify what is getting hung up here as nothing is shown in the midserver logs, the WMI service is running correctly and no errors are returned.
Also recently started getting an issue where randomly a scan will try to authenticate in the powershell probe using the listed credentials in ServiceNow and this locks the AD account for some reason. After unlocking it, we end up back with the same original issue above with it hanging up. I've also added in local admin credentials which we pushed out via a group policy for PCs and via JAMF for Macs so credential tests pass fine. The issue is the same where the UNIX – Classify rule runs and gets stuck (multiprobe instead of WMI is the only difference).
The only difference I see between the issue with windows machines and mac systems is that the windows machines get a scan status saying “Completed 1” where as the mac systems scans show “Scan 0 of 1” on the devices tab of the discovery.
Attempted Fixes:
Tried modifying the business rule for discovery – complete to the OOB version. Same issue.
Also disabled the winrm_ssl port probe to see if this was an issue with scanner method and use just HTTP instead, still gets hung up at the same phase.
Also tried running test probes, everything works fine there. Ensured the ports and all communication are open between test devices and the midservers and with service now (from what I could check)
I’ve added in our updated jamfagent and sysadmin credentials for the probes to use in WMI and SSH. Still hangs up at the same spot.
Conclusion:
It seems to me that perhaps the scripting that controls the process of the classification phase may be not functioning correctly. I’ve tried adjusting the Discovery – Complete business rule as mentioned to the OOB version of the code ServiceNow provides, but the same issue occurs regardless. This is beyond my expertise as I’m not familiar with how service now handles the java script as its all segmented and not available to pull into an IDE or anything of that sort to test things.
CAN ANYONE HELP WITH THIS? I AM STUMPED!
