Discovery takes long time to complete on Windows Servers

[amolpunekotwal]

Hello Experts,

We have some windows servers where discovery takes 7-8 hrs to complete. Only error I see in the pattern logs are about SQL Server Cluster commands failing as below. Interestingly if we run discovery from different mid server with same ip it gets completed within 5 minutes.  What could be difference between these 2 mids? Network path? Any configuration setting which we needs to validate?

Also even if we some command does not work why it takes so much time declare it failure? As i see most of the commands has 300 seconds timeout value.

We are unable to find out the root cause of this. Please let us know if anyone has experienced the same issue and shed some light.

2024-02-29 23:28:45: Executing WMI query on host: removed original ip with namespace: root/Microsoft/SqlServer and query: SELECT Name FROM __NAMESPACE

2024-02-29 23:28:47: Error during execution of Windows command: executeQuery -Namespace root/Microsoft/SqlServer -Query "SELECT Name FROM __NAMESPACE" due to com.snc.automation_common.integration.exceptions.InvalidCommandException: executeQuery : Could not get query SELECT Name FROM __NAMESPACE on namespace=root/Microsoft/SqlServer. Original Exception: Invalid namespace

At line:1 char:5

+ & { executeQuery -Namespace root/Microsoft/SqlServer -Query "SELECT N ...

+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException

+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,executeQuery

2024-02-29 23:28:47: Execution time: 2899 ms

Thanks & Regards,
Amol Kotwal

[DevLeoric]

We are going through the same error, did anyone find a fix??

[AJ-TechTrek]

Hi @amolpunekotwal ,

1. Why One MID Is Slow and Another Is Fast
Even though the MID Server uses the same patterns, the following differences can cause delays

Area	Possible Difference	Impact
Network Path	Firewall routing, packet loss, or higher latency from MID to target	Commands time out or retry, drastically extending Discovery
DNS Resolution	One MID might have slow or misconfigured DNS	Adds delays to every connection attempt
WMI/WinRM Connectivity	Firewalls or security policies cause fallback/retries	Discovery waits until timeout before moving on
MID Server Resources	CPU/memory load, Java heap size, or thread pool	Slow execution and slower ECC Queue processing
Credential Affinity	MID might be attempting multiple credentials due to ordering	Adds multiple failed connection attempts before the right one is used
Antivirus / EDR	Scanning every PowerShell/WMI call on one MID	Significant delays per command

2. Why a Failing Command Can Stall Discovery
* In Discovery patterns, each step has a timeout (often 300 seconds).
* If a step retries multiple times (e.g., due to multiple namespaces/credential attempts), the time multiplies.
* In cluster discovery, the pattern might iterate through multiple nodes, repeating the failing step per node.
* The default retry logic in the MID Probe framework doesn’t always short-circuit after one failure—it can cascade delays across the whole schedule.

3. Things to Check & Compare Between MIDs
Run these checks on both MIDs and compare:

Network/Connectivity
* ping <target> — latency and packet loss.
* tracert <target> — routing differences.
* Test-NetConnection -ComputerName <target> -Port 135 (WMI) and SQL ports.
* Test WMI namespace manually:
Get-WmiObject -Namespace root\Microsoft\SqlServer -Class __Namespace -ComputerName <target>
* 
If this fails instantly on the fast MID but hangs on the slow one, you have a network/firewall delay.

MID Server Configuration
* MID Server logs: agent0.log and wrapper.log — look for slow step execution or thread blocking.
* Check MID Server capabilities — both should have the same windows, power_shell, wmi capabilities.
* Java heap size (-Xmx setting in wrapper.conf) — insufficient memory can throttle MID performance.
* Thread pool settings (mid.pool.size parameter in agent.conf).

Credential Behavior
* Check the Windows credential order in Discovery > Credentials.
* Use the “Test Credential” option from the slow MID to see if it succeeds instantly or after retries.
* If multiple credentials match, reorder so the correct one is first.

Security / AV
* If EDR/Antivirus is running on the slow MID, it may intercept every WMI or PowerShell call.
Exclude C:\ServiceNow\MID Server from scanning and test again.

4. Reducing Delay When Commands Fail
If the namespace (root/Microsoft/SqlServer) doesn’t exist on the target, you can avoid long timeouts by:
1. Editing the SQL Server pattern and wrapping the WMI query step in a conditional check for the namespace.
2. Lowering the step timeout in the pattern from 300s to something smaller (e.g., 60s) for this query.
3. If you’re not discovering SQL Server clusters, disable the cluster-specific pattern steps.

5. Recommended Next Steps
1. Test WMI namespace manually from both MIDs.
2. Compare network latency and routing.
3. Check DNS performance (nslookup target).
4. Review MID Server resource usage and config.
5. Adjust pattern timeouts for failing SQL Server cluster steps.

Please appreciate the efforts of community contributors by marking appropriate response as Mark my Answer Helpful or Accept Solution this may help other community users to follow correct solution in future.
 

Thank You
AJ - TechTrek with AJ - ITOM Trainer
LinkedIn:- https://www.linkedin.com/in/ajay-kumar-66a91385/
YouTube:- https://www.youtube.com/@learnitomwithaj
Topmate:- https://topmate.io/aj_techtrekwithaj (Connect for 1-1 Session)
ServiceNow Community MVP 2025