Discovery Pattern 'MSSql DB On Windows' Launching Unrelated Process

[aen612]

What happened

The OOB 'MSSql DB On Windows' pattern was handed a 'cmdb_running_process' record with a stale PID. SQL Server had restarted, and Windows had reassigned that PID to a different process. The pattern resolved the PID live, got back the other process, and used it without checking that the executable name matched what it expected.

Step 4 of the pattern, "get version from command line if process exist", builds a command from $process.executablePath. With the wrong process bound, that step launched an unrelated binary on the target host.

The sequence

Pre-pattern data supplied a stored record:

pid=9600
name=sqlservr.exe
command=H:\MSSQL\MSSQL16.MSSQLSERVER\MSSQL\Binn\sqlservr.exe
listening_on=:1433:1434:

Process Detection resolved that PID against the host:

Select * From Win32_Process WHERE (ProcessId = 9600)

A different process came back. The pattern set $process to it anyway:

setAttribute(process,ProcessDTO [executable='<other-agent>.exe',
  executablePath='C:\Program Files\...\<other-agent>.exe', pid='9600', ...])

Step 4's only precondition is $process.executablePath Is Not Empty. That passed, and the pattern ran:

"C:\Program Files\...\<other-agent>.exe" -v

The command timed out after 70 seconds. The pattern logged a tolerated failure and continued.

Result

A process was created on the host, owned by the Discovery service account, with a command line that does not match how that application is normally launched. The command timeout ends the WMI wrapper but not the spawned process, so it stays running until someone removes it.

The MSSQL instance CI also picked up no Version value, since step 4's regex expects Microsoft SQL Server in the output.

Why it is intermittent

On our host the two processes had adjacent PIDs, meaning they start close together during boot. Which one gets which PID varies. The stored record is correct after one boot and wrong for another.

The pattern only has to run in the window between a reboot and the next process probe. The probe itself works fine and corrects the record on its next pass, so most runs are unaffected.

Not specific to this pattern

Any pattern that binds $process from a stored PID and then executes something based on it can do this. 'MSSql DB On Windows' is just where we found it, and the binary that inherited the PID is incidental.

Fix

Process Detection should compare the executable name returned by Win32_Process against the name in the source record before binding $process. Both values are available at that point.

Tightening the discovery schedule narrows the window but does not close it, since a host can reboot at any point in the cycle.
I've submitted a Case with ServiceNow Support. If they fix it, great. If not, I'll update the pattern to verify the returned process and fail if it doesn't match.

[ayushraj7012933]

Hi @aen612 

This looks like a PID reuse issue on the Windows server.

The PID itself cannot be treated as a permanent identity for a process. When SQL Server restarts, Windows can assign the same PID to another process. So, for example, the Discovery record may still have:

PID = 9600
Name = sqlservr.exe

But when Discovery runs:

Win32_Process WHERE ProcessId = 9600

Windows may return a completely different process because that process has now taken PID 9600.

The main issue is that the pattern appears to use the process returned by WMI without first checking whether the executable name matches the process stored in cmdb_running_process.

This becomes a problem in the next step, where the pattern uses $process.executablePath to execute the version command. Since the returned process has a valid executable path, the condition passes and Discovery ends up running something like:

"C:\Program Files\...\other-agent.exe" -v

Recommended fix

I would handle this by adding a validation between Process Detection and the command execution step.

The logic should be:

1. Get the process using the stored PID.

2. Compare the executable/name returned by WMI with the expected executable from the cmdb_running_process record.

3. If they match, continue with the pattern.

4. If they don't match, consider the stored process record stale and do not execute the returned executable.

5. Log the PID and executable mismatch so it can be investigated.

For example:

Expected: sqlservr.exe
Returned: other-agent.exe
PID: 9600

In this case, Discovery should not bind the returned process to $process and should not execute its executablePath.

I would also recommend not modifying the OOB pattern directly. If you need an immediate workaround, clone the pattern and add the validation there. At the same time, raise a ServiceNow Support case because this validation should ideally be handled in the OOB pattern/process detection logic.

Increasing the Discovery schedule frequency can reduce the possibility of stale process data, but it is not a permanent fix because PID reuse can still occur between Discovery runs.

So the key point is: PID alone should not be used to identify the process before executing its executable path. The process name/executable should also be validated.

[aen612]

I submitted a case when I wrote the original post. Looks like ServiceNow Support is going to create a PRB. 

In the meantime I'll either modify the pattern to add a verification step, or clone and add the step. Support has offered to assist with that, so will follow whatever approach they recommend. I'll report back when I have that complete.

[aen612 · Accepted solution]

Support created a PRB, and suggested to update the OOB pattern. I've updated that pattern ('MSSql DB On Windows') adding a step that checks if the running process contains 'sql' and to exit if not.