High CPU Usage on Web/App Server - Service Mapping - Find.exe "sybase"

[Will Patterson]

I just left a Major Incident call where I was notified that a process (find.exe "sybase") was running on 2 Windows Servers in a pair, that were initiated by the Elevated Privilege (Domain Admin) account used for Discovery and Service Mapping.

I had been running a Service Map Discovery on the related Application Service earlier this afternoon (active error from 12:02PM CT) and assumed that the Service Mapping had concluded. There were 2 identical processes on each of these 2 servers. I am not referring to the MID Server, but the host machines running a production website. The processes were still running on those servers well into the evening causing 100% CPU usage.

1. Has anyone ever experienced this behavior before?
2. Is there anything that can be done to prevent this from happening again in the future?
3. I was under the impression that Discovery and Service Mapping were relatively low-risk, was I wrong?

I'm also concerned as to why this process would continue to run for several hours without some kind of safeguard in place to say if it's run for more than an hour to stop it and report the delay in process completion back to the log for investigation..

Any advice, recommendations, or steps to avoid this happening again are greatly appreciated.

Also any tips on a way to restore faith in Service Mapping to the individuals throughout the organization and to leadership will be very valuable.

Off to continue my research for a possible root cause, other than what I have now of: "Service Mapping kicked off a process that got hung and was never killed."

[Ashutosh Munot1]

Hi,

Ideally what i have seen is few processes like powershell consumes most of the CPU but after processing we kill that as well and if something is waiting long it is killed as well.

On Each schedule you have a cancel window like if the schedule runs for 2hrs then cancel then whole schedule field name is max run time.

Thanks,

Ashutosh

[Will Patterson]

Ashutosh,

I appreciate the response, but you're referring to Discovery. This was not a Powershell command that was running on the MID server, this was a "find.exe" process running on the target hosts that was initiated from Service Mapping.

Unless I'm mistaken, I've never seen anywhere in any Service Mapping screens the option to create a schedule, nor the option for max run time. I have seen this and actively use it for Discovery schedules. Since Discovery uses IP addresses and could run for a longer period of time, it makes sense to include the option to kill Discovery after a set amount of time since you could set up a schedule to run on an entire network. However, Service Mapping does not function that way. When you initiate Top-Down Discovery on an Application Service, it is only running commands on the inter-connected systems of a single service.

Thanks,

Will

[Ashutosh Munot1]

Hi,

Right.

Sorry for the confusion though.

I was just looking at the pattern:

See it uses findstr but still an issue right.

Thanks,
Ashutosh

[Will Patterson]

I've found a HI ticket that references an almost identical issue. This one was reported 7 days ago but in this case references PostgreSQL rather than Sybase, but the cause and symptoms are the same:

https://hi.service-now.com/kb_view.do?sysparm_article=KB0827975

I will continue to follow this ticket for further updates. 

Thanks,

Will