High CPU Usage on Web/App Server - Service Mapping - Find.exe "sybase"

[Will Patterson]

I just left a Major Incident call where I was notified that a process (find.exe "sybase") was running on 2 Windows Servers in a pair, that were initiated by the Elevated Privilege (Domain Admin) account used for Discovery and Service Mapping.

I had been running a Service Map Discovery on the related Application Service earlier this afternoon (active error from 12:02PM CT) and assumed that the Service Mapping had concluded. There were 2 identical processes on each of these 2 servers. I am not referring to the MID Server, but the host machines running a production website. The processes were still running on those servers well into the evening causing 100% CPU usage.

1. Has anyone ever experienced this behavior before?
2. Is there anything that can be done to prevent this from happening again in the future?
3. I was under the impression that Discovery and Service Mapping were relatively low-risk, was I wrong?

I'm also concerned as to why this process would continue to run for several hours without some kind of safeguard in place to say if it's run for more than an hour to stop it and report the delay in process completion back to the log for investigation..

Any advice, recommendations, or steps to avoid this happening again are greatly appreciated.

Also any tips on a way to restore faith in Service Mapping to the individuals throughout the organization and to leadership will be very valuable.

Off to continue my research for a possible root cause, other than what I have now of: "Service Mapping kicked off a process that got hung and was never killed."

[SiD2]

Hi,

I just want you to confirm me some things. As you told issue occurs in service mapping, what is the node that connects you to sybase node in your map?

There must be some other pattern which makes a connection to Sybase pattern using sybase endpoint. I'm asking as I suspect one library for this "find" issue, but want to make sure if that is the same running at your end. As there are multiple patterns which have a connection to sybase, wanted to know the flow at your end.

You can check the pattern log of the previous node which makes connection to sybase. In that connection pattern i want to make sure whether this suspected pattern is the reason.

You can send me the screenshot of the pattern log of the previous node to this sybase if it is fine for sharing here.

The KB you referred w.r.t postgres was also as a result of my fix with some other customer.

[Will Patterson]

Hi,

Thanks for your response. Unfortunately when I try to go into the Discovery Log for the Application Service it says no results. I'm not sure what happened to them. From my research into the various CIs on the Service Map, there is nothing related that is running Sybase. We are a Microsoft shop, and as a result have 95% or more of our databases in MS SQL.

The only nodes in this map are the entry point (http), the Load Balancer (F5) that connects to 2 servers running the site and the IIS sites running on them. The last connection is to a MS SQL database. The connection at the bottom left is deprecated and the server is no longer online. I'm not sure why this node is being mapped, other than possibly somewhere in the code of the application it is still pointing to that server. 

The find.exe processes were running on the 2 servers in the image that host the IIS site.

[SiD2]

I couldn't pinpoint to the location where issue is happening. As you said and from the map I don't see any sybase related patterns executed.

May I know, why in your title it is mentioned sybase? If some sybase related nodes or discovery is happening in your env, as said, there is an initial suspected location was w.r.t sybase related pattern only.

From these patterns from map I no where observe "find" getting executed, its always "findstr" used in these patterns.

Reg the log, discovery erases the logs after few days. You can try doing a discovery again and let me know if Sybase or any other patterns being executed, we can check the occurrence of "find" in that.

We are correcting the occurrences of "find" with "findstr" to avoid these hang issues, and that is where u even see the KB with some other pattern.

If we can know the patterns from log after your rediscovery, we can trace, otherwise I could only help this much.

[Will Patterson]

Honestly, I'm not sure which pattern this is from. The servers in question do not even run a database or database instances of any kind.

Unfortunately, the most amount of information that I have is that I got a call from my Operations Center reporting 2x "find.exe sybase" applications running on two hosts and that they were initiated by the Domain Admin account that I use for Discovery. I immediately told them to kill the processes. There was no output within ServiceNow that showed that Discovery or Service Mapping was still running, so these processes apparently were hung and never killed after SM finished.

I will continue to review my discovery patterns to see if I can find what was kicked off and why and report back to this post when I find it.

Thanks,

Will