MID Server runs too many Discovery processes at once

kirkr
Tera Expert

I am attempting to figure out why our mid servers run at 100% CPU. Currently we have 2 Windows MID servers as a Load Balanced Cluster, both with 4 x 2.40 GHz cores and 8 GB of memory.

I've confirmed that both servers have 25 threads under the conf.xml

<parameter name="threads.max" value="25"/>

This happens differently in 2 scenarios I've tried to detail below, but I'm not sure if I should just expect this to happen or if something is awry.

In scenario 1, both servers run tons of Powershell processes at once and max out the CPUs on both servers

In scenario 2, 1 server runs a ton of Powershell processes at once and maxes out the CPU, while the other server is barely used

Looking at the ECC Queue for a Discovery run, they get the same number of tasks to run for each MID server

Does anyone have any ideas?

Scenario 1 - Load balanced cluster

Doing a Discovery Schedule with the following settings causes both MID servers to max out at 100% CPU

Discover: Configuration items

MID Server selection method: Use Behavior

MID Server cluster: MID Cluster01

Scenario 2 - Behavior

Doing a Discovery Schedule with the following settings causes 1 MID server to max out by running tons of Powershell tasks at once and the other MID server spans them out.

Discover: Configuration items

MID Server selection method: Use Behavior

Behavior: Windows & SNMP

Behavior used in a Discovery Schedule

discovery-behavior.png

MID Server 1 (DEV01) during Disovery

mid-discovery03.png

MID Server 2 (DEV02) during Discovery

ProcessesPerformance
mid-discovery01.pngmid-discovery02.png
1 ACCEPTED SOLUTION

doug_schulze
ServiceNow Employee
ServiceNow Employee

Yep, in today's discovery as it always been Java loves it some CPU (and Memory) you are going to find that with your configuration the mid (Java) is going to take as much CPU that your host has to offer to do its work.   You are already at 25 threads so there really isn't a point in going any lower, however, you might consider adding more CPU power if you are seeing that your discoveries aren't finishing in the time you need. I've seen folks with 16 CPU Quad core hosts and the CPU never goes over 60% (Dependant on the tasks).   Look at this way, you're getting your monies worth out of the processing power allocated



As to the differences between the behaviors and the load balancing cant speak too much to since behaviors are really only used for protocol selection nowadays with other features being delegated to other components such as the Clusters and PowerShell


View solution in original post

2 REPLIES 2

doug_schulze
ServiceNow Employee
ServiceNow Employee

Yep, in today's discovery as it always been Java loves it some CPU (and Memory) you are going to find that with your configuration the mid (Java) is going to take as much CPU that your host has to offer to do its work.   You are already at 25 threads so there really isn't a point in going any lower, however, you might consider adding more CPU power if you are seeing that your discoveries aren't finishing in the time you need. I've seen folks with 16 CPU Quad core hosts and the CPU never goes over 60% (Dependant on the tasks).   Look at this way, you're getting your monies worth out of the processing power allocated



As to the differences between the behaviors and the load balancing cant speak too much to since behaviors are really only used for protocol selection nowadays with other features being delegated to other components such as the Clusters and PowerShell


Vini
Giga Guru

Hi  kirkr,

 

As doug.Schulze suggest, have you increased CPU from 8 to 16 processors? is there any improvement on CPU Utilization?.

 

Thanks & Regards,

Vinil.