MID Server Resource Reservation

This feature allows resources to be reserved before a task is assigned to a thread. If the resources the task needs aren’t available, then that task waits in the MID Server work queue while other tasks are assigned to the thread.

For Discovery, the MID Server tasks are the probes or patterns it executes. While probes are waiting to be executed on the MID Server, they provide resource requirements (CPU, memory, and so on) and if they cannot be met, the probe waits in the work queue until the resources are available.

注:

MID Server Resource Reservation delays probe execution until resources are available. This is based on extremely flexible criteria. The MID Server Resource Reservation feature is for advanced users and should not be configured until a problem is identified. This configuration depends entirely on the problem details.

Using the MID Server Resource Reservation allows for better use of threads since the threads are not assigned a task it cannot complete.

Features:

• Resource reservations apply to a single MID Server only
• Resources can depend on system or MID Server properties
• Resources can depend on probe parameters, allowing you to limit the number of active probes and patterns against a given IP
• Reports resource usage
• Extensible (the customer defines their own resources)
• Scriptable

Benefits:

• Keeps PowerShell probes from impacting execution of other probes
• Can be used to limit the impact Discovery has on a target system
• Can be used to limit the number of ”large” probes being executed by a MID Server at the same time
• Can be used to throttle Discovery (to minimize impact on the instance)

Use MID Server Resource Reservation

A resource is just a name and a count. Define the name of the resource, which probes use it, and how many of the resources those probes should use.

始める前に

Role required: admin

• Make sure the MID Server property mid.probe.wait.resources is set to true to enable MID Server Resource Reservation. Changing this property requires restarting the MID Server.
• Choose a resource name. Anything in { } in the name is replaced by the probe parameter with that name. This name is typically used for per-host resources, for example, ssh{source} means a separate resource named “ssh” is available on every host being discovered. (“source” is the name of the probe parameter that contains the IP address being probed.)
• Decide how to specify the number of the resource used by probes or patterns.
  • Every probe uses a constant number of the resource: Create a “simple value” resource.
  • The number of resources used depends on a system property: Create an “expanding” resource.
  • The number depends on multiple factors: Create a “scripted” resource.
  • Expanding: Anything in { } is replaced by the system or MID Server property with that name. Logical operators are allowed, for example, "{mid.windows.management_protocol}".toLowerCase() == "winrm" ? 1:0
  • Scripted: the script is evaluated. The return value is expanded.

手順

1. Create a new resource.
   1. Navigate to Discovery Definition > Limited Resources> Simple Value and click New.
   2. Enter a resource name.
   3. Enter the number of resources used and click Save.
2. Define which probes use the resource.
3. Navigate to the desired tab and select from the list.
   • Used by Topic tab: Includes all probes with that topic. Click Invert Topic to include all probes without that topic. For example: SSHCommand with Invert Topic, means all probes except for SSH. Heartbeat and queue messages are always excluded.
   • Used by probes tab: Includes a list of probes. Click Invert Probes List to include all probes without that probe. Horizontal Discovery probe means apply to all patterns.
   • Used by patterns tab: Includes a list of patterns. You can also click Invert Pattern list.
4. Define availability on the MID Server.
5. Navigate to Discovery Definition > Limited Resources > MID Resources.
   • Resource: This is the reference to the resource.
   • Available: Number available. Anything inside brackets, is replaced by the system or MID Server property.
   • MID Server: MID Server to apply this to. (Empty) means all.
6. Set the resource order.
   1. Click in the Order field of each resource.
   2. Type in the number.
   3. Click the check mark to Save.

注:

Less valuable resources should have a lower order – these resources are reserved and held until higher order resources can be obtained. For example, both the total PowerShell sessions (resource is “PowerShell”) and the number of simultaneous sessions to any host (resource is “PowerShell{host}”) are limited. Therefore, PowerShell{host} resource should have a lower order. Obtaining PowerShell first would impact all other PowerShell probes. Obtaining “PowerShell{host}” first only impacts other PowerShell probes to that host.

7. Review the Resource State that you have set up.
   1. Run a Discovery.
   2. Observe Resource State.
   3. Click Get resource state from the MID Server page or view in an ECC input payload.

      Result of Get resource state:

      • PowerShell{source} has additional entries for every {source} with a current allocation.
      • “Reserved by” may show multiple resources if the probe requires more than one resource.

      Result of ECC input payload:

      • The resource_wait attribute only exists if the probe had to wait for a resource.
      • The time displayed is in milliseconds.
      • The probe may have waited on multiple resources, with different wait time for each.

How MID Server Resource Reservation works

In addition to the work queue, the MID Server contains a waiting queue. When picking a task for execution, the waiting queue is always checked first. If no task in the waiting queue can execute, then the work queue is tried. Probes and patterns in the waiting queue reserve all necessary resources until they’re able to execute.

The first probe or pattern in the queue is offered available resources. It takes any of the lowest order resource that is available. After getting the required number of lowest order resources, the probe or pattern goes to the next lowest order resources, and so on. The order allows the user to configure which resources are more or less important. Lowest order resources are gathered first because holding these resources has less impact on execution of other probes and patterns.

Tables

mid_limited_resource

• Defines the resources used by a probe or pattern.
• Extended by mid_limited_resource_value, mid_limited_resource_expanded, and mid_limited_resource_script each containing a single additional field.
  • mid_limited_resource_value adds a single field named “value” to the base table.
  • mid_limited_resource_expanding adds a field named “expanding.”
  • mid_limited_resource_script adds a field named “script”.

mid_resource

• Sets the resources available on a MID Server.
• Values from this table are copied to ecc_agent_property.
• A business rule on this table creates the corresponding MID Server properties.
• It has a reference to a resource and the number available. If the number available is inside { } then it’s the name of a system property, MID Server config, or MID Server property (all three places are checked). So {mid.powershell_api.session_pool.max_size} is the value of that MID Server config that sets the size of the PowerShell session pool.

表 : 1. Base: mid_limited_resource table

Label	Column	Type	Size	Information
Resource Name	name	String	100	Unexpanded name of the resource
Active	active	Boolean		Allows temporary disable
Invert Topic	invert_topic	Boolean
Invert Probe List	invert_probe_list	Boolean
Invert Pattern List	invert_pattern_list	Boolean

注:

There are m2m tables that associate a mid_limited_resource record with topics, probes, and patterns. The “invert_” fields change the list from inclusion to exclusion.

表 : 2. Extensions: mid_limited_resource_value, mid_limited_resource_expanding, and mid_limited_resource_script tables

Label	Column	Type	Size	Information
Value	value	Integer
Expanding	expanding	String	1000	A slightly extended version of the availability number because it accepts logical and ternary operations, for example, "{mid.windows.management_protocol}" == "WinRM" ? 1 : 0. If the management protocol is WinRM this evaluates to 1, otherwise it’s 0.
Script	script	String	4000	The script is evaluated. If the result is a string, then it’s expanded

表 : 3. mid_resource table

Label	Column	Type	Size	Information
Active	active	Boolean		Allows temporary disable
Available	available	String	255	Number of this resource available on MID Server
MID Server	ecc_egent	Reference		Reference to MID Server or empty for all
Order	order	Integer		Order in which resources are allocated
Resource	resource	Reference		Reference to the resource