Horizontal Pattern probe
Summarize
Summary of Horizontal Pattern probe
The Horizontal Pattern probe is a key component in ServiceNow Discovery that enables horizontal discovery using patterns. It works alongside the Horizontal Discovery sensor to update the CMDB based on identification rules. In the ECC Queue, messages from this probe appear under the name “Pattern Launcher” followed by the pattern name.
Show less
For customers creating custom device or process classifiers that use patterns, specifying this probe in the classifier record is essential. However, modification of the probe or sensor is not required.
Payload Splitting
When discovering devices with large data sets such as load balancers or network switches, the MID Server may collect extensive data payloads. To manage this efficiently, the Horizontal Pattern probe splits payloads into smaller chunks, creating multiple ECC Queue records.
Payload splitting occurs in two phases:
- First, the MID Server checks overall payload size against configured size and count limits, prioritizing size.
- Second, it evaluates each Configuration Item (CI) for the number of related items, splitting those with many related items into separate ECC queue pages.
This splitting can result in a large number of ECC queue pages; for example, a network switch with 50,000 related items and a chunk size of 100 would generate 500 ECC queue pages for that single CI.
Key MID Server Properties for Payload Splitting
- mid.discovery.maxtotalitemssizeperpage: Maximum size (default 3 MB) of all items per ECC queue page; checked first.
- mid.discovery.maxcicountperpage: Maximum number of CIs per ECC queue page (default 500).
- mid.discovery.maxrelationcountperpage: Maximum number of relations per ECC queue page (default 10,000).
- mid.discovery.maxrelatedcountperpage: Maximum related items per chunk (default 100); each chunk forms a separate ECC queue page including a copy of the parent CI.
- mid.discovery.shouldminifypatternpayload: When enabled (false by default), trims JSON whitespace in payloads to reduce size.
Tuning Payload Splitting
To optimize Discovery performance and reduce excessive ECC queue pages, customers can adjust MID Server properties based on specific issues:
- If a CI has many related items causing many ECC queue pages, increase mid.discovery.maxrelatedcountperpage.
- If payload size limits cause splitting, increase mid.discovery.maxtotalitemssizeperpage and the MID Server JVM heap size.
- If increasing heap size is not feasible, enable JSON minification by setting mid.discovery.shouldminifypatternpayload to true.
- If splitting is due to high CI or relation counts per page, increase mid.discovery.maxcicountperpage or mid.discovery.maxrelationcountperpage respectively.
These properties are set per MID Server, so changes must be applied to all MID Servers involved in the Discovery schedules. Care must be taken to avoid setting values too high, which could cause out-of-memory errors. Adjust values according to your environment’s needs.
Discovery uses the Horizontal Pattern probe to launch patterns for horizontal discovery.
If you create your own device or process classifier and you want to use patterns for discovery, you must specify this probe in the classifier record. You don't need to modify this probe or the Horizontal Discovery sensor.
Splitting payload
When Discovery uses patterns to discover certain devices such as load balancers or network switches, the MID Server can collect large amounts of data. To better handle this, the Horizontal Pattern probe splits the payload into chunks and create multiple ECC Queue records.
Payload splitting happens in two phases. First, the MID Server checks the total payload against size and count limits. The system checks the size limit first. Second, the MID Server evaluates each configuration item (CI) for the number of related items. When a CI has more related items than the chunk size, the system splits those items into separate ECC queue pages. Each page contains a copy of the parent CI with a subset of its related items.
When both phases interact, the number of ECC queue pages can grow significantly. For example, a network switch with 50,000 related items and a chunk size of 100 creates 500 ECC queue pages for that single CI.
| Property | Default value | Description |
|---|---|---|
| mid.discovery.max_total_items_size_per_page | 3,145,728 bytes (3 MB) | Sets the maximum size limit of all items (CIs, related items, relations, and references) per ECC queue page. The system checks this threshold first. |
| mid.discovery.max_ci_count_per_page | 500 | Sets the maximum number of CIs per ECC queue page. |
| mid.discovery.max_relation_count_per_page | 10,000 | Sets the maximum number of relations per ECC queue page. |
| mid.discovery.max_related_count_per_page | 100 | Sets the maximum number of related items per chunk. Each chunk becomes a separate ECC queue page containing a copy of the parent CI. |
| mid.discovery.should_minify_pattern_payload | false | Enables JSON minification for pattern payloads. When set to true, trims JSON whitespace from the pattern payload to reduce its size. |
Tuning payload splitting
| Issue | Action | Related property |
|---|---|---|
| A CI with many related items creates an excessive number of ECC queue pages | Increase the related items chunk size | mid.discovery.max_related_count_per_page |
| Payload size is the splitting bottleneck | Increase the size limit and the MID Server JVM heap size | mid.discovery.max_total_items_size_per_page |
| Payload size exceeds the limit but increasing the JVM heap size is not feasible | Enable payload minification to reduce the JSON payload size | mid.discovery.should_minify_pattern_payload |
| A high CI count per page is causing splitting | Increase the CI count limit | mid.discovery.max_ci_count_per_page |
| A high relation count per page is causing splitting | Increase the relation count limit | mid.discovery.max_relation_count_per_page |
- These properties are per MID Server. Apply changes to all MID Servers involved in the affected Discovery schedules.
- If necessary, increase mid.discovery.max_total_items_size_per_page and the MID Server JVM heap size accordingly. Setting these values too high can cause out-of-memory errors on the MID Server and instance nodes. Set values appropriate for your environment.
- Setting mid.discovery.should_minify_pattern_payload to true can reduce payload size without requiring heap size increase.
- For environments where a CI has many related items, but fewer total CIs, the primary bottleneck is likely mid.discovery.max_related_count_per_page. Start by increasing that property.