Understanding Probe Results Cache returning as "Processed"

[omarali]

The ECC queue input payload field contains the execution results of a specific Discovery probe or an Orchestration activity. Therefore, it is considered invaluable source of debugging information when it comes to troubleshooting a failed Discovery or Orchestration activity or an unexpected results of those activities including errors and warnings returning from the target network devices.

Discovery uses XML documents for the payload, and more often than not, these are considerably large documents. However, sometimes when you look at the payload field of an input ECC queue record you only see the word 'processed' there.

Then, if you try to view the payload as a formatted XML document by clicking on the XML button, you will only to get the error message: This page contains the following errors: error on line 1 at column 39: Extra content at the end of the document [as seen in the screenshot below].

Let's start with what the word 'Processed' actually means. Discovery performance is improved by caching probes results on the instance and only processing results that have changed. Probe results that have not changed do not need sensor processing, and therefore, the sensor does not run. The cache is turned on by default for base system probes and sensors whose output is unlikely to change. The word 'Processed' means that the probe results of that specific configuration item (CI) are cached and have already been 'Processed' and the payload hasn't changed since. The error seen later on, when clicking on the XML button, simply means that the word 'processed' is not a valid XML document.

Sometimes, you will need to look at the original payload for debugging reasons but, by default, after 7 days the ECC queue records are rotated and the only thing you will see in the payload field of the cached probe is the word 'Processed'. To get the original payload again you need to clear the cached probe result and run discovery again.

Clearing cache for a probe or configuration item pair

Where are the cached results are stored and how can I clear cache for a specific Probe or CI pair? The probe results cache records are stored in the Probe Results Cache [discovery_probe_results_cache] table in a Key/Value pair format. Which may seem a bit ambiguous at first sight:

The secret to understand these values is to know that they are base64 encoded in the following way:

• The "Key" value is the concatenation of the probe sys_id in base64 and the CI sys_id in base64.
• The "Value" value is the MD5 checksum of the probe result encoded in base64.

This is an efficient way to compare new payload result against the cached value without actually having to save the entire payload as the payload of some probes could be huge.

To find the cached result of the probe or CI:

1. Grab the sys_id of the probe you want to find its cached result. For example: The sys_id of 'Linux — Installed Software 'is 2C93A59C0A0A0A8B00D5F029CCAF5573
2. Convert the hex value of the sys_id to base64 using an online tool. For example: http://tomeko.net/online_tools/hex_to_base64.php?lang=en

   *You should get a string like: LJOlnAoKCosA1fApzK9Vcw==

3. Grab the sys_id of the ci you want to find the probes results for and convert it to base64 (eg one of my linux test servers sys_id is D4AE1DEC4F3E960027B350AF0310C756 and its base64 string is 1K4d7E8+lgAns1CvAxDHVg==
4. Concatenate the base64 strings of steps 2 & 3.

You will get the following string: LJOlnAoKCosA1fApzK9Vcw==1K4d7E8+lgAns1CvAxDHVg== which is the 'key' value found in the Probe Results Cache [discovery_probe_results_cache] table for my linux test server for the payload returned by the 'Linux — Installed Software' probe.

Having a better understanding of how probe results cache works will help you be a better ServiceNow admin. Don't fret when you see 'Processed' in the payload field instead the actual payload, as this can be remedied by clearing the cache and reassessing the cached result.   I hope that after reading this post you now have a better understanding of how the probe results cache works and what it is for. As well as why the word 'Processed' is seen in the payload field instead the actual payload. Most importantly, you now know how to delete the cached result of a specific Probe or CI, so you can get the original payload to be able to troubleshoot related issues.

Cheers,

Omar