Diving Into ECC Queue Attachments...

[SlightlyLoony]

I've discussed the ECC Queue before; it's the way that messages are sent from the Service-now instance to MID servers, and vice versa. Today I'm going to talk about a detail concerning these ECC Queue messages — specifically, how very large messages from the MID server to the instance are handled.

Certain kinds of probes may return a lot of information. For instance, the list of running processes on a big, busy server can be many thousands of lines long. Another example is big "core" routers; these may have many thousands of routes. For most probes, this data gets stored in the payload field of the ECC Queue — but sometimes the data package is just too big to be conveniently stored in a database field.

On the Hummingbird UAV pictured above, there's a similar problem: it can carry payloads that won't fit in its standard small payload compartment. They solved the problem by coming up with an attachment for the Hummingbird (that "pod" you see hanging below it). Discovery solves the problem in an analogous way — it stores large payloads in attachments to the ECC Queue record.

Let's start by looking at a normal ECC Queue record (screenshot at right). This is the response to a PortScanner probe, and you can see the data returned is directly stored in the payload field. Most of the probe responses you see will look much like this.

Now at left you can see an ECC Queue record where the payload data was stored in an attachment. It's pretty obvious what's going on: I've highlighted the big clue in green, where it says . Toward the top of the screenshot, I've highlighted the [view] link that you can click on to actually see the contents of the attachment in a pop-up window.

There are some additional details worth knowing about:

• A size threshold determines whether a particular payload is stored as an attachment. By default, that threshold is 512,000 bytes, but you can control it with the property glide.soapprocessor.large_field_patch_max. Note that this controls this behavior for all SOAP transactions, so be sure you know what you're doing before you muck around with this.
• Ordinarily these ECC Queue attachments are deleted as soon as the information has been processed. This saves database space and increases performance, both good things — but if you're trying to troubleshoot a problem, it can be annoying. There's a property in Discovery (Discovery Definition→Properties) called Save ECC Queue Attachments that lets you turn this behavior off, either temporarily or permanently.
• All of the data transmitted over the network between the MID server and the Service-now instance is compressed. Data stored directly in the ECC Queue payload field is not compressed; data stored in attachments is compressed.

Whew! Now you know all about Hummingbird Payload Modules and ECC Queue attachments...