Can a "Unclassed Hardware" CI-bound DI ever re-match to the real CI once richer payload arrives?

[lyubenkonov]

This is a purely theoretical question with simplified conditions.

Setup:
A third-party integration (say Qualys) brings in a discovered item (DI) with almost nothing in the source data, only a Qualys Host ID.

Two CI lookup rules exist:

• Qualys Host ID (order 100)
• Serial Number (order 200)

There is an actual CI in the CMDB for this asset, but it has only a serial number, no Qualys Host ID.

On the first run nothing matches, so IRE creates a placeholder in Unclassed Hardware carrying only the Host ID. The DI now points at that placeholder via Qualys Host ID: the CI matching rule shows Qualys Host ID, but the DI stays Unmatched, since the match is to a placeholder in the Unclassed Hardware class.

Question:
Later, Qualys sends a richer payload for the same asset that now includes the serial number matching the real CI, and the "Re-apply CI Lookup Rules on the Changed Discovered items" job is enabled and runs.

Since the DI is already bound to the placeholder by its Host ID, wouldn't Qualys Host ID (order 100) keep winning before Serial (order 200) is ever evaluated? Would it ever be possible for the DI to reach the real CI on serial number on a later reapply?

If not, is removing or reconciling the placeholder the only way the DI would ever match the real CI?

(Note: reordering the rules is not the fix I'm asking about.)

[andy_ojha]

Hey there,

In theory this should work out.  

The part I am not certain about, if the Qualys Host List job or Qualys Host Detection, actually bringing in Serial number for the asset in Qualys?  Perhaps that is a new object sent in the payload for the Qualys VR jobs?

Interesting -> I do see Qualys sending "SERIAL_NUMBER" now, that was not there before from what I recall.

There is logic in-place that can help here.

Once a change in the Discovered Item > Source data is detected, it should set a flag on the respective Discovered Item record.  Each Discovered Item pertains to a given asset in Qualys (DIscovered Item > Source ID will show the corresponding Qualys Host ID).

That flag (Re-evaluate CI) will be set to True, once a change to an object is detected (like DNS, etc.).   There is another field on the Discovered Item (Initial Source Data), this would capture the first snapshot of the raw context that may be helpful as well.

Then a scheduled job can be used to crawl through those 'flagged' Discovered items and re-evalaute them against your SecOps CI Lookup Rules again.

This page has more detail on that
- https://www.servicenow.com/docs/r/security-management/vulnerability-response/re-evaluate-discovered-...

One activity to look you can do, is look at your list of Discovered Item records at the moment and see how many have the "Re-evaluate CI" flag set to True

The scheduled job likely has not ran (Re-apply CI Lookup Rules on the Changed Discovered items), as it is set to On-Demand by default

[lyubenkonov]

Hello Andy,

Yes exactly, Host List is what brings in SERIAL_NUMBER from the Agent-scanned assets from Qualys.

As for your explanation, agreed, we'll end up re-applying the CI Lookup Rules either manually or via "Re-apply CI Lookup Rules on the Changed Discovered items." But that's again where my question comes:

-> Let's say that the Discovered Item already matches the Unclassed Hardware CI via the Qualys Host ID rule, which runs before the Serial Number rule. On re-apply, the Host ID rule runs first and matches the Unclassed Hardware CI yet again, so the chain maybe stops there and the Serial Number rule further down is never reached?

My preference would be that an "order 200" rule matching a real, properly classed CI takes precedence over an "order 100" rule that only matches an Unclassed Hardware CI. Or once the first match is found, is that final regardless of CI class?

[andy_ojha]

Hey there -- totally understand

Tough to say without seeing the data we have populated in CMDB (i.e. Serial number hydration), but I think we'd agree that Serial Number, objectively is more authoritative over Qualys Host ID in this situation (factors that may influence this otherwise, would be if we are using the Qualys Service Graph Connector, etc).

I think it'd be reasonable to move your Serial number rule up to run first, and that would cover us when an asset initially comes in (try to find a real geniuine CI with that value), as well as during later re-evaluations.   The Unclassed HW CIs created by VR-Qualys, do not stamp Serial Number values onto the CIs it creates, like it does for for Qualys Host ID.

Then you won't have to fight with trying to exclude CI Classes per rule (scripted condition) or globally across all CI Lookup rules (sn_sec_cmn.ignoreCIClass), to try to influence the rules to exclude matching to CIs in an unresolved class (E.g. Unclassed HW).  Meaning, we won't accidental just match to another Unclassed HW CI like we would with the Host ID rule if we bump the Serial Number rule to run first.

As each CI Lookup Rule runs, there is logic behind the scenes that forces the actual rule being evaluated to try to pick a genuine real CI over an unresolved class CI (e.g. Unclassed HW, Incomplete IP).  This helps more if we have duplicate CIs and the rule matches to multiple CIs, it will pick one of the genuine CIs first, over Unclassed HW, Unmatched CI, Incomplete IP, etc.  But as you said, this may not help if we keep that Host ID rule running first in the order.

Bumping your Serial number rule up to run first, would seem like a durable starting point, and it'd be safe as it would really just match on genuine / real CIs in CMDB first, and if no match is found then move to your next rule (e.g. FQDN, Host ID, etc).

There may be further tuning to consider on the downstream rules as well - but let me know if that helps and sounds reasonable to you, to bump Serial number to run first.  Feels like best bang for buck to start with, for the specific situation of Serial number + and richer data coming in after the first import from Qualys.