Vulnerability Item Creation based on Unmatched CI

Community Alums · ‎10-25-2023

Hi everyone.

I need a layman's term understanding. I have watched videos, but it is not very clear.

From Qualys,

The system will search using QID in the Discovered Item table, if no record is found, it will invoke the CI Look up rule and then if it also doesn't find the CI in the CMDB based on the ci look up rule, my understanding is that it will create a record in the CMDB table as unclassified hardware and also create a record in the discovered item table with the state as unmatched.

Questions:

1. My understand is that, when discovered item state is matched, that is only when VIT should or can be created.

2. At what point does the VIT get created after these two records are created (Unmatched Discovered item record and Unclassified hardware CMDB record) due to not finding the CI

3. From discovered item table, which field contains the QID is it the source data and if I were to search, will I be searching based on keywords on the table?

I will have more questions based on the response I receive.

Thank you in advance.

andy_ojha · ‎10-26-2023

Hey there - can appreciate the complexity here and need to break down these concepts

As you mentioned, there a few videos floating around, sharing this one below if you have not already stumbled on it:

https://youtu.be/-Y-P9hD0hUI?si=JNI7w_za4Sh2NANc

A few things to touch on based on what you have mentioned:

A) Yes, the very first thing that happens when either an asset in the Host List job or an asset in the Host List Detection job comes over -> is to see "have we seen this host from Qualys before", by looking at the Discovered Items and searching the Source ID field which correlates to the Qualys Host ID

B) A Discovered Item gets created, regardless of whether a Qualys Host matches to a CI in the CMDB, or a new CI is created in the CMDB as a match could not be found

C) Unclassed Hardware is NOT the only target CI Class used -- that is just one of the target classes that are possible, and happens to be the best case scenario when Qualys provides IP Address plus another attribute like DNS, FQDN, NetBIOS, etc. If Qualys only provides an IP for the host (e.g. no DNS, no FQDN) -> and we do not find a matching CI, a CI is created in the Incomplete IP CI Class (see that YouTube video linked above)

-------------------------------------------------

1. My understand is that, when discovered item state is matched, that is only when VIT should or can be created.
---> Not true
---> If Qualys vulnerability detections are provided for a host, in VR -> Detections and VITs are created (regardless of that Host matching to a CI in the CMDB or not matching (this is when we create a new CI)

2. At what point does the VIT get created after these two records are created (Unmatched Discovered item record and Unclassified hardware CMDB record) due to not finding the CI

---> What really happens, is first the system looks to see if an existing DETECTION record exists
-> If a corresponding DETECTION does exist already, it gets updated (e.g. Last found date, etc)
--> If a corresponding DETECTION does not exist -> one gets created
--> When the DETECTION is created, then another check occurs to see if a Vulnerable Item (VIT) exists for it
--- If a VIT exists (and is not Closed) --> that DETECTION gets mapped to to the existing VIT
--- If a VIT does not exist --> a new VIT is created and that DETECTION is linked to the VIT

The process above, occurs regardless of whether the Qualys Host matches to a CI in CMDB, or did not match and a new CI is created

3. From discovered item table, which field contains the QID is it the source data and if I were to search, will I be searching based on keywords on the table?

-> Need to clarify, are we talking about the QID (Qualys KB / Vuln) or QID (Qualys Host ID, that maps to the IP Interface on the host)?

-> The Discovered Item will have three key fields you'll want to review :
i) Source ID -> This stores the 9 digit Qualys Host ID for the given host Qualys sends over
- NOTE that in Qualys, this is used to map to a given IP interface for an asset
- If a Qualys Host has multiple IP interfaces, you will see multiple Discovered Items (as the Source IDs are different)
- The Discovered Items here should all reference the same target CMDB CI

ii) Initial Source Data - Stores the raw content (in JSON) of the payload Qualys sends over for the Host (e.g. all the identifying attributes, and host /environment details like OS, Tags, etc. details --> nothing related to vulnerabilities though)

iiI) Source Data - Stores the raw content of the payload Qualys sends over for the host - but this field is special, if certain attributes change over time -> like Name, FQDN, etc -> this field gets updated
- It can be useful to compare Initial Source Data and Source Data over time as attributes on the Qualys host Change
- Additional logic / automation exists when certain attributes change *just FYI*

----------------------------------------------------------------------

The QID (Qualys KB or vuln) is not linked in the Discovered Item; the DI just represents the Host / Asset objective of Vulns...

The QID (Qualys KB / vuln) is linked to the Detection and Vulnerable Item layers...

Community Alums · ‎10-28-2023

You have exceeded my expectations. I am so grateful for your response.

Please see my outlined process.

Qualys scans the network for vulnerability.
Qualys sends vulnerability found with the CI to ServiceNow
ServiceNow checks the discovered item table using Source_ID which is the same as Host ID from Qualys.
1. If found, it will create a
  1. Matched discovered item record.
  2. VIT.
  3. Detection

If not found in the discovered item table, it will invoke the ci look-up rules to find the matching CI in the CMDB table with attributes such as IP ADDRESS , DNS, FQDN
1. If found in CMDB, it will create a
  1. matched discovered item record using the source ID.
  2. VIT.
  3. Detection
2. If not found, it will create an unknown CI record in the CMDB table based on either of the below details.
  1. Unclassified hardware Class = if it has all the NetBIOS, DNS, IP Address, FQDN
  2. Incomplete IP Class = If it no FQDN, DNS
3. Once the CMDB ci record has been created, it will create a record in the Discovered Item as unmatched associating the unknown ci record created.
  1. QUESTION HERE: WILL VIT AND DETECTION NOW BE CREATED BASED ON STEP ABOVE?
4. These things will now follow.
  1. Assignment Rule
  2. Remediation Task Rule
  3. Remediation target rule
  4. Vulnerability Calculator(E.G Risk Score )

Questions:

Based on my process above:

What parameter/attribute does the system use to search from the discovered item table when trying to find if the system has seen the host before?
How can I incorporate the detection process you outlined above into step #3?
QUESTION HERE: WILL VIT AND DETECTION NOW BE CREATED BASED ON STEP ABOVE?
I have a serious issue right now; based on the data sent from Qualys, some of it did not create VIT, and some did. I am trying to search using the QID sent from Qualys from the Discovered item table.
- What could possibly be the reason why some VITS are created and some are not.

Community Alums · ‎10-28-2023

You have exceeded my expectations. I am so grateful for your response.

Please see my outlined process.

Qualys scans the network for vulnerability.
Qualys sends vulnerability found with the CI to ServiceNow
ServiceNow checks the discovered item table using Source_ID which is the same as Host ID from Qualys.
1. If found, it will create a
  1. Matched discovered item record.
  2. VIT.
  3. Detection

If not found in the discovered item table, it will invoke the ci look-up rules to find the matching CI in the CMDB table with attributes such as IP ADDRESS , DNS, FQDN
1. If found in CMDB, it will create a
  1. matched discovered item record using the source ID.
  2. VIT.
  3. Detection
2. If not found, it will create an unknown CI record in the CMDB table based on either of the below details.
  1. Unclassified hardware Class = if it has all the NetBIOS, DNS, IP Address, FQDN
  2. Incomplete IP Class = If it no FQDN, DNS
3. Once the CMDB ci record has been created, it will create a record in the Discovered Item as unmatched associating the unknown ci record created.
  1. QUESTION HERE: WILL VIT AND DETECTION NOW BE CREATED BASED ON STEP ABOVE?
4. These things will now follow.
  1. Assignment Rule
  2. Remediation Task Rule
  3. Remediation target rule
  4. Vulnerability Calculator(E.G Risk Score )

Questions:

Based on my process above:

What parameter/attribute does the system use to search from the discovered item table when trying to find if the system has seen the host before?
How can I incorporate the detection process you outlined above into step #3?
I have a serious issue right now; based on the data sent from Qualys, some of it did not create VIT, and some did. I am trying to search using the QID sent from Qualys from the Discovered item table.
- What could possibly be the reason why some VITS are created and some are not.

andy_ojha · ‎10-30-2023

Hey there - your analysis looks right.

For Qualys and SecOps VR - there is a wrinkle that was introduced recently for Cloud based hosts (e.g. Amazon AWS, Azure, etc) - if not match occured, it creates in the Cloud Resource CI Class

https://www.servicenow.com/community/secops-forum/vulnerability-response-creating-ci-s-of-multiple-c...
See this post above for the quirks here

Questions:

Based on my process above:

What parameter/attribute does the system use to search from the discovered item table when trying to find if the system has seen the host before?
- It really boils down to two key fields on the Discovered item (think of it like Primary keys in a database, or in ServiceNow speak -> coalesce (deduplication values):
  - The Integration Source (Source field that references -> `sn_sec_int_impl`)
  - The Source ID
How can I incorporate the detection process you outlined above into step #3?
- The same (high level) details previously shared about the Detection and Vuln Item lookups / creation, would apply on your Step 3
I have a serious issue right now; based on the data sent from Qualys, some of it did not create VIT, and some did. I am trying to search using the QID sent from Qualys from the Discovered item table.
- What could possibly be the reason why some VITS are created and some are not.
Searching on the DI to start may not really help here - it does verify whether we have accounted for that Host to start with
First - can you confirm, that in the Detections table (sn_vul_detection) - you see the identified vulnerability finding?
- Keep in mind, there is an optional filter that handles 'Closed' detections
- Generally, you do not want to create net new Detections / Vulnerable Items for 'Closed' findings (when they are created their State just goes to Closed) - rather, you would want to update existing records in ServiceNow as they get Closed out
When we say some created VITs and some didn't - need to clarify
- Are we saying that Detections are present, but not linked to a VIT?
- Or - Detections are not present when we expect them to be there
When we say "based on the data sent" -- are you opening up the raw payload (XML) file that ServiceNow fetched from the Qualys API - and cross referencing that to data in ServiceNow?
- Discovered Item table
- Detection table
- Vulnerable Item table

What filtering are we using in our API requests?
- What severity values are we filtering for (3,4,5)?
- Are we doing any sort of special filtering on top of this - i.e. with Search Lists, IP Ranges, Host Tags, etc.

This type of data challenge is hard to diagnose without reviewing examples, log files, trying adhoc imports - your best bet is to create a NOW Support Case - if you have not already created one...