Remediate Duplicate "Discovery items" for Customer Usage (Licensing)

[Brian43]

We are attempting to calculate our customer usage for the last 90 days in the Security Operations module, Discovered items (sn_sec_cmn_src_ci table).  The KB0861920 indicates that what is counted is; a server, a desktop, a laptop, a printer, a firewall, and any other IP-enabled device that the vulnerability scanner scans.

When we review the results, there are duplicate entries for various devices.  Many are showing the same name, serial number, FQDN, IP Address, etc.  Here is an example:

We have 17 F5 BIG-IP devices in our environment.  There are over 400 "Discovered items" listed in our report on "Discovered items in the Last 90 Days".  When filtered by IP address, there are only 9 devices showing.  Their IP address, name, FQDN, etc. are all the same.

For Customer Usage, the report appears to show that we have over 400 devices, when, in fact we are only scanning 9 devices.

How do I remediate these duplicates and merge them together to show the true count for customer usage?

[Chris McDevitt]

Hi,

Some background:

When the source data comes in from the scanner, the first thing the SN VR API does is check the DI table for a matching Source ID in the source record. 

-- IF the API FINDS a matching Source ID, it uses that DIs CI to create the VIT (along with the Vulnerability).

-- IF the API FAILS to FIND the matching Source ID, then the CI Lookup Rules are consulted.

--- IF the CI Lookup Rules FIND a match in the CMDB THEN the API (1) Create the VIT with that CI and (2) insert a record using the Source ID, into the DI table.

--- IF the CI Lookup Rules FAIL to Find a match in the CMDB THEN the A{O (1) create a new CI (Incomplete IP or Unclassified Hardware) (2) creates a VIT with the new CI (3) insert a record using the Source ID, into the DI table.

The Source ID comes from the Scanner, and it indicates to the SN VR API that this incoming record is different.

To answer your question, no, the CI Lookups are not causing the DI records to be created. 

As for duplicates... from a human perspective, they may look like duplicates. From the perspective of a machine, they represent unique occurrences of scanning.

As for licensing, the best person to discuss this with is your sales representative. 

[Mike G1]

Thank you for the confirmation.

What if the same (or nearly same) records come in but with different source ID values? For simplicity, I am focusing on the hardware scans.

Some scans include the domain in the FQDN & HOSTNAME fields along with values for credential assessments, and last scan start/end while other scans populate only the simple hostname as FQDN & HOSTNAME. These nearly same records have different source ID values.

Some scans with only the simple hostname have different IP address and different source ID values.

(These scans with different source ID values successfully match to the correct CI value. I would think that accurate CI data is the intended outcome, but apparently multiple DI records impact licensing cost.)

[jlaps]

I think we are seeing the same sort of issue in our instance. We have ~6000 CIs, but over 10,000 DIs. If we group those DIs by CI, we get very close to what we believe our correct unique device number is. 

I do not understand why the different SOURCE IDs is coming into play, as I am not doing any of the Qualys side stuff... SN is wanting a ton more license money from us, so I am trying to figure out how they are counting and why our numbers are so far off.

The SOURCE for all of our DISCOVERED ITEMS is "QUALYS" but the SOURCE ID is different; I can see some DIs have 16 different DIs all rolling up to the same CI, but SN is counting it as 16 devices instead of 1, and I am not understanding if this is a SN problem, or a Qualys problem.

[Chris McDevitt]

So.... Not a Quays expert but....

Each unique source id should represent a unique occurrence in Qualys. If you are running Authenticated and Unauthenticated scans against the same host then that may account for some of the differences. If you are running unauthenticated scans and a host has multiple IP addresses, this too will be represented as unique occurrences in Qualys.

I would start with Qualys to understand how scanning is occurring and see if you can unify the scans into a single entity. 

[MGanon]

As I understand it, the sn_sec_cmn_src_ci table is simply an integration table so duplicates would be allowed. As long as the data in the sn_sec_cmn_src_ci does not create duplicate CI records then the integration using the sn_sec_cmn_src_ci is functioning as designed.

ServiceNow calculates the licenses from the Vulnerability Response Usage dashboard populated from these 2 tables:

• Discovered Items table: sn_sec_cmn_src_ci
• Container usage count table: [sn_vul_container_vr_container_counts]

Both tables and the dashboard are installed from 3 separate ServiceNow Store apps/plugins.

Ensure these ServiceNow Store apps/plugins are installed:

• Vulnerability Response Licensing and Usage [sn_vul_licensing]
• Vulnerability Response and Configuration Compliance for Containers [sn_vul_container]
• Qualys or Rapid7 Integration for Security Operations