Remediate Duplicate "Discovery items" for Customer Usage (Licensing)

[Brian43]

We are attempting to calculate our customer usage for the last 90 days in the Security Operations module, Discovered items (sn_sec_cmn_src_ci table).  The KB0861920 indicates that what is counted is; a server, a desktop, a laptop, a printer, a firewall, and any other IP-enabled device that the vulnerability scanner scans.

When we review the results, there are duplicate entries for various devices.  Many are showing the same name, serial number, FQDN, IP Address, etc.  Here is an example:

We have 17 F5 BIG-IP devices in our environment.  There are over 400 "Discovered items" listed in our report on "Discovered items in the Last 90 Days".  When filtered by IP address, there are only 9 devices showing.  Their IP address, name, FQDN, etc. are all the same.

For Customer Usage, the report appears to show that we have over 400 devices, when, in fact we are only scanning 9 devices.

How do I remediate these duplicates and merge them together to show the true count for customer usage?

[Chris McDevitt]

Hi,

Add the field Source ID to the Discovered Items list view. What you should discover is that these are not duplicates. Each unique Source ID represents a unique "device" coming from your vulnerability scanner. What is most likely happening is that you conducting unauthenticated scans and the F5 is sending the traffic through those 9 IPs to 400 unique devices behind the F5. So, the count is probably accurate. 

I would work with your sales rep and show them the KB and your report and see what they say. 

😉

[Avery Brown]

Hi Chris, 

Question on the point you made regarding the Source ID.. So we're experiencing a similar issue in my org, and when we pull in Source ID, there are numerous entries with the same Source ID. Is there a way to reconcile those cases? For some additional clarity we have multiple 'Integrations' with our scanning tool, Tenable, and it looks like the Discovered Items with duplicate entries for Source ID are coming from different integrations. (The integrations are just looking at different queries in Tenable, they're hitting the same endpoint).

[Chris McDevitt]

@Avery Brown 

I see....

It looks like you have duplicates because you are accessing the same Data Source with different Integrations and you getting back the same Source ID... correct? You can tell them apart by looking at the Source ID and the Source (aka Integration) like so:

Background:

When an Integration run (i.e. the Integration part above) the data comes in and the first thing it does is looks for the Source ID field AND the Source (aka Integration) to see if there is a match. If there is a match, it uses that matches Configuration Item for the CI. If there is no match, it runs through the CI Lookup Rules using the Source (aka Integration) (so it knows which rules to run), and when it is done, it writes its finding to the Discovered Items modules Source ID and Source (aka Integration) fields and the process starts over.

So... you can not consolidate those DI because of the multiple Sources (aka Integrations). (The next VR run will bring them right back in)

I would work with my Sales Rep and ask them to only report on the "primary" Source (aka Integration) only. (The "primary" is the one with the most unique DI)

[Mike G1]

We have multiple Discovery Items (DI) records that correctly associate to the same CI. The main DI differences are different source ID values and FQDN/HOST_NAME format. Some DI records include the domain for both FQDN and HOST_NAME while others populate only the hostname for the FQDN and HOST_NAME fields.

Should these records be considered "duplicate" records? Will they impact license cost charged by ServiceNow?

Is it possible that a faulty CI Lookup Rule is causing the issue? I expect that the CI lookup rules are called after the data populates the DI table, but I was asked to investigate.