Best approach for Dynamic Filter/Visualization when Script Include processes large datasets

Pushpaka
Kilo Expert

3 weeks ago

Hi Community,

I have a requirement to identify duplicate Hardware Model Library records and show the affected CMDB Hardware Product Models in a Visualization/Report.

Scenario:

  • Hardware Model Library table: sn_hamp_hw_product_model
  • CMDB table: cmdb_hardware_product_model
  • Reference field: cmdb_hardware_product_model.normalized_model
  • Duplicate criteria need to be checked separately:
    1. Name + Model Number
    2. Name + Manufacturer
  • We need the sys_ids of the affected cmdb_hardware_product_model records for reporting/visualization.

I have a Script Include that correctly identifies the records when executed from Scripts - Background. However, the Hardware Model Library contains a large number of records, so the Script Include performs a large data scan and can take significant time.

The requirement is to use the result dynamically in a Visualization/Report Dynamic Filter.

My concern is that a Dynamic Filter executes synchronously, so running the complete duplicate-detection logic every time the visualization loads may not be practical for a large dataset.

Question:

What would be the recommended ServiceNow architecture for this scenario?

I am looking for an  recommended approach for large datasets, rather than simply optimizing the query.

Any guidance on the recommended  approach would be appreciated.

0 Helpfuls
  • 275 Views
1 REPLY 1

vaishali231
Kilo Sage

3 weeks ago - last edited 3 weeks ago

hey @Pushpaka 

For a large dataset, I would not recommend putting the complete duplicate detection logic inside the Dynamic Filter, since it runs synchronously.

A better approach would be:

  • Run the duplicate check using a Scheduled Script Execution.
  • Identify the affected cmdb_hardware_product_model records for both conditions:
    • Name + Model Number
    • Name + Manufacturer
  • Store the affected CMDB record sys_ids in a custom result table.
  • Use this result table for the report or visualization.
  • Refresh the result periodically, or use an incremental approach when the relevant Hardware Model Library records are changed.

This keeps the heavy processing outside the report execution. The Dynamic Filter only needs to query the already calculated results, which should perform much better for a large dataset.

So, I would recommend a precompute and store approach rather than recalculating duplicates on every visualization load.

 

*********************************************************************************************************

If this response helps, please mark it as Accept as Solution and Helpful.

Doing so helps others in the community and encourages me to keep contributing.

Regards

Vaishali Singh
Linkedin - https://www.linkedin.com/in/vaishali-singh-2273361bb











.







0 Helpfuls