In such a scenario of huge number of duplicate CIs, first priority is prevention of duplicates. For that, identification rules need to be reviewed and refined. I have explained this in the following article:
https://medium.com/@ashoksasidharan/essential-servicenow-cmdb-concepts-demystifying-ci-identificatio...
De-duplication task is the right way to handle duplicate CIs and must be used for all relevant CI classes. But in the case of a huge number of duplicate CIs, you can consider the following approach to check if there are any records which you can handle in bulk without having to process de-duplication tasks:
i. Make a list of CI classes which have the highest number of duplicate records
ii. Check with configuration manager and CI class owners to find out if any of these classes are actually not used for any business objective. Some classes may have got populated automatically from discovery or other sources, but they may not be serving any purpose.
iii. For the classes which are identified as not required to support any business objective, you can take actions to archive or delete the records in bulk after verifying again with stakeholders (also additionally verify that they are not referenced in any task records).
Note: The process best practice is to retire CIs first, then archive after the organization's defined archival period and delete the archived records after the organization's defined data retention period.
