SGO-Dynatrace - "DUPLICATE_PAYLOAD_RECORDS" error for SGO-Dynatrace Processes Data Source

[s-jacobs]

Hello CMDB Forum,

We have been evaluating the Dynatrace Service Graph Connector for Observability. I have a support case open for this, but we are seeing these errors in the Import Set for Dynatrace Processes. 

"error":"ABANDONED","message":"Abandoned due to too many errors"

"error":"DUPLICATE_PAYLOAD_RECORDS","message":"Found duplicate items in the payload (index 1 and 2), using className [cmdb_ci_app_server_websphere] and fields [cell,name,node,sys_class_name]. Remove duplicate items from the payload"}

I've reviewed the response body from Dynatrace and there are not actually duplicate records. I think how this Service Graph Connector reads the body and transforms it is not actually functioning correctly. 

If I temporarily disable the identification rule for "cmdb_ci_app_server_websphere" and re-run the import job, it creates CIs with too generic of a name ("IBM WebSphere AS@<hostname>"  where <hostname> is a server the application runs on). There can be many instances of WebSphere AS running on a server; I believe this is where the "duplicates" specified in the error message come from. The CIs populated by horizontal discovery in this class have a JVM name listed in the Name attribute instead. 

It also doesn't populate the Node or Cell attributes like horizontal discovery does, even though those are attributes listed in the JSON response body from the Dynatrace API call. These are also part of the identification rule for the IBM WebSphere AS class. 

My assumption is that how the Dynatrace API presents information wouldn't be too wildly different between Dynatrace instances, so I'd be surprised if we are the first to run into this issue. Has anyone had a similar experience, and if so, how did you resolve?

[AJ-TechTrek]

Hi @s-jacobs ,

Great detailed question — you’re actually hitting something quite classic when integrating Dynatrace (or any external monitoring tool) via Service Graph Connector (SGC) into the CMDB, especially around WebSphere AS CIs which are often multi-instance and have complex identifiers.

As per my understanding Why this happens:

1. The Dynatrace SGC for Observability fetches process data from Dynatrace (which typically identifies WebSphere AS processes by the executable + runtime context).
2. Your identification rule for cmdb_ci_app_server_websphere uses fields like

* cell
* node
* name
* sys_class_name

A. If Dynatrace payload doesn’t include unique values for these fields (e.g., cell or node are empty or same for multiple records)

B. The Import Set detects them as duplicate payload records: same class, same identification keys.
So even if the raw JSON has multiple logical instances, if those distinguishing fields are empty or same, IRE sees them as duplicates.

Solution can be below as per my understanding might help you in this.

1. Verify raw payload → staging table
Go to:
System Import Sets → Admin → Import Log
* Look for the staging table for Dynatrace Processes
(e.g., x_dynat_dynatrace_process or similar)
* Confirm: do the cell and node fields actually get values from the JSON?
If missing → the problem is upstream: transform map is not mapping them.

2. Fix the Transform Map / Mapping Assist
In the transform map for the Dynatrace Process payload:
* Make sure JSON fields → staging fields → target fields (cell, node, name) are correctly mapped.
* Sometimes content pack OOB doesn’t map nested JSON properly.
Tip: Use “Auto Map Matching Fields” or manually add:
* Source JSON → cell
* Source JSON → node

3. Adjust Identification Rule if necessary
If Dynatrace cannot reliably provide cell and node, you must:
* Edit the identification rule → remove cell and node as mandatory identifiers
OR
* Add another distinguishing attribute Dynatrace does provide (e.g., JVM name, instance ID)
Removing too many identifiers risks merging distinct instances into one CI.
So do this carefully, ideally in lower environment first.

4. If you must ingest anyway:
* Temporarily adjust the identification rule to identify only by name + sys_class_name.
* Import once to get data in → then manually correct / split in CMDB.
Not ideal: may cause inaccurate CI merge.

5. Validate Import Set run:
Re-run import job.
* Confirm:
* No duplicate payload errors
* CIs created with correct cell and node attributes populated.

6. For the generic CI name issue:
“IBM WebSphere AS@” happens when:
* IRE doesn’t get enough to uniquely identify → fallback to generic naming template.
* Root cause: missing identifiers or transform issues.
Fix: Make sure name is populated with a meaningful JVM name / instance name from payload.

Please appreciate the efforts of community contributors by marking appropriate response as Mark my Answer Helpful or Accept Solution this may help other community users to follow correct solution in future.
 

Thank You
AJ - TechTrek with AJ - ITOM Trainer
LinkedIn:- https://www.linkedin.com/in/ajay-kumar-66a91385/
YouTube:- https://www.youtube.com/@learnitomwithaj
Topmate:- https://topmate.io/aj_techtrekwithaj (Connect for 1-1 Session)
ServiceNow Community MVP 2025

[s-jacobs]

Hello @AJ-TechTrek, 

I sincerely appreciate your detailed response, thank you. 

I checked the for an import log, but there doesn't appear to be one created. Assuming that would also be on the Import Log related list from the Import Set as well. 

I have recreated the outbound HTTP call made via the Service Graph Connector via Postman and can confirm that the Cell and Node fields are returned in the JSON response body from Dynatrace. These live in the WEB_SPHERE_NODE_NAME and WEB_SPHERE_CELL_NAME key/value pairs. There is also a WEB_SPHERE_SERVER_NAME key with the JVM name. 

As you noted, modifying identification rules is not ideal. I think the value behind Service Graph Connectors is being able to easily connect to third-party data sources without much effort in modifying the transform maps that the Service Graph Connector uses once it installs them. Am I wrong for expecting this to work better out of the box?