Self-solved: IH-ETL - No existing records updated after mapping changes, same import set loaded

[Michal Sadowski]

Hi IntergrationHub ETL Superstars,

We've built an IntegrationHub ETL application that pulls data from a legacy inventory system via Data source of REST (IntegrationHub) type in JSON format.

One issue we have is that after class mappings are adjusted (e.g. a new transformation is created that maps to another field or existing mapping is changed) and data is loaded from the same import set or a new one based on the same dataset, the integration does not perform any update operations on the target records.

Any ideas why this may be the case?

I know with Intune-SG and other IH-ETL based integrations, you would expect the system to simply update the records on a new load of the same import set, after mappings are adjusted.

Not working for us for some reason.

Any pointers appreciated.

Update: The root cause of this was mapping a largely static date from the legacy system to Source Recency Timestamp. This resulted in ServiceNow logging Last Scan for each Ci in sys_object_source table that was largely the same for most records with every load and ignoring the rest of IRE process at this step, assuming no new information is being passed. Solving this was as easy as removing any mapping from Source Recency Timestamp. What this does is logging Last Scan as the date of the import, which is dynamic and every new is considered "new information" that goes to IRE for further processing.