Hi, our teams perform scheduled data ingestion from ServiceNOW to Databricks based on the sys_updated_on field. However, for records in the software installation table, when deduplication process is run, the sys_updated_on field is not updated.
As a result, the latest deduplicated version of the record can't be ingested into Databricks.
To resolve this issue, as a mediator between ServiceNOW and Databricks side, I need more context to understand how this deduplication process works, and provide suggested solutions to both sides.
Can anyone provide me relevant information/knowledge/potential workaround regarding this? Thanks!
