Is it possible to control the row order visited by a Robust Import Set Transformer?

[Seong-HyunR]

Our product uses a Robust Import Set Transformer to ingest CI data from Kafka, and we noticed an issue which comes up when there are two rows of a single import set which feature the same CI but have conflicting data.

For example, we might have an import set which includes two rows for the CI "SYSG", one with the timestamp "2025-04-08" and the operational status of "Retired," and the other with a timestamp "2025-01-02" and the operational status of "Operational." The expected behavior is that the entry with the later timestamp should be ingested after the other, so that the most up-to-date entry is reflected in the CMDB. However, we haven't been able to find any way to designate an order for the rows within a single import set. 

We were able to find information regarding this feature within standard Transform Maps ("OrderBy" option in Advanced Options for Mapping), but this does not seem to be available in Robust Transformers. Is there any way to control the row order within an import set for a Robust Import Set Transformer?

[Shivalika]

In that case @Seong-HyunR 

You should sort the imported data itself by created date group by CI and only allow the latest one to go through transformation.  

Kindly mark my answer as helpful and accept solution if it helped you in anyway. This will help me be recognized for the efforts and also move this questions from unsolved to solved bucket. 

Regards,

Shivalika 

My LinkedIn - https://www.linkedin.com/in/shivalika-gupta-540346194

My youtube - https://youtube.com/playlist?list=PLsHuNzTdkE5Cn4PyS7HdV0Vg8JsfdgQlA&si=0WynLcOwNeEISQCY

[Seong-HyunR]

How might we go about doing this?

[amila]

How you receive data from a Kafka topic? Do you use stream connect or receive data some other means? 

If you use Stream connect ETL consumer, it creates an import set with the rows in same order as of Kafka partitions it consume and process them in that order. In an asynchronous import set, rows are processed with the order defined in row number for any type of transformer.

https://www.servicenow.com/docs/bundle/yokohama-integrate-applications/page/administer/integrationhu...