Robust Transform to multiple targets

barcar · ‎10-10-2023

I think I must be missing something with Robust Transforms. Looking for some pointers...

My scenario is a single Flow Action Data Stream source which I want to transform into 2 target tables with different transforms.

I thought the RTE would allow me to transform data from a single import set into multiple targets with different transform maps for each.

I started off in Integration Hub Import and was able to connect my data source, target table and transform successfully. But the "Add a table" button on the Map to Target step is now disabled so I can't create a second target there.

That led me to look at the underlying ETL Definition and I tried to add a new ETL Entitity there for the second target. That worked but, when I open the integration in Integration Hub it throws the message "Error while fetching integration instance report: Multiple target entity found for etl definition <sys_id>".

What am I missing here? Is this possible?

Side-bar question... In a regular transform map you can perform scripted actions for a field rather than a direct mapping. Is that not possible in the RTE? For example, if I want to timestamp a record with the last import date/time?

Extra side-bar question... If one of the fields in my data stream contains a JSON which I want to expand, split and transform into a third target table, is that possible in the same integration?

KevinBellardine · ‎10-10-2023

I've run into the issue you're describing when attempting to use the same entity as the target entity in more than one entity mapping. I solved it by creating an additional temporary entity to serve as a staging area to load data. It creates some additional overhead because I've got to create 2 entity fields for every value I want to map, but it gives me a lot more freedom.

Side Bar: What you're looking for is called an Entity Operation. I've found the script operations to be the most straightforward to use. They run after the data has been loaded into an Entity but before any Entity Mapping involving that entity runs. For example, you can pass a multi-input operation two values, do some logic, and then output a new value to a single field on the Entity.

Extra Side Bar: Yes, you can. The robust transform should just treated it as a nested JSON, so as long as your path is configured correctly you can grab the data. Arrays can be a little more tricky and I'd refer to the documentation for that, but you essentially just use a [*] in the path to indicate the beginning of an Array.

View solution in original post

Manoj89 · ‎10-10-2023

Hi,

When working with Robust Transform Map think Robust Entity fields as variables, first you have to declare it and you can directly load some value into it or can transform and load the imported value using "Robust Transform Engine Entity Operations", and then you can map those date to as many tables as you want using "Robust Transform Engine Entity"

KevinBellardine · ‎10-10-2023

I've run into the issue you're describing when attempting to use the same entity as the target entity in more than one entity mapping. I solved it by creating an additional temporary entity to serve as a staging area to load data. It creates some additional overhead because I've got to create 2 entity fields for every value I want to map, but it gives me a lot more freedom.

Side Bar: What you're looking for is called an Entity Operation. I've found the script operations to be the most straightforward to use. They run after the data has been loaded into an Entity but before any Entity Mapping involving that entity runs. For example, you can pass a multi-input operation two values, do some logic, and then output a new value to a single field on the Entity.

Extra Side Bar: Yes, you can. The robust transform should just treated it as a nested JSON, so as long as your path is configured correctly you can grab the data. Arrays can be a little more tricky and I'd refer to the documentation for that, but you essentially just use a [*] in the path to indicate the beginning of an Array.

Angelo Cortesi · ‎08-17-2024

Enjoying the topic. I did this demo showing RTE loading data to multiple target CI classes ,in case it helps

https://www.linkedin.com/feed/update/urn:li:activity:7230577972175360002/?originTrackingId=PRhHB1bYT...