IntegrationHub ETL Script not working

[Soumita1]

Hi All,

This question is related to the IntegrationHub ETL also with Robust Transform Map. The requirement is such that the while "Prepare and Preview data" in the Prepare Source Data for mapping, I have created a new field which is called "DeDuplicate". The use of this field is to put in the unique serial numbers and empty the rows for duplicate serial numbers.

So, the u_serial_number column has duplicate for the values 9 and 4.

In the DeDuplicate column just the unique value should be copied except for 2 9s and 2 4s. So, I created a script for the DeDuplicate.

Script is below:

(function(batch, output) {
var input, count=0;
for (var i = 0; i < batch.length; i++) {
gs.info("Soumita dd2 i: "+batch[i].input);
for(var j = 0; j < batch.length; j++){
// gs.info("Soumita dd2 j: "+batch[j].input);
if(batch[i].input == batch[j].input){
count= count+1;
gs.info("Soumita count dd2: "+count);
 }
 }
}
})(batch, output);

The duplicate column is not giving me unique values and when I do gs.info I just get the i loop is running 8 times instead of 10 as there are 10 records.

[#2]

A few things to note:

1) The script operation batches requests.  These batches will use unique inputs to improve performance.  If you have 10 serial numbers but only 8 are unique you will send a batch of 8.  This is controlled by the use_unique_input_sets field on sys_rte_eb_operation (note: your table in this case will be the child table sys_rte_eb_script_operation).  You will need to set this value to false for your script operation.

2) It doesn't look like you are setting an output.  At the end of your for (var i...) loop you need to set the value you want outputed.  In this case something like output[i] = count.

3) Please keep in mind that all script are batched. IE: if you have 1 million records in your import set table it won't send all 1 million to this script at the same time.  It will batch them into smaller numbers.  This will presumably break the functionality you are looking for as my guess is you want to check all unique values across the entire import.  This task is better aimed as business rule or scheduled job once data is in the CMDB.

[heptane]

Try ammending your IRE instead.