Transform map script to only update on current import/transform, and insert for new?

[bobby24]

I have a transform Map in which I had made a field as Coalesce, so that only matching records gets updated when our daily import job runs. The transform map runs and inserts the new records & merges some of them based on the coalesced field.

The next time we run an import/transform manually or the daily job runs, the previous imported data will still be in the import table. I want to ignore all data from the previous import so that data is not appended to the previous items that were transformed, and only focus on the newly imported data. What would be the best method to do this?

[vinothkumar]

You can provide in the onBefore transform script by using the query itself and my suggestion is you can still limit the data from the source by adding the necessary query in your data source

if(action == 'update'){

ignore = true;

}

[bobby24]

So if I put that in the onBefore script, does that prevent the next transform from updating each record based on the coalesced field? Not sure if I am being clear enough, or maybe I don't understand how that script you posted works.

[vinothkumar]

Yes Bobby, OnBefore transform script will run before the transformation is being done on the target table.

Since we can able to get the type of action in the transform script, so this script will ignore the transformation if the action is being update. Suppose if there are any new record then that action will be considered as Insert, so it will work

[bobby24]

Ok, so this keeps all items from updating. Maybe I am not clear on what I am doing. I coalesce on ID, so all rows from the first import set/transform that contain ID 1 are merged etc.. If I use the onBefore transform script, these won't be merged because the additional records are seen as 'updates'.

This would prevent those items from being updated on the 2nd import, but doesn't work for the initial daily load. Is there a way ignore records that match yesterday's load date somewhere?