Run scheduled data import script only when data is loaded

[Utpal Dutta]

I want to run Scheduled data pre import script to delete all the records from the target table before importing new data from the data source. 

Issue: I want to know if this pre import script will run even if the data is not imported? Eg: While data import there maybe connection issues as this data is coming from a data source. In this case the pre import script should not delete the data from the target table.

Please let me know how this pre import script runs or please suggest any other way to delete target records before loading new records in a safe way.

Thanks,

Utpal

[Ankur Bawiskar]

@Utpal Dutta 

your data load is based on ftp, https etc?

there are few objects available in those script. one of those is import_set. Using that you can check if import was successful or not and based on that proceed with deletion

Scheduled data import scripting options 

Why not have the deletion then in onStart transform script of that transform map?

If my response helped please mark it correct and close the thread so that it benefits future readers.

[Ankur Bawiskar]

@Utpal Dutta 

Thank you for marking my response as helpful.

If my response helped please mark it correct and close the thread so that it benefits future readers.

[Utpal Dutta]

Thanks @Ankur Bawiskar ,

Can you suggest if we are using JDBC type of data load then do we need to check import_set with pre import script or the script only will run after successful successful import to delete records from target table?

[Ankur Bawiskar · Accepted solution]

@Utpal Dutta 

I would suggest you try first with pre-import and add some logs and verify

But I would recommend using onStart since they will trigger once data is successfully imported into staging table. So in that way you are safe

If my response helped please mark it correct and close the thread so that it benefits future readers.