Issue using Flow Designer to load and transform data from a Data Source

[Javeria]

I have a data source with 3 transform maps attached to it and a scheduled job triggered from flow. The flow first fetches data from a third party and uploads the attachment to the data source. Once that’s done, the scheduled job is run from the flow via a script action.

The issue is that the REST call runs multiple times in the same flow (using a do-while loop). Before uploading the next file, we need to delete the previous attachment; otherwise, the load step fails with an error saying no/more than 1 file attached. I tried adding a postscript on the scheduled job to delete attachments, but that isn’t working.

Is there a way to always pick the most recent file from the data source for load & transform?

As an alternative, I tried using GlideImportSetLoader and GlideImportSetTransformerWorker, but that throws errors in the scoped app (HR Integrations). We also want to avoid putting this code in a global script include and then calling it from the HR scope.

Any guidance or best practices would be appreciated.

@Ankur Bawiskar 

[Javeria]

@Ankur Bawiskar  Exactly!
I’ve added a 5-second wait to let the data fetch complete, but that’s not a reliable approach. We need a proper way to confirm when the data import has actually finished then delete file and I was thinking of using a wait-for condition on the sys_import_set table until the state is set to processed, but since that table isn’t accessible in Flow, I can’t use it there.