Issue using Flow Designer to load and transform data from a Data Source

[Javeria]

I have a data source with 3 transform maps attached to it and a scheduled job triggered from flow. The flow first fetches data from a third party and uploads the attachment to the data source. Once that’s done, the scheduled job is run from the flow via a script action.

The issue is that the REST call runs multiple times in the same flow (using a do-while loop). Before uploading the next file, we need to delete the previous attachment; otherwise, the load step fails with an error saying no/more than 1 file attached. I tried adding a postscript on the scheduled job to delete attachments, but that isn’t working.

Is there a way to always pick the most recent file from the data source for load & transform?

As an alternative, I tried using GlideImportSetLoader and GlideImportSetTransformerWorker, but that throws errors in the scoped app (HR Integrations). We also want to avoid putting this code in a global script include and then calling it from the HR scope.

Any guidance or best practices would be appreciated.

@Ankur Bawiskar 

[Ankur Bawiskar]

@Javeria 

I will suggest before adding file to Data source, always delete the existing one and then attach new one.

with this you will always have latest file on data source.

If my response helped please mark it correct and close the thread so that it benefits future readers.

[Javeria]

I did that, maybe some raise condition is happening file is deleting before loading. What if I go with  schedule job logic in flow and have wait condition on state field to wait flow unless state is processed?

[Javeria]

@Ankur Bawiskar 
That's the question: how to know if load/transform is completed then only I can delete file. In the flow in wait for condition I am unable to access sys_import table to check state if it's processed.
Also below is my script include code:

    loadData: function(dataSourceID) {
        var dsGR = new GlideRecord('sys_data_source');
        if (!dsGR.get(dataSourceID)) {
            gs.error("Invalid Data Source sys_id: " + dataSourceID);
            return null;
        }
        var loader = new GlideImportSetLoader();
        var importSetRec = loader.getImportSetGr(dsGR);
        loader.loadImportSetTable(importSetRec, dsGR);
        importSetRec.state = "loaded";
        importSetRec.update();
        var importSetId = importSetRec.getUniqueValue();
        return importSetId;
    },
    transformData: function(importSetID) {
        var transformWorker = new GlideImportSetTransformerWorker(
            importSetID,
            'transform map sys_id' 
        );
        transformWorker.setBackground(false);
        transformWorker.start();
        return ImportSetID
    }

[Ankur Bawiskar]

@Javeria 

you said 1st flow pulls and adds file to data source.

in that flow itself you can delete the file.

But I believe you are saying by the time the same flow runs again you are not sure if earlier import has been completed or not?

If my response helped please mark it correct and close the thread so that it benefits future readers.