Best practice on importing lots of files via SFTP

[Max Nowak]

Hi,

I have the requirement to regularly import lots (>150) of CSV files via SFTP into the ServiceNow instance. The way I originally thought this could work is by using data sources and scheduled imports, but as far as I know, I'd have to create a data source and scheduled import for each CSV file, which would mean a lot of work.

Is there another, simpler way, to reduce the number of objects I'd have to create in ServiceNow? Ideally, I'd just like to provide a folder on the SFTP server, an import set table, and a transform map to import all files in the folder periodically. I really don't want to create data sources and scheduled imports for all files.

Thanks for your help,

Max

[Sulabh Garg]

Hello Max,

In your case, you can write a script on SFTP server (SFTP team can help you on this) to move the file from one folder to say "Archive" folder on sftp once consumed by ServiceNow (with timestamp added in filename). And make sure you have the new file with same name as mentioned in data source for next scheduled run.

That way you will have one folder on SFTP and you can handle with one data source and transform map with all consumed file in archival folder for future reference.

Please Mark ✅ Correct/helpful, if applicable, Thanks!! 

Regards

Sulabh Garg

[Max Nowak]

Hi Sulabh,

but how would I be able to import a whole folder via SFTP? As far as I know, you have to specify the full file path in a datasource, meaning you can only import one file per data source.

[Sulabh Garg]

No, You cannot import whole folder via SFTP and you are right you can only import one file per data source, However you can dynamically place new file with same name on the SFTP folder to be consumed by data source and remove (move the file) which is already consumed to some other folder (e.g.: say Archive folder).

e.g.: If you have one file named "ABC.csv" on one SFTP folder and once data source consumed the file, you can place the new file on the same SFTP folder (and move the "ABC.csv" on Archive folder) with the same name as of earlier file to be picked by data source in next scheduled run.

Hope it helps!!

[Sulabh Garg]

Also looking at your original ask, you will have >150 csv files at particular time on SFTP folder? Or It will be dynamic place on SFTP folder?

Please share the use case.