ServiceNow Learning 70: Time Saving Feature "Concurrent Import Sets"

[Shamma Negi]

Hi All,

I have come across a requirement wherein we had to load 100,000 data every day but have to find out the best way to achieve this which takes less time and doesn't impact the performance.

Hence, I come across this solution:

Concurrent imports

Split incoming data into multiple import sets and transform the import sets concurrently to reduce processing time.

Running a concurrent import can be helpful when order does not matter and imports take a long time due to large data sets with time-consuming scripts. If order matters, you can split the import into multiple partitions to ensure that each partition is processed in order.

Note: Concurrent imports add processing and monitoring overhead. Use them only with large data sets.

Enable concurrent imports only after fine-tuning all other parameters, such as database indexes and transformations.

Live Example:

When I didn't check the concurrent import checkbox and tried to import and transformed the data to the target table it took 7+ hours to get the transformation done.

When I checked the concurrent import checkbox and did the schedule import then it took 1hour 25 minutes which is very less compared to the time taken before.

Let's see below what approach Concurrent import takes to reduce the huge time. 

Scheduling concurrent imports

You enable concurrent imports by selecting Concurrent Import on the Scheduled Data Import form. For instructions, see Schedule a data import.

When the schedule runs a concurrent import, the system pulls the data from databases, Excel spreadsheets, CSV files, or other sources to a temporary staging table, and then transforms the data from the staging table to the target table.

When you run a concurrent import, the system creates multiple import sets, up to the value of the glide.scheduled_import.max.concurrent.import_sets system property (default = 10). For example, a two-node cluster produces four import sets, and a ten-node cluster produces ten import sets.

Import Set Transformer job

Each active node runs two Import Set Transformer jobs every minute, and those jobs poll the Concurrent Import Sets Jobs queue, pick import sets from the queue, and transform those import sets. All jobs run concurrently, depending on the availability of worker threads.

Concurrent Import Set record

Each concurrent import creates a Concurrent Import Set record. The form view shows all related import sets, concurrent import set jobs, and transform histories.

Hope this helps.

I hope this article helpful. Please mark it as helpful and bookmark if you like it.

Regards,

Shamma

[RahulY00040]

Hi @Shamma Negi , Import Set Transformer Job processes Async Import Sets? Can you please elaborate?

Thanks,
Rahul

[Shamma Negi]

It processes the Import set Async as it does all the work at te backend without waiting for anything. It automatically breaks into small chunks of Import sets and process it simultaneously. Without waiting for anything in frontend.