Bestpractice of inserting huge number of records

Go to solution
Makoto Uchiyam1
Tera Contributor

‎05-18-2023 11:59 PM

Hi experts,

 

I would like to perform a record insert on another table based on information from one table in Servicenow instance. There are more than 10 million records that need to be created, and if I were to do this with the following simple script, it would take more than 100 hours in total. Please advise on how to speed this up.

I couldn't find GlideRecord API to insert multiple records at once.

 

var grInput = new GlideRecord("<Input Table>");
grInput.addQuery(<Condition>);
grInput.query();
while (grInput.next()) {
    var grOutput = new GlideRecord("<Output Table>");
    grOutput.initialize();
    // Set parameters
    grOutput.xxx = gr.Input.xxx;
    grOutput.yyy = grInput.yyy;
    grOutput.zzz = grInput.zzz;
    grOutput.insert();
    }
}

 Regards,

0 Helpfuls
  • 2,889 Views
1 ACCEPTED SOLUTION

Go to solution
Markus Kraus
Kilo Sage
In response to Makoto Uchiyam1

‎05-19-2023 02:40 AM

Unfortunately (at least to my knowledge) there is no "batch mode" for background scripts. Especially because of the rollback function it would advice to split the source data.

Note: As long as you intend to use a background script (for "rollbackability"), there is no way for you to speed this up, as every transaction is executed by a user. The only workaround is - as i mentioned - to fire up the background scripts using multiple sessions.

 

Theoretically you could also try the following if you have a integration hub professional subscription:

Create a Data Source which uses Data Stream Source (this would support pagination) and then configure the data source to do the splitting for you. 

Are you able to create such a data stream in the flow designer? It would look something like this:

MarkusKraus_0-1684489204853.png

 

View solution in original post

5 REPLIES 5

Go to solution
Markus Kraus
Kilo Sage
In response to Makoto Uchiyam1

‎05-25-2023 12:02 AM

Hm I think you will have to go with an external script - if you insist on rollback ability.

In this case I would suggest you to do the following:
Open up a background script and open the browser debugger console. Investigate what exactly is being sent when you execute a plain "gs.info('hello world')" script. 
Write a small python script (hint: ask chatgpt - it's a great starting point) which does the login and executes a (most likely post) request which contains your script. 

The only thing you have to do is to create this script in a paramitrized way so that you can split the data.

 

To my knowledge, there is no way to acomplish your requriements with pure OOTB servicenow scripts/functionality.

0 Helpfuls