Number of time onStart transform event executes.

asoni1 · ‎08-14-2019

As per the docs for the onStart transform event ->"When: The onStart event script is processed at the start of an import run, before any data rows are read."

My understanding was it will execute only once despite any number of records I might insert to the import set. But when I tried it, I saw it executes once for each record insertion in import set.

Is my understanding correct that all the event executes as many number of times as input records?

Thanks

ARG645 · ‎08-21-2019

Great discussion so far.

As you might already know, WebService Import sets have two modes, 1.Synchronous 2.Asynchronous

An import set with a Mode of Synchronous will transform the data as soon as it is inserted (provided that the transform map already exists). This import set will also have a default State of Loading. By default, all Synchronousimport sets will automatically be modified to Processed (the state is set to process) at midnight. As a result, when a new insert happens to the same table after the previous importset is marked processed, a new Synchronous import set will be created. Web service import set mode

An import set with a Mode of AsSynchronous will be in a state of Loading until all the records are inserted. Once all the records are inserted [We dont know how to detect it in your case] we have to run the job manually or run it via script [script is what you need]. To run it via a script, you can write an Hourly/minutely scheduled data import. You can find how to create a scheduled data import in the link: Schedule a data import

Now that you already developed logic to make your importset mode as asynchronous. You can find a scheduled job named "Asynchronous Import Set Transformer", this may be inactive by default. You can activate it/create a copy of it and change the repeat interval to every 10 minutes[your desired iinterval] and change the script in such a way that it only checks for your one particular data source.

Try the below script in the scheduled job

transformAsyncIset();

function transformAsyncIset() {
var igr = new GlideRecord("sys_import_set");
igr.addQuery("mode", "asynchronous");
igr.addQuery("state", "loading");//Because in your case the state will never change to "loaded"
igr.addQuery("table_name","YOUR IMPORT SET TABLE NAME");
igr.query();
  while(igr.next()) {  
    sTransform(igr);
    igr.setValue('state','loaded');
    igr.update();//Set the state to Loaded , so that it wont be captured by the job again and again. 
  }
}

function sTransform(igr) {
  var mapGR = getMap(igr.table_name);
  if(mapGR.next()) {
    var t = new GlideImportSetTransformerWorker(igr.sys_id, mapGR.sys_id);
    t.setProgressName("Transforming: " + igr.number + " using map " + mapGR.name);
    t.setBackground(true);
    t.start();
  }
}

function getMap(sTable) {
  var mapGR = new GlideRecord("sys_transform_map");
  mapGR.addQuery("source_table", sTable);
  mapGR.addActiveQuery();
  mapGR.query();
  return mapGR;
}

Please mark my eligible responses as helpful/correct if applicable.

Thank you,

Aman Reddy Gurram

View solution in original post

asoni1 · ‎08-14-2019

That might be possible I am doing something wrong as I am new to all this and learning on the go.

I have added "gs.log" to check in system logs. Suppose, I just write "gs.log('print hello')" in onStart event of a transform script.

And then i insert say 5 records using script, it should display "print hello" in log only once??? But strangely that's not the case....

AbhishekGardade · ‎08-14-2019

Check out this code:

var gr = new GlideRecord('u_item_option_new');

gr.addQuery('u_cat_item',source.u_cat_item);

gr.query();

while(gr.next()){

gs.log('Record value is : ' + gr.getValue('u_record_name'));

and in system logs it goes like :

Record value is : "x"

Record value is : "y"

Record value is : "z"

}

Please mark as Correct Answer/Helpful, if applicable.
Thanks!
Abhishek Gardade

Thank you,
Abhishek Gardade

ARG645 · ‎08-14-2019

It may not be your script. It may be because the transform map is running three times. How are you inserting records in the import-set via ansible ? Does all the three import set records that are going through transform map belong to the same import set ? Please check, if all three belong to a different update set, then your transform map will get triggered 3 times.

asoni1 · ‎08-14-2019

I am inserting records to import set by making an Import API Call from ansible to service now import set.

All the records that I am inserting goes to the same import set.

asoni1 · ‎08-14-2019

I think you might be correct. I am not sure if it makes sense but Ansible is inserting multiple rows but one row at a time and transform map is processing that particular row at a time. Therefore, using any event is not making much of a difference.

One thing, is there anything like event pipeline in service now, or using global variable in transform map?

Thanks

Aman