Number of time onStart transform event executes.

asoni1 · ‎08-14-2019

As per the docs for the onStart transform event ->"When: The onStart event script is processed at the start of an import run, before any data rows are read."

My understanding was it will execute only once despite any number of records I might insert to the import set. But when I tried it, I saw it executes once for each record insertion in import set.

Is my understanding correct that all the event executes as many number of times as input records?

Thanks

ARG645 · ‎08-21-2019

Great discussion so far.

As you might already know, WebService Import sets have two modes, 1.Synchronous 2.Asynchronous

An import set with a Mode of Synchronous will transform the data as soon as it is inserted (provided that the transform map already exists). This import set will also have a default State of Loading. By default, all Synchronousimport sets will automatically be modified to Processed (the state is set to process) at midnight. As a result, when a new insert happens to the same table after the previous importset is marked processed, a new Synchronous import set will be created. Web service import set mode

An import set with a Mode of AsSynchronous will be in a state of Loading until all the records are inserted. Once all the records are inserted [We dont know how to detect it in your case] we have to run the job manually or run it via script [script is what you need]. To run it via a script, you can write an Hourly/minutely scheduled data import. You can find how to create a scheduled data import in the link: Schedule a data import

Now that you already developed logic to make your importset mode as asynchronous. You can find a scheduled job named "Asynchronous Import Set Transformer", this may be inactive by default. You can activate it/create a copy of it and change the repeat interval to every 10 minutes[your desired iinterval] and change the script in such a way that it only checks for your one particular data source.

Try the below script in the scheduled job

transformAsyncIset();

function transformAsyncIset() {
var igr = new GlideRecord("sys_import_set");
igr.addQuery("mode", "asynchronous");
igr.addQuery("state", "loading");//Because in your case the state will never change to "loaded"
igr.addQuery("table_name","YOUR IMPORT SET TABLE NAME");
igr.query();
  while(igr.next()) {  
    sTransform(igr);
    igr.setValue('state','loaded');
    igr.update();//Set the state to Loaded , so that it wont be captured by the job again and again. 
  }
}

function sTransform(igr) {
  var mapGR = getMap(igr.table_name);
  if(mapGR.next()) {
    var t = new GlideImportSetTransformerWorker(igr.sys_id, mapGR.sys_id);
    t.setProgressName("Transforming: " + igr.number + " using map " + mapGR.name);
    t.setBackground(true);
    t.start();
  }
}

function getMap(sTable) {
  var mapGR = new GlideRecord("sys_transform_map");
  mapGR.addQuery("source_table", sTable);
  mapGR.addActiveQuery();
  mapGR.query();
  return mapGR;
}

Please mark my eligible responses as helpful/correct if applicable.

Thank you,

Aman Reddy Gurram

View solution in original post

ARG645 · ‎08-14-2019

It executes only once.

Please provide screenshots, code etc to show us what you are observing.

asoni1 · ‎08-14-2019

Thanks. Please find below sample code that i tried to execute.

var gs = new GlideRecord('u_item_option_new');

gs.addQuery('u_cat_item','=',source.u_cat_item);

gs.query();

gs.log('Record value is : ' + gs.getValue('u_record_name'));

and in system logs it goes like :

Record value is : "x"

Record value is : "y"

Record value is : "z"

and so on....

Sorry, I can't share screenshot at the moment as it is disabled for me.

ARG645 · ‎08-14-2019

it appears as if you are using OnBefore transform script. But either way, gs is a reserved keyword.

Change your script to below(This will not fix your problem)

var gr = new GlideRecord('u_item_option_new');

gr.addQuery('u_cat_item','=',source.u_cat_item);

gr.query();

 

gs.log('Record value is : ' + gr.getValue('u_record_name'));

 

and in system logs it goes like :

Record value is : "x"

Record value is : "y"

Record value is : "z"

Make sure you are not using an onbefore transform script

asoni1 · ‎08-14-2019

Thanks for the response.

I am not using gs in my code, i just used it here. I tried this code in onBefore, onStart events and its behaving the same way.

ARG645 · ‎08-14-2019

Defies the fundamentals of the transform script. There has to be something that's causing this behavior.