Number of time onStart transform event executes.

asoni1 · ‎08-14-2019

As per the docs for the onStart transform event ->"When: The onStart event script is processed at the start of an import run, before any data rows are read."

My understanding was it will execute only once despite any number of records I might insert to the import set. But when I tried it, I saw it executes once for each record insertion in import set.

Is my understanding correct that all the event executes as many number of times as input records?

Thanks

ARG645 · ‎08-21-2019

Great discussion so far.

As you might already know, WebService Import sets have two modes, 1.Synchronous 2.Asynchronous

An import set with a Mode of Synchronous will transform the data as soon as it is inserted (provided that the transform map already exists). This import set will also have a default State of Loading. By default, all Synchronousimport sets will automatically be modified to Processed (the state is set to process) at midnight. As a result, when a new insert happens to the same table after the previous importset is marked processed, a new Synchronous import set will be created. Web service import set mode

An import set with a Mode of AsSynchronous will be in a state of Loading until all the records are inserted. Once all the records are inserted [We dont know how to detect it in your case] we have to run the job manually or run it via script [script is what you need]. To run it via a script, you can write an Hourly/minutely scheduled data import. You can find how to create a scheduled data import in the link: Schedule a data import

Now that you already developed logic to make your importset mode as asynchronous. You can find a scheduled job named "Asynchronous Import Set Transformer", this may be inactive by default. You can activate it/create a copy of it and change the repeat interval to every 10 minutes[your desired iinterval] and change the script in such a way that it only checks for your one particular data source.

Try the below script in the scheduled job

transformAsyncIset();

function transformAsyncIset() {
var igr = new GlideRecord("sys_import_set");
igr.addQuery("mode", "asynchronous");
igr.addQuery("state", "loading");//Because in your case the state will never change to "loaded"
igr.addQuery("table_name","YOUR IMPORT SET TABLE NAME");
igr.query();
  while(igr.next()) {  
    sTransform(igr);
    igr.setValue('state','loaded');
    igr.update();//Set the state to Loaded , so that it wont be captured by the job again and again. 
  }
}

function sTransform(igr) {
  var mapGR = getMap(igr.table_name);
  if(mapGR.next()) {
    var t = new GlideImportSetTransformerWorker(igr.sys_id, mapGR.sys_id);
    t.setProgressName("Transforming: " + igr.number + " using map " + mapGR.name);
    t.setBackground(true);
    t.start();
  }
}

function getMap(sTable) {
  var mapGR = new GlideRecord("sys_transform_map");
  mapGR.addQuery("source_table", sTable);
  mapGR.addActiveQuery();
  mapGR.query();
  return mapGR;
}

Please mark my eligible responses as helpful/correct if applicable.

Thank you,

Aman Reddy Gurram

View solution in original post

asoni1 · ‎08-21-2019

Hi Mike,

when i insert records to import set using API/WebService call, it inserts the records synchronously and triggers the transform map as soon as 1 record is inserted. Thats where the problem i am facing

Now i have managed to write a business rule to change the mode to asynchronous but the state in import set still shows as loading. How do i change that to loaded ? i dont want to do that manually.