jason_petty
Tera Expert

The method below was originally published on Oct 25, 2012, is an outdated process

Please see the following post for help! >   Loading data from an email attachment


I had a number of people respond to this post with the same problem and together with my colleague I re-wrote the instructions to make it work.

I had a customer that wanted to have IBM Maximo product send an email to ServiceNow and have a CSV/XLS file attached to the email automatically create a new Data Source, load the data, then transform it all automatically just from receiving the email. Here is what I did to get that to work:


1. Create a Data Source Manually first so you can set up the Import Set Table and the Transform Map that you will use later. This is necessary since you need a Transform Map defined before you send the email inbound to ServiceNow. This Data Source can be deleted after creating the Import Set Table and Transform Map.
2. Get a sample CSV/XLS file and attach it to the Data Source and load the data at least once.
3. Define your Transform Map and set it up so the data is getting transformed into the right table in the format that you need.
4. Create a user that you will use for basic authentication with no roles and has a password.
5. Create a REST Message that points to your own instance for the REST endpoint: https://instancename/sys_import.do as the url named "Inbound Import Set Poster"
6. Check the box for "Use basic auth" and set the username and password from step #4
7. Edit the "get" function and add the following function parameters:

8. Create an Inbound Email Action with the following settings:

HERE IS THE CODE TO COPY:



if (current.getTableName() == "sys_data_source") {

      current.name = email.subject.substring(3);
      current.import_set_table_label = email.body.import_set_table_label;
      current.import_set_table_name = email.body.import_set_table_name;
      current.file_retrieval_method = email.body.file_retrieval_method;
      current.format = email.body.format;

      current.update();

var r = new RESTMessage('Inbound Import Set Poster', 'get');
r.setStringParameter('tableLabel', encodeURIComponent(current.name));
r.setStringParameter('customDataSource', current.sys_id);
              r.setStringParameter('tableName', current.import_set_table_name);
r.execute();
}


9. The Condition can be anything you want but the Target table needs to be sys_data_source.
10. Enable the inbound email settings in your instance to receive email and process them. See WIKI http://wiki.servicenow.com/index.php?title=Configuring_Email for this.
11. Now send an email like this to the email instancename@unix.service-now.com (The "unix" part of this avoids any email formatting issues with email clients):

  1. To: instancename@unix.service-now.com
  2. Subject: DS:Test Data Source 1
  3. Body:


              import_set_table_label:Test Import Set Table
              import_set_table_name:u_test_import_set_table
              file_retrieval_method:Attachment
              format:CSV

NOTE: CSV would be Excel if you are using an XLS file

12. Wait until the email is processed and it should have created your Data Source with whatever is after the "DS:" in the subject
13. Make sure the import_set_table_name is set to whatever you named your import set table when you created your Data Source in step 1 (This is very important since the Transform Map is linked to this table)
14. The format is set to either Excel or CSV.
15. You need to attach the file you want to attach to the Data Source.

>


The method above was originally published on Oct 25, 2012, is an outdated process

Please see the following post for help! >   Loading data from an email attachment


68 Comments
payerle
Tera Contributor

Hi Jason,



I know this is an old thread, but still very valuable, can you post the updated script and where it goes with the inclusion of the redirect?   What we see stops after the RestMessage r.execute();     Would love to see how this finishes off as it appears to be the last piece I need.   Manual import works from this, but automated transform from email does not.



Kirk, did Jason happen to send this to you that you can share?



Thanks everyone!


haimn
Kilo Contributor

Hello



its create the data source with the transform map but it not makes the transformation. maybe needs to execute the transform button?


payerle
Tera Contributor

Hey everyone, I got this working successfully recently, without using a mid-server.   Maybe this will work for you.



Make sure you have a dedicated report from your system, in my instance its Dell KACE.   You can capture whatever fields you want, but you need to pick a primary key in SN and coalesce on that field.   I picked the bios serial number, and created a new field for that data on the cmdb_ci_computer table.   Other fields I grabbed - Ram, mac address, user name, warranty end date, system name, chassis type, name, model, and manufacturer.



From KACE, you should have it email your report in a CSV format to your SN instance.  



You will need to import the data once manually so you can create the transform map.


Source table: imp_computer


Target: cmdb_ci_computer


Run business rules: no



Next, create a scheduled data import, and use the manual data source that you imported manually for the first run.   Set the timer to run sometime in the future (this won't matter because the inbound action script will kick off the run immediately)



Then you need an inbound action script.   This fortunately does all of the work for you, no extra scripts needed.   This should capture the attachment, create a new data source of the attached file, update the scheduled import set based on this, then kick off a process to import the data, and run the scheduled import.


Keep in mind I have sourced the inbound action from several different posts to make this work.   Not my code, I don't claim it as mine   (also, couldn't find the code tag, sorry!)




Inbound action:


Target table: sys_data_source


Action: record action


script:


[CODE]


// set up the data source. The system will automatically add the attachment for us.


gs.log ('TDI001 - Setting up data source');


current.name = "SNKaceImport" + gs.nowDateTime(); //append date time to name of data source for audit


current.import_set_table_label = "Computer";


current.import_set_table_name = "imp_computer";


current.type= "File";


current.format = "CSV";


current.file_retrieval_method = "Attachment";



var myNewDataSource = current.insert();



gs.lg ('TDI002 - Data source inserted with sys_id - ' + myNewDataSource);



//point the scheduled import record to the new data source


var gr2 = new GlideRecord ('scheduled_import_set');


gr2.addQuery('sys_id', '693eeccc135d62000d087981xxxxxxx'); //sys id of the Scheduled Data


gr2.query();


if (gr2.next()) {


gs.log ('TDI003 - Found Scheduled Import definition');


gr2.data_source = myNewDataSource;


gr2.update();


gs.log ('TDI004 - Scheduled Import definition updated to point to data source just added');


//Packages.com.snc.automation.TriggerSynchronizer.executeNow(gr2);   <----- old line!


SncTriggerSynchronizer.executeNow(gr2);     //<------ this works!


} else {


// add error conditions to email somebody that this has occurred


gs.log('TDI005 - ERROR - Unable to locate scheduled import definition. Please contact your system administrator');


}


gs.log ('TDI007 - Inbound email processing complete');


[/CODE]




Let me know if this doesn't work, I might be able to help with your problems.


Tom Brown
Giga Guru

We kept the same data source, wrote it to remove the existing attachment and add the new one, then trigger the synchronizer:



//   First section deletes files attached to data source.


//   Otherwise, the second part won't overwrite, but will add on more attachments every day.


//


var gr = new GlideRecord('sys_attachment');


gr.addQuery('table_name', 'sys_data_source');


gr.addQuery('table_sys_id','913d34a4f9fedd047a219da3b79c7778');


gr.query()


while (gr.next()) {


  gr.deleteRecord();


  }


//


//   Note that the data source sys_id below is the one to which the email attachment will be affixed.


//


var emailSysId=sys_email.sys_id;


GlideSysAttachment.copy('sys_email', emailSysId, 'sys_data_source', '913d34a4f9fedd047a219da3b79c7778');


//


//   Trigger start of scheduled import


var rec = new GlideRecord('scheduled_import_set');


rec.get('name', 'PeopleSoft Import');


SncTriggerSynchronizer.executeNow(rec);


payerle
Tera Contributor

The only thing I need to do and am struggling with is: When we import computers from our KACE environment, we would like to mark them as 'installed' for the status.   Then, if the item did not get imported (because it was deleted from KACE), then it would mark that item as 'retired'.   Any thoughts on how to do this?   It's a weird thought process because we are trying to retire things that don't get updated.   Possibly could be done with another business rule, but would rather have this run after transform, so it doesn't run every day without importing data.



Any help would be appreciated, thanks!


haimn
Kilo Contributor

Thanks Matthew,



now it does execute the transform but it stack on 'Loading' state.


the email log shows that the excel spreadsheet uploaded but it desappear from the form and not forwarding to the Data Source record that created by the inbound action.


payerle
Tera Contributor

What step are you in?   I would need a little more information based on your system setup and your import data.


haimn
Kilo Contributor

user sent an email with excel file (contains uniqe name).


there is an inbound action contains the script above.


the script creating record on sys_data_source table and set all the values from the script well.


the excel file from the email is not attached to the record from some reason.


then this line 'SncTriggerSynchronizer.executeNow(gr2);'   creates import set but the import set stack on 'Loading' state and not complete the transform.


i think its happend because the excel file is not attaching to the data source.


haimn
Kilo Contributor

here is my script





if (current.getTableName() == "sys_data_source") {



  current.name = email.subject.substring(0) + ' ' + gs.nowDateTime();


  current.import_set_table_label = 'TEST';


  current.import_set_table_name = 'u_test';


  current.file_retrieval_method = 'Attachment';


  current.type = 'File';


  current.format = 'Excel';


  current.sheet_number = '1';


  current.header_row = '1';




  var myNewDataSource = current.insert();


  gs.log ('TD002 - Data source inserted with sys_id - ' + myNewDataSource);


  var gr2 = new GlideRecord ('scheduled_import_set');


  // gr2.addQuery('sys_id', '693eeccc135d62000d087981xxxxxxx'); //sys id of the Scheduled Data


  gr2.query();


  if (gr2.next()) {


  gs.log ('TDI003 - Found Scheduled Import definition');


  gr2.data_source = myNewDataSource;


  gr2.update();


  gs.log ('TDI004 - Scheduled Import definition updated to point to data source just added');


  SncTriggerSynchronizer.executeNow(gr2);


  } else {


  gs.log('TDI005 - ERROR - Unable to locate scheduled import definition. Please contact your system administrator');


  }


  gs.log ('TDI007 - Inbound email processing complete');


find_real_file.png


Tom Brown
Giga Guru

Since the import and transform method finds correlation ids, I struggle with how you might use that to retire items that are not in the list.   I'm tossing around an idea of a business rule on the transform history table that looks for new entries for that particular import set.   When it sees a new one that meets criteria, it could do a GlideRecord search for active configuration items that were not updated "today" and set those to 'inactive' or whatever.