Best practice for inbound emails - how do you do it?

[JK1]

Hello everyone,

I would like to hear how do you deal with reply emails in your instance. As you know, there is an issue with pointing the system to the correct record. A Good example would be :

1. I send email, to ServiceNow mailbox, but several other addresses are recipients as well

2. Ticket is created and SN sends back an mail with refI. D and/or number

3. But the users start to reply to the original email (from step 1) so system starts creating new record for that. We end up with 2 records for same thing

We attempted to solve this by doing lookup over the Subject of the email, where we strip the prefixes and suffixes. Still, this is not good enough as some people change the subject ( you cant manage the users so easily ). Or you have emails from automated systems ( for ex. Akamai have such one ) , where every new email has different subject, but its for same thing ( ticket usually ).

So, how do you deal with such situations ?

Cheers,

Joro

[Dubz]

Hi Georgi,

In terms of users replying to the initial email that raised the ticket, the only viable solution to this is to have your first line guys respond to the ticket as quickly as possible so that the thread begins properly with the ticket reference in the subject line. If you send the Incident Opened notification to everyone cc'd you just end up with multiple threads of the same conversation on the same ticket which is probably more confusing that having multiple duplicates.

To deal with the inevitable duplicates we've been working on a merge function so we can thread in the additional comments from a duplicate incident to the activity log of the master. I'm hoping to lock this down to only be available for new tickets as merging two tickets that have been open a while will get messy. We don't have it working yet though so i have no handy code to share with you on that front.

We also have issues with 3rd parties sending notifications from their own system with their own subjects. We're building a table populated with the various ticket references used by the 3rd parties we deal with, then we have a field on the incident form that stores the 3rd parties reference so the inbound action can lookup against this table and match email to incident based on the syntax of a 3rd parties ticket reference. This isn't a particularly scalable solution due to similarities between ticket reference formats but it's hopefully going to mitigate some of the chaff we get from vendors letting us know about maintenance and those inconsiderate folks whose notifications completely change the subject line when they respond!

Ultimately though, inbound emails are always going to be a giant pain-in-you-know-where. There's just too much variance in format and you can't rely on consistent behaviour from humans or considerate behaviour from systems, moving customers towards a service portal is the best option but also it might be useful to have separate email addresses provided to customers and to vendors so you can create distinct rules for each but even that gets messy when you have customers who are also vendors etc!

Good luck!

[JK1]

Hi Dave,

yeah, its indeed pain in you know where

This is endless battle with mind mills. I really hope, that ServiceNow will allow some basic usage, for free, for the machine learning/AI . May be this will help us in such complex situations where usually human interaction is needed.

I have some code snippet (very basic) and table structure. If you want I can share it here

Cheers,

Joro

[Dubz]

Yeah that'll be great until the AI becomes self-aware and starts to get frustrated at the abuse it receives from users. It wouldn't surprise me if an angry ITSM AI became the eventual arbiter of human-kinds demise.

[JK1]

Hopefully ServiceNow (the company not eh tool itself) will allow us to touch the AI/ML at least a bit - something like "taste" it at least. Because now its "look, but dont, touch, touch, but dont swallow, etc. " - what I mean, its paid, paid   paid, no free stuff (at least like 200 transactions, or 100 MB/day, or whatever, but something ).

Here is the snippet I am developing (honestly, its far from complete but 85-90% of the cases its does the trick).

Please, excuse me for some bad/not used lines. There are some redundand fields (btw you can use both or one of them), like u_index and the table structure u_sdcall_subject (in general few fields there, number, subject(cleaned up) - this is the better way for me to match the subject to real record. Add and active field as otherwise you will update old tickets - like I am doing atm ).

You can start from or even use this snippets for that. Note that the Reply is only checking for watermark and/or ticket number. If none - drop. Inbound New is having and the check. I assume its not very correct but I was in hurry back then. I will post the new, revamped snippet as soon as i finish it. Let me know if any questions.

Inbound on NEW only :

-----------------------------

gs.include('validators');

(function(current){

var sd = new Paysafe_SDCaller_Utils();

//Checking if email is in forbidden list

if(sd.checkForForbidden(email.from.toLowerCase())) {

sd.log("InboundAction: Found forbidden email. Quit...");

return;

}

var cleanSubject = sd.cleanUpEmailSubject(email.subject.trim());

var cleanBody = sd.cleanUpSignature(email.body_text);

if(email.from != ""){

sd.log("InboundAction Email not empty. Create tkt with subject : " + cleanSubject);

var existingCall = sd.getRecordFromSubject(cleanSubject);

if(existingCall) {

sd.log("InboundAction Found for update!");

var rec = new GlideRecord('new_call');

if(rec.get(existingCall)){

rec.u_journal_1 = "Update from: " + email.origemail + "\n\n" + cleanBody;

rec.update();

//Adding the relation to email tbl

sd.setTargetRecord(sys_email.uid, existingCall);

}

} else {

current.u_active = true;

current.u_index = cleanSubject;

//For STRY STRY0010259 [Service Desk Caller] - status UNKNOWN and dash

//BUG with user and company. If user doesnt have related company, company will be null hence no BS

if(sd.getVendorCompany(email.from)) {

current.company = sd.getVendorCompany(email.from);

current.description = "New ticket received from: " + '\n' + email.origemail + "\n"   + cleanBody;

current.short_description = cleanSubject;

current.u_html = email.html;

current.call_type = "triage";

current.caller = email.from;

current.u_bs = current.company.u_bs;

current.contact_type = "email";

current.u_assignment_group = current.u_bs.support_group;

} else {

current.company.setDisplayValue("UNKNOWN");

current.description = "New UNKNOWN ticket received from: " + '\n' + email.origemail + "\n"   + email.body_text;

current.short_description = cleanSubject;

current.call_type = "triage";

current.caller = email.from;

current.contact_type = "email";

current.u_state = '-100';

}

current.notify = 2;

if (email.body.assign != undefined && email.body.assign != "")

current.assigned_to = email.body.assign;

if (email.importance != undefined && email.importance != "") {

if (email.importance.toLowerCase() == "high")

current.priority = 1;

}

var sysid = current.insert();

sd.addCallToRelationship(sysid , cleanSubject);

sd.log("InboundAction New SD CALL ticket created : " + current.number);

}

} else {

sd.log("InboundAction Nothing to insert/update!");

}

})(current);

Script Include:

----------------------------

/*

    Class:

    Paysafe_SDCaller_Utils

    *Author:*

  Joro Klifov

    *Company:*

  Paysafe Group

    *Date created:*

    29.05.2017

    *Type:*

    Script Include Prototype

    *Table:*

    new_call

    *Global:*

    FALSE

*/

var Paysafe_SDCaller_Utils = Class.create();

Paysafe_SDCaller_Utils.prototype = {

initialize: function() {

},

/*

@debug variable

@param bool

*/

debug: true,

/*

@log method

@param String

@returns void

Used to log debug messages

*/

log: function (message) {

var name = 'Paysafe_SDCaller_Utils';

if(this.debug) {

gs.log(message,name);

}

},

/*

@cleanUpEmailSubject method

@param String

@returns Object

Used to clean the mail subject from all prefixes and returns the cleaned up subject, later used for short description

*/

cleanUpEmailSubject: function(subject) {

try {

      var subjArr = [];

this.log('cleanUpEmailSubject: cleaning up ' + subject);

var prefixes = ['re:','re.:','отн.:', 'отн:', 'aw:', 'aw.:', 'fw:', 'fw.:', 'fwd:', 'fwd.:','re: ','re.: ','отн.: ', 'отн: ', 'aw: ', 'aw.: ', 'fw: ', 'fw.: ', 'fwd: ', 'fwd.: '];

//double replace for RE: FW:

for(var i = 0; i < prefixes.length; i++) {

subject = subject.replace(prefixes[i],'');

}

this.log('cleanUpEmailSubject: cleaned up ' + subject);

//return subject.replace(/\w{2,3}\:\s{0,1}|\w{2,3}\.\:\s{0,1}|\W{2,3}\:\s{0,1}|\W{2,3}\.\:\s{0,1}/gmi, "");

return subject.replace(/([r]|[a]|[f]|[w]|[о]|[а]|[\S]).{1,2}\.?\:\s?/gmi, "").replace(/^\s{1}|\s{1,3}$/gmi, "");

}

catch(ex){

this.log("Exceprion occured in cleanUpEmailSubject() " + ex);

return;

}

},

/*

@cleanUpSignature method

@param String

@returns String

Used to replace body from signature via RegEx with empty string

*/

cleanUpSignature: function(body){

this.log("cleanUpSignature: cleaning up body from \n" + body + "\n" + body.replace(/^Paysafe Group.*\n{0,3}.*\n.*transmission\./gmi, "").replace(/\[cid\:.*\]/gmi, "").replace(/\n{5,10}/gmi, ""));

return body.replace(/^Paysafe Group.*\n{0,3}.*\n.*transmission\./gmi, "").replace(/\[cid\:.*\]/gmi, "").replace(/\n{5,10}/gmi, "");

},

/*

@getRecordFromSubject method

@param String

@returns reference

Used to query for existing record via its short description = subject

*/

getRecordFromSubject: function(subject) {

try {

this.log('getRecordFromSubject: Subject:   ' + subject);

var existingRecord = new GlideRecord('u_sdcall_subject');

existingRecord.addQuery('u_subject', subject);

//

//TODO: Add additional field active in the table so only active records to be updated

//

existingRecord.query();

if(existingRecord.next()) {

this.log('getRecordFromSubject: found an record for subject ' + subject + "   AND SYSID " + existingRecord.u_sdcall);

return existingRecord.u_sdcall;

}

this.log('getRecordFromSubject: couldn\'t find an record for subject ' + subject);

return null;

} catch (ex) {

this.log('getRecordFromSubject: Exception ' + ex);

return null;

}

},

/*

@addCallToRelationship method

@param reference

@param String

@returns void

Used to create the relationship between the subject (from email) and the record (SDCALL reference)

This is done in relationship table

*/

addCallToRelationship: function (sdcall, subject) {

this.log('addCallToRelationship ' + sdcall + ' for ' + subject);

var existingRecord = new GlideRecord('u_sdcall_subject');

existingRecord.initialize();

existingRecord. u_sdcall = sdcall;

existingRecord.u_subject = subject;

existingRecord.insert();

},

/*

@getMyVendor method

@param String

@returns bool

Search for if vendor exists.

!!! DEPRICATED !!!

*/

getMyVendor: function (vend) {

try {

//cleanVend strips all from th email but domain, we use it to search in the table for such name

//cleanedVendDom is stripping all from email but the part from @ on. Then search in the table for such email/1/2

//We search with cleaned domain and with email, respectfully by domain1/2 and email1/2/3

vend = vend.toLowerCase();

var mail = vend.toLowerCase();

//var cleanVend = vend.replace(/^(?:[^@\n]+@)?([^\.\/\n]+)/, '');

var cleanVend = vend.replace(/(^.+@)/, "").replace(/(\.+?.{1,3})/, "");

//commented as moved to table based forbidden mails

//if( cleanVend.toString() === 'skrill' || cleanVend.toString() === 'fireeyecloud' ) return false;

var cleanVendDom = vend.replace(/(@.*)/g, '');

var vd = new GlideRecord('core_company');

vd.addEncodedQuery('u_paysafe_vendor=yes^u_active=true^u_email=' + vend + '^ORu_email_2=' + vend + '^ORu_email_3=' + vend + '^ORu_domain=' + cleanVend + '^ORu_primary=' + cleanVend + '^ORu_secondary=' + cleanVend);

vd.query();

if(vd.next()){

this.log("Found provider/vendor " + vd.name);

return true;

}

else {

this.log("This provider/vendor " + cleanVend + " , and original values passed = " + vend + " is not on the list ");

return true;

}

}

catch(ex){

this.log("Execption in getMyVendor() " + ex);

return;

}

},

/*

@getVendorCompany method

@param String

@returns Object

Search for if vendor exists.

Returns the user's company (vendor/provider)

*/

getVendorCompany: function(user) {

try {

var comp = new GlideRecord('core_company');

comp.addEncodedQuery('u_email=' + user + '^ORu_email_2=' + user + '^ORu_email_3=' + user);

comp.query();

if(comp.next()){

this.log("getVendorCompany => user " + user + "   " + comp.u_domain + "   " + comp.name);

return comp.sys_id;

}

var usr = new GlideRecord('sys_user');

usr.addQuery('email', user);

usr.query();

if(usr.next()){

return usr.getValue('company');

}

else {

this.log("This passes user => " + user +   " is not on the list ");

return "UNKNOWN";

}

}

catch(ex){

this.log("Exception in getMyVendor() " + ex);

return;

}

},

//Get company for user that is not company related( read it vendor/provider)

getUserCompany: function(user) {

try {

var comp = new GlideRecord('core_company');

comp.addEncodedQuery('vendor=true^u_primary=' + user + '^ORu_secondary=' + user);

comp.query();

if(comp.next()){

this.log("getUserCompany => user : " + user + " , vendor : " + comp.u_domain + " , vendor name : " + comp.name);

return comp.sys_id;

}

else {

this.log("This passes user => " + user +   " is not on the list ");

return "UNKNOWN";

}

}

catch(ex){

this.log("Exception in getMyVendor() " + ex);

return;

}

},

//Populate the BS depending on the company

getBS: function(comp){

try   {

var result = {};

var bs = new GlideRecord('core_company');

bs.addQuery('u_paysafe_vendor', 'yes');

bs.addQuery('name', comp);

bs.query();

while(bs.next()){

result.push(bs.u_bs);

}

gs.log("Return from getBS is : " + result);

return result;

}

catch(ex){

this.log("Exception in getBS() " + ex);

return;

}

},

// !!! NOT USED !!!

//Create new vendor in table notknown

setUnknownVendor: function(mail){

try {

var vend = new GlideRecord('u_sdcall_unknown_provider');

var cleanVend = mail.replace(/(^.+@)/,'').replace(/(\.+?.{1,3})/,'');

vend.initialize();

vend.u_name = cleanVend;

vend.email = mail;

vend.insert();

}

catch(ex){

this.log("setUnknownVendor error " + ex);

}

},

getUnknownVendor : function(vend){

try {

var vendor = new GlideRecord('u_sdcall_unknown_provider');

vendor.addQuery('u_email', vend);

vendor.query();

if(vendor.next()){

return vendor.u_name;

}

else {

return this.setUnknownVendor(vend);

}

}

catch(ex){

this.log("getUnknownVendor " + ex);

}

},

/*

@setRelatedUser method

@param String

@returns void

      Set the company for a user with none. For UI action/frontend/user interaction decision

*/

setRelatedUser: function(email){

var domUser = email.replace(/(^.+@)/, "").replace(/(\.+?.{1,3})/, "");

var company = this.getUserCompany(domUser);

if(company){

var user = new GlideRecord('sys_user');

user.addQuery('user_name', email);

user.query();

if(user.next()){

user.company = company;

user.update();

}

}

},

/*

@setTargetRecord method

@param String

@param String

@returns void

Used to set taret record & table in sys_email table records (emails)

takes two params - the UID of the sys_email object and the sys_id of the SDCALL record

*/

setTargetRecord: function(u, id){

var email = new GlideRecord('sys_email');

email.addQuery('uid', u);

email.orderByDesc('sys_created_on');

email.query();

if(email.next()){

this.log("setTargetRecord: Found email to update " + email.subject);

email.target_table = 'new_call';

email.instance = id;

email.receive_type = 'reply';

email.update();

}

},

/*

@checkForForbidden method

@param String

@returns boolean

Used to determine if a email or its domain are forbidden to create SDCALL records

takes one param - the email to be checked for

*/

checkForForbidden: function(email){

var rec = new GlideRecord('u_sdcall_forbidden');

rec.addEncodedQuery('u_email=' + email + '^ORu_name=' + email + '^ORu_domainLIKE' + email.replace(/(^.+@)/, "").replace(/(\.+?.{1,3})/, ""));

rec.query();

if(rec.hasNext()){

this.log("Found match! Emaail or its domain is forbidden to create SDCALL tickets!");

return true;

}

this.log("Email OK for create SDCALL ticket");

return false;

},

type: 'Paysafe_SDCaller_Utils'

};