Reply to badly processed email or unknown email sender

r_gissarl_ · ‎08-24-2016

Hi,

We recently experienced problems with our customers who changed their sending email addresses. Investigating this problem showed us that we had a lot of badly processed emails.

I would like to understand why some emails stays in the inbox and are never processed, and why some are processed and "moved" to the received mailbox. I don't find information on that subject in the wiki. If you have a link, it would be nice.

The goal is to send a reply to the user if the mail has not been correctly processed (unknown email address, no watermark), to inform the user that the ticket has not been updated.

I found a way of doing this directly in an email notification :

But it doesn't work with emails that are never 'moved' in the received mailbox. Do you have an acheiving my goal ?

Note that we don't use inbound mail to create automaticaly tickets, we use it to update comments on incidents or changes.

r_gissarl_ · ‎09-06-2016

Hi,

finally I share my solution as it seems to me to be nice. Moreover, this could answer to other users in the same case (we wanted the sender of a non or bad processed email to receive a warning email also in case of unknown email).

First, the prerequisites :

you have to enable Guest user (so you might have to modify some out of the box inbound actions, if you don't want that guest user create incidents or request),
you will have to order your inbound actions as some of them will have to be triggered in a defined order and stop the process if they match conditions
- To avoid this particular prerequisite, you can write a code in a new inbound action that will check the logs of previous inbound actions filtered by field source = EMAIL.(sys_id of the email received) and search for 'Processed' in the beginning of the message log. If the log shows 'Processed' then your received email has been correctly treated, if not you can fire the email.failed event as shown below. This inbound action must have a very high order to be executed as last one. But it is not my use case here.

That being writed, here is my solution :

enable Guest user (set active to true and locked out as false),
create an event with Event registration :

Description used to send back to the unkniwn user an email to inform him that his mail has not been processed

Event name email.failed

Fired by Email sent by an unknown user

3.create an inbound action to trigger the event in the case that an unknown user tries to send an email to your instance :

Active true

Condition gs.getUserID() == '5136503cc611227c0183e96598c4f706'

Description Inbound action to trigger the processing of a mail received by unknown user. It is used to send a notification to this user to warn him that his email has not been processed.

Event name email.read

Name email received from Guest

Order 0

Script

// if guest then fire email.failed event

gs.eventQueue("email.failed", sys_email, email.from, sys_email.sys_id);

event.state = 'stop_processing';

Class Inbound Email Actions

Domain global

Target table sys_email

As you can see, the inbound action has order 0, so it will be the first inbound action triggered. And it includes the line "event.state = 'stop_processing';" that will stop all other inbound action to be processed if the condition matches (if the sender is the out of the box guest user).

4. Create a script action, to change to state of the received email to "error" when the event email.failed is queued :

Active true

Event name email.failed

Name Change mail status on event email.failed

Order 100

Script

// used when an unknown user send a mail, we send a reply back informing that the email has not been processed. This script action forces the email in error state to trigger the notification.

var email = new GlideRecord("sys_email");

email.get(event.parm2);

email.state = "error";

email.update();

Synchronous false

Class Script Action

Domain global

5. Create a script action to prevent cases of sender replying with a known watermark and the record's watermark is in a table of the inbound actions' target tables :

Active	true
Event name	email.read
Name	Stop processing if the sender is guest
Order	100
Script	// used when an unknown user sends a mail before inbound actions, we send an event 'email.failed'. And stops email_read to avoid inbound actions to be triggered. (because if watermark is valid and is referring to a target table in an active inboud action, Guest will be allowed to update). var email = new GlideRecord("sys_email"); email.get(event.parm1); if(email.user_id == undefined \|\| email.user_id == "") { //if sender is unknown (so it is guest) gs.eventQueue("email.failed", email, email.user, event.parm1); gs.log("Stop processing email, because Guest user is detected and isn't authorized to create or update records","EMAIL."+event.parm1); event.state = 'stop_processing'; }
Synchronous	true
Class	Script Action
Domain	global

Indeed, this script action was not included in the first version of this post. But it is necessary if you don't want that guest users are able to update records. Because without this script action when a received mail comes from an unknown user BUT if :

the mail has a known watermark (the email is a considered as a reply)
And this watermark is referring to a record in a target table of an inbound action
- Then the unknown user is auhorized to update the record and the system doesn't send an automatic reply to inform him that he used an unknown email

6. You will have to work on your own to order your own inbound action (change order field). Be sure to add event.state = 'stop_processing'; at the end of the each active inbound action's script. This means that you will have only one inbound action triggered by received email, which is my case (if you don't want to do that, you can script the workround I detailled for the second prerequisite). Because we will have to create a new inbound action which will be the last one to fire the event email.failed. This will indicate that the received email has not been processed by any other inbound mail.

7. create a new inbound action, with a higher order (in my case 1 000 000 is much higher than others) :

Active true

Event name email.read

Name email.failed if no other inbound action

Order 1 000 000

Script

// if no other inbound action is triggered then fire email.failed event

gs.eventQueue("email.failed", sys_email, email.from, sys_email.sys_id);

event.state = 'stop_processing';

Class Inbound Email Actions

Domain global

Target table sys_email

8. Create a message (system localization -> Messages) to define the subject of the notification (it will be usefull to find if messages are sent in loop) :

Key subject_error_mail

Language English

Message Courriel non traité | Error in your email processing

9. Create a mail script to set the subject of the notification :

Name set_subject_error_mail

Newlines to HTML false

Script

email.subject = gs.getMessage('subject_error_mail');

10.Create the notification the system will send to the sender when an error or ignored state shows on the received mail :

Name Notification for error in processing

Table sys_email

Type EMAIL

Active True

Description If an email is in error or ignored state, the notification informs the sender

Send when Record is created or updated

Weight 0

Inserted True

Updated True

Conditions

Mailbox is Received

State is Error

OR State is Ignored

Advanced condition

var skipmsg = false;

// define exception words in subject

var subjectStrings = ["out of the office",

"out of office",

"out-of-office",

"out-of-the-office",

"auto reply",

"automatic reply",

"réponse automatique",

"r&eacute;ponse automatique",

"reponse automatique",

"réponse auto",

"r&eacute;ponse auto",

"reponse auto"]

// define exception words in body

var bodyStrings = ["out of office",

"out of the office",

"je suis absent",

"i'm on leave",

"je suis en congés",

"je suis en cong&eacute;s",

"je suis actuellement absent",

"i'm currently on leave",

"je suis actuellement en congés",

"je suis actuellement en cong&eacute;s"]

// check in subject for exception words

if (current.subject != undefined) {

       var subject = current.subject.toLowerCase();

       for(i=0; i!=subjectStrings.length ; i++) {

               if (subject.indexOf(subjectStrings[i]) > -1) {

                       gs.log("Skipping notification for error in processing incoming mail due to out of office type message in the subject [sys_id] : " + current.sys_id + " with subject : " + subject, "EMAIL." + current.sys_id);

                       skipmsg = true;

               }

       }

}

// check in body for exception words

if (skipmsg == false) { // no need to check if skipmsg is already to true

       if (current.body != undefined) {

               var body = current.body.toLowerCase();

               for(i=0; i!=bodyStrings.length ; i++) {

                       if (body.indexOf(bodyStrings[i]) > -1) {

                               gs.log("Skipping notification for error in processing incoming mail due to out of office type message in the body [sys_id] : " + current.sys_id + " with subject : " + current.subject, "EMAIL." + current.sys_id);

                               skipmsg = true;

                       }

               }

       }

}

// Checking how much error mail have been sent to this user today, if more than 10 then no more mails

if (skipmsg == false) { // no need to check if skipmsg is already to true

       var subject = gs.getMessage('subject_error_mail');

       var sender = "";

       var emailQuery = new GlideRecord('sys_email');

       if (current.user_id == undefined || current.user_id == "")

               sender = current.user;

       else

               sender = current.user_id.email;

       emailQuery.addQuery('recipients',sender);

       emailQuery.addQuery('subject',subject);

       emailQuery.addQuery('sys_created_on', '>', gs.beginningOfToday());

       emailQuery.query();

       var nb = emailQuery.getRowCount();

       if (nb > 9) {

               gs.log("From notification - Notification for error in processing - no more error warning email sent to this user " + sender + " as it seems to flow the instance", "EMAIL." + current.sys_id);

               skipmsg = true;

       }

}

if (skipmsg)

       answer = false;

else

       answer = true;

Users/group in fields User, User ID

Send to event creator True

Subject Error

User Message

Bonjour / Hello,

Une erreur a été détectée dans le traitement de votre mail intitulé ${subject}. Le système n'a pas pu le traiter et l'information n'a pas pu être transmise. Veuillez vous rapprocher de votre contact opérationnel ou contractuel. Vous pouvez aussi nous transmettre l'information par l'ouverture d'un incident.

An error has occured in the processing of your email ${subject}. The system didn't treated it and the information has not been transmitted. Please get in touch with your operationnal or contractual contact. You can also open an incident.

Service centre.

${mail_script:set_subject_error_mail}

As you can see, the advanced condition checks a number of parameter in order to be sure that the system will not enter in an infinite loop (reply to out of office message or an attempt to flood the system). You can add some strings in your language to check exception words in case of out of office or any other scenario. To do this, add strings in the subjectStrings to check the subject or bodyStrings to check in the body of the mail.

Hope this will be helpful to somebody !

View solution in original post

Michael Fry1 · ‎08-24-2016

Your post is right on. Emails that don't get processed stay in the Inbox queue, those that do get processed move to Received. No email notifications get triggered because nothing happens. The workaround we did, activated the Guest account, and add the following code to the top of the inbound actions where needed. If the users email doesn't match a user in the system, the inbound action fails and triggers an event.

if (gs.getUserID() == '5136503cc611227c0183e96598c4f706') { // Guest

gs.eventQueue("email.failed", sys_email, email.from, email.from);

} else {

r_gissarl_ · ‎08-24-2016

Hi Michael,

This is indeed really usefull.

On your side, what does happen to the email state ? Because when I had this in an inbound action, the email is processes but with the 'Processed' state and not with "error" or "ignored" (it is executed in order 1 000 000) ?

Michael Fry1 · ‎08-24-2016

It's marked as Processed, because an inbound action ran, though no target is set.

We ended up switching to the New Call plugin so we accept all emails, and then process them to Incident or Request from there. The current issue is that emails we can't recognize, nothing is sent back to emailer letting them know we don't recognize their email address. Simple fix of many.

r_gissarl_ · ‎09-06-2016

Hi,

finally I share my solution as it seems to me to be nice. Moreover, this could answer to other users in the same case (we wanted the sender of a non or bad processed email to receive a warning email also in case of unknown email).

First, the prerequisites :

you have to enable Guest user (so you might have to modify some out of the box inbound actions, if you don't want that guest user create incidents or request),
you will have to order your inbound actions as some of them will have to be triggered in a defined order and stop the process if they match conditions
- To avoid this particular prerequisite, you can write a code in a new inbound action that will check the logs of previous inbound actions filtered by field source = EMAIL.(sys_id of the email received) and search for 'Processed' in the beginning of the message log. If the log shows 'Processed' then your received email has been correctly treated, if not you can fire the email.failed event as shown below. This inbound action must have a very high order to be executed as last one. But it is not my use case here.

That being writed, here is my solution :

enable Guest user (set active to true and locked out as false),
create an event with Event registration :

Description used to send back to the unkniwn user an email to inform him that his mail has not been processed

Event name email.failed

Fired by Email sent by an unknown user

3.create an inbound action to trigger the event in the case that an unknown user tries to send an email to your instance :

Active true

Condition gs.getUserID() == '5136503cc611227c0183e96598c4f706'

Description Inbound action to trigger the processing of a mail received by unknown user. It is used to send a notification to this user to warn him that his email has not been processed.

Event name email.read

Name email received from Guest

Order 0

Script

// if guest then fire email.failed event

gs.eventQueue("email.failed", sys_email, email.from, sys_email.sys_id);

event.state = 'stop_processing';

Class Inbound Email Actions

Domain global

Target table sys_email

As you can see, the inbound action has order 0, so it will be the first inbound action triggered. And it includes the line "event.state = 'stop_processing';" that will stop all other inbound action to be processed if the condition matches (if the sender is the out of the box guest user).

4. Create a script action, to change to state of the received email to "error" when the event email.failed is queued :

Active true

Event name email.failed

Name Change mail status on event email.failed

Order 100

Script

// used when an unknown user send a mail, we send a reply back informing that the email has not been processed. This script action forces the email in error state to trigger the notification.

var email = new GlideRecord("sys_email");

email.get(event.parm2);

email.state = "error";

email.update();

Synchronous false

Class Script Action

Domain global

5. Create a script action to prevent cases of sender replying with a known watermark and the record's watermark is in a table of the inbound actions' target tables :

Active	true
Event name	email.read
Name	Stop processing if the sender is guest
Order	100
Script	// used when an unknown user sends a mail before inbound actions, we send an event 'email.failed'. And stops email_read to avoid inbound actions to be triggered. (because if watermark is valid and is referring to a target table in an active inboud action, Guest will be allowed to update). var email = new GlideRecord("sys_email"); email.get(event.parm1); if(email.user_id == undefined \|\| email.user_id == "") { //if sender is unknown (so it is guest) gs.eventQueue("email.failed", email, email.user, event.parm1); gs.log("Stop processing email, because Guest user is detected and isn't authorized to create or update records","EMAIL."+event.parm1); event.state = 'stop_processing'; }
Synchronous	true
Class	Script Action
Domain	global

Indeed, this script action was not included in the first version of this post. But it is necessary if you don't want that guest users are able to update records. Because without this script action when a received mail comes from an unknown user BUT if :

the mail has a known watermark (the email is a considered as a reply)
And this watermark is referring to a record in a target table of an inbound action
- Then the unknown user is auhorized to update the record and the system doesn't send an automatic reply to inform him that he used an unknown email

6. You will have to work on your own to order your own inbound action (change order field). Be sure to add event.state = 'stop_processing'; at the end of the each active inbound action's script. This means that you will have only one inbound action triggered by received email, which is my case (if you don't want to do that, you can script the workround I detailled for the second prerequisite). Because we will have to create a new inbound action which will be the last one to fire the event email.failed. This will indicate that the received email has not been processed by any other inbound mail.

7. create a new inbound action, with a higher order (in my case 1 000 000 is much higher than others) :

Active true

Event name email.read

Name email.failed if no other inbound action

Order 1 000 000

Script

// if no other inbound action is triggered then fire email.failed event

gs.eventQueue("email.failed", sys_email, email.from, sys_email.sys_id);

event.state = 'stop_processing';

Class Inbound Email Actions

Domain global

Target table sys_email

8. Create a message (system localization -> Messages) to define the subject of the notification (it will be usefull to find if messages are sent in loop) :

Key subject_error_mail

Language English

Message Courriel non traité | Error in your email processing

9. Create a mail script to set the subject of the notification :

Name set_subject_error_mail

Newlines to HTML false

Script

email.subject = gs.getMessage('subject_error_mail');

10.Create the notification the system will send to the sender when an error or ignored state shows on the received mail :

Name Notification for error in processing

Table sys_email

Type EMAIL

Active True

Description If an email is in error or ignored state, the notification informs the sender

Send when Record is created or updated

Weight 0

Inserted True

Updated True

Conditions

Mailbox is Received

State is Error

OR State is Ignored

Advanced condition

var skipmsg = false;

// define exception words in subject

var subjectStrings = ["out of the office",

"out of office",

"out-of-office",

"out-of-the-office",

"auto reply",

"automatic reply",

"réponse automatique",

"r&eacute;ponse automatique",

"reponse automatique",

"réponse auto",

"r&eacute;ponse auto",

"reponse auto"]

// define exception words in body

var bodyStrings = ["out of office",

"out of the office",

"je suis absent",

"i'm on leave",

"je suis en congés",

"je suis en cong&eacute;s",

"je suis actuellement absent",

"i'm currently on leave",

"je suis actuellement en congés",

"je suis actuellement en cong&eacute;s"]

// check in subject for exception words

if (current.subject != undefined) {

       var subject = current.subject.toLowerCase();

       for(i=0; i!=subjectStrings.length ; i++) {

               if (subject.indexOf(subjectStrings[i]) > -1) {

                       gs.log("Skipping notification for error in processing incoming mail due to out of office type message in the subject [sys_id] : " + current.sys_id + " with subject : " + subject, "EMAIL." + current.sys_id);

                       skipmsg = true;

               }

       }

}

// check in body for exception words

if (skipmsg == false) { // no need to check if skipmsg is already to true

       if (current.body != undefined) {

               var body = current.body.toLowerCase();

               for(i=0; i!=bodyStrings.length ; i++) {

                       if (body.indexOf(bodyStrings[i]) > -1) {

                               gs.log("Skipping notification for error in processing incoming mail due to out of office type message in the body [sys_id] : " + current.sys_id + " with subject : " + current.subject, "EMAIL." + current.sys_id);

                               skipmsg = true;

                       }

               }

       }

}

// Checking how much error mail have been sent to this user today, if more than 10 then no more mails

if (skipmsg == false) { // no need to check if skipmsg is already to true

       var subject = gs.getMessage('subject_error_mail');

       var sender = "";

       var emailQuery = new GlideRecord('sys_email');

       if (current.user_id == undefined || current.user_id == "")

               sender = current.user;

       else

               sender = current.user_id.email;

       emailQuery.addQuery('recipients',sender);

       emailQuery.addQuery('subject',subject);

       emailQuery.addQuery('sys_created_on', '>', gs.beginningOfToday());

       emailQuery.query();

       var nb = emailQuery.getRowCount();

       if (nb > 9) {

               gs.log("From notification - Notification for error in processing - no more error warning email sent to this user " + sender + " as it seems to flow the instance", "EMAIL." + current.sys_id);

               skipmsg = true;

       }

}

if (skipmsg)

       answer = false;

else

       answer = true;

Users/group in fields User, User ID

Send to event creator True

Subject Error

User Message

Bonjour / Hello,

Une erreur a été détectée dans le traitement de votre mail intitulé ${subject}. Le système n'a pas pu le traiter et l'information n'a pas pu être transmise. Veuillez vous rapprocher de votre contact opérationnel ou contractuel. Vous pouvez aussi nous transmettre l'information par l'ouverture d'un incident.

An error has occured in the processing of your email ${subject}. The system didn't treated it and the information has not been transmitted. Please get in touch with your operationnal or contractual contact. You can also open an incident.

Service centre.

${mail_script:set_subject_error_mail}

As you can see, the advanced condition checks a number of parameter in order to be sure that the system will not enter in an infinite loop (reply to out of office message or an attempt to flood the system). You can add some strings in your language to check exception words in case of out of office or any other scenario. To do this, add strings in the subjectStrings to check the subject or bodyStrings to check in the body of the mail.

Hope this will be helpful to somebody !