SMTP connection status alerts

[Darth Jed]

I have a problem that I'm trying to solve and I'll be surprised if no one else has experienced this problem. We have emails from our instance being routed through a relay so they appear come from a mailbox on our domain. When something changes on our network that inadvertently causes the SMTP connection to fail, ServiceNow quietly stops sending mail and the only indicator of trouble is the lack of email. Yes, I know there is a gauge that displays connection status, but that is an on demand check, not any sort of real time monitoring of the connection that will alert the admins wherever we happen to be in the instance.

I would like to incorporate some sort of popup alert message that will notify any admins (that are logged in) about the SMTP connection being down. I've been reading and building prototypes for a few days now and have parts of the desired result working, but I can't tie everything together due to the limitations of the system. I feel like I'm missing something and I'd like to call on all of you scripting gurus (Mark Stanger (Crossfuze) Travis Toulson Geoff Cox Jim Coyne (CompuCom) Jim Pisello (Crossfuze) out there to see what you might have for solutions.

My goals:

1. The alert needs to reach an admin wherever they are in the system.

2. The alert should not put a load on the system for the rest of the users.

3. The alert should be triggered when the glide.smtp.status property displays 'Cannot connect' in the value. I have a scheduled job that detects the property value easily enough.

4. I like the idea of the alert being able to be acknowledged so it doesn't harass the admins until the connection is restored.

My attempts so far have used a variety of solutions others have provided on the community or the ServiceNowGuru site.

Does anyone have a solution they have developed for this problem or have an idea of how I could pull this together?

[Jim Coyne · Accepted solution]

Here's my attempt.   It's actually fairly simple and heavily based on Mark Stanger (Crossfuze)'s "Outlook-style Module Counts" ServiceNowGuru article - http://www.servicenowguru.com/system-ui/outlook-style-module-counts/.   In fact, I used it to generate the Module part of the solution.

There are 3 parts to it:

1. UI Script:

Name: Custom - SMTP Server Problem Warning

Active: Checked

Global: Checked

Script:

if (window.name == "") addLoadEvent(u_smtpServerProblemWarning);



function u_smtpServerProblemWarning(message){

   if (message != undefined) {

   //add a warning about the SMTP server

   var options = {}; //create a new object to store all of the message attributes

   options.type = "error";

   options.text = "<span style='font-weight:bold;color:red;'>" + message + "</span>";

   options.sticky = true;

   new NotificationMessage(options);

   }

}

This will display the UI Notification message.   I was going to make it a little more generic, maybe another day...

2. Script Include:

Name: u_RecordCount

Active: Checked

Client callable: Checked

Script:

var u_RecordCount = Class.create();

u_RecordCount.prototype = Object.extendsObject(AbstractAjaxProcessor, {

   recordCount: function() {

   //Perform a query based on the table and query passed in

   var result = 0;

   var count = new GlideAggregate(this.getParameter("sys_table"));

   count.addAggregate("COUNT");

   count.addEncodedQuery(this.getParameter("sys_query"));

   count.query();

   if (count.next()) {

   result = count.getAggregate("COUNT");

   }

   return result;

   }

});

This is used to return the number of records in a particular query, used by the Module below.   It is based on the ModuleCount Script Include from Mark's article, but uses GlideAggregate instead.

3. Module:

Title: Email - SMTP Status

Order: 200

Application menu: System Properties

Link type: HTML (from Arguments)

Active: Checked

Arguments:

<img src='/images/send.gifx' oncontextmenu='return showModuleContext(event, "8867cceddc6e2100dc62facef0cff51c")'/><span id='modCount8867cceddc6e2100dc62facef0cff51c'></span>

<script>

refreshNumber8867cceddc6e2100dc62facef0cff51c();

function refreshNumber8867cceddc6e2100dc62facef0cff51c() {

     //Set the module title and link

     var mod8867cceddc6e2100dc62facef0cff51c = gel('modCount8867cceddc6e2100dc62facef0cff51c');

     //Only construct the link if it does not exist yet

     if(!gel('8867cceddc6e2100dc62facef0cff51clink')){

           var menu_link = document.createElement('a');

           menu_link.id = '8867cceddc6e2100dc62facef0cff51clink';

           menu_link.innerHTML = "Email - SMTP Status";

           menu_link.title = "";

           menu_link.href = 'sys_properties_list.do?sysparm_userpref_module=8867cceddc6e2100dc62facef0cff51c&amp;sysparm_query=name=glide.smtp.status%5eEQ%5e&amp;sysparm_view=';

           menu_link.target = 'gsft_main';

           menu_link.className ='menu';

           menu_link.onclick = function() {refreshNumber8867cceddc6e2100dc62facef0cff51c()};

           mod8867cceddc6e2100dc62facef0cff51c.appendChild(menu_link);

     }



   //Query for the count of records for the module

     var count = new GlideAjax('u_RecordCount');

     count.addParam('sysparm_name', 'recordCount');

     count.addParam('sys_table', 'sys_properties');

     count.addParam('sys_query', decodeURI('nameSTARTSWITHglide.smtp.status%5evalue=Cannot%20connect%5eEQ'));

     count.getXML(u_displayModuleCount8867cceddc6e2100dc62facef0cff51c);

}



function u_displayModuleCount8867cceddc6e2100dc62facef0cff51c(response){

     var answer = response.responseXML.documentElement.getAttribute('answer');

     var menu_link = gel('8867cceddc6e2100dc62facef0cff51clink');

     if(answer > 0){

     menu_link.innerHTML = "&#60;font color='Red'&#62;Email - SMTP Status - Problem&#60;/font&#62;";

     top.u_smtpServerProblemWarning("Problem detected with the SMTP Server");

     }

     else {

     menu_link.innerHTML = "&#60;font color='Green'&#62;Email - SMTP Status - OK&#60;/font&#62;";

     }

     window.setTimeout('refreshNumber8867cceddc6e2100dc62facef0cff51c()', 900000);

}

</script>

The code for the module was generated by Mark's solution and then I modified it in a couple places for it to work independently.

The basic idea is the Module is HTML with some script included.   The script calls the Script Include to see if there are any System Properties where the name is "glide.smtp.status" with a value of "Cannot connect".   If there is, the function in the UI Script is called to display the message to the admins.

The UI Script is loaded for everyone, but it does not really do anything.   It just makes the function available and is only called by the code in the Module.   Nice thing about that is the Module can be protected by roles so it will only be loaded for users with the "admin" role.   I had to load the function via the UI Script becuase the UI Notification would not work from within the Navigator.

The code in the module will run on initial login, whenever the Navigator is refreshed and every 15 minutes.   You can change that on the window.setTimeout line of the module, right at the bottom (in milliseconds).

If the AJAX call returns a value > 0, the module title is changed and the message is displayed.

The only thing that I did not address is the "doesn't harass the admins" part.   Clicking on the module will show the system property - I guess you could change the text slightly so the messages are not continuously triggered.   That's what I've been doing while testing.   I see you had something where you would check records to see if the admin acknowledged the message already.   Problem with that is the records would have to be cleared for them to get notified in case of a future issue.

The idea is actually kinda neat, BUT the big problem is there would have to be at least 1 admin logged in to see the message.   A better solution would be to use a Web Service to alert the admins independently of the app.   Maybe fire off a text message or an email somehow using some external means.

Works fine in Dublin, seems to work OK in Eureka (but you might want to remove the image at the top of the HTML for the module if using UI14).   Have not tested in other versions.

I've attached XML files for the 3 records.

[Darth Jed]

Thank you, Jim! I'll give this a try as well.

[Jacob_Andersen]

Very cool, Jim Coyne (CompuCom)!   I can see a lot of customers benefiting from this.

[Jim Coyne]

Thanks.   Again, it's not 100%, but a good start I think.

[Darth Jed]

Jim,

  Here's something that future readers will probably want to know. The glide.smtp.status property is actually on the sys_status table. I was looking for it in the wrong place. I modified lines 13 and 24 of the module script to display and query the sys_status table for the value like "Cannot connect" as shown below.

           menu_link.href = 'sys_status_list.do?sysparm_userpref_module=8867cceddc6e2100dc62facef0cff51c&sysparm_query=name=glide.smtp.status%5eEQ%5e&sysparm_view=';

     count.addParam('sys_query', decodeURI('nameSTARTSWITHglide.smtp.status%5evalueLIKECannot%20connect%5eEQ'));

This allows the module to function correctly. The UI Notification didn't appear to work for me until I logged back into my Calgary instance. Overall, this is a fantastic solution! I can live without the acknowledgement feature since the message is triggered on such a low frequency. On a related note for any other readers that are looking for something like this but with different functionality, I did find a post with three other possible solutions. Email Monitoring and Heartbeat — ServiceNow ELITE.com They didn't meet my needs, but I thought I'd share them anyway.

Thanks again, Jim and Travis. I knew that someone had a solution for this.