Start Monitoring your ServiceNow instance with Predictive AIOps

[charleselite]

UPDATE: NEW ON XANADU -> Setting up Syslogs Monitoring is even easier than documented below.  Watch this video for details: 

You might have seen that in our August Release, we announced the new capability to monitor ServiceNow syslogs using Health Log Analytics. Pair this with the existing option to stream Mid Server logs, you can now get the entire ServiceNow platform monitored and start detecting anomalies using HLA.

ServiceNow Health Log Analytics predicts IT issues before they affect your users. The application helps you solve problems faster by collecting, analyzing, and correlating machine-generated log data in real time. It discovers any anomaly, or deviation from normal behavior, as it happens and alerts you of possible issues.

In addition to predictive alerting, you can leverage the Log Viewer after setting this up, making browsing your system and mid server logs significantly easier and faster. Log Viewer in SOW enables you to browse the logs by timestamp or time range, to search for particular log text, and to visualize the frequency of anomaly occurrences in a particular time period. If you discover an important metric in the log data, you can use it to define a Log Analytics alert rule.

It only takes 30 minutes to set up and you’ll start getting results within days. Here is what you need to do:

Once your are done with this setup, read more on how you can improve Tagging and take immediate action on anomaly alerts here: Advanced Tagging for Syslog Anomalies w. HLA 

Prereqs:

1. Make sure your instance is on a compatible version: 
   Xanadu
   Washington DC Patch 4
2. Install the Health Log Analytics Plugin from the store, and make sure it is up to date.
3. Confirm with Account team on entitlements for HLA

Enable Sys_Log Data Streaming to HLA

1. Create a new Data Input for Sys_logs by Navigating to Health Log Analytics > Data Input > Data Inputs
2. Create new and select Glide Sys Log Retriever

3. Give the record a name like: ServiceNow Sys Logs

OPTIONAL: Add a filter Level is not one of Debug, Trace. Filtering these out will prevent extra alerting on logs that are activated for specific troubleshooting efforts. Ie. You will be focused on logs you aren’t traditionally looking at.

4. Click Start Data Input

NOTE: If you get an error message, you might be missing api access from HLA to the SysLog Table. To resolve:

1. Navigate to All > Activity Subscriptions > Administration > Script includes.
2. Right-select any table header and then select Import XML.
3. Upload the script file that is attached

Complete! HLA is configured to now consume and analyze your Instance logs, you will start seeing alerting in

Recommended Steps: Review Data Input Mapping and Source Type Structures for your SysLog streaming.

5. Navigate to Health Log Analytics > Mapping > Data Input Mapping
6. Open the record for your ServiceNow Syslog Data Input
7. UPDATE: New Data Mapping record preview (TBD shipped in Xanadu Release and Newest version of HLA) : Copy and Paste this script in your data mapping record.

   function map(sample, metadata) {
       var cmp = null,
           stp = 'glide_sys_log';
   
       if (metadata.autoParserOutcome && metadata.autoParserOutcome["sys_class_name"]) {
           var src=metadata.autoParserOutcome["source"] || null;
           var className = metadata.autoParserOutcome["sys_class_name"];
   
           switch (className) {
               case "syslog":
               case "sys_metric_trigger_log":
               case "syslog_app_scope":
               case "wf_log":
               case "pa_job_log_rows":
               case "sys_notification_execution_log":
               case "import_log":
                   cmp = src;
                   break;
               case "syslog_email":
                   cmp = className;
                   break;
               case "pa_diagnostic_log":
                   cmp = src;
                   stp = className;
                   break;
               default:
                   break;
           }
       }
   
       var log = JSON.parse(sample);
       if (log.message && log.message.startsWith("identification_engine")) {
           cmp = "identification_engine";
       }
   	
   // Add more mapping logic here that is specific to your instance to group log sources together.
       var cmpMapping = {
           '^sn_': 'SNTableName',
           '^x_': 'CustomTable',
           '^UADownloader': 'UADownloader',
           '^ImportSetTransformer': 'ImportSetTransformer',
           '^DataCollector': 'DataCollector',
           '^com\\.': 'CustomApps',
           '^\\*\\*\\*': 'Scripts',
   		'^Loader': 'Loader'
       };
   
       for (var pattern in cmpMapping) {
           if (new RegExp(pattern).test(cmp)) {
               cmp = cmpMapping[pattern];
               break;
           }
       }
   
       return {
           'applicationService': 'ServiceNow Glide Sys Log',
           'component': cmp,
           'sourceType': stp,
       };
   }

8. Confirm there are no errors and Test Mode is set to OFF and PUBLISH this record

8. Navigate to the Source Type Structure either by using the related list Source Types  from the Data Input Mapping record.

Or navigate to: Health Log Analytics > Source Type Structures

9. Open the Source Type structure labeled: glide_sys_log
10. Scroll down to the related list for Key/Value Mapping
11. Change Classification of _user and table_name to Automatic Root Cause (ARC Only)

12. Test and publish.

What this does is coaches the AI analyzing your logs to help detect anomalies and alerts on these extra values. The User and Table Name is helpful in determining where or who is triggering the anomalous behavior in the logs.

Enable Mid Server Log Data Streaming to HLA

1. Create a new Data Input for Sys_logs by Navigating to Health Log Analytics > Data Input > Data Inputs
2. Create new and select Mid Server

Note: This is still in BETA mode, which means we will need to do a little extra work on data mapping and source type structures.

3. Fill out the form simply by selecting with Mid Server you want to stream logs from

NOTE: you will want to setup a data input for each mid server in your instance. But I recommend you complete one mid server setup, end to end first, then add others

4. Save and Publish/Start your data input.

5. Navigate to the data mapping record for your new Mid Server Data Input by using the related link on the data input record

Or Navigate to Health Log Analytics > Mapping > Data Input Mapping

6. Open the record for your Mid Server Data Input

7. Update the custom JS function to map it to the right Application Service and a Source Type Structure.

I recommend something like this:

function map(sample, metadata) {
var midName = metadata.eventStream.midName;

  return {
	'applicationService': "Mid Servers",
    'component': midName,
	'sourceType': "Basic_MidServer",
  };
}
// Do not write code here

8. Save, and Test this. If no errors go ahead and Publish.

Note: After a couple of minutes your new Source Type Structure will be automatically created

9. Navigate to the newly created source type structure either by using the related list Source Types  from the Data Input Mapping record.

Or navigate to: Health Log Analytics > Source Type Structures

10. Open the Source Type structure labeled: Default_MidServer_logs

11. From here we are going to create a Custom JS function to map out the key values from the Mid Server Logs

Add this to the Custom JS Function section:

function construct(sample, output) {
    var severityRegex = /\b(ERROR|INFO|WARNING|DEBUG)\b/i;
    severityMatch = sample.match(severityRegex);
    if (severityMatch) {
        output.severity = severityMatch[0].toUpperCase();
    } else {
        output.severity = "INFO";
    }
}

12. Test and validate there are no errors.
13. Scroll down to the related list for Key/Value Mapping and make sure the labels are properly assigned to the appropriate outputs

14. Publish this Source Type Record.       
15. Create Event Rule to bind Mid Server Anomaly Alerts 
    1. Navigate to Event Management > Rules > Event Rules
    2. Create New
    3. Add Filter - based on your app service name from Step 7 Above: sn_application_service_name = Mid Servers   
    4. Add Manual Attribute: ci_identifier =  {"name":"${sn_component}"} 
    5. Override CI Binding to match to the class: ServiceNow MID Server 

Repeat steps 1-8 for each Mid server.

This will stream all mid sever logs, and attach them to the same Application Service and Source Type structure function.

Note: There is an opportunity here to separate your mid servers into different Application Services based on their functions. For example:

• Discovery Mid Servers
• Orchestration Mid Servers
• ACC Mid Servers

Creating different Application Service ‘buckets’ in the Data mapping script will provide your with more targeted anomaly alerting and is helpful if you have a larger deployment of Mid Servers.

Conclusion

After a few days the AI behind HLA will start detecting anomalies and generate alerts that you can view in Express List to start taking action on. You are on your way to proactively monitoring your ServiceNow platform!

You will be able to see logs in the SOW Log Viewer almost immediately and also have the option to setup manual alerting on Logs from that screen.

If you want to take it a step further, add NowAssist for ITOM to decode the log alerts into plain English! Check out this post on how GenAI + Log alerts enhances alerting even further:

Turning Proactive Problem Solving into Reality with GenAI and ServiceNow Logs