Advanced Tagging for Syslog Anomalies w. HLA

[charleselite]

Harnessing the power of HLA to monitor your platform logs for anomalies is truly transformative. Imagine being able to detect hidden issues before they become major problems!

Read this blog post to get started: Start Monitoring your ServiceNow instance with Predictive AIOps 

One challenge you might face is the sheer volume and variety of alerts. While HLA effectively filters through the noise, you can take it a step further with Event Management and AiOps Experience by applying advanced tagging on anomaly alerts. These tags allow you to sort, filter, make informed decisions, and automate processes like never before.

HOW TO START ADVANCED TAGGING FOR HLA ALERTS:

STEP 1: Setup your Syslog Streaming: Start Monitoring your ServiceNow instance with Predictive AIOps 

STEP 2: Take note of all the fields you mark in the Source Type Structure as Automatic Root cause or ARC.

** Step 11 from Syslog Streaming Blog:

STEP 3: We are going to create an advanced field mapping script in Event Management to use those ARC properties as tags.

To setup Advanced tagging via Field Mapping:

1. Navigate to Event Management > Rules > Field Mapping
2. Create new Record

   1. Name: Log Analytics Advanced Tagging

      Source: Log Analytics

      Order: 1000

      Mapping Type: Advanced mapping using script

      Filter: Resource starts with ServiceNow Glide Syslog**

      ** Double check your setup, and make sure your App Service from HLA is set or update the filter in this record so you only want to run this against your ServiceNow system logs.

      Script:

      /**
         eventGr - GlideRecord representing the event.
         origEventSysId - Id of the event.
        The GlideRecord event parameter is a temporary object, and therefore does not contain the id of the original event.
         fieldMappingRuleName - The name of this field mapping rule.
       */
      (function eventFieldMappingScript(eventGr, origEventSysId, fieldMappingRuleName) {
          // Make any changes to the alert which will be created out of this Event
          // Note that the Event itself is immutable, and will not be changed in the database.
          // You can set the values on the eventGr, e.g. eventGr.setValue(...), but don't perform an update with eventGr.update().
          // To abort the changes in the event record, return false;
          // Returning a value other than boolean will result in an error
      
          try {
              var eventJsonString = eventGr.getValue('additional_info');
              var eventJson = JSON.parse(eventJsonString);
      
      
              var rawJson = new GlideRecord('em_extra_data_json');
              if (rawJson.get(eventJson.sn_json_id)) {
                  var finalJson = JSON.parse(rawJson.getValue('json'));
      
      
                  if (finalJson && finalJson.incident && finalJson.incident.topAlerts && finalJson.incident.topAlerts) {
                      var properties = finalJson.incident.topAlerts[0].properties;
                      var selectedProperties = ['_user', '_script_record', 'record_sys_id', 'table_name', '_scope', 'source', "sys_class_name", 'job_name', 'source_package']; // List of selected properties
      
      
                      if (!eventJson) {
                          eventJson = {};
                      }
      
                      for (var i = 0; i < selectedProperties.length; i++) {
                          var key = selectedProperties[i];
                          if (Object.prototype.hasOwnProperty.call(properties, key)) {
                              var prefixedKey = 't_' + key;
                              var value = properties[key];
      
                              eventJson[prefixedKey] = value;
      
                              // Stringify and set the updated eventJson object within the loop
                              eventGr.setValue('additional_info', JSON.stringify(eventJson));
                          } else {
                              // Use this for Debugging 
      // gs.info("Property not found: " + key);
                          }
                      }
                  }
              }
      
              return true;
          } catch (e) {
              gs.error("The script type mapping rule '" + fieldMappingRuleName + "' ran with the error: \n" + e);
              return false;
          }
      })(eventGr, origEventSysId, fieldMappingRuleName);

3. Review line 26 in the script, and make sure you add all the properties you marked as ARC in your source type structure record.
4. Save Record.

Results:

All your Log Analytics Alerts will now have your key properties as alert Tags.

WHAT CAN YOU DO WITH THIS:

The first quick, easy value here is the ability to create filters in express list, which will give you a focused and targeted view of Anomalies from your syslog.  Here are some examples:

• View all anomaly alerts related to your inbound/outbound integrations.

HOW: Tag: _user, Contains, Integration Accounts

• View all anomaly alerts related to specific applications or pillars across the platform (ITOM, ITSM, HR, SecOps, SPM, Etc.)

HOW: Create a filter where Tag= Source_package or _Scope and Value = your targeted applications (or set of applications)

Ultimately you can take this to consume data easier, and find more actions and provide feedback to the AI engine behind HLA.

Take this further by using these tags to take action! Automation opportunities, based on Job names, application, etc. The opportunities are endless.

Share your ideas on what you can do with advanced tags on your Anomalies alerts below!