Sending AWS CloudWatch Alarms to ServiceNow through a MID server

[vNick]

Within the event management module of ServiceNow there has long been the ability to ingest CloudWatch alarms from AWS.  In this post I'm going to explain how to achieve this same outcome, but using a MID server to route the data.  Why is this needed?  Some customer instances of ServiceNow are configured to only allow access from specific IP ranges so that external users are not able to access their system.  In such configurations, accepting events directly to the instance from a public cloud provider like AWS is not possible.  Special thanks to my colleague Vinh Tran for assistance with some of these items.

The components involved in this solution are 1) an AWS SNS topic and subscription, 2) an AWS Lambda application, 3) a ServiceNow Listener Transform Script, and 4) a ServiceNow event rule.  Obviously a MID server is also needed here, but I will not be detailing the setup here.

AWS Configuration

Let's start with the AWS components.  You will first need to create a Lambda application.  You can wait on specifying triggers until the next step.

AWS Lambda Application

The environment variables are important as they are used in the Python code of the application.  The password is anticipated to be base64 encoded.  The instance URL needs to be in the form of http://{IP address of your MID server}:{port}/api/mid/em/inbound_event?Transform={name of your transform script}.  We will see the {port} configuration later in this post.  The "transformHeader" is the same as the transform script you specify in the URL. 

Here is the code of the Lambda app:

import json
import logging
import os
import urllib3
import base64

logger = logging.getLogger()
logger.setLevel(logging.INFO)

INSTANCE = os.environ['instance']
HEADER = os.environ['transformHeader']
SNUSER = os.environ['user']
SNPWD = os.environ['pwd']

MYPWD = base64.b64decode(SNPWD).decode('utf-8')

def lambda_handler(event, context):
    logger.info("Event: " + str(event))
    #message = event
    message = json.loads(event['Records'][0]['Sns']['Message'])
    logger.info("Message: " + str(message))
    
    # Create the http session
    http = urllib3.PoolManager()

    # Set the ServiceNow instance or MID server endpoint to receive events
    url = INSTANCE

    # Eg. User name="admin"
    user = SNUSER
    
    # base64 password
    pwd = MYPWD
    
    auth = user + ":" + pwd
    
    # Set proper headers
    headers = {"Content-Type":"application/json","Accept":"application/json","Transform":HEADER}
    headers.update(urllib3.util.make_headers(basic_auth = auth))
    
    # Encode the payload
    encoded_data = json.dumps(message).encode('utf-8')
    
    # Do the HTTP request
    r = http.request('POST', url, body=encoded_data, headers=headers)
    
    # Log the output of the request
    logger.info("Data: " + str(r.data))
    logger.info("Status: " + str(r.status))

The Lambda app is expecting to receive CloudWatch alarms from a SNS topic, so you make the app be a subscriber to a SNS topic where you have configured alarms to be sent.

AWS SNS Topic Subscription

That is about it on the AWS configuration.  Setting up the subscription will add a trigger to the Lambda app.

MID Server Additional Configuration

Let's review what needs to be done to ensure the MID server can accept the alarms.  First, you need to ensure you have defined a MID Web Server Context (MID Server -> Extensions -> MID Web Server).

MID Web Server Context

Be sure to specify a port that is not already in use on the MID server.  This is the port you specify as part of the URL provided to the Lambda app above.  The user and password specified on this record are also the ones provided to the Lambda app so it can authenticate to the MID server web listener.

The final MID server setup step is to create a MID WebService Event Listener Context as shown below, and associate it with the MID web server context you just created above (Event Management -> Event Listener (Push) -> MID WebService Event Listener).

MID WebService Event Listener Context

It is also important to verify the MID server OS (presumably Windows) is allowing inbound traffic on the port you specified for the MID web server above.  In lieu of turning of the entire firewall, you might test disabling the policies while doing initial configuration or if traffic doesn't seem to reach the endpoint you're specifying (run the following command to disable "netsh advfirewall set allprofiles state off").

MID Server Transform Script

One of the final steps is to now create a transform script that will be used by the MID web server to process the incoming CloudWatch alarms.

Listener Transform Script

While the code in the script is not terribly long, it would be quite a bit to consume in the length of this post.  As such, I have posted an update set that will create the MID script include and this listener script record.  The "Header value" on this record is the same as used in the Lambda app.

The final step is to create an event rule or rules to process the alarms.  While there are some out of the box rules for AWS CloudWatch, I created an update set that will process specifically for EC2 instance alarms.  This can server as a good starting point should you need to create others based on how this process is bringing in data to the event table.  The update set is in the same repo as the MID transform script update set.

This same process could be replicated for other cloud providers where it's not feasible to obtain all of the possible IP address from all the data centers they have around the world.  Should you need to scale this setup due to a large volume of CloudWatch alarms, consider putting multiple MID servers behind a load balancer (easy in the public cloud environments) and have the Lambda app send data to the load balancer.

[Miran Alam]

Hi Nick,

I appreciate this post since it has helped me to an idea on how I can implement this..

I am currently doing a integrations between X and ServiceNow. I need to use the Listener transform script but I am not good at coding... So I have copied your transform script and made it my own, tho I dont understand under which row/funtion I can map these fields.. Can you give me an example ?

The requirements are exactly the same as yours but from another system.

Can you give me an example on where in the code I do the mappings for the fields below?

Also, this code generates a alert on the event table right - And puts all the additional info in the "additional info" field?

[vNick]

Hello - In my example, I could have transformed the data in the Lambda script before sending to ServiceNow and made it a nice JSON format supported natively by event management which would have allowed me to normalize the data with just an Event Rule.  However, I chose to send the raw data and process it in a transform script.

You can see other examples (if you haven't loaded my update set) and see how they are populating the "event" record based on the payload sent in.  It really depends on what is being sent.

If you can format the payload prior to sending and put most values into the "additional_info" object, then you probably don't need to complicate it with a transform script.  Just send directly to the MID listener endpoint (http://{MID_Server_IP}:{MID_Web_Server_Port}/api/mid/em/jsonv2) and process with an event rule.

https://docs.servicenow.com/bundle/paris-it-operations-management/page/product/event-management/task/configure-listener-transform-script.html

https://docs.servicenow.com/bundle/paris-it-operations-management/page/product/event-management/task/configure-em-context-extension.html#event-collection-extension

[Miran Alam]

Alright, thanks!

I have uploaded the update set in my dev instance, are there more then one mid server script include in it?

[vNick]

No - the update set just has the 1 new one for AWS through a MID (the one for going directly to the instance is part of the baseline instance).  There should be others there too as part of the baseline (like BMC Truesight and others).

[Miran Alam]

Hi Nick, if you look at the picture I sent above... I am now mapping the values I will recieve from the other system.. So the values at left in the picture is from System X and right is on ServiceNow.

I have done this exactly like "bmctruesight" OOTB script:

if(keyVal == "data") { // data field is a JSON object
var eventObject = jsonObject.getJSONObject("data");
var dataKeys = eventObject.keySet().toArray();
//ms.log("dataKeys: " + JSON.stringify(dataKeys));
for(var dataKey in dataKeys) { // Go over the data field (JSON) keys
var dataKeyVal = dataKeys[dataKey];
var dataValue = eventObject.get(dataKeyVal);
if(dataKeyVal == "Event")
event.setField("message_key", dataValue);
if(dataKeyVal == "msg")
event.setField("description", dataValue);
if(dataKeyVal == "HostName")
event.setField("node", dataValue);

Can you show me a sample script to continue where I can map all the "additional_info" fields or should I continue on the if statement? (See my picture on the first comment)

Much appreciated.

[Miran Alam]

We’ll be receiving the event in JSON format, that’s for sure. The only difference is that instead of the value (text) in the additional_info, we’ll have a JSON string there that needs to be further processed. Can this be handled in an event rule or do I need a transform script?

[vNick]

If you keep going down in the code of the transform script, you'll see where it converts everything into the additional info field.  Yes, you'd need to do this in a transform vs event rule.

// Create additional_info field
	this.updateAdditionalInfo(event, "", jsonObject, additionalInfo);
        // Iterates over Additional information JSON object and adds all nested objects' fields as fields of the Additional information object
        var notDone = true;
        while (notDone) {
              notDone = false;
              for (var key in additionalInfo) {
                  if (Object.prototype.toString.call(additionalInfo[key]) == '[object Object]') {
                        notDone = true;
                        this.updateAdditionalInfo(event, key, additionalInfo[key], additionalInfo);
                        additionalInfo[key] = "";
                    }
                }
            }

[Miran Alam]

Thank you Nick!

So is this the place where I should map all the additional info fields (See picture 2 in event rule). All my fields in Pic 1 with "additional_info" should be written in the even rule right.. And the transform script iterates over additional information?

Pic 1.

Pic 2.

[Miran Alam]

Hi Nick,

I am trying to do this but the Node and Resource is not getting populated from the MacAddress in Additional Info.. Am I missing something?

Also I want to say thank you for helping me out and for this article. I have made a lot progress 🙂 

 function updateAdditionalInfo(event, field, jsonObject, additionalInfo) {
        for (var key in jsonObject) {
            var newKey = key;
            if (field != "") {
                newKey = field + '_' + key;
            }
            //You can do some transformation here and set fields on the event
            //             if (key == "Source")
            //                 event.node = jsonObject[key];

            if (key == "MacAddress") {

                event.node = jsonObject[key];
                event.resource = jsonObject[key];
				gs.info(event.node + jsonObject[key]);
            }
            additionalInfo[newKey] = jsonObject[key];

[vNick]

If you're getting the event in with all the additional info populated, then it would be easier to just assign the necessary value to the node and/or resource in the event rule used to process it.  Difficult to troubleshoot code in the context of the forum.