AZURE / EVENT Management

Jonny Lord · ‎12-17-2018

Hey Guys,

We have integrated many monitoring tools successfully into our new ServiceNow instance.

I'm stuck with my Azure Events coming into ServiceNow - i have them coming in via a web hook with the configuration suggested here: https://docs.servicenow.com/bundle/london-it-operations-management/page/product/event-management/task/azure-events-transform-script.html

I can confirm that Azure alerts are coming into ServiceNow but are not populating the Event table.
They are infact populating the logs files with the event message...immediately followed by another log entry with the same timestamp with the source as "com.glide.ui.ServletErrorListener" and payload as;

JavaScript evaluation error on:
(function process(/*RESTAPIRequest*/ request, /*RESTAPIResponse*/ response) {
var body = GlideStringUtil.getStringFromStream(request.body.dataStream);
var queryParams = request.queryParams;
var source = queryParams['source'] ? queryParams['source'][0] : '';
var headers = request.headers;
delete headers["authorization"]; // dont save the base64 user/password
var gr;
if (gs.getProperty('cloud.event.debug') == 'true') {
gr = new GlideRecord('em_connector_push_debug');
gr.body = body;
gr.source = source;
var hstr = JSON.stringify(headers);
gr.headers = hstr;
gr.query_params = JSON.stringify(queryParams);
gr.url = request.uri;
gr.insert();
}

gr = new GlideRecord('em_connector_push');
gr.addActiveQuery();
gr.addQuery('type', 1);
    gr.addQuery('source', 'azure');
gr.orderBy('order');
gr.query();
var results = {};var exception;
while (gr.next()) {
var procSource = gr.getValue('source');
var procName = gr.getValue('name');

   var script = gr.getValue('script');
   gs.debug('Executing event processing script: ' + procName);
   var evaluator = new GlideScopedEvaluator();
   evaluator.putVariable('request', request);
   evaluator.putVariable('body', body);
   evaluator.putVariable('status', 200);
    var res = evaluator.evaluateScript(gr, 'script');
   if(evaluator.getVariable('status')!==200){
    exception=new sn_ws_err.ServiceError().setStatus(evaluator.getVariable('status'));
    exception.setMessage(procName+" : "+ res);
    throw exception;
   }


    results[procName] = res;
   return results;
   //}
}

exception=new sn_ws_err.ServiceError().setStatus(400);
exception.setMessage('No Transform script found for given header or source ');
throw exception;
})(request, response);
: org.mozilla.javascript.JavaScriptException: [object ServiceError] (sys_ws_operation.648f0381672132001e91c44d2685ef6e.operation_script; line 40): org.mozilla.javascript.gen.sys_ws_operation_648f0381672132001e91c44d2685ef6e_operation_script_4974._c_process_1(sys_ws_operation.648f0381672132001e91c44d2685ef6e.operation_script:40)
org.mozilla.javascript.gen.sys_ws_operation_648f0381672132001e91c44d2685ef6e_operation_script_4974.call(sys_ws_operation.648f0381672132001e91c44d2685ef6e.operation_script)
org.mozilla.javascript.ScriptRuntime.doCall2(ScriptRuntime.java:2650)
org.mozilla.javascript.ScriptRuntime.doCall(ScriptRuntime.java:2590)
org.mozilla.javascript.optimizer.OptRuntime.call2(OptRuntime.java:42)
org.mozilla.javascript.gen.sys_ws_operation_648f0381672132001e91c44d2685ef6e_operation_script_4974._c_script_0(sys_ws_operation.648f0381672132001e91c44d2685ef6e.operation_script:1)
org.mozilla.javascript.gen.sys_ws_operation_648f0381672132001e91c44d2685ef6e_operation_script_4974.call(sys_ws_operation.648f0381672132001e91c44d2685ef6e.operation_script)
org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:563)
org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3428)
org.mozilla.javascript.gen.sys_ws_operation_648f0381672132001e91c44d2685ef6e_operation_script_4974.call(sys_ws_operation.648f0381672132001e91c44d2685ef6e.operation_script)
org.mozilla.javascript.gen.sys_ws_operation_648f0381672132001e91c44d2685ef6e_operation_script_4974.exec(sys_ws_operation.648f0381672132001e91c44d2685ef6e.operation_script)
com.glide.script.ScriptEvaluator.execute(ScriptEvaluator.java:279)
com.glide.script.ScriptEvaluator.evaluateString(ScriptEvaluator.java:118)
com.glide.script.ScriptEvaluator.evaluateString(ScriptEvaluator.java:82)
com.glide.script.fencing.GlideScopedEvaluator.evaluateScript(GlideScopedEvaluator.java:309)
com.glide.script.fencing.GlideScopedEvaluator.evaluateScript(GlideScopedEvaluator.java:214)
com.glide.script.fencing.GlideScopedEvaluator.evaluateScript(GlideScopedEvaluator.java:201)
com.glide.rest.service.custom.CustomService.runScript(CustomService.java:95)
com.glide.rest.service.custom.CustomService.execute(CustomService.java:82)
com.glide.rest.handler.impl.ServiceHandlerImpl.invokeService(ServiceHandlerImpl.java:36)
com.glide.rest.processors.RESTAPIProcessor.process(RESTAPIProcessor.java:271)
com.glide.processors.AProcessor.runProcessor(AProcessor.java:483)
com.glide.processors.AProcessor.processTransaction(AProcessor.java:205)
com.glide.processors.ProcessorRegistry.process0(ProcessorRegistry.java:178)
com.glide.processors.ProcessorRegistry.process(ProcessorRegistry.java:167)
com.glide.ui.GlideServletTransaction.process(GlideServletTransaction.java:31)
com.glide.sys.Transaction.run(Transaction.java:2038)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)

What should the header be based on events coming in via a Listener transform script?
The header name and header value fields are none mandatory and i have no clue what they need to be, if in fact this is the issue. Any other direction would be great.

Thanks Guys.

Jonny Lord · ‎12-19-2018

Awesome, i will check this out.
Thanks!

View solution in original post

Jonny Lord · ‎12-18-2018

Hi Nick/all,

Ive checked and triple checked that article and I've even had someone qualify what i have done in case i was going mad.
When i browse to the System logs i can see Azure events coming in and immediately (to the second) followed up with exception error within a bunch of text (in my original post). So an example would be me receiving 300 events and getting 300 errors with the below.

The exception i see is;
exception=new sn_ws_err.ServiceError().setStatus(400);
exception.setMessage('No Transform script found for given header or source ');
throw exception;
})(request, response);
: no thrown error

-------------------------------------------------------------------
My transform script is below (it came out of the box)

(function process(/*RESTAPIRequest*/ request, body) {
   gs.info('inside em azure processor, body: ' + body + ', headers: ' + JSON.stringify(request.headers));
   try {
       var requestBody = JSON.parse(body);
       var severity = requestBody.status;
       var context = requestBody.context; //Get the alert context
       var condition = context.condition; //and seperate the trigger information
       var triggerBody = "Metric Type: " + condition.metricName + "\n"; //Parse the condition into a readable string
       triggerBody += "Metric Unit: " + condition.metricUnit + "\n";
       triggerBody += "Metric Value: " + condition.metricValue + "\n";
       triggerBody += "Threshold: " + condition.threshold +"\n";
       triggerBody += "Window Size: " + condition.windowSize + "\n";
       triggerBody += "Time Aggregation: " + condition.timeAggregation + "\n";
       triggerBody += "Operator: " + condition.operator + "\n";

       var resolutionState = 'New';
       // Set severity to warning if activated and to info and Closing state otherwise
       if(severity == 'Activated')
           severity = '4';
       else {
           severity = '5';
           resolutionState = 'Closing';
       }

       //Create the new event record
       var gr = new GlideRecord('em_event');
       gr.initialize();
       gr.event_class = 'Microsoft Azure';
       gr.source = 'Microsoft Azure';
       gr.type = context.resourceType;


       // Prepare additional_info JSON
       var additional_info = JSON.parse(JSON.stringify(requestBody.context));
       additional_info['status'] = requestBody.status;
       additional_info['object_id'] = context.resourceId;
       additional_info['timestamp'] = context.timestamp;
       var properties = requestBody.properties;
       additional_info['$type'] = properties.$type;
       additional_info['metricUnit'] = condition.metricUnit;
       additional_info['metricValue'] = condition.metricValue;
       additional_info['threshold'] = condition.threshold;
       additional_info['windowSize'] = condition.windowSize;
       additional_info['timeAggregation'] = condition.timeAggregation;
       additional_info['operator'] = condition.operator;
       var region_id = "";
       if(context.resourceRegion) {
           // we remove spaces and make itlower cases because this is how the region saved by discovery
           region_id = context.resourceRegion.replace(/\s/g, "").toLowerCase();
           additional_info['resourceRegion_id'] = region_id;
       }
       additional_info['original_body'] = requestBody;

       //remove name field (if exsits)- that's the alert name and it breaks the binding that tries to bind to this ci name
       if (additional_info.hasOwnProperty('name')) {
           additional_info.alert_name = additional_info.name;
           delete additional_info.name;
       }


       gr.additional_info = JSON.stringify(additional_info);
       gr.description = triggerBody;
       gr.time_of_event = new GlideDateTime();
       gr.severity = severity;
       gr.metric_name = condition.metricName;
       gr.resource = context.resourceName;
       gr.resolution_state = resolutionState;
       gr.message_key = context.resourceType + "-" + context.resourceName + "-" + condition.metricName + "-" + region_id + "-" + context.subscriptionId;
       gr.insert();

   } catch (er) {
       gs.log(er);
       status=500;
       return er;
   }
   return "success";
})(request, body);
------------------------------------------------------------------------------------------------

Can anyone tell me what is going wrong?
Header Name: Azure Events
Header Value: True
URL Parameter Value: azure.
Active
via Instance
https://{Instance-name}.service-now.com/api/global/em/inbound_event_azure
(obviously replacing my instance name)
I used this link because when i used https://<<INSTANCE>>/api/global/em/inbound_event?source=genericJson i received absolutely nothing into service now. not even error logs.

Thanks!

vNick · ‎12-18-2018

Is it possible for you to paste in the payload coming from Azure (masking any proprietary data you need to)?

Jonny Lord · ‎12-19-2018

Hey Nick - Again, thanks for taking the time out to assist here:

The payload from Azure into the logs is as follows;
The inappropriate contents i will replace with the word Example.
Immediately after this arrives i get the below errors after the page break ive mentioned above.

Level - Information
Source - ***Script
Creation date -
Message below;

inside em azure processor, body: {"WorkspaceId":"Example","AlertRuleName":"USE2-USCN-CITRIX-WIN-FREE SPACE IN MB-CRITICAL","SearchQuery":"Perf | where ( CounterName == \"Free Megabytes\" ) | where (Computer contains \"Example\" or Computer contains \"Example\" or Computer contains \"Example\" or Computer contains \"Example\" or Computer contains \"Example\" or Computer contains \"Example\" or Computer contains \"Example\" or Computer contains \"Example\" or Computer contains \"Example\" or Computer contains \"Example\" or Computer contains \"Example\" or Computer contains \"Example\") | where InstanceName !contains \"HarddiskVolume\" and InstanceName !in (\"T:\",\"_Total\") | summarize AggregatedValue = avg(CounterValue) by Computer, InstanceName, bin(TimeGenerated, 5m) | where AggregatedValue < 500","SearchResult":{"tables":[{"name":"PrimaryResult","columns":[{"name":"Computer","type":"string"},{"name":"InstanceName","type":"string"},{"name":"TimeGenerated","type":"datetime"},{"name":"AggregatedValue","type":"real"}],"rows":[["Example","F:","2018-12-19T10:25:00Z",59],["Example","F:","2018-12-19T10:30:00Z",59]]}]},"SearchIntervalStartTimeUtc":"2018-12-19T10:27:02","SearchIntervalEndtimeUtc":"2018-12-19T10:32:02","AlertThresholdOperator":"Greater Than","AlertThresholdValue":0,"ResultCount":2,"SearchIntervalInSeconds":300,"length":"3211","expect":"100-continue","cookie":"JSESSIONID=959EB5136F3B1724442B319DE898D86F; glide_user_route=glide.ec56de7ea57db3a18fe6b348245f4bf0; Example=2709642250.33088.0000","x-forwarded-proto":"https","x-forwarded-host":"Example","x-correlationcontext":"RkkKACgAAAACAAAAEAAG1st1od/3S5DnEeMoxk3cAQAQADbS/aRCc/9Nge8iRBp8d2M=","host":"Example","content-type":"application/json; charset=utf-8","connection":"Keep-Alive","x-forwarded-for":"13.106.57.181","user-agent":"IcMBroadcaster/1.0"}

vNick · ‎12-19-2018

So a few things based on this data... first is that it doesn't look like a "Classic Metric Alert", which is the only one we support with the transform script right now (in K and L).

When I use postman to send to the generic JSON endpoint (https://<instance>.service-now.com/api/global/em/inbound_event?source=genericJson) I get it successfully (screenshot below), but it requires authentication by default so you probably will have to modify it a bit as Azure can't handle the authentication piece. You could add some logic at the top of the inbound_event script to do something like check the subscription ID, though your payload doesn't appear to have anything like that.

This is where you could edit (or duplicate into your own to not adjust the out of the box content), which is the "Inbound Event POST" resource on the "Inbound Event" Scripted REST API. You would have to uncheck the "Requires authentication" and then add logic into the code to validate the payload is coming from a valid Azure source.

Jonny Lord · ‎12-19-2018

Awesome, i will check this out.
Thanks!