Pending events stuck in a Ready state / Create Rest call for ServiceNow Report

[Hernan5]

We recently had an issue with our production instance having multiple events stuck in a Ready state. As a way to get ahead of this in case, this issue happens again. I've added a report to the Event Management Overview dashboard in ServiceNow that displays the number of pending events in a Ready state to monitor pending events.
 
I've been tasked with looking to see if we can get a REST call to this report, so we are notified once the number of events has exceeded a certain number of pending events in a Ready state. This is not something I have done before. If this is something you are familiar with, please share any insight. 

Also, what approach would you take to have an incident created automatically if the pending events in a Ready state exceed a certain number of pending events in a ready state?

Thank you in advance! 

[Vasantharajan N]

Step 1: Set new custom system property to mention the threshold for the event queue in Ready state.

Step 2: Write a script to query sysevent table with event sate in Ready and check the count and compare it with the step 1 system property value. if the value is above then create ticket.

Step 3: Schedule the Job with script developed in the step 2, to monitor the event queue.

[Hernan5]

Hello Vasantharajan,

Can I use 'diagnostics.condition.events' (system property name) instead of creating a new custom system property? If so, can you share what the code would look like for step 2 that you mentioned? 

[Vasantharajan N]

Please try the below script. Please make the necessary change in the incident create/update logic 

var thresholdCond = gs.getProperty("diagnostics.condition.events");

// Last 2 hours event count whose state is Ready
var eventGr = new GlideRecord("sys_event");
eventGr.addEncodedQuery("state=ready^sys_created_onONLast 2 hours@javascript:gs.beginningOfLast2Hours()@javascript:gs.endOfLast2Hours()");
eventGr.query();

var pendingEventCount = eventGr.getRowCount();

var isExceededThreshold = eval(pendingEventCount + thresholdCond);

if (isExceededThreshold) {
    var short_description = "ServiceNow Alert : Pending Event queue count exceeded " + thresholdCond;
    
	//Check any active incident exist in the system
    var incGr = new GlideRecord("incident");
    incGr.addQuery("short_description", short_description);
	incGr.addActiveQuery();
	incGr.orderByDesc("sys_updated_on");
	incGr.setLimit(1);
    incGr.query();
    if (incGr.next()) {
		//Add worknotes to the ticket 
		incGr.work_notes = "Current Pending Event count "+pendingEventCount;
		incGr.update();
    } else {
		//If no active ticket exist then create a new ticket.
        incGr.initialize();
        incGr.short_description = short_description;
        incGr.description = "Current Event count to be processed is " + pendingEventCount;
        incGr.assignment_group = "sys_id_of_the_group";
        // Include other mandatory field required to create incident

        //
        incGr.insert();
    }
}