What happens to SNMP Traps if the MID Server cannot connect to the ServiceNow Instance?

duanew · ‎07-03-2018

We receive SNMP Traps at the MID Server which are then forwarded to the Instance as Events. What happens if the MID Server connection to the Instance is down?

Does the MID Server "store and forward" the traps?

Narayan2 · ‎07-03-2018

Hi,

Mid server does not store data so when ever there is no connection between Service now instance and mid server the data will be lost. Essentially the collected data isn't really stored locally other than in logfiles

Below is the process how Mid server works:

   - Every 15 seconds, the MID Server checks the ECC queue for jobs that are assigned to it.
   -If a job exists in the ECC Queue for that MID Server , the MID Server sets the status to "I'm working on it"
   - Does the work that is requested.
   - Reports the findings of the job back to the ECC queue.

Please, mark correct or useful if I helped you

Warm Regards,

Narayan Munagala

Engineer

Board Line +91-20-6728-5000 | Direct +91-20-6728-5147 | Mobile +91-9158467666

|www.DxSherpa.com|

View solution in original post

tim_broberg · ‎07-04-2018

Hmm... I haven't tried it, but my expectation is that the event would get translated into an ecc_queue message which would get stuffed in an .xml file in agent/work/monitors/ECCSender/output_* until it gets sent successfully to the instance.

- Tim.

robertgeen · ‎07-04-2018

Tim,

I don't disagree it probably does end up using the ecc queue but they specifically call out API in their diagrams so better safe then sorry. Also the trap may store and forward if that is the case as it's a separate extension of the MID Server software. Either way if anyone does test this let us all know what you find.

tim_broberg · ‎07-04-2018

That is diplomatic of you, and I appreciate that, but it looks like you're just right and I'm just wrong.

I'm no expert on mid server extensions, snmp, or events, but I do have the source code and I have battled a few network retry issues in this code this year.

I see where it sends the events to the instance, where it captures the error, and what it does: logs "Failed to send event:" at the low level and then ("SNMPTrapListener: (" + fConnectorName + "): Failure sending the event to the Instance: " + event) at the snmp trap listener level.

So, at this point, all you could do is scrape the agent logs.

That seems pretty dodgy to me. (...although, while we're in the neighborhood, there's no guaranteed delivery on the original trap either, right? This is global to all events sent from mid to instance, however, not just snmp traffic.)

I logged PRB1293340 to have the mid server guys at least look at the issue and make a determination.

- Tim.

ori2 · ‎07-05-2018

The TRAP that are sent are stored on a in memory queue directly (Not to the ECC queue) , from this queue those traps are sent to the instance by bulks. The limitations on the mid queue are the size (configurable)- if it will reach the queue max size the traps will be lost or if the mid server powered off.

robertgeen · ‎07-05-2018

Hello ori,

Thanks for the information from the DEV side of the house. So when it builds up the data in the memory queue does it make API calls to the ServiceNow instance in bulk and thus bypassing ecc queue completely? Either way thanks for the information it's great to get to see the inside workings of this. @duanew if you can update the correct answer to ori. Thanks.