One of the most powerful features of ServiceNow is the Discovery plugin and Orchestration plugin. With these plugins, you can discover CIs and automate things within your network when you install the MID Server.
Discovery and Orchestration are magical features. Once you set them up properly, they work like a charm. If you would like to keep the magic alive, maybe you could imagine that we use Minions in our systems. You may skip this blog because it would destroy your fantasy (Or you could read it until the end anyway).
The MID Server in the agentless architecture
The Management, Instrumentation and Discovery (MID) Server is a Java application that sits in your local network and runs as a Windows service or UNIX daemon within your network. Based on the configuration that you set for the MID Server, it connects to your ServiceNow instance via https with the specified username and password. There is no way for an external source to initiate a communication with the MID Server. The MID Server initiates the connection to the ServiceNow instance via https. There is no need to open or tunnel any other external ports to the MID Server and your network can remain safe within your firewall.
Traditionally, client/server architecture requires network to open and expose certain specials dedicated ports to the services. Exposing external open ports pose a risk and can be a nightmare for network administrators.
The role of MID Server in the activity flow
During a discovery activity or orchestration activity, the following sequence of events happen:
- Probe Process
- The Probe Process creates a new record to ECC Queues table (where the "queue" field is "output")
- MID Server process
(The MID Server regularly polls the ECC queue on the ServiceNow instance)- MID Server picks up the record from ECC Queues table (where the "queue" field is "output")
- MID Server process the ECC record
- MID Server executes the commands from the internal network.
- MID Server receive the result from the execution of the command
- MID Server constructs a new ECC record
- MID Server writes the new ECC record back to the ECC Queues table (where the "queue" field is "input")
- Sensor Process
- The Sensor Process picks up the new ECC record from the ECC Queues table (where the "queue" field is "input")
- The Sensor Process process the new ECC record.
Here is an example of the orchestration activity flow:
ECC Queue
The key component between the probe/sensor process and the MID server is the ECC queue.
Generally, the ECC Queue table is the holding place where different components of ServiceNow communicates with each other. The ECC Queue table is also used for internal Event Management. For MID Server services (which uses the probe and sensor process), the ECC Queue table can be understood as the communication channel between ServiceNow instance and the MID Server.
Example of the two ECC Queue records that interacts with the MID Server. a) Output queue: the record with "Queue" field that shows "output", are ECC records for MID Server to pick up for execution. b) Input queue: the record with "Queue" field that shows "input" is written by the MID Server after it finishes processing. |
MID Server services
A number of services are built on top of MID Server. Discovery and Orchestration are two of the main ones. JDBC probes and Scheduled Imports also uses the MID Server services via the Probe and Sensor process. Hence the understanding of how MID Server works, is useful in design consideration and enable you to troubleshoot most issues resulting from using the MID Server.
Limitations and design considerations
Just as much as we would like everything to be done immediately, there are just some MID Server limitations we must consider.
- The discovery and orchestration activities are not designed for real time.
Based on the design, there could be a number of delay along the process. There are also polling and parsing time between the instructions. Hence a split second instruction at the MID Server could take up to two seconds depending on network, I/O, CPU processing power between the ServiceNow instance and the MID Server. - The commands to be executed by the MID Server should not take too long.
If the commands to be executed by the MID Server takes too long, the MID Server might get hogged up and this could be a bottleneck performance issue. - The results from the execution from the MID Server should not be too long.
If the results are too big, this could also cause performance bottleneck at the MID Server and ServiceNow processing.
Troubleshooting the MID Server
The MID Server process goes through a few stages and as a result there are some opportunities for things to go wrong. If there were to be any issues with services using the MID Server, you could analyze the various stages to isolate where the issue is.
You could check the ECC Queue table for the respective records.
- Check the "state" and "processed" field of the ECC record on the output queue to see if it has been processed. You can also compare the difference between the "created" field and the "processed" field to see how long it actually takes to be processed.
- If there are a lot of ECC records on the output queue in ready state, maybe the MID Server is overloaded or overwhelmed.
- If there are a lot of ECC records on the input queue in ready state, maybe the ServiceNow instance is not able to process the messages fast enough.
- You could also open the respective ECC record and review the XML of the payload. By looking at the ECC record on the input queue, we can see the actual results that comes back from the MID Server processing before the sensor process processes them. This is useful to isolate whether there is an issue with the MID Server processing or an error in the sensor script.
Here is an example of the XML payload result from an orchestration activity for PowerShell to Get-Date:
Fantasy Busters
I hope the information is useful. Unfortunately, we do not employ any Minions in our systems. Maybe one day we might replace MID Servers with Minions. Whatever it is, the work of the MID Server continues to be magical.
Resources
For more information, please see:
Everything you wanted to know about Discovery
Discovery Agentless Architecture
ServiceNow KB: Troubleshooting a MID Server that is down or not responding (KB0535180)
1 Comment
