This guide provides a comprehensive walkthrough for integrating custom metrics into the Metric database for a new Configuration Item (CI). In our scenario, we aim to start recording temperature measurements from a datacenter rack equipped with instruments and servers essential for our service. The facilities services offer this measurement via a REST API, which we can query. We will create a CI and begin collecting metrics from it.
Metrics can encompass a variety of data points you wish to monitor, such as the number of users logged in or the number of transactions within the last five minutes. This data is valuable for anomaly detection and can be utilized in the alert process to aid in troubleshooting.
This walkthrough specifically addresses the integration of a custom metric for which a standard integration is not available. For standard metric integrations, please refer to ACC-M and/or the Integration Launchpad in the Service Operation Workspace.
Prerequisites
- Red Hat Linux 9 for a MID Server, 2 vcpu, 4 Gb, 256 Gb ssd.
- ServiceNow Install ITOM Health with the following components:
Overview of the steps involved:
- Step 1 Setup MID Server
- Step 2 Enable Metric intelligence on MID
- Step 3 Create a CI
- Step 4 Send in a Metric and setup a CI binding rule
- Step 5 Setup a cron job
- Step 6 View results
- Troubleshooting
Step 1 Setup MID Server
Create a user for the MID server to use.
Setup a user in the Servicenow instance with mid_server role. This role will expand it the roles listed in the screenshot. Make sure this user is not locked or needs to change its password on first log in. If you make some mistakes in the next commands the user may get lock out and you may need to clear that lock if you try again.
Press “New” on the top right corner. Enter username “miduser1”, right click the menu bar and press “save”. Now you can add a role named “mid_server”, this will expand in more granular roles when saved. Then use “Set Password” and set a secure password. Copy and past this password to a secure location, you’ll need it later. Make sure to uncheck the “Password needs reset”. And please be reminded that a user can get “Lock out” when you mis type the password in a later step. Then you can unlock the user here.
Log in into the Linux VM and create a user for the mid server run run as.
sudo useradd servicenowuser
And set a new password:
sudo passwd servicenowuser
Save the download link for the MID Server
Use the link to download the MID server package to the Linux VM (this link will change for newer MID Server version so use the URL as per the previous step):
wget https://install.service-now.com/glide/distribution/builds/package/app-signed/mid-linux-installer/202...
Install the Mid Server package:
sudo rpm -ivh --nodeps mid-linux-installer.washingtondc-12-20-2023__patch1-02-28-2024_03-09-2024_0815.linux.x86-64.rpm
Then configure the MID server with this one liner, please change the instance URL usernames and password as you previously configured them.
sudo /opt/servicenow/mid/agent/installer.sh -silent -INSTANCE_URL https://INSTANCE.service-now.com/ -USE_PROXY N -MUTUAL_AUTH N -CERTIFICATE_REVOCATION N -MID_USERNAME miduser1 -MID_PASSWORD "PASSWORD" -MID_NAME MetricsMID -APP_NAME MetricsMIDApp -APP_LONG_NAME LinuxMetricMidServer -NON_ROOT_USER servicenowuser
In order for the servicenowuser user to be able to restarts a Linux service polkit needs to be setup. Use the following instructions:
sudo vi /etc/polkit-1/rules.d/10-midserver.rules
And add the following content:
polkit.addRule(function(action, subject) {
if (action.id == "org.freedesktop.systemd1.manage-units") {
if (subject.user == "servicenowuser") {
return polkit.Result.YES;
}
}
});
Make the file readable by polkit user 'polkitd' by changing it's ownership
sudo chown polkitd /etc/polkit-1/rules.d/10-midserver.rules
Restart polkit
sudo systemctl restart polkit
After a few minutes the Mid Server should be available for validation
For background information on MID server installation, for example if you want to use another flavour then Red Hat. The polkit part may not apply.
Step 2 Enable Metric intelligence on MID
This step will setup the MID Server to be ready to receive metrics begin pushed in via a REST API call.
The previous step had you locate the MID Server in the instance. Use “Setup Metric Intelligence”:
This will start the Metric Intelligence Extension. Wait for it to get started:
Click on the name “MI Metric Extension MIDServer name” to setup the REST listener:
This will also create a API key:
If you click the MID Web Server API Key you can find the key:
Store the key value. You’ll need it later.
Side note: Another method to find the API key later is here:
And look for mid_webserver_api_key
Go to the MID Server page again and locate the started Mid Web Server:
After a while the MID Web Server has started, a port number can be found here (make a note of it):
Connections can be setup securely which if accessed remotely is a must for security reasons. Since we are going to approach the port locally, we’ll not do this for simplicity for now.
The MID Server is now ready to receive REST calls.
The steps above are documented here:
Step 3 Create a CI
We will create a CI on which we register our Metrics. In our example we will be measuring the temperature on a rack in a data center. We will create a rack CI.
Class manager
Click on Open Hierarchy and locate Equipment Rack and click on it:
Click on CI List and New
Note the cmdb table name:
Enter “cmdb_ci_container_rack.list”
And mark this as a favorite so we can easily get back to it later:
Click on Rack1 and we will be adding some related lists so we can see the Metrics easily later:
Add “Metric Time Series Models” and “Metric To CI Mappings” to the Selected list and press save.
So the CI related list view will look like:
Now we have created a CI and setup its views so we can ingest metrics.
Note: If you pick a CI class with no identification rule you must create it in the CI class manager.
Step 4 Send in a Metric and setup a CI binding rule
Since we don’t have a real temperature measuring device, we are going to use the weather temperature in Amsterdam as our proxy source for temperature metrics.
On the Linux MID server create a script metric_single.sh with the follow content:
vi metric_single.sh
Change the apikey as per the one you created when setting up the mid server in a previous step.
#!/bin/bash
CITY="Amsterdam"
# Fetch the weather data in JSON format
RESPONSE=$(curl -s "http://wttr.in/${CITY}?format=j1")
# Check if the response contains valid JSON data
if echo "$RESPONSE" | jq -e . > /dev/null 2>&1; then
# Extract the current temperature from the JSON response
TEMPERATURE=$(echo "$RESPONSE" | jq -r '.current_condition[0].temp_C')
echo "The current temperature in Amsterdam is ${TEMPERATURE}°C."
else
echo "Failed to get the temperature data."
fi
current_epoch=$(date +%s%3N)
url="http://localhost:8097/api/mid/sa/metrics"
apikey="8e7626923bc3ce10507e9daf55e45a12"
json_data='[{"metric_type": "rack_temperature", "node": "Rack1", "value": '"${TEMPERATURE}"', "timestamp": '"${current_epoch}"', "ci2metric_id": { "node": "Rack1" } ,"unit": "celcius", "source": "EnvMonitor"}]'
curl -v "${url}" -H "Content-Type: application/json" -H "Accept: application/json" -H "Authorization: Key ${apikey}" -d "${json_data}"
Note the key components of the JSON payload
Metric_type = rack_temperature
Node = Rack1
Source = EnvMonitor
Make it executable:
chmod u+x metric_single.sh
Run this command twice.
./metric_single.sh
Validate the result is a http code 200 so we successfully pushed the Metric into the MID Server.
[azureuser@LinuxMIDServer ~]$ ./metric_single.sh
The current temperature in Amsterdam is 17°C.
* Trying ::1:8097...
* Connected to localhost (::1) port 8097 (#0)
> POST /api/mid/sa/metrics HTTP/1.1
> Host: localhost:8097
> User-Agent: curl/7.76.1
> Content-Type: application/json
> Accept: application/json
> Authorization: Key 8e7626923bc3ce10507e9daf55e45aaa
> Content-Length: 157
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Fri, 05 Jul 2024 19:27:16 GMT
< Content-Length: 0
<
* Connection #0 to host localhost left intact
[azureuser@LinuxMIDServer ~]$
Now we have to setup an Event rule to bind the Metric to the CI. Go to:
Find the event that represents the metric sample we pushed in:
Click on the date and start the Event Rule Designer:
Add the Name
Adding the metric type so we can control which Metric we exactly want to allow.
No changes needed in Transform and Compose as well as Threshold.
Setup the Binding so that the Node field is used to identify the CI in the Equipment Rack class (cmdb_ci_container_rack)
If you need to find the Event rule go here:
Step 5 Setup a cron job
To have the metric being push every 5 minutes we need to setup a cron job. This is the easiest way to do for this so
sudo contab -e
And add the following content.
*/5 * * * * /home/azureuser/metric_single.sh
And check if it worked:
sudo crontab -l
*/5 * * * * /home/azureuser/ metric_single.sh
Step 6 View results
We had marked “cmdb_ci_container_rack.list” as a favorite so we can review our Rack1 CI. And see can see that we have data starting to appear in Metric to CI and a Time Series Models.
Metric intelligence we start to learn the pattern of this metric in the coming days.
If you want to learn about the statistical models go here:
To see the Metric in Insights Explorer setup a new View with the Rack CI in it.
Hit the + sign and give the view a name
Now click on the CI and select the rack_temperature metric and drag it to the “Drop Metric here”
Over the next few days watch the metric and toggle the “Show Bounds” to see what they do. Note not all metrics lend themselves to Anomaly detection, see the statistical models link a bit back in this document. The bounds will change with every analysis run every 24h. And tweak the Time setting to see a daily or weekly few when there is enough data.
You can experiment with different sources with this same mechanism (If you have something more relevant to your business use that by all means). Many time this involved getting an API key for the source API. It is of course possible to push metrics in from a source directly into the MID Service push API if the source can adhere the the JSON structure required.
In next blog in this series: Walkthrough guide for AIOPS and Metrics 2: Configure Metric Explorer and AIOps Dashboard
Notes
- The first time a new metric is pushed is creates the necessary structures in the MID Servers and instance and the data will be discarded. The second value will be recorded.
- For the ML to work on metrics, they need to be push in via a MID Server. The MID Server plays a role in the anomaly detection.
- There is no Metric Database on the PDI instances via developer.servicenow.com (no Metric Database)
- Check if the Metric database (Clotho) is provisioned and is working (stats.do). See below
Troubleshooting
- See if the Metric Database is functioning (stats.do)
- If you want to reset the registration of a Metric go to “Metric To CI Mapping” and delete it
- Watching a MID Server log file on Linux
tail -f /opt/servicenow/mid/agent/logs/agent0.log.0
- Restarting the MID Server on Red Hat Linux
[azureuser@MidMetricsBlogs ~]$ sudo systemctl | grep -i mid
MetricsMIDApp.service loaded active running LinuxMetricMidServer
or
sudo systemctl --all | grep -i mid
[azureuser@MidMetricsBlogs ~]$ sudo systemctl status
MetricsMIDApp.service - MetricsMIDApp.service - LinuxMetricMidServer
Loaded: loaded (/etc/systemd/system/MetricsMIDApp.service; enabled; preset: disabled)
Active: active (running) since Fri 2024-06-28 20:15:54 UTC; 7min ago
Process: 19910 ExecStart=/opt/servicenow/mid/agent/bin/mid.sh start sysd (code=exited, status=0/SUCCESS)
Main PID: 19989 (wrapper-linux-x)
Tasks: 113 (limit: 23157)
Memory: 645.5M
CPU: 42.618s
CGroup: /system.slice/MetricsMIDApp.service
├─19989 /opt/servicenow/mid/agent/bin/./wrapper-linux-x86-64 /opt/servicenow/mid/agent/bin/../conf/wrapper.conf wrapper.syslog.ident=MetricsMIDApp wrapper.pidfile=/opt/servicenow/mid/a>
└─20009 /opt/servicenow/mid/agent/jre/bin/java -Djava.util.logging.config.file=properties/glide.properties -Dsun.net.maxDatagramSockets=65535 -Dcom.sun.jndi.ldap.object.disableEndpoint>
Jun 28 20:15:50 MidMetricsBlogs systemd[1]: Starting LinuxMetricMidServer...
Jun 28 20:15:50 MidMetricsBlogs mid.sh[19910]: Starting LinuxMetricMidServer...
Jun 28 20:15:54 MidMetricsBlogs mid.sh[19910]: Waiting for LinuxMetricMidServer......
Jun 28 20:15:54 MidMetricsBlogs mid.sh[19910]: running: PID:19989
Jun 28 20:15:54 MidMetricsBlogs systemd[1]: Started LinuxMetricMidServer.
[azureuser@MidMetricsBlogs ~]$
sudo systemctl restart MetricsMIDApp.service
- MID Server not installing:
Validating the Mid-user details...
ERROR: Could not authenticate or authorise user 'miduser1' on the ServiceNow instance
Likely the user is locked due to too many bad logins, uncheck the "Locked out" box and save.
- Removing the MID Server service in case you need to run the MID Server install command again
sudo systemctl stop MidMetricNew.service
sudo systemctl disable MidMetricNew.service
sudo rm /etc/systemd/system/MidMetricNew.service (Run this twice)
sudo rm /usr/lib/systemd/system/MidMetricNew.service (Run this twice)
sudo systemctl daemon-reload
sudo systemctl reset-failed
