The Agent Client Collector Monitoring app (or "ACC-M") can be extended to use a multitude of monitoring scripts and plugins. This is an example of how to do so using a Nagios plugin.
Preamble
This process will largely follow the same steps I outline in my previous article, Adding a Custom Plugin to ACC-M. In addition, I will leverage automation for bundling up the files and uploading them to my ServiceNow instances as featured in this article: Pipelining Custom ACC Plugins. The Agent Client Collector natively supports running Nagios-style scripts/executables (or "plugins"). A bit of adjustment is required in order to pass in sensitive parameters such as credentials and I will cover that in this article.
This article assumes you are a ServiceNow admin/developer comfortable with the Linux command line and the Git source management system.
NOTE: the steps and code provided in this example are provided on an as-is basis, with no support nor explicit/implicit guarantees.
Create Git Repository
The Nagios plugin which I'll be configuring is "Check TrueNAS - Extended Play" (https://github.com/StewLG/check_truenas_extended_play). The first step in the process of bringing it into my ServiceNow instance is to create a Git repository (or "repo") to hold the required files. I recommend naming the Git repo the same as the plugin will be named within your instance. I derived the name based on how the standard ACC-M plugins are named, e.g. "monitoring-plugin-truenas".
Populate Git Repository
After creating my Git repo, I populate it by following the directions in Pipelining Custom ACC Plugins, e.g.
- create a "bin" directory, into which the scripts and executables are placed -- in this case the Python script "check_truenas_extended_play.py"
- create an "allow_list" directory, into which you put a properly-formatted JSON allow list file, named "check-allow-list.json"
- create a "plugin.json" file which enumerates the plugin name and directories to include in the archive
Create Pipeline
Once the Git repo is populated, create a CI/CD pipeline similar to the one in Pipelining Custom ACC Plugins which will bundle and upload the files for the plugin. Use the pipeline to populate the corresponding Plugin record in your ServiceNow instance.
Confirm Plugin Creation
After at least one successful run of your CI/CD pipeline, you should see a record under Agent Client Collector->Plugins which matches your plugin name from your Git repo (e.g. "monitoring-plugin-truenas"). You are now ready to define one or more checks which use the scripts and executables contained in your new plugin.
Define ACC Check
Navigate to Agent Client Collector->Check Definitions. Click "New" to create a new check definition and populate the record as follows:
Name: follow the naming pattern used by the included plugins; for my example I'll name my check "app.truenas.check-zpool"
Check type: "Events" for commands which return an OK/WARN/CRIT type result, "Metrics" for commands which return numeric time series data
Command auto generation: unless you have a need to manually generate the check command, keep this box checked
Command prefix: populate this field with the base command to be executed by this check; this is commonly the name of the script or executable -- checks which use secure parameters are a special case, refer to the next section for the specific information on how to use those with a Nagios plugin
Check Parameter Definitions: this related list contains the command line parameters required by the script/executable
Check Secure Parameter Definitions: this related list contains any sensitive command line parameters such as credentials or other authentication tokens; see the following section for information on how to use these with a Nagios plugin
Plugins: ensure that the plugin which you created in the prior step is referenced in this list -- it is what prompts the monitored agents to retrieve the needed archive file containing the scripts/executables
Secure Parameters With Nagios Plugins
Secure parameters are not passed into a check command on the command line so that they do not appear in agent or instance logs or the ECC queue. Instead, they are passed via standard input ("stdin"), in the same order in which they are listed in the check definition. Sensu plugins recognize this mechanism automatically, but the same is not true for Nagios plugins. One approach for consuming these secure parameters for use with a Nagios plugin is to create a wrapper script which reads from stdin and then passes the values through on the command line. An example of such a wrapper, written as a Bash script, is as follows:
#!/bin/bash
# semi-generic wrapper for a Nagios plugin which requires sensitive parameters
# should support any Nagios plugin which takes -u <user> and -p <password>
# expects variables from stdin in the order "cred_user_name", "cred_password"
# extract credentials from stdin
read cred_user_name
read cred_password
# get current directory
current_dir=`dirname $0`
# extract plugin name
plugin_name=$1
# shift args
shift
# call plugin, add -u <cred_user_name> -p <cred_password>
${current_dir}/${plugin_name} -u ${cred_user_name} -p ${cred_password} $@I save the wrapper as "nagwrap.sh" in the "bin" subdirectory of my repo. Merging the update into the main branch invokes my pipeline which automatically updates the record in my instance.
The wrapper expects two secure parameters, "cred_user_name" and "cred_password", to be passed in that order. Referring to the Secure Parameter Definition list, I ensure those two parameters are defined in the proper order. Those names are important, as it causes the check to extract the user_name and password fields from the credential record which I specify later in the setup process.
To invoke the wrapper as part of the check command, I prepend it to the actual plugin command in the "Command prefix" field, e.g. "nagwrap.sh check_truenas_extended_play.py".
NOTE: Be sure to include the wrapper name in the allow_list/check-allow-list.json file under your repo.
Create/Update Check Policy
To activate the new check on monitored CIs, I create a new Policy record. First navigate to Agent Client Collector->Policies. Then click "New". Particular fields of interest are as follows:
Name: name the policy in a manner consistent with the existing ones, for example I used "hallam-nas-checks"
Monitored CIs: define a filter statement or script which enumerates the desired list of CIs to run the check
Checks: select the desired check(s) to be run under this policy
Proxy Settings: for checks which will not run locally on an Agent, specify which Agent(s) should be running the check on your behalf
Credentials: for checks which take a username/password (such as my example), specify a Credential or Credential Alias record which contains the needed info to authenticate the check.
Once this new policy is published, using the "Publish" button or the "Save" button followed by the "Republish" button, the new check will be executed on target Agents. The Agent Client Collector natively understands the standard Nagios plugin output syntax and will forward results back via Events (for OK/WARN/CRIT type "Event" checks) or time series data (for "Metrics" type checks).
