When an incident is created and assigned to a technician, one of the first tasks the technician must complete is to look through log files for errors (at least when the incident relates to infrastructure or data center applications). This requires the tech to swivel chair to login to the system, find the correct log file, possibly FTP or SCP the file off the system to their own computer or copy and paste to the incident. All of this adds time to the process of resolving the incident.
In this short article I hope to provide a simple option for automating this type of log retrieval and automatically add these types of files to an incident. The features involved in the discussion below include event management, CMDB, and orchestration.
NOTE: This example could be expanded to work with both Windows and Linux. However, this code and workflow only work on *nix based platforms and assume the presence of certain utilities to function correctly.
Let's take a look at the script that will run on the server to retrieve the log file (in this example, an Apache error log file).
#!/bin/sh
# Use the apachectl command to find the apache root and log directory
httprootvar=`apachectl -V | grep ' -D HTTPD_ROOT'`
errorlogvar=`apachectl -V | grep ' -D DEFAULT_ERRORLOG'`
# Remove the quote around the path from the apachectl results, and remove all spaces
httprootclean1=`echo $httprootvar | tr -d '"'`
httprootclean2=`echo $httprootclean1 | tr -d [:space:]`
errorlogclean1=`echo $errorlogvar | tr -d '"'`
errorlogclean2=`echo $errorlogclean1 | tr -d [:space:]`
# Clean the values to only include the path values
httproot=`echo $httprootclean2 | sed -r 's/-DHTTPD_ROOT=//'`
errorlog=`echo $errorlogclean2 | sed -r 's/-DDEFAULT_ERRORLOG=//'`
errlogdir=$httproot'/'$errorlog
# Use curl to send the log file to a ServiceNow incident record
curl "https://${activityInput.instance}.servicenow.com/api/now/attachment/file?table_name=incident&table_sys_id=${activityInput.inc_sys_id}&file_name=apache_error_log" \
--request POST \
--header "Accept:application/json" \
--user '${activityInput.username}':'${activityInput.password}' \
--header "Content-Type: text/plain" \
-F "apache_error_log=@$errlogdir"Log retrieval script
In the script above, you can see we are leveraging "apachectl", "tr", "sed", and "curl" as utilities for finding and parsing specific pieces of data. If your Apache servers do not have these utilities, then the script will need to be modified to retrieve the necessary data (in this case, the location and name of the log file).
The curl command is using a ServiceNow attachment API to "POST" the log file to a specific incident that we are passing in via the orchestration workflow we'll discuss shortly. To being, however, we will have to create a custom orchestration activity that will be used as part of the workflow (at least until Integration Hub supports SSH).
Certain inputs will be required when defining the activity because the attachment API needs these parameters to successfully execute.
Custom SSH Activity Inputs
You then copy and paste the script above into the "Execution Command" section and you can see how the inputs we just defined are used within the curl command and make more sense now in the context of why they are defined in the script. For some orchestration best practices, see Steve Bell's blog post here.
Once you have the activity completed, you can then create a simple workflow to leverage it. Below is a quick workflow where I use the custom activity as part of an event remediation workflow. A couple items that are unique to this type of workflow are the first and last activity (not including being and end). These "Set Values" activities exist because we want to use this orchestration workflow to be triggered as a result of an event management alert where a frontline technician can take the initial action to get the log file. When you trigger remediation from event management, a specific task type is created (em_remediation_task) and we do not want it to just be left open forever (which could be the case if we did not properly handle it as part of this workflow).
Workflow Using New Custom Activity
A final piece of the workflow puzzle is the setting of the variable. This is very important as we need to pass the new custom activity the proper inputs based on the alert we are trying to address (and its associate incident).
var instanceName = gs.getProperty('instance_name');
workflow.scratchpad.instanceName = instanceName;
// Find CMDB relationship from apache to actual server running apache to
// obtain IP address to connect to
var relgr = new GlideRecord('cmdb_rel_type');
relgr.addQuery('name', 'Runs on::Runs');
relgr.query();
if (relgr.next()) {
var gr = new GlideRecord('cmdb_rel_ci');
gr.addQuery('parent', current.cmdb_ci);
gr.addQuery('type', relgr.sys_id);
gr.query();
if(gr.next()){
var chgr = new GlideRecord('cmdb_ci_linux_server');
chgr.addQuery('sys_id', gr.child);
chgr.query();
if(chgr.next()) {
workflow.scratchpad.targetip = chgr.ip_address;
}
}
}
// Set incident sys_id to pass to get log activity
workflow.scratchpad.incID = current.alert.incident;
// Look for specific credential alias to get user and password
// Otherwise, explicitly define user and password values
var tagGR = new GlideRecord('sys_alias');
tagGR.addQuery('id', 'attachmentuser');
tagGR.query();
if (tagGR.next()){
var credgr = new GlideRecord('basic_auth_credentials');
credgr.addQuery('tag', tagGR.sys_id);
credgr.query();
if(credgr.next()) {
workflow.scratchpad.username = credgr.user_name;
// decrypt password
var encr = new GlideEncrypter();
var encrpwd = credgr.password;
workflow.scratchpad.password = encr.decrypt(encrpwd);
}
} else {
workflow.scratchpad.username = 'changeme';
workflow.scratchpad.password = 'changeme';
}Set Variables Step in Workflow
An important step to note within this code that sets variables is the section tries to find a credential entry based on a specific tag, "attachmentuser". I used this option so that an entry could be made into the standard credentials table with the password encrypted and all that you get with that baseline table. If for some reason you do not have access to create an entry, you can set the values in the script where it currently has changeme/changeme. If you choose to use a tag other than "attachmentuser", then just update this script to look for that tag value.
Entry in Credential Table (Basic Auth Credential)
At this point we are ready to start using the orchestration workflow (assuming you published it). Based on how we defined the workflow to be associated with an EM Remediation Task, we can use this it in a couple of places. As part of an alert action rule, or as part of a CI Remediation option, both of which are in the event management module. In both cases, be sure to specify a proper filter so that the action only applied to the correct CI type (an Apache web server in this case).
CI Remediation Definition (class is Apache Web Server)
That's it! You should now be able to leverage this remediation option as part of normal ServiceNow alert management. While I hope you could build this yourself as a result of this article, I have also created option to download it (with instructions) from the developer site where the share site now resides. To recap, we performed 3 high-level steps: 1) create a custom activity, 2) create a workflow to leverage the activity, and 3) create trigger options for use within the ServiceNow platform. Quick, effective incident resolution is key to driving better MTTR metrics, and leveraging either full closed loop remediation capabilities (like restart the Apache service) or intermediate options like this log retrieval can have a major influence on improving performance.
