In this article I’ll cover the last of the Predictive Intelligence frameworks, regression. We’ll start with the primary use case and then dive into guidelines on how to tune your regression model.
What is Regression?
Regression is a supervised machine learning algorithm that is used to predict continuous values such as the dollar value of a house, the number of widgets produced in an assembly line, or the number of minutes it will take to receive a call back from a call center. For example, we can use regression to predict the value of a house using the follow attributes: number of rooms, quality of local schools, and age of house. The value of the house is called a dependent variable because the price is influenced by the independent variables such as the number of rooms, crime rate, etc.
In ServiceNow we apply regression to predict the Estimated Time to Resolve (ETTR) based on inputs such as short description and description. This allows the agent to set the expectation with the employee/customer on how long it will take to resolve the issue. In the below example I am using regression to predict the Max ETTR for onboarding a new employee.
(fig1 – HR Workspace ETTR)
Note – In this article I’ll be using an HR example, but regression can be applied to ITSM, SecOps, ITBM, GRC, and CSM use cases also. You're also not restricted to our OOTB models. For example, you could use regression to predict the amount of time to close a request for a catalog item. The data just needs to be in the platform and has to follow the regression data type guidelines.
How do I model regression using Predictive Intelligence?
Let’s start by reviewing an OOTB regression model for HR. If you’re using ITSM you can build you own regression model patterned after the HR model.
- Go to filter > Predictive Intelligence > Regression > Solution Definitions.
- Select the HR Case Resolution time solution to open the model definition. Notice the configuration has four steps.
(fig2 – Regression Definition)
Predictive Intelligence was designed to allow platform users to leverage the power of machine learning without the need of a data scientist. Defining a Predictive Intelligence model consists of the following four steps:
STEP 1 – Select the word corpus. The word corpus is used to convert any text into numerical vectors for use in our regression algorithm. For details about word corpus see this article.
STEP 2 – Select the table that has the value that you wish to predict. In this example we are using HR Case and predicting Actual Resolution time. If you were doing this for ITSM, you would select the incident table and business resolve time. In this example Actual Resolution time is the “dependent variable”, in that its value is influenced by other variables.
STEP 3 – Select the input data used to predict the Actual Resolution time. In this case we have several fields such as short description, description, assignment group, etc. These values are known as “independent variables” because they are used to predict the actual resolution time. If you are not sure which input fields to pick, you can use the correlation analysis script from this article.
STEP 4 – Define the historical data used to train the model. You will want to restrict this based on time and use it to exclude any outliers that may cause the model to fail (we’ll see this in the tuning section).
Once you select the desired input and output fields we will hit submit and train.
(fig3 – Submit & Train)
Evaluating your Regression Model
Once the model has completed training we can review the regression statistics to understand the performance of the model. To review the model performance, follow the below steps.
- Go to filter > Predictive Intelligence > Regression > Solutions.
- Click “true” on your trained solution to review the regression statistics.
(fig4 – select trained model)
3. Review SMAPE, MAE, Range Accuracy and Average Interval Width
Predictive Intelligence provides SMAPE and MAE loss functions to evaluate your regression model’s performance. SMAPE and MAE both measure the deviation of predicted values from the target/actual values used in the training set; a lower SMAPE and MAE score is generally better as we want to minimize the error between predictions and the target/actual values.
(fig5- MAE)
The regression algorithm computes the Range Accuracy and Average Interval Width; these metrics helps us to understand the upper and lower bounds of the prediction. Range accuracy shows the percentage of target/actual values that fall within the calculated bounds, the higher the range accuracy value the better. The average interval width shows the distance between the upper lower and lower bounds, the smaller the value the better.
(fig6-range)
To test the model click on the “test solution” tab and fill out the input fields. In my case my input fields are short description and description as seen below.
(fig7-test)
The prediction output provides a point estimate (what the model thinks the value is) and an interval with the percent probability (in this case 95%) that the true value lies within the lower and upper bounds.
The point estimate is predicting 9.57 days to handle this particular “time and attendance” case.
The confidence level of 95, means there is a 95% chance that the value is between 4.37 days and 14.77 days.
So how do we validate this prediction? Since the data is in the platform you can run a query where short description contains “time and attendance inquiry”and description contains “e-time” then see if the actual resolve times makes sense by plotting a bar chart of all the results.
For more details about the different regression model statistics please see our documentation.
Tuning your Regression Model
So armed with all the knowledge that we’ve covered so far you log into your instance and create a regression model, train it, and surprise you get an error! The most common one being “SE0035:Training terminated due to Exception. SE0075:Solution training failed as either the data used is not sufficient or the input field(s) is not predictive of the output field.”
(fig8-error)
To begin to address this error we need to understand our data. To do that we’ll go through a process data scientist call Exploratory Data Analysis (EDA). All that means is a process of visualizing our data. We’ll do this using a simple bar chart and boxplot report in ServiceNow.
If you recall the primary purpose of our regression model is to predict the Estimated Time to Resolve (ETTR) using the short description and description fields as input fields and the actual resolve time as the output field. Let’s start the EDA by asking four questions to see if the data is a good fit for regression.
Question 1 – Do I have mixed languages or special characters in my data?
Poor data quality can cause poor model performance. Often mixed language, HTML tags, and special characters create noise which drag down our model’s precision. To check the quality of your data, follow these steps:
- Go to filter > regression > solution definitions > (your) regression model.
- On step 4, click the filter link to review the training data set.
(fig9 – training data)
3. Follow the guidelines below to assess and correct your training data.
Confirm Language - When you click the training set link make sure that all you input and output fields are in the same language. If you have mixed languages eliminate those records or adjust your filter so the training data is all in one language.
Remove Noise from your data - Make sure that you don’t have any special characters or HTML tags in your input & output data. Also your data should be “non-empty”. You can write Java Script to remove undesired characters and you can store the cleansed data into a new column. Finally, review the Stop Words; if there are stop words that aren’t included in the default Stop Word list add them to reduce noise.
For more details on assessing data quality you can refer to the AI Academy session “AI Fundamentals” , this session focused on classification but the guidelines also apply to regression, similarity, and clustering.
Question 2 – What are my top HR Services?
We ask this question so we can see if we have class imbalance. A class imbalance occurs if the majority of cases fall into one or two HR Services. A class imbalance will throw the “not sufficient data” error when training our model. To see if I have a class imbalance you can create a bar chart grouping the cases by HR Service. Note - if your use case is ITSM you can select service or assignment group.
I can see that my cases are distributed among several HR Services and my top five HR Services are “Time and Attendance”, “Other Benefits Inquiry”, “Disability”, “Pay Inquiry”, and “Employee Record Update Request”. This looks good, one HR Service does not dominate the case load.
(fig10-class distribution)
If you did have class imbalance you will need to select a training set that normally distributes the cases among different HR services. If you can’t get a training set that normally distributes the cases/incidents you may need to create different predictive intelligence models and call them using flow designer or a business rule. For example, it’s common in ITSM to create a Predictive Intelligence model for different geographies and then call a model using IF..THEN..ELSE logic in a business rule or Flow Designer. For example, IF location = North America THEN use the North America PI model ELSE use the Asia model.
Question 3 – What is the distribution of the actual resolution time?
The actual resolution time is our dependent variable, its value is dependent on inputs such as short description and description. In our regression model we are trying to fit the “actual resolution times” of all the cases to a line. The line represents the ETTR. If the actual resolution times vary widely for all the different HR Services or there are outliers the model will fail to train. To understand the “actual resolution time” distribution you can create a box plot grouped by assignment group and use the actual resolution time as the measured value to generate a box plot that looks like the below in fig11.
(fig11-boxplot assignment groups)
The box plot breaks the data up into quartiles and shows the skew of the distribution, see fig12. For example, the interval between Q4 (Quartile 4) to Q3 (Quartile 3) represents 25% of the data. Everything below Q3 represents 75% of the data. Everything in the box represents 50% of the data. The blue dot is our mean and the line is our median. When I hover over the Payroll/Time & Attendance LATAM box plot I can see that 50% of the values fall between 1.05 days (lower quartile) and 11.24 days (upper quartile) with 25% of the data falling between 11.24 days (upper quartile) to 24.12 days (maximum).
(fig12-quartiles)
I can use the box plots in a couple ways. First, I can use this information as part of my testing process. In the Predictive Intelligence Test Solution tab I can test a case normally handled by “HR – Payroll/Time & Attendance – LATAM” (fig13).
(fig13- test LATM)
The point estimate is 9.66 days with a 95% confidence that the actual value falls between the lower bound of 2.94 days and upper bound of 16.38 days. When I look at my box plot 50% of the case data for this assignment group falls between 11.24 days and 1.05 days for resolution. I can conclude that this prediction looks reasonable.
The second way we can use the box plot is to define a regression model that focuses on actual resolution times that capture 75% of the data, which would be everything below Q3 (Quartile 3). In fig14 I’ve drawn some rough lines trying to capture the range of Q3, which is between 2 days and 15 days.
(fig14 – Q3)
I can use the actual resolution time range of 2 to 15 to filter my regression training set and increase the performance of my model, because 75% of the data fits in this range.
(fig15-regression filter)
Question 4 – Is there enough variability in my input data?
For regression to work the input data needs to have variety and not to be homogeneous. Homogeneous inputs can occur if you are using templates to populate the short description or description.
For example, if you have 100k cases where 50% of the short descriptions are “Disability Inquiry Urgent - <name>” then you have a homogeneous training set which will either fail training or not produce good results. If the majority of your data is of the homogeneous type you may be better off using an Assignment Rule and providing a calculated average of the ETTR.
My hope is that these guidelines will help you with your regression modeling in Predictive Intelligence. If you have any questions, feel free to post on the community so we can share our learnings together.
Best Regards, Lener
