on - edited on by
ChrisSteinke2
You're sold on using Predictive Intelligence to route work more efficiently, help drive faster resolution, and recommend major incidents before they become a serious problem. AWESOME! Then, you run your first classification, similarity, or cluster solution definition and the results aren’t what you expected. What do you do? Part 1 of this article will break down some of the most common tuning patterns we see in the field for classification. Part 2 will provide similar tuning recommendations for similarity and clustering.
Classification is used to auto-populate fields on case/incident forms to assist in intelligent routing to the most efficient assignment group. Just to be clear, machine learning will never be 100% right (aka precise) all the time. The goal is for the machine learning model to be right more than their human counterparts. Using real world data, classification models with a precision higher than 70% and coverage higher than 80% are typically more effective than agents looking at thousands of incidents/cases per month.
Figure1 represents a good classification model. The blue circles represent our assignments groups, the larger the circle the greater the distribution. What's distribution? Let's look at the blue bubble "Deskside Support". This assignment group represents 22% of the incidents from the training set. What makes this a good model is that all the blue bubbles which represents assignments groups are in the upper far right quadrant. Which means they have good precision and good coverage. Note: the x-axis changes to “recall” in Orlando. (This is a big picture you may need to scroll right.)
In Figure2 we notice that there are a couple of outliers that are not in the far upper right quadrant. Those are assignment groups that could be excluded from the model as they have a very little distribution and are most likely rare cases where a human is required to determine when an incident goes to that assignment group. Our Predictive Intelligence documentation lists out the steps on how to exclude classes from the model.
Below Figure3 represents a model that needs a lot of work. You’ll notice many of the assignment group bubbles with large distributions are scattered all over the classification graph, with several in the low precision and low coverage sections.
So how do you move all those assignment group bubbles to move to the upper right, like our ideal model in Figure1?
Starting in Orlando, we have the ability to set the target precision and coverage for a solution definition. For most customer, setting a target solution level precision should be the first step when they start using predictive intelligence. By setting the target level precision, predictive intelligence will automatically change recall and coverage values to get all classes close to the target precision of 90% (see Figure4).
What if target metrics aren’t helping? I’ve listed a few approaches that can help you move the model to a more ideal state below.
Techniques to improve the classification model precision & coverage:
- Understand the quality of the data. If there are lots of assignment groups that mean the same thing with different spellings or if there are empty/null assignment groups this will impact the precision of model. You’ll want to either exclude those from the training set or clean up your data.
- “More is not always better”. Sometimes training using 200k records vs a smaller selection of 80k records from the past 3 months may drive down precision & coverage. If your precision is low try training with a smaller data set first (i.e. 30k records), increase the number of records to see if that increases the precision. In some cases, 200k records may reduce precision because there is a chance for more bad data in that training set.
- Make sure there is good distribution of the assignment groups. If 90% of the incidents are skewed toward “Deskside Support” then you don’t need a machine learning model to tell you where new incidents will land. Also, you don’t want to use inputs with lost of unique values. For example configuration item, which may have millions or more records may make it harder for the machine learning algorithm to predict if you us CI as an input field.
- Look at individual class level settings and see if a higher precision can be selected for that class.
- If you started with the OOTB classification solution definition and it’s giving a low precision/coverage then try analyzing the incident/case data to determine what other inputs may help the ML predict a more precise result. Running lists/reports with these different input combinations should shed light on whether additional inputs such as category, subcategory, or location might help improve the precision/coverage of the model.
- Simplify your model. Because predictive intelligence is easy to configure many customers will add lots of inputs to the solution definition without understanding that “more is not better”. If you have lots of inputs and your model has low precision/coverage, then strip out all the inputs and start again with just short description. Update and retrain and then add input #2 and so forth until you get the desired precision.
- You might not be able to predict assignment group with just one model. So, plan to use multiple models. One model may not be able to predict all the assignment groups. You can call different solutions definitions in multiple passes. In Figure3, I used four classification models to predict the assignment group.
- Finally use Performance Analytics(PA) with Predictive Intelligence. PA has helpful OOTB indicators such as re-assignment count that can help identify areas where you want to focus Predictive Intelligence.
If you’d like to learn more about how to interpret and tune similarity and clustering models make sure to check back on the Analytics, Intelligence, and Reporting Community.
Thanks for reading. Stay health and safe!
thanks CM_X. Your comment reminded me of a quick tip for those who are already working with Similarity. One thing that I find that is not obvious is "setting and saving the similarity threshold". Go into application navigator > predictive intelligence > similarity > solutions > select your trained solution. If you test your similarity solution and it says "below threshold" try changing and saving the similarity threshold.
Hi Venkat,
What's the training error for your solutions? You can access it by app navigator > predictive intelligence > classification > solutions. Click the solution that failed, and scroll to the bottom and look for training progress. Please share a screen shot of your error.
Also when you say you have 744 unique records and you bumped it up to 185k, how did you do that? Did you duplicate the data over and over? If so if there is little variability your models will fail.
Finally are you on a Personal Developer instance (PDI) or sub prod? You may have some flexibility with your PDI on minimums.
Best, Lener
Hello
I am working on an assignment group classification solution and still have not gotten the results I want. Our data quality is causing quite a challenge, and I am having a rough time coming up with the right conditions to eliminate records that are skewing my results, but I am still working on that.
One thing I see is that your screenshots in this post don't match my Solution Visualization and I'm sure it has to do with the release. We are up to the Quebec release and here is a screenshot of mine. Instead of charting Precision and Coverage, in Quebec it charts Precision and Recall. So I figure my goal should be just as you say, except I would want my results to be in the upper left quadrant. Wouldn't this be correct?
You see the giant bubble in the upper left corner? That represents a bunch of assignment groups that have 100% precision and zero coverage. According to documentation, these are classes we should never include in class prediction.
- What does the 114 represent in the slider (114 out of 229)? I thought it may already be eliminating those 100/0's but it's not.
- When I change any of the class confidence records individually, this bubble chart does not update. Is that how it's supposed to be?
- When I set a specific target metric (on Precision for example), again, the bubble chart does not update but the class confidence records do update. Is that also how it's supposed to be?
- I haven't tried applying Gradient Boost yet. Can I expect that to change the bubble chart visualization so I can see the impact?
Basically I am having a hard time seeing the impact of any changes I make because I don't think this bubble chart updates, and I just want to be sure I'm not missing a step.
I haven't tried PredictabilityEstimate yet but I may try that next. In my rough analysis of my data, I don't think another input field will help my results, but it's worth trying this to find out. I'm basically trying to find ways to eliminate some of that data showing up as 100/0 with the condition builder in my solution definition, and then to even out the coverage.
Do you have any other advice that would help me?
Thanks again for all the time you have put into your articles and posts. I greatly appreciate it all!
Hi Lisa, I also enjoyed your K20 PA presentation. You're correct I wrote many of the PI tuning articles in the Paris release and am working in updating all of them to Rome. The solution visualization changed in Quebec and has recall on the X axis and precision on the Y axis.
Having your model performance with classes in the upper quadrant is still a good rule of thumb, however it's important to ask if you need high recall? If you adjust for higher recall your precision will go down. In general, you want high recall when you want the sensitivity of the model to predict true positives to be higher. For example, this may be important if you want your PI model to make sure all potential Denial of Service incidents are going to the SecOps team. Here is a great article on Recall vs Precision.
The solution visualization should update when you change your precision & recall values. I just tested on my Quebec instance, and confirmed that the bubbles do move. The only thing I can think of is you might of missed hitting the OK button when you changed the values (see fig1).
fig 1 (make sure to hit apply)
Going back to your specific questions.
- What does the 114 represent in the slider (114 out of 229)? I thought it may already be eliminating those 100/0's but it's not.
[Lener] the 114 represents the classes (pulled from ALL classes even the 100 0's) to be displayed
- When I change any of the class confidence records individually, this bubble chart does not update. Is that how it's supposed to be?
[Lener] Make sure to hit OK in fig1.
- When I set a specific target metric (on Precision for example), again, the bubble chart does not update but the class confidence records do update. Is that also how it's supposed to be?
[Lener] As I mentioned when I tested my Quebec instance my solution visualization did update. My patch level is below. If it still doesn't update for you please open a ticket and add me to the watch list (lener.pacania@servicenow.com) and I'll take a closer look with the support engineer.
Build name: Quebec
Build date: 09-07-2021_2219
Build tag: glide-quebec-12-09-2020__patch8-09-02-2021
- I haven't tried applying Gradient Boost yet. Can I expect that to change the bubble chart visualization so I can see the impact?
I've tested XGboost with a few Fortune 50 customers and have gotten really great results. The default classification algorithm is LogR. If the precision is still low I would definitely try XGBoost and compare. The solution visualization should update.
In general other classification advice:
1. Make sure the field you are predicting has a normal distribution, that is it's not all skewed to a couple assignment groups. You can see this by plotting a bar chart on the assignment group of your training set.
2. Make sure you do not have empty, nulls, weird characters in your input or output fields as that will cause the model not perform.
3. The Word Corpus is auto-generated for classification. However, I have seen improvements by creating a word corpus with filters that match you're model. You may wish to change to TF-IDF and experiment to see if that improves precision/recall.
4. Finally, in addition to support you have an entire field team that supports you. Please reach out to your sales team and they will pull a specialist architect in (I believe yours would be Chris Steinke) to be a sounding board for some of these issues.
HTH -Lener
A few customers have asked me for the older pre-Paris business rules to implement predictive intelligence, so I am including here for reference. Figure 1 has the Business Rule definition. Figure 2, is the code snipped that shows you what you should paste into the advanced tab. I am also attaching the script.
Note - as of San Diego, the business rule to call your predictive intelligence solutions is "Default Task Based Prediction". It calls all active PI solutions and looks for events against the task table.
(figure 1 - Business rule definition)
/*************************************************************************
*** Call Solution Definition ***
*** Older BR to call Predictive Intelligence **
*************************************************************************/
(function executeRule(current, previous /*null when async*/) {
var solutionNames = ["ml_incident_categorization", "ml_x_snc_global_global_incident_assignment_br", "ml_incident_categorization", "ml_x_snc_global_global_incident_priority"];
var predictor = new MLPredictor();
var info = "";
solutionNames.forEach(function(solutionName) {
var solution = predictor.findActiveSolution(solutionName);
if (!solution)
return;
/* The next line of code is doing the prediction and updating current record. */
/* If no prediction is done, we don't build prediction info message. */
if (!predictor.applyPredictionForSolution(current, solution))
return;
/* If user doesn't have 'itil' role, we don't build prediction info message. */
if (!gs.hasRole('itil'))
return;
/* Building prediction info message */
var fieldName = solution.getPredictedField();
var fieldLabel = current.getElement(fieldName).getED().getLabel();
var predictedDisplayValue = current.getDisplayValue(fieldName);
var msg = gs.getMessage("Predicted {0} for {1}.", [predictedDisplayValue, fieldLabel]);
if (info.length > 0)
info += " ";
info += msg;
});
/* Print out prediction info message on screen. */
if (info.length > 0) {
var incidentUrl = "<a href='"+current.getLink()+"'>"+current.number+":</a>";
gs.addInfoMessage(incidentUrl + " " + info);
}
})(current, previous);(figure 2 - older business rule code)
San Diego Update - Many of you may display the prediction in a blue box using the above code in your UI Action on your form or in a business rule for Agent Workspace. As of San Diego we now automatically show the predictions for several of the out of the box classification models in the Service Operations Workspace and for the HR Agent Workspace (see below).
Service Operations Workspace
HR Agent Workspace
Pls share the script for getting models displayed in blue boxes as shown in the screen shot.
Script Include is hr_CaseAjax that displays the message, which I highlighted in the screenshot below.
Regards,
Brian
