Issue Auto Resolution tuning options
Summarize
Summary of Issue Auto Resolution tuning options
The Issue Auto Resolution (IAR) tuning options in the NLU Workbench enable ServiceNow customers to adjust their model's behavior to better align with their business priorities. Customers can tune the model for precision, automation, or a balance of both, influencing how confidently the model predicts resolutions for incidents. This tuning helps achieve the right trade-off between accuracy and the number of incidents automatically resolved.
Show less
Key Features
- Tuning Goals: In the Analyze step of IAR tuning, users can select from three tuning objectives:
- Precision: Model predicts only when confidence is high, resulting in fewer errors but fewer resolved incidents. This is the default and recommended setting for ITSM models.
- Automation: Model predicts at a lower confidence threshold, resolving more incidents but with potentially higher error rates.
- Balance: Strikes a compromise between precision and automation, balancing error rates with resolution coverage.
- Match Rate: Represents how often the model correctly predicts the intent relative to total predictions, averaged across all intents except NOINTENT.
- IAR Coverage: Indicates the percentage of incidents resolved based on predictions exceeding the confidence threshold, acknowledging some errors may occur.
- Interactive Comparison: Customers can select different tuning options to compare projected Match rate and IAR coverage results, enabling informed decision-making.
- Intent-to-Topic Mapping: After tuning, users can map more intents to Virtual Agent topics via the IAR Admin Console to enhance automation.
Using the Tuning Options
To tune the Issue Auto Resolution model, users with the nluadmin role navigate to All > NLU Workbench > Models, select the Issue Auto Resolution tab, and choose the desired model. After providing feedback, users proceed to the Analyze step where they can adjust tuning goals and observe projected outcomes.
Once an optimal tuning goal is selected, users save the choice and publish the tuned model to apply the changes.
Benefits for ServiceNow Customers
- Customizable Automation: Tailor the model to match specific operational priorities, such as minimizing errors or maximizing automated incident resolution.
- Data-Driven Decisions: Use projected metrics to understand the impact of tuning changes before deployment, reducing risk.
- Scalable Incident Management: Increase efficiency by automating incident resolutions confidently aligned with business tolerance for errors.
- Enhanced Virtual Agent Integration: Map intents to Virtual Agent topics post-tuning for improved automation workflows.
When you are tuning your Issue Auto Resolution model in NLU Workbench, you can adjust the output for several goals: precision, automation, or a balance of the two. Compare how your choice of tuning options affect match rate and coverage, before committing.
Summary usage
By default, Issue Auto Resolution tuning in NLU Workbench optimizes for precision. Depending on your business requirements, you can also tune the model for other objectives. In the Analyze step of Issue Auto Resolution tuning, the tuning goals list enables you to adjust for Precision, Automation, or Balance. As you select one of these options, the projected Match rate and IAR coverage percentages change accordingly, so you can compare possible outcomes.
- Navigate to .
- Select the Issue Auto Resolution tab, then select the model name. The tuning experience opens to step 1 (Feedback) initially.
- Provide feedback, then select the Analyze button. Step 2 (Analyze) opens.
- In the section Here are your tuning options and projected results, using the list You can tune for precision, automation, or balance, select options to see projected scenarios. You can also select the link Learn about tuning goals to open the following window.
Precision
When tuned for precision, the IAR model makes predictions only when its confidence is relatively high. This results in lower error rates, but also in fewer incidents resolved.
Precision is the recommended tuning option for the IAR ITSM model, so this option is selected by default.
Automation
When tuned for automation, the IAR model makes predictions at a lower confidence threshold. This results in more predictions, so more incidents are resolved. However, higher error rates are possible.
Balance
When tuned for balance, the IAR model attempts to strike a balance between precision and automation.
Match rate
The match rate is defined as the number of Incidents where the intent was predicted correctly, divided by the number of predictions for that intent. This ratio is averaged across all intents except for NO_INTENT.
IAR Coverage
Coverage is defined as the percentage of Incidents that would be resolved because the model was able to make predictions above its confidence threshold. The predictions may contain some errors.
Using tuning options
Select several different tuning options to compare projected results. Depending on the option you select, the system presents scenarios for projected Match rates and IAR coverage rates. Also the system displays how much these rates change according to your selection.
Review further information in the Here's a detailed breakdown section of Analyze. Here you can drill down into results that are specific for each intent in the model.
Note that the intents are grouped into mapped and unmapped intents, depending on whether they have been mapped to Virtual Agent topics. After providing feedback in IAR Tuning, you may wish to activate some intent-to-topic mappings. To do so, expand See unmapped intents, then select the Map more intents button. This opens the IAR Admin Console.
When you have decided the optimum tuning option for your requirements, select the Save choice button in the Learn about tuning goals window. Then, select the Tune and publish model button to advance to the next step.