Cross-model Conflict Review
Summarize
Summary of Cross-model Conflict Review
Cross-model Conflict Review helps ServiceNow customers identify and resolve conflicting intents within or across Natural Language Understanding (NLU) models. Such conflicts arise when intents have overlapping or nearly identical utterances, including cases where utterances marked as "Not relevant" closely resemble those assigned to intents. Addressing these conflicts before model publication improves NLU model accuracy and performance.
Show less
Key Features
- Conflict Detection: Runs analysis on one or two NLU models to detect overlapping intents and utterances, including conflicts across different models or applications.
- Conflict Severity: Classifies conflicts as Critical or Moderate, enabling prioritization starting with critical issues.
- Resolution Actions: Allows users to ignore conflicts, delete duplicate utterances, or edit utterances to clarify intent distinctions.
- Review Management: Maintains a list of conflict reviews for ongoing monitoring and resolution, displaying details such as model versions, training dates, and conflict counts.
Using Cross-model Conflict Review
- Installation: Available via the NLU Workbench - Advanced Features app on the ServiceNow Store; requires activation of the corresponding plugin.
- Roles: Access requires nluadmin or admin roles; nlueditors assigned to models can resolve conflicts within those models.
- Running Analysis: Users select one or two models for conflict analysis, then run the process to generate detailed conflict reports.
- Resolving Conflicts: Review overlapping utterances and intents, decide on appropriate actions such as deleting conflicting utterances, then retrain the model to apply changes.
Practical Steps for Conflict Resolution
- Navigate to the Cross-model Conflict Review interface in the NLU Workbench.
- Run conflict analysis on selected models to identify overlaps.
- Examine conflict details, including intent names and shared utterances, to determine the best resolution.
- Delete or edit conflicting utterances to clarify intent distinctions.
- Retrain the model to incorporate changes and improve prediction accuracy.
- Mark conflicts as reviewed or ignored once addressed to maintain accurate conflict tracking.
Benefits for ServiceNow Customers
By leveraging Cross-model Conflict Review, customers can proactively detect and correct intent overlap issues that degrade NLU accuracy. This results in more precise intent classification, better user experience, and smoother deployment of NLU models across applications.
Identify conflicting intents within or across models so you can take corrective actions, resolve such conflicts, and improve your NLU model performance.
Summary usage
As the number of intents within a model increases, two intents may overlap in scope. This may occur when training utterances in one intent are almost identical to utterances in another intent. There may also be conflicting intents across models and even applications.
Utterances may also be marked as Not relevant, meaning that no intent should be predicted. When these irrelevant utterances are too similar to utterances assigned to an intent, they are displayed in Conflict Review. For more information, see Irrelevance detection in NLU.
To address and fix these issues, Cross-model Conflict Review runs an analysis on your models. Use the analysis to identify and resolve these issues prior to model publication and deployment.
Installation
Cross-model Conflict Review is part of the NLU Workbench - Advanced Features app available on the ServiceNow® Store.
To use Cross-model Conflict Review, ensure that the NLU Workbench - Advanced Features (com.snc.nlu.workbench.advanced) plugin is active on your instance. For more information, see Install NLU Workbench - Advanced Features and Activate the NLU Workbench.
Roles
To access Cross-model Conflict Review, use the nlu_admin or admin role. When assigned to a model, the nlu_editor can resolve conflicts in that model.
Running the analysis
The Conflict Review screen shows a list view of all conflict reviews created in your instance. When a review is completed, it's added to a running list of reviews. In this example scenario, you're creating the first review in your instance, so when it's completed, it's shown in the count as 1 of 1 reviews. As more conflicts arise over time, you will see multiple reviews in the list.
Conflict reviews are analyzed on either one or two NLU models. When you run an analysis on a single model, the system shows intents and utterances that are only in that model. When you run an analysis on 2 models, the system shows intents and utterances that are in both models.
Conflict reviews always run on the last trained version of the model(s) they analyze.
Conflict reviews have two types: Critical and Moderate. The standard approach is to begin with the critical ones.
- Ignore the conflict
- Delete an identical or nearly identical utterance from one of the intents
- Edit the utterances to make them more distinct from each other
In this example scenario, you're resolving a conflict where two different intents contain the exact same utterance.
1. Navigate to .
2. Select Run analysis.
3. In the Model(s) field on the Choose one or two models to analyze for conflicts screen, select two NLU models for the analysis. In this example scenario, you choose the demo_hardware_issue and demo_it_request models.
4. Select Run Analysis.
The Conflict Review screen refreshes to show the analysis, including the two models you selected for analysis, the counts of Critical and Moderate conflicts under review, the number of reviews that have been completed, and the run date for the analysis. If you point to the far right column on the screen you will see options to rerun the analysis, or delete it and start all over.
5. In the Model(s) column, select your two paired models so you can drill down into the review.
The screen refreshes to show the details of the conflict review. Note the summary of the two models you chose for the analysis, their latest training dates, the types of conflicts they hold, and the version time stamp of the analysis. Note also the 0 of 1 count, which indicates that this is the first conflict review created in this instance. As the system detects more conflicts over time, and those conflicts are reviewed, the count increases.
If you were to determine the utterances in the image below are fine as they are, then you should click Ignore. Clicking Ignore tells the system you've completed your review, so it marks it as reviewed and moves on to the next conflict review. However, in this scenario, you don't ignore the conflict, because intents that share the same utterance are a conflict worthy of review.
To determine how you will resolve this conflict, consider the 2 intent names and the identical utterances they share. Consider which intent is more likely to use the laptop is really slow utterance. If you compare the 2 intent names closely, you might realize that a laptop that's really slow isn't the same as a laptop that doesn't work. However, a laptop that's really slow is indeed a laptop issue. So in this example scenario, you decide to dig deeper into the intents to scan the context of their other utterances.
Note also that when an intent uses unique utterances, it helps the system to more accurately predict which utterances belong to it. Hence, you will need to edit or delete the utterance from 1 of the 2 intents. In this example scenario, you decide to dig into the #laptop_not_working intent.
6. Select laptop_not_working.
The #laptop_not_working Intent screen appears, showing its current 3 utterances. You make the decision to delete the laptop is really slow utterance from the #laptop_not_working intent.
7. Click the Delete this utterance trash can icon.
The Confirm Delete screen appears.
8. Select Delete.
The Confirm Delete screen disappears, and the Utterances count drops from 3 to 2 because you've deleted the laptop is really slow utterance from the intent.
9. Select Train.
A banner appears on the Intent screen, confirming the model is successfully trained.
10. Select Conflict Review in the navigator.
Result:
The Conflict Review list screen appears, showing your conflict review analysis is complete, and that it's been reviewed.