Irrelevance detection in NLU

  • Release version: Xanadu
  • Updated August 1, 2024
  • 5 minutes to read
    • Summarize
    Summarized using AI
    This content was generated using new OpenAI-powered functionality. Results are provided on an as is basis and are not guaranteed to be accurate or complete.

    Summary of Irrelevance detection in NLU

    The Irrelevance detection feature in ServiceNow’s NLU improves model accuracy by training the model to recognize and avoid making predictions on user utterances that do not correspond to any intent. This keeps Virtual Agent chats focused and prevents irrelevant inputs from triggering incorrect intent matches.

    Show full answer Show less

    Utterances marked as Not relevant are included in model training so that when similar inputs are encountered, no intent is predicted. This feature is available exclusively for Virtual Agent models within the NLU Workbench.

    Access and Roles

    • Available for users with nluadmin, admin, or nlueditor roles (with editing rights limited to assigned models).
    • Access via All > NLU Workbench > Models under the Virtual Agent tab, then navigate to the "Keep chats focused" card to reach Irrelevance detection.

    Managing Irrelevant Utterances

    Irrelevant utterances can be added through multiple methods:

    • Virtual Agent chat logs: Use Expert Feedback Loop to mark utterances as Not relevant for one or all models.
    • Manual input: Directly type utterances into the Irrelevance detection interface.
    • Importing: Upload CSV or XLSX files with empty Intent columns to designate irrelevant utterances.

    All irrelevant utterances are managed in the Irrelevance detection table, showing their source and association.

    Behavior and Limits

    • Irrelevant utterances can be linked to a specific model or marked as irrelevant to all models.
    • Each model includes up to 200 irrelevant utterances for training; adding new ones removes the oldest if the limit is reached.
    • A model cannot have more irrelevant utterances than regular training utterances.

    Conflict Handling

    Irrelevant utterances take precedence over conflicting training utterances, causing the model to avoid predicting intents for these inputs. Conflicts appear in:

    • The Cross-model Conflict Review module (requires NLU Workbench - Advanced Features application).
    • The Conflicts tab within intents, where irrelevant utterances appear as a separate NOINTENT “intent.”

    Irrelevant utterances cannot be edited or deleted directly in conflict views; they must be modified via the Irrelevance detection interface.

    Best Practices and Additional Details

    • Include approximately 10% irrelevant utterances in test sets to evaluate model handling of non-intent inputs.
    • When importing data, leaving the Intent column empty designates utterances as irrelevant.
    • Irrelevant utterances can be reassigned to intents if new intents are created, turning them into normal training utterances. Remember to save feedback and retrain the model after reassignment.
    • Training the model is required to apply changes related to irrelevant utterances.
    • Irrelevant utterances should have distinct content from intent-associated utterances to avoid confusion.

    Keep Virtual Agent chats focused with Irrelevance detection. Use the Irrelevance detection feature to train your NLU model to avoid making predictions for utterances that are not relevant.

    Summary information

    The Irrelevance detection feature improves the prediction accuracy of NLU models by training them to ignore certain utterances. These utterances from your users may not apply to any intent, so should not get any prediction.

    To ensure that models do not predict an intent when they are not supposed to, you can mark utterances as Not relevant. These marked utterances are included as part of model training. When the published model encounters similar utterances from your users, no intent is matched or predicted.

    The Irrelevance detection table is where you can manage utterances marked as Not relevant.

    Roles, Usage, and Navigation

    Use the nlu_admin or admin role to access Irrelevance Detection. The nlu_editor role is also able to access Irrelevance Detection, but must be assigned to a model to edit that model's contents.

    Irrelevance Detection is available for Virtual Agent models only.

    1. Navigate to All > NLU Workbench > Models. The Virtual Agent tab opens by default.
    2. Scroll down the list of Virtual Agent models to the Boost your model performance section.
    3. Scroll horizontally to locate the card Keep chats focused, and select its button Go to irrelevance detection.In NLU Workbench, on the Virtual Agent tab, the card Keep chats focused is highlighted.
    Note:
    The instance URL for this feature is <instance-name>.servicenow.com/now/nlu-workbench/irrelevant-utterances.

    Adding utterances to Irrelevance detection

    There are several methods for adding utterances to Irrelevance detection:

    • The Virtual Agent chat log: In the Expert Feedback Loop in NLU Workbench, when you review an utterance collected from the Virtual Agent chat log, you can mark it as Not relevant. The system will ask whether it should be irrelevant to a particular model, or to all models. After adding, these utterances may display as NO_INTENT in Expert Feedback Loop.

      For more information on Expert Feedback Loop in NLU, see NLU Expert Feedback Loop.

      These utterances have a Source of VA Chat Logs in the Irrelevance detection table.

    • Manual input: In Irrelevance detection, type your utterance in the Type utterances here field, then select Add. The system will ask whether it should be irrelevant to a particular model, or to all models.

      These utterances have a Source of Manual in the Irrelevance detection table.

    • Importing: When you use a CSV or XLSX (Excel Workbook) file to import training utterances and their intents, you can indicate irrelevant utterances by leaving the Intent column empty. These utterances may display as NO_INTENT in areas such as Expert Feedback Loop and Conflict Review.

      For more information on importing utterances and intents, see Create an NLU model from a CSV file.

      These utterances have a Source of Manual in the Irrelevance detection table.

    Behavior of irrelevant utterances

    There are two types of irrelevant utterances: those associated to one specific model, or those irrelevant to any model. A model can have a maximum of 200 irrelevant utterances associated to it.

    Following are details of how these two types and the 200-count maximum interact.

    When a model is submitted for training, at most 200 irrelevant utterances are submitted with it. First, irrelevant utterances directly associated to that model are submitted. Then, utterances which are designated as not relevant to any model are submitted. The total of these types does not exceed 200.

    If a model has 200 irrelevant utterances associated to it, and a new irrelevant utterance is added, then the model's oldest utterance is dropped. The new irrelevant utterance could be of either type (not relevant to the specific model, or not relevant to any model).

    A model cannot have more irrelevant utterances than normal training utterances.

    Conflict review

    If an utterance is marked as irrelevant, and there is a similar utterance in an intent, the model does not make a prediction for that utterance. In other words, irrelevant utterances take precedence over training utterances.

    Because irrelevant utterances impact the model's predictions, they are displayed as conflicts when they overlap with training utterances.
    There are two locations where conflicts with irrelevant utterances are highlighted:
    • The Cross-model Conflict Review module. For more information about this module, see Cross-model Conflict Review.
      Note:
      The Cross-model Conflict Review module is available with the NLU Workbench - Advanced Features application from ServiceNow® Store. For more information, see NLU Workbench - Advanced Features.
    • The Conflicts tab of an intent. For more information on conflicts in an intent, see Resolve intent issues.
    For the purpose of reviewing conflicts, irrelevant utterances are displayed as though they are in their own intent, named NO_INTENT.

    Note that irrelevant utterances cannot be edited or deleted on the conflict page. Copy the irrelevant utterance from the conflict page to the Irrelevance detection page to modify or delete the utterance.

    More information

    • In test sets, aim to include about 10% of test utterances as irrelevant. This helps to assess how your model handles utterances that should not have an intent predicted. When you import training or test utterances from a CSV or XLSX (Excel Workbook) file, you can indicate that an utterance is irrelevant by leaving the Intent column empty.
    • When testing models against test sets, results are considered Correct if no intent is predicted for an irrelevant utterance.
    • Utterances that were marked as Not relevant can be re-assigned later. For example, if a new intent is created, existing irrelevant utterances can be manually assigned to the new intent. They then become part of the normal training utterances.

      To re-assign an utterance in Irrelevance detection, expand the list in the Corrected intent column and select the appropriate intent. Be sure to select the Save feedback button after re-assignment. Also be sure to retrain the model to incorporate these updates into the model.

    • Irrelevant utterances are not associated to particular intents within a model. They are either associated to one model, or marked as not relevant to any model.
    • The utterances which are marked as not relevant to any model are submitted as part of the training data for each model. In other words, these apply to all models.
    • Model training is necessary to incorporate Not relevantutterances. Training any model adds newly marked utterances to all models.
    • Utterances can be deleted or edited in the Irrelevance detection table.
    • Irrelevant utterances should have content that is different from utterances associated to an intent.