NLU model settings

  • Release version: Washingtondc
  • Updated February 1, 2024
  • 2 minutes to read
    • Summarize
    Summarized using AI
    This content was generated using new OpenAI-powered functionality. Results are provided on an as is basis and are not guaranteed to be accurate or complete.

    Summary of NLU model settings

    This guide provides instructions for adjusting the settings of your NLU model within ServiceNow's Natural Language Understanding feature. You can modify aspects such as the model's name, description, and confidence threshold to enhance its performance in predicting intents based on user utterances.

    Show full answer Show less

    Key Features

    • Model Configuration: Access and edit the model's name, short description, and business area through the Settings page. Note that language, purpose, and scope cannot be changed without creating a new model.
    • Punctuation Handling: The default setting ignores punctuation to reduce variance in intent prediction. It is recommended to keep this option enabled for optimal results.
    • Confidence Threshold Settings: Adjust the confidence threshold, which determines the minimum score required for an intent prediction. A well-set threshold balances the risk of false positives and missed intents.
    • Threshold Adjustment Options: Choose between automatic and manual threshold settings. Automatic settings dynamically optimize based on test results, while manual settings allow for user-defined thresholds.
    • Recommendations: The system may suggest improved thresholds during testing, provided certain criteria are met. Applying these recommendations can enhance prediction accuracy.

    Key Outcomes

    By effectively managing your NLU model settings, you can improve intent prediction accuracy, reduce false positives, and ensure your model aligns with business requirements. Regular adjustments based on testing can lead to more precise outcomes for user interactions, thereby enhancing the overall AI experience within your ServiceNow environment.

    Change your NLU model's name, description, or confidence threshold on the Settings page of the model overview.

    Access the model's settings by navigating to All > NLU Workbench > Models. Select the tab for your model's application, then your model's name. On the model's overview, select the Model settings tab. Model settings on the model's overview page

    Model settings

    In the upper section of the model settings page, you can change the model's name, short description, and business area. You cannot change the model's language, purpose, or scope. To make a model with a different language, purpose, or scope, see Creating models.

    By default, the Ignore punctuation check box is active. Ignoring punctuation makes it so that there is less variance between predicted intents and confidence scores for utterances with slightly different punctuation. For best results, keep the check box active.

    Model threshold settings

    Here you can adjust how the confidence threshold works in your model.

    A threshold is a confidence score represented by a percentage. The confidence threshold of a model determines what intents from that model will be predicted for a given utterance. For example, if the model threshold is 65%, then an intent will be predicted for an utterance only when the intent has a confidence score that is at least 65%. Setting a threshold that is too low may increase the false positives by predicting intents that should not be a match for an utterance. On the other hand, a model threshold that is too high may filter out intents that you do want to get predicted. Finding the ideal threshold improves your model's ability to predict intents correctly.

    There are two types of model threshold settings:
    • Automatic - Allow the system to choose the optimal confidence threshold for your model. The value is updated dynamically based on test results. This happens in the Test and publish your model phase, where your model's default test set is used.
    • Manual - You can manually set the confidence threshold. The system may also recommend a better threshold for the model during testing. You can choose to accept recommendations.

    Prebuilt models come with a tuned threshold. The confidence threshold on prebuilt models was chosen specifically for that model.

    Test results include a model threshold recommendation only if they meet the following requirements:
    • The test set has a Test Coverage score of at least 60%, with at least 5 test utterances per intent. For more information, see Test set creation and management.
    • The test set has at least 100 utterances.
    • The model is not a prebuilt model.
    • The recommended threshold would have better results than the current threshold.
    Test results page with the bar charts for the current and recommended thresholds.

    Test results with a recommended threshold contain a second graphic. The second graphic shows the prediction percentages with the recommended thresholds applied.

    Applying the threshold recommendation may improve the prediction percentages of your model. Select Apply recommendations to change the threshold. The system automatically retrains the model, and the test results show the prediction percentages with the new threshold.