Test set creation and management

  • Release version: Zurich
  • Updated July 31, 2025
  • 4 minutes to read
    • Summarize
    Summarized using AI
    This content was generated using new OpenAI-powered functionality. Results are provided on an as is basis and are not guaranteed to be accurate or complete.

    Summary of Test set creation and management

    ServiceNow customers using NLU models for Virtual Agent or AI Search can leverage a default test set created automatically with each model. This test set is essential for evaluating and improving the model’s performance and accuracy over time. The test set starts empty and must be populated with utterances and expected intents to provide meaningful testing results.

    Show full answer Show less

    Accessing and Managing the Test Set

    • Access the default test set via the NLU Workbench by navigating to the Models section and selecting your specific model and its Test set tab or Test Coverage tile.
    • Populate the test set by adding utterances manually through the interface or by importing them from CSV files or other models.
    • The Expert Feedback Loop feature allows importing actual user utterances from Virtual Agent chat logs, enhancing the test set’s relevance.

    Test Coverage and Quality

    For effective model evaluation, ensure your test set covers at least 60% of the model’s enabled intents with a minimum of five test utterances per intent. This level of coverage helps produce accurate performance metrics and optimizes the confidence threshold during batch testing.

    It is recommended that about 10% of test utterances be marked as “not relevant” to assess how the model handles irrelevant inputs without predicting intents.

    Using and Managing Test Sets

    • Test sets are used in the Test and Publish phase and support multi-model batch testing.
    • Default test sets are automatically created during instance upgrades for existing models and duplicated when models are copied.
    • Test utterances must be distinct from training data and match the model’s language.
    • Default test sets cannot be deleted independently of their models.

    Downloading and Moving Test Sets

    • Test sets can be downloaded as CSV files, containing utterances and expected intents but not their sources.
    • They can be moved between instances using update sets, which include test utterances, expected intents, and sources when the NLU model is added.
    • Note that exporting an NLU model as CSV does not include the default test set.

    Use the default test set of your NLU model to test the model's performance and accuracy. Manage your test set over time by building or updating its content in the NLU Workbench.

    Note:
    To test your model, install the ServiceNow® Store application NLU Workbench - Advanced Features. For more information, see Install NLU Workbench - Advanced Features.
    When you create an NLU model for Virtual Agent or AI Search, a default test set is created and associated to the model. You can use the default test set to evaluate the model’s performance. Initially the test set is empty, ready to be populated with your content.

    Access your default test set

    Access your default test set with one of the following methods.
    • Navigate to All > NLU Workbench > Models. Select the tab for your model's application, then the name of your model from the list. On the model's overview page, find the Build and Train your model card and select its View phase button. Then select the Test set tab. On the NLU model overview page, the Test set tab is highlighted.
    • Navigate to All > NLU Workbench > Models. Select the tab for your model's application, then the name of your model from the list. On the model's overview page, select the Test Coverage tile. On the model overview page, the Test Coverage tile is highlighted.
    • Navigate to All > Multi-model Batch Testing > Test sets tab. Find the name of your model. Default test sets are labeled as Default.An entry in Multi-model Batch Testing's Test sets table.

    Add content to your default test set

    Add utterances and their expected intents to build and manage your test set over time. You can add content to the default test set with the following methods:

    • Add test utterances and their expected intents manually. From the model's overview page navigate to Build and train your model > Test set tab. Type your input into the Type a test utterance here field, select an appropriate intent, then select the Add button.

      These test utterances are assigned a source of Manual.

    • Import test utterances and their expected intents from a CSV file or from other models. To import content to a default test set, from the model's overview page navigate to Build and train your model > Test set tab. Select Import test utterances.

      Imported test utterances are assigned a source of Manual.

    • The Expert Feedback Loop feature lets you add actual user utterances from Virtual Agent chat logs to the test set.

      These test utterances are assigned a source of Expert Feedback. For more information, see NLU Expert Feedback Loop.

    Test Coverage

    The Test Coverage score is the percentage of a model's enabled intents that have test utterances in the default test set. Before testing your model, ensure that there is at least 60% coverage. The higher the Test Coverage score, the more accurate the performance testing results.

    Your test coverage needs to be at least 60%, with at least 5 test utterances per intent, in order for the system to provide an optimal confidence threshold during batch testing. For more information about the confidence threshold, see NLU model settings.

    Aim to have about 10 percent of a model's test utterances marked as "not relevant", meaning that there is no intent associated. This helps assess how the model handles irrelevant utterances which should not have any intent predicted. For more information about irrelevant utterances, see Irrelevance detection in NLU.

    Use the test set

    To use the default test set from the Test and publish your model phase, see Test and publish your model.

    To use the test set in Multi-model Batch Testing, see Multi-model Batch Testing.

    Characteristics of default test sets

    When an instance is upgraded, default test sets are created for any existing models that don't already have them.

    When you copy a model using Duplicate this model, the original's default test set is copied into the new model. For more information, see Duplicate an NLU model.

    The utterances in the test set shouldn't be the same as the utterances in the training set.

    Default test sets can't be deleted separately from their models.

    Test set utterances should be in the same language as their model.

    Test sets are available for Virtual Agent or AI Search models.

    Downloading or moving default test sets

    Default test sets can be downloaded or moved as follows.

    • Default test sets can be separately downloaded in CSV format. To download the test set, from the model's overview page navigate to Build and train your model > Test set tab. Select Download test set.
      Note:
      Test sets that are downloaded from Download test set contain test utterances and their expected intents, but not the sources.
    • Default test sets can be moved with update sets. When you add an NLU model to an update set, its default test set is added, including test utterances, expected intents, and sources. For more information, see Add an NLU model to an update set.
    • When using the Export model as CSV function in the All existing models table, the default test set is not included. For more information, see Export an NLU model.