Scalable Evaluation Framework for AI Search and Virtual Agent

Bisondamiano
Tera Contributor

‎09-14-2026 09:40 AM

I would like to store a list of ~100 prompts along with grounded truths, then repeatedly execute those prompts into AI Search / Virtual Agent to view and compare the synthesized responses in a lower environment. In between test rounds, I may update knowledge or other content sources. What is the best practice to achieve this?

 

I've looked into the following:

  1. Now Assist Data Kit / Skill Kit - great for maintaining the prompt dataset and ground truths to then run evaluations for skills
  2. Automated evaluations - great for running evaluations for AI agents
  3. Assistant Designer > Testing - great for manually testing individual prompts
  4. Analytics for AI Search and Assistant Designer - great for aggregate KPIs and surfacing glaring issues

But is there a supported method to evaluate AI Search / Virtual Agent at scale in a lower environment over multiple iterations?

0 Helpfuls
  • 514 Views
1 REPLY 1

sanjaygupta
Tera Guru

a week ago

I didn’t find a supported ServiceNow feature that runs about 100 saved prompts through AI Search or Virtual Agent, then scores the generated answers against ground truths across several rounds. The documented tools each cover part of the problem, so combine them and keep the answer review as a defined manual step.

Tool What it covers for your use case
AI Search Tuning evaluationCompares result relevancy for a set of search terms. It can compare two search applications, such as AI Search against Zing text search or two AI Search applications. Reviewers rate each result 2, 1 or 0.
NLU Workbench test sets and Multi-model Batch TestingBatch tests utterances against expected intents for the NLU models used in Virtual Agent or AI Search. This measures intent accuracy, not synthesized answers. Full test set features need the NLU Workbench - Advanced Features Store app.
Virtual Agent Designer automated testsCaptures a topic conversation and saves it as an Automated Test Framework (ATF) test, which you can run regularly. This suits fixed topic flows, not generated responses.
Now Assist Data Kit and Skill KitGood for holding your prompt dataset and ground truths, as you noted.

A workable process for each test round:

  • Keep the 100 prompts and ground truths in one place, such as a Data Kit dataset, with a version label for each round.
  • Install the AI Search Tuning app and create an evaluation using ais_admin. Load the same search terms each round so retrieval quality is comparable.
  • After you update knowledge or content sources, run a new evaluation and compare its search quality card with the previous one.
  • Run the same prompts through Assistant Designer testing and record the synthesized answer against each ground truth in a scoring sheet. Use a fixed rubric, such as correct, partially correct or incorrect, and a named reviewer.
  • Use NLU batch testing and ATF tests for the intent and topic flows, so a content change does not break them.
  • Keep the content changes between rounds in an update set or other tracked record, so you can link score changes to specific updates.

Treat any script that submits prompts and compares answers automatically as a custom build. The documentation does not describe a supported interface for this, so test it carefully in the lower environment before depending on it. Also expect some variation in generated answers between runs, so judge trends over several rounds rather than single results.

Sources:

0 Helpfuls