ServiceNow Virtual Agent powered by Now Assist can significantly enhance self-service experiences, so it is equally important to evaluate the quality of those interactions to ensure users receive accurate, effective, and reliable support. Organizations typically use several methods to assess Virtual Agent performance.
- Traditional Reviews: Manual conversation reviews, surveys, and spot checks that provide insights but are time-consuming and difficult to scale.
- Conversational Insights: ServiceNow Conversation insights application provides inferred CSAT and metrics such as resolution, empathy, and effort to measure user satisfaction.
- Now Assist Evaluation Skills: Automates conversation quality assessment, provides reasoning behind scores, and supports continuous improvement of Virtual Agent performance.
Now Assist Evaluation Skills and Conversation Insights application together provides a complete picture of virtual agent effectiveness, from system performance to end-user satisfaction. While Now Assist evaluation skills help in measuring Virtual Agent design and model performance whereas Conversation insights app focuses on measuring customer satisfaction and effort. This article focuses on Now Assist Evaluation skills activation, configuration and usage details.
Now Assist Evaluation Skills Activation and details
- Intent Accuracy – Verifies whether the Virtual Agent correctly understood the user's request.
- Slot Filling – Evaluates how accurately required information is captured.
- Conversation Flow – Identifies repetitive loops or conversations that fail to progress.
- Context Retention – Measures the agent's ability to remember earlier user inputs.
- Coherence – Ensures responses are logical and well-structured.
- Truthfulness & Hallucination Detection – Validates that responses are grounded in available knowledge and context.
- Conciseness – Checks whether responses are clear and avoid unnecessary detail.
- Topic Classification – Categorizes conversations for reporting and trend analysis.
Steps involved in Conversations Evaluation
Not every Virtual Agent conversation is evaluated. By default, ServiceNow evaluates a 10% sample of daily conversations, with a limit of 200 conversations per day. This limit can be increased/decreased through a system property if needed (sn_na_conv_eval.maxEvaluateCount). Since each evaluation skill consumes 1 Assist and all eight skills may be executed for a conversation, sampling helps optimize Assist consumption while still providing meaningful insights.
Additionally, some conversations are excluded from evaluation, including Conversations involving restricted Knowledge Base articles, HR-related conversations and Very short conversations.
A transcript is generated from each eligible conversation and sent to the LLM for evaluation. Once processing is complete, the results are stored in the Evaluation and Metrics tables for analysis and reporting.
Evaluation Methods
We can choose one of the following evaluation approaches:
- Scheduled Evaluation: A daily scheduled job evaluates conversations completed during the day. Job name: Execute Evaluations
- Real-Time Evaluation: A Flow triggers the evaluation immediately after a conversation ends.
- Historical Evaluation: A one-time batch process can be used to evaluate past conversations up to 100 conversations by adding a conversation records query (sys_cs_conversation) to the "sn_na_conv_eval_evaluation_set" table.
Note: Scheduled and real-time evaluation methods are mutually exclusive. Only one approach should be used at a time.
Where to view the results of Evaluation
