--- sourceDocument: Australia Enable AI sourceDocumentLink: https://www.servicenow.com/docs/r/australia/intelligent-experiences Release : - australia ft:locale : - en-US ft:publication_title : - Australia Enable AI ft:clusterId : - platai bundleId : - platai workflow : - Platform --- # Conversational evaluator # Conversational evaluator {#ariaid-title1} Release version: Australia Updated September 23, 2026 ![](https://www.servicenow.com/docs/portal-asset/ico-clock) 2 minutes to read Summarize ![AI sparkle icon](https://servicenow.com/docs/portal-asset/ai-sparkle-icon) Summarized using AI This content was generated using new OpenAI-powered functionality. Results are provided on an as is basis and are not guaranteed to be accurate or complete. ## Summary of Conversational evaluator The Conversational Evaluator in AI Control Tower provides ServiceNow customers with a measurable way to assess the quality and business impact of Virtual Agent conversations. It tracks Virtual Agent performance over time by combining quality metrics with value-based measurements like estimated time savings. This enables customers to connect conversation quality directly to operational efficiency and cost savings. The tool is accessible from the Evaluation tab within AI Control Tower. Show full answer Show less ## Key features * **Auto-evaluation framework:** Automatically scores sampled Virtual Agent conversations against defined quality metrics and generates an overall user satisfaction score. * **Customizable quality benchmarks:** Allows rating conversations to define your own quality standards. * **Exclusion rules:** Automatically excludes unsuitable conversations from evaluation (e.g., those transferring immediately to live agents or very short conversations). These rules can be extended to other criteria such as geographic location. * **Comprehensive evaluation metrics:** Scores on request completion, intent accuracy, slot filling, truthfulness, context retention, conciseness, smoothness (deadlock avoidance), and coherence. * **Value metrics:** Estimates business impact by calculating total evaluations, average scores, evaluation trends, conversation efficiency, time savings, user acceptance, and estimated cost savings based on hours saved converted into currency value. * **Conversation grouping:** Groups conversations into small, medium, and large sizes by word count to analyze efficiency and savings by conversation size or content type (knowledge, catalog, etc.). ## Practical benefits for ServiceNow customers * Monitor and improve Virtual Agent performance using quantifiable quality metrics and benchmarks tailored to your environment. * Identify how Virtual Agent interactions save live agent time and reduce operational costs through detailed value metrics. * Exclude irrelevant conversation types from evaluation to focus on meaningful data and insights. * Leverage insights to optimize Virtual Agent scripts and content for better customer experience and business outcomes. * Validate AI-generated content carefully, as generative AI may produce incomplete or inaccurate information. Conversational evaluator measures the quality of Virtual Agent conversations in AI Control Tower. ## Conversational evaluator overview {#mv-conversational-evaluator__section_overview} Conversational Evaluator gives you a measurable view of conversation quality so you can track Virtual Agent performance over time. A conversation that saves time for a live agent is treated as a successful conversation. Conversational Evaluator combines quality scores with value metrics, such as estimated time savings, so you can connect conversation quality to business impact. You access Conversational Evaluator from the Evaluation tab in AI Control Tower. Note: Generative AI may produce inaccurate or incomplete information. Always validate AI-generated content. ## Benefits {#mv-conversational-evaluator__section_erf_5xh_tkc} Conversational evaluator provides the following benefits: * Track Virtual Agent performance across a defined set of quality metrics. * Define your own quality benchmark by rating conversations. * Estimate the hours and cost that Virtual Agent conversations save. {#mv-conversational-evaluator__ul_dkw_2fb_tkc} ## How it works {#mv-conversational-evaluator__section_how_it_works} Conversational evaluator uses an auto-evaluation framework to score Virtual Agent conversations. It works as follows: 1. The system identifies qualified conversations by excluding conversation types that aren't suitable for evaluation. 2. The system selects a random sample of qualified conversations for auto-evaluation. 3. The auto-evaluation framework scores each sampled conversation against a set of quality metrics and rolls the metric scores up into a user satisfaction score. 4. The system calculates value metrics, such as estimated time savings and estimated Virtual Agent efficiency, based on the evaluated conversations. {#mv-conversational-evaluator__ol_ekw_2fb_tkc} ## Conversations excluded from evaluation {#mv-conversational-evaluator__section_exclusions} The following conversation types are excluded from auto-evaluation by default: * Conversations that use an inaccessible or empty knowledge base * Conversations that transfer immediately to a live agent * Short conversations * Conversations started by custom triggers {#mv-conversational-evaluator__ul_fkw_2fb_tkc} A script include controls these exclusion rules. You can extend the script include to exclude other types of conversations, such as conversations from a specific geographic location. ## Evaluation metrics {#mv-conversational-evaluator__section_metrics} The auto-evaluation framework scores conversations on the following metrics, which roll up into the User Satisfaction score: * Request completion * Intent accuracy * Slot filling * Truthfulness * Context retention * Conciseness * Smoothness --- deadlock avoidance * Coherence {#mv-conversational-evaluator__ul_gkw_2fb_tkc} ## Value metrics {#mv-conversational-evaluator__section_value_metrics} The Value view shows metrics that estimate the business impact of Virtual Agent conversations, including Total evaluations, Average auto-evaluation score, and Evaluation score trend. Conversations are grouped into small, medium, and large sizes based on the number of words in each conversation. For example, a conversation of about 200 words or fewer is a small conversation. You can view conversation efficiency and time savings by conversation size or by content type, such as knowledge and catalog. The value metrics also include a user acceptance score and an estimated cost saving calculation that converts hours saved into a currency value. **Related tasks** * [Review conversation evaluation scores](https://www.servicenow.com/docs/hke449v7Pu7qyy90soTtQQ "Review auto-evaluation scores for Virtual Agent conversations in AI Control Tower to track conversation quality over time.") * [Calculate Virtual Agent cost savings](https://www.servicenow.com/docs/utxRpdzXPGamkQov0~Fpug "Estimate cost savings from Virtual Agent conversations by entering an hourly rate in the Value view of Conversational Evaluator.") **Related topics** *