---
sourceDocument: Australia Enable AI
sourceDocumentLink: https://www.servicenow.com/docs/r/intelligent-experiences

 Release :

    - australia

ft:locale :

    - en-US

ft:publication_title :

    - Australia Enable AI

ft:clusterId :

    - platai

bundleId :

    - platai

workflow :

    - Platform


---

# Evaluation metrics and calculations

# Evaluation metrics and calculations {#ariaid-title1}

Release version: Australia  
Updated March 12, 2026  
![](https://www.servicenow.com/docs/portal-asset/ico-clock) 3 minutes to read
Summarize  
![AI sparkle icon](https://servicenow.com/docs/portal-asset/ai-sparkle-icon) Summarized using AI  
This content was generated using new OpenAI-powered functionality. Results are provided on an as is basis and are not guaranteed to be accurate or complete.  

## Summary of Evaluation Metrics and Calculations

This document details the evaluation metrics and calculations used to assess virtual agent conversations within ServiceNow.
It focuses on how metrics are defined, their significance, and the process of calculating adjusted scores to align automated evaluations with human judgments over time.
Show full answer Show less  

## Key Features

* **Metrics:** The following key metrics evaluate conversation performance:
  * **Request Completion:** Measures the agent's ability to fulfill user requests accurately.
  * **Intent Accuracy:** Assesses how well the agent understands user requests.
  * **Slot Filling:** Evaluates the agent's ability to extract necessary information from user responses.
  * **Smooth Flowing Conversation:** Ensures the conversation progresses without repetitive responses.
  * **Context Retention:** Measures the agent's ability to use information from the conversation.
  * **Truthfulness:** Checks for genuine responses, avoiding inaccuracies or hallucinations.
  * **Conciseness:** Evaluates the agent's effectiveness in providing clear and relevant responses.
  * **Coherence:** Assesses the logical structure and clarity of responses.
  * **User Satisfaction:** A weighted average of all metrics to gauge overall satisfaction.
* **Calculations:** Two types of deviations (Upper and Lower) are calculated to adjust scores based on human evaluations, ensuring alignment with user expectations.
* **Auto-evaluation Scores:** The user satisfaction score is derived from both automated and human evaluations, allowing for a comprehensive understanding of performance.

## Key Outcomes

By applying these metrics and calculations, ServiceNow customers can effectively evaluate the performance of their virtual agents, ensuring they meet user needs and enhance overall satisfaction. The adjusted scores provide a reliable measure that reflects both automated and manual assessments, allowing for continuous improvement in service delivery.  
Metrics against which conversations are evaluated and calculation of adjusted scores.

## Metrics {#ai-eval-metrics__section_vdx_rxr_hgc}

The Select metric list shows all the metrics against which each conversation is evaluated for the selected date range. You can filter the evaluation trend based on each metric. The following metrics are available:{#ai-eval-metrics__table_oxb_rkb_yfc__entry__2}

| Metric | Description |
|-|-|
| Request Completion | Measures the virtual agent's ability to complete user requests by accurately identifying the user's intent and gathering all required information (slot filling). |
| Intent Accuracy | Shows the virtual agent's ability to comprehend user requests, resulting in relevant responses. |
| Slot Filling | Shows the virtual agent's capability to interpret user responses and extract structured answers to the required questions. |
| Smooth Flowing Conversation (Deadlock avoidance) | Checks if the virtual agent responds dynamically, successfully moving the conversation forward without repetition. |
| Context Retention | Shows if the virtual agent succeeds in retaining and using information provided during the conversation, including request interpretation and slot filling. |
| Truthfulness (Hallucination Prevention) | Shows if the virtual agent generated genuine responses grounded in conversation, excluding fabrication or memory and comprehension failures. |
| Conciseness (Redundancy Avoidance) | Checks the virtual agent's ability to avoid superfluous or verbose and generic responses, which doesn't contribute to the core intent of the conversation. |
| Coherence | Checks for clear logical flow, structure, and organization of the virtual agent's responses. |
| User Satisfaction | Weighted average of all the other metrics on which the conversation was evaluated. |
[ ]

{#ai-eval-metrics__table_oxb_rkb_yfc}  
Note:  
All the metrics are rated on a scale of 3 or 5, and are finally scaled up to 5.

## Calculations {#ai-eval-metrics__section_qqz_xdl_ngc}

Calculation of deviations and Adjusted Score:  
To align the auto-evaluation scores with human judgment over time, a deviation is calculated and used to produce an adjusted score on metric level.

* Upper DeviationCondition: If the number of human-labeled scores that are higher than the auto-evaluated scores in the last 6 months is more than 30.

  Calculation: The top 90% of these cases are taken and the difference
  (delta) between the human score and the auto-evaluated score is averaged. This delta is the Upper Deviation.
* Lower DeviationCondition: If the number of human-labeled scores that are lower than the auto-evaluated scores in the last 6 months is more than 30.

  Calculation: The top 90% of these cases are taken and the difference
  (delta) between the human score and the auto-evaluated score is averaged. This delta is the Lower Deviation.
* Adjusted Score  
  The final Adjusted Score is calculated based on the availability of the deviations.
  * If at least 30 distinct evaluations of both upper and lower deviations are labeled for a given metric, Error band is calculated as SUM(Avg labeling score -- LLM score)/Distinct evaluations. This error band is added to Auto-Eval score to get Adjusted Score.
  * If neither deviation is available, then Adjusted Score = Auto-Eval Score
  {#ai-eval-metrics__ul_gsx_msz_mgc}
{#ai-eval-metrics__ul_zxh_3wr_hgc}  
Calculation of Auto eval user satisfaction score, Human user satisfaction score, and Upper and Lower deviation on Evaluation level:

* Auto eval user satisfaction score: For a given evaluation, get all the scores for each metric that are LLM generated and calculate SUM(metric score \* metric weight)/SUM(metric weights).
* Human user satisfaction score: For a given evaluation, if at least one metric is labeled, it's considered to calculate the human user satisfaction score. If labeled, the labeling score is used, or else LLM score is used. Calculated as SUM(metric score \* metric weight)/SUM(metric weights).
* Gap: Gap is calculated as (Human user satisfaction score -- Auto eval satisfaction score).
* Upper Deviation: If the Gap is positive and there are more than 30 records, the error band is calculated at top 90% by SUM(Positive Gap) / Distinct evaluations. This error band is added to the Auto eval user satisfaction score.
* Lower Deviation: If the Gap is negative and there are more than 30 records, the error band is calculated at top 90% by SUM(Negative Gap) / Distinct evaluations. This error band is added to the Auto eval user satisfaction score.
* Adjusted user satisfaction score is calculated as SUM(Gap) / Distinct evaluations.
{#ai-eval-metrics__ul_otr_stz_mgc} Note:  
* The evaluator provides aggregated score per chat, even if there are multiple different requests made by user.
* Performance Analytics indicators are used to calculate the average score over time. If you run batch jobs on historical data, then by the definition of Performance Analytics indicators, these evaluations are counted on the evaluation date in aggregated scores and not counted for scores on the actual chat date.
{#ai-eval-metrics__ul_hs1_rsm_ngc}