---
sourceDocument: Brazil Enable AI
sourceDocumentLink: https://www.servicenow.com/docs/r/intelligent-experiences

 Release :

    - brazil

ft:locale :

    - en-US

ft:publication_title :

    - Brazil Enable AI

ft:clusterId :

    - platai

bundleId :

    - platai

workflow :

    - Platform


---

# Evaluation metrics and calculations

# Evaluation metrics and calculations {#ariaid-title1}

Release version: Brazil  
Updated September 10, 2026  
![](https://www.servicenow.com/docs/portal-asset/ico-clock) 3 minutes to read
Summarize  
![AI sparkle icon](https://servicenow.com/docs/portal-asset/ai-sparkle-icon) Summarized using AI  
This content was generated using new OpenAI-powered functionality. Results are provided on an as is basis and are not guaranteed to be accurate or complete.  

## Summary of Evaluation metrics and calculations

This content explains the evaluation metrics used to assess virtual agent conversations in ServiceNow, along with the methodology for calculating adjusted scores.
These metrics help measure the virtual agent's effectiveness in understanding and responding to user requests, which is critical for improving automated service interactions.
Show full answer Show less  

## Evaluation Metrics

ServiceNow evaluates virtual agent conversations using multiple metrics, each rated on a scale that is normalized to 5. Customers can filter trends based on these metrics within a selected date range. The key metrics include:

* **Request Completion:** Measures how well the virtual agent fulfills user requests by identifying intent and gathering required information.
* **Intent Accuracy:** Assesses the agent's ability to correctly understand user requests to provide relevant responses.
* **Slot Filling:** Evaluates how effectively the agent extracts structured answers from user responses.
* **Smooth Flowing Conversation (Deadlock avoidance):** Checks the agent's ability to dynamically progress the conversation without repetitive loops.
* **Context Retention:** Measures the agent's capacity to remember and utilize information throughout the conversation.
* **Truthfulness (Hallucination Prevention):** Ensures responses are genuine and based on conversation context, avoiding fabrications.
* **Conciseness (Redundancy Avoidance):** Verifies that responses are clear and not unnecessarily verbose.
* **Coherence:** Evaluates logical flow and organization of responses.
* **User Satisfaction:** A weighted average of all other metrics, representing overall conversation quality.

## Adjusted Score Calculations

To better align automated evaluation scores with human judgments, ServiceNow calculates deviations and adjusts scores accordingly:

* **Upper Deviation:** Calculated if more than 30 human-labeled scores are higher than auto-evaluated scores over the last 6 months. It averages the top 90% of positive differences.
* **Lower Deviation:** Calculated if more than 30 human-labeled scores are lower than auto-evaluated scores, averaging the top 90% of negative differences.
* **Adjusted Score:** Combines the error band derived from deviations with the auto-evaluation score to better reflect human evaluations. If deviations are unavailable, the adjusted score equals the auto-evaluation score.

## User Satisfaction Score Calculations

User satisfaction scores are computed at the evaluation level as follows:

* **Auto-eval User Satisfaction Score:** Weighted average of LLM-generated metric scores.
* **Human User Satisfaction Score:** Weighted average using human-labeled scores when available, otherwise auto-eval scores.
* **Gap:** Difference between human and auto-eval satisfaction scores.
* **Upper and Lower Deviations:** Calculated similarly to metric-level deviations when more than 30 records exist, applied to adjust auto-eval user satisfaction scores.
* **Adjusted User Satisfaction Score:** Reflects these adjustments to align with human judgments.

## Practical Implications for ServiceNow Customers

These evaluation metrics and adjustment calculations enable customers to:

* Monitor and analyze virtual agent performance comprehensively through multiple quality dimensions.
* Understand how automated evaluations compare to human assessments, ensuring trustworthiness of scores.
* Leverage adjusted scores to identify areas for improvement in virtual agent conversations.
* Use Performance Analytics indicators to track conversation quality trends over time, with clear guidance on interpreting batch job data.  
Metrics against which conversations are evaluated and calculation of adjusted scores.

## Metrics {#ai-eval-metrics__section_vdx_rxr_hgc}

The Select metric list shows all the metrics against which each conversation is evaluated for the selected date range. You can filter the evaluation trend based on each metric. The following metrics are available:{#ai-eval-metrics__table_oxb_rkb_yfc__entry__2}

| Metric | Description |
|-|-|
| Request Completion | Measures the virtual agent's ability to complete user requests by accurately identifying the user's intent and gathering all required information (slot filling). |
| Intent Accuracy | Shows the virtual agent's ability to comprehend user requests, resulting in relevant responses. |
| Slot Filling | Shows the virtual agent's capability to interpret user responses and extract structured answers to the required questions. |
| Smooth Flowing Conversation (Deadlock avoidance) | Checks if the virtual agent responds dynamically, successfully moving the conversation forward without repetition. |
| Context Retention | Shows if the virtual agent succeeds in retaining and using information provided during the conversation, including request interpretation and slot filling. |
| Truthfulness (Hallucination Prevention) | Shows if the virtual agent generated genuine responses grounded in conversation, excluding fabrication or memory and comprehension failures. |
| Conciseness (Redundancy Avoidance) | Checks the virtual agent's ability to avoid superfluous or verbose and generic responses, which doesn't contribute to the core intent of the conversation. |
| Coherence | Checks for clear logical flow, structure, and organization of the virtual agent's responses. |
| User Satisfaction | Weighted average of all the other metrics on which the conversation was evaluated. |
[ ]

{#ai-eval-metrics__table_oxb_rkb_yfc}  
Note:  
All the metrics are rated on a scale of 3 or 5, and are finally scaled up to 5.

## Calculations {#ai-eval-metrics__section_qqz_xdl_ngc}

Calculation of deviations and Adjusted Score:  
To align the auto-evaluation scores with human judgment over time, a deviation is calculated and used to produce an adjusted score on metric level.

* Upper DeviationCondition: If the number of human-labeled scores that are higher than the auto-evaluated scores in the last 6 months is more than 30.

  Calculation: The top 90% of these cases are taken and the difference
  (delta) between the human score and the auto-evaluated score is averaged. This delta is the Upper Deviation.
* Lower DeviationCondition: If the number of human-labeled scores that are lower than the auto-evaluated scores in the last 6 months is more than 30.

  Calculation: The top 90% of these cases are taken and the difference
  (delta) between the human score and the auto-evaluated score is averaged. This delta is the Lower Deviation.
* Adjusted Score  
  The final Adjusted Score is calculated based on the availability of the deviations.
  * If at least 30 distinct evaluations of both upper and lower deviations are labeled for a given metric, Error band is calculated as SUM(Avg labeling score -- LLM score)/Distinct evaluations. This error band is added to Auto-Eval score to get Adjusted Score.
  * If neither deviation is available, then Adjusted Score = Auto-Eval Score
  {#ai-eval-metrics__ul_gsx_msz_mgc}
{#ai-eval-metrics__ul_zxh_3wr_hgc}  
Calculation of Auto eval user satisfaction score, Human user satisfaction score, and Upper and Lower deviation on Evaluation level:

* Auto eval user satisfaction score: For a given evaluation, get all the scores for each metric that are LLM generated and calculate SUM(metric score \* metric weight)/SUM(metric weights).
* Human user satisfaction score: For a given evaluation, if at least one metric is labeled, it's considered to calculate the human user satisfaction score. If labeled, the labeling score is used, or else LLM score is used. Calculated as SUM(metric score \* metric weight)/SUM(metric weights).
* Gap: Gap is calculated as (Human user satisfaction score -- Auto eval satisfaction score).
* Upper Deviation: If the Gap is positive and there are more than 30 records, the error band is calculated at top 90% by SUM(Positive Gap) / Distinct evaluations. This error band is added to the Auto eval user satisfaction score.
* Lower Deviation: If the Gap is negative and there are more than 30 records, the error band is calculated at top 90% by SUM(Negative Gap) / Distinct evaluations. This error band is added to the Auto eval user satisfaction score.
* Adjusted user satisfaction score is calculated as SUM(Gap) / Distinct evaluations.
{#ai-eval-metrics__ul_otr_stz_mgc} Note:  
* The evaluator provides aggregated score per chat, even if there are multiple different requests made by user.
* Performance Analytics indicators are used to calculate the average score over time. If you run batch jobs on historical data, then by the definition of Performance Analytics indicators, these evaluations are counted on the evaluation date in aggregated scores and not counted for scores on the actual chat date.
{#ai-eval-metrics__ul_hs1_rsm_ngc}