---
sourceDocument: Australia Enable AI
sourceDocumentLink: https://www.servicenow.com/docs/r/australia/intelligent-experiences

 Release :

    - australia

ft:locale :

    - en-US

ft:publication_title :

    - Australia Enable AI

ft:clusterId :

    - platai

bundleId :

    - platai

workflow :

    - Platform


---

# Evaluation flow for batch evaluations

# Evaluation flow for batch evaluations {#ariaid-title1}

Release version: Australia  
Updated March 12, 2026  
![](https://www.servicenow.com/docs/portal-asset/ico-clock) 4 minutes to read
Summarize  
![AI sparkle icon](https://servicenow.com/docs/portal-asset/ai-sparkle-icon) Summarized using AI  
This content was generated using new OpenAI-powered functionality. Results are provided on an as is basis and are not guaranteed to be accurate or complete.  

## Summary of Evaluation flow for batch evaluations

The batch evaluation process allows Eval admins to assess up to 100 completed virtual agent conversations simultaneously using a predefined query.
This streamlines the evaluation process by creating evaluation records and applying the same logic used in single-conversation evaluations, ensuring consistency and efficiency at scale.
Show full answer Show less  

## Key Features

* **Triggering Mechanism:** Initiated when the evaluation state changes to In Progress with a designated conversation type.
* **Random Sampling:** Randomly selects up to 100 conversations based on the specified query, while excluding those already evaluated and HR-scoped interactions.
* **Transcript Construction:** Builds a detailed transcript from conversation messages, ensuring relevant knowledge articles and catalog items are appropriately included.
* **Evaluation Skills:** Asynchronously invokes a range of evaluation skills to assess conversation quality and performance.

## Key Outcomes

By implementing batch evaluations, ServiceNow customers can expect:

* Increased efficiency in processing multiple evaluations, saving time and resources.
* Improved data integrity through the avoidance of duplicate evaluations.
* Detailed insights into conversation quality through comprehensive scoring and rationale provided by the evaluation metrics.  
Batch evaluation enables Eval admins to evaluate up to 100 completed virtual agent conversations at once, based on a saved query.
Flow name: Execute Batch Evaluation.

The flow creates evaluation records and invokes Now Assist skills for each eligible conversation, mirroring the single-conversation evaluation logic, but at scale. It enforces HR scope exclusions, topic/category validation, transcript construction rules, early
live-agent exclusions, and asynchronous scoring through skills.

Batch evaluations are performed using the following logic:  
Trigger

* Table: Evaluation set \[sn_na_conv_eval_evaluation_set\]
* Condition: State changes to In Progress and Evaluation type = Conversation
{#ai-eval-flow-batch__ul_pbd_hrl_ngc}  
Inputs

* Evaluation Set record with:
  * Query filter: A query that targets conversations to be evaluated (for example, sys_cs_conversation filters).
  * Evaluation type: Conversation
  * State: In Progress (to start)
  {#ai-eval-flow-batch__ul_hvp_lrl_ngc}
* LLM/Skills: Chat Topic Classifier, plus the evaluation skills listed after this.
{#ai-eval-flow-batch__ul_qkj_krl_ngc}  
High-level behavior

* Reads the query filter and randomly samples up to 100 conversations.
* Skips already-evaluated conversations.
* Excludes HR-scoped interactions.
* Uses Chat Topic Classifier to validate evaluation eligibility and extracts Topic and Category.
* Builds a transcript with controlled inclusion of Knowledge articles and catalog sources, and applies early live agent exclusions.
* Creates an Evaluation record and asynchronously invokes all selected evaluation skills, writing scores and rationale to metrics.
{#ai-eval-flow-batch__ul_pfc_trl_ngc}

Sequence of execution:  
Action 1: If the query filter isn't empty

* Purpose: Guard clause.
* Logic: Look up the Evaluation Set record and check the query filter field.
* If the query filter is present: Proceed to Action 2.
* If empty: Stop and optionally log No query provided.
{#ai-eval-flow-batch__ul_b45_msl_ngc}  
Action 2: Randomize conversations

* Purpose: Select a bounded, random sample of conversations from the provided query.
* Logic:
  * Execute the query to get matching conversation records.
  * Randomly select up to 100 conversations.
    * If \>100 matches, cap at 100.
    * If \<100, select all.
    {#ai-eval-flow-batch__ul_f4j_xsl_ngc}
  * Validate the query; if invalid, return false and an empty or partial array.
  {#ai-eval-flow-batch__ul_lnd_wsl_ngc}
* Outputs:
  * success: true/false
  * conversation_ids: array of sys_ids (max 100)
  {#ai-eval-flow-batch__ul_vyz_ctl_ngc}
* If success = true: Proceed to Action 3; otherwise, stop and log the validation error.
{#ai-eval-flow-batch__ul_xwk_tsl_ngc}  
Action 3: Look up the evaluation table to check prior evaluation

* Purpose: Avoid duplicate evaluations.
* Logic: For each conversation sys_id, check sn_na_conv_eval_evaluation for existing records indicating that it's already evaluated or is in progress (implementation choice: state not in canceled/failed).
* If not previously evaluated: Proceed to Action 4 for that conversation.
* If already evaluated: Skip this conversation, optionally log Already evaluated.
{#ai-eval-flow-batch__ul_pls_j1m_ngc}  
Action 4: Look up the interaction record

* Purpose: Enforce HR scope exclusion.
* Logic: Resolve the interaction related to the conversation. If its application scope contains hr, skip the conversation.
* If the scope doesn't contain hr: Proceed to Action 5.
{#ai-eval-flow-batch__ul_zxy_41m_ngc}  
Action 5: buildTranscript

* Purpose: Construct the final, minute-level transcript and determine downstream skill set and guardrails.
* Steps:
  * Aggregate all conversation messages.
  * Tag user messages as \[User\]: and virtual agent messages as \[Virtual Agent\]:.
  * Knowledge articles:
    * If genius results reference Knowledge articles, query the Knowledge article and replace the genius snippet with the entire article body.
    * Annotate with \[Virtual Agent\]: Help articles for user query: and wrap content within Article_Start and Article_End.
    * Constraints:
      * If the KB is HR-scoped or inaccessible, don't evaluate (skip conversation).
      * Truncate the article body to a maximum of 10,000 words.
      * If the KB content source is attached files (PDF/Word/Txt), fall back to the genius result instead of full file content.
      {#ai-eval-flow-batch__ul_tyy_rbm_ngc}
    {#ai-eval-flow-batch__ul_n3q_nbm_ngc}
  * Catalog Items:
    * If genius results reference catalog items, query sc_cat_item and build a string: catalog name, short description, description.
    * Annotate with \[Virtual Agent\]: Please choose one of the below options: and include citation order.
    {#ai-eval-flow-batch__ul_nh4_zbm_ngc}
  * Live Agent Exclusions:
    * If the first user message requests a live agent, skip evaluation.
    * If a live agent is invoked within the first 120 words, skip evaluation.
    {#ai-eval-flow-batch__ul_yph_2cm_ngc}
  {#ai-eval-flow-batch__ul_s2y_3bm_ngc}
* Outputs:
  * ExecuteEvaluation: true/false (post-guardrail outcome)
  * Chat transcript
  * Knowledge articles referred
  * Catalog items referred
  * First live agent occurrence: Sys_id of the conversation message (if present)
  * Skills to invoke:
    * Coherence Chat Evaluation
    * Conciseness Chat Eval
    * Context Retention
    * Inadequate Slot Filling Chat Eval
    * Intent Accuracy Chat Eval
    * Smooth Flowing Conversation Chat Eval
    * Truthfulness Hallucination Chat Eval
    {#ai-eval-flow-batch__ul_wtx_lpm_ngc}
  * Additional logs
  {#ai-eval-flow-batch__ul_z33_ggm_ngc}
* If ExecuteEvaluation = true: Proceed to Action 7; otherwise, skip the conversation.
{#ai-eval-flow-batch__ul_tkd_fbm_ngc}  
Action 6: If Block

* Purpose: Branch to record creation.
* Logic: If ExecuteEvaluation from Action 6 is true, go to Action 8.
{#ai-eval-flow-batch__ul_vzs_spm_ngc}  
Action 7: Chat Classifier Eval

* Purpose: Validate whether the conversation should be evaluated and extract high-level labels.
* Logic:
  * Build a lightweight transcript from sys_cs_message for classification input.
  * Invoke Chat topic classifier skill with the transcript.
  * Receive:
    * Execute evaluation: true/false
    * Topic Name
    * Category: IT or HR
    {#ai-eval-flow-batch__ul_mnn_1bm_ngc}
  {#ai-eval-flow-batch__ul_in2_x1m_ngc}
* If Execute evaluation = true: Proceed to Action 6.
* If false: Skip conversation and log the classifier decision.
{#ai-eval-flow-batch__ul_vr1_v1m_ngc}  
Action 8: Create or Update evaluation record

* Purpose: Persist an evaluation entry for this conversation.
* Table: sn_na_conv_eval_evaluation
* Field population:
  * Document conversation: Conversation reference
  * State: processing
  * Topic: from Action 5
  * Category: from Action 5
  * KB Referred: from Action 6
  * Catalog Referred: from Action 6
  * First live agent occurrence: from Action 6
  * Type: chat summarization
  * User: initiating user for the conversation
  * Message log: Additional logs from Action 6
  {#ai-eval-flow-batch__ul_j4m_ypm_ngc}
* On success: Proceed to Action 9.
{#ai-eval-flow-batch__ul_s5y_vpm_ngc}  
Action 9: For Loop over skills

* Purpose: Execute each selected evaluation skill.
* For each skill in the list from Action 6:
  * Action 10: invokeApiDefinition
    * Inputs: Skill Name, Conversation, Transcript, Evaluation Id
    * Behavior:
      * Invoke the Now Assist skill asynchronously.
      * The post-processor writes results into sys_generative_ai_response_validator.
      * Extract JSON response fields:
        * Score
        * Reason for Score
        * Examples supporting the reasoning
        {#ai-eval-flow-batch__ul_rph_mrm_ngc}
      * Create child metric records in sn_na_conv_eval_evaluation_metrics linked to the parent evaluation.
      {#ai-eval-flow-batch__ul_ins_3rm_ngc}
    {#ai-eval-flow-batch__ul_bmm_hrm_ngc}
  * Action 11: WaitPause seven seconds before proceeding to the next skill to manage rate limits or throttling.

  {#ai-eval-flow-batch__ul_dgc_2rm_ngc}
{#ai-eval-flow-batch__ul_wmf_brm_ngc}