---
sourceDocument: Australia Enable AI
sourceDocumentLink: https://www.servicenow.com/docs/r/australia/intelligent-experiences

 Release :

    - australia

ft:locale :

    - en-US

ft:publication_title :

    - Australia Enable AI

ft:clusterId :

    - platai

bundleId :

    - platai

workflow :

    - Platform


---

# Explore

# Exploring AI system monitoring in AI Control Tower {#ariaid-title1}

Release version: Australia  
Updated June 30, 2026  
![](https://www.servicenow.com/docs/portal-asset/ico-clock) 5 minutes to read
Summarize  
![AI sparkle icon](https://servicenow.com/docs/portal-asset/ai-sparkle-icon) Summarized using AI  
This content was generated using new OpenAI-powered functionality. Results are provided on an as is basis and are not guaranteed to be accurate or complete.  

## Summary of Exploring AI system monitoring in AI Control Tower

AI Control Tower enables ServiceNow customers to continuously monitor the quality and safety of AI systems---both ServiceNow AI and external AI---by automatically scoring live AI interactions.
This ongoing evaluation is crucial because AI models can degrade or change in production over time, causing potential issues that might otherwise go unnoticed until reported by users.
The system leverages an OpenTelemetry-based data model and large language model (LLM)-based judges to assess AI performance against configurable metrics, providing early detection of regressions and detailed insights for investigation.
Show full answer Show less  

## Key Features

* **Automated Scoring:** Samples of live AI sessions are scored automatically using LLMs fine-tuned for specific quality and safety criteria.
* **Configurable Metrics and Weights:** Organizations can tailor the evaluation by selecting which quality and safety metrics to include and assign weights to reflect priorities.
* **Unified Framework for ServiceNow and External AI:** Both internal and external AI systems are evaluated within the same framework, though data collection methods differ (automatic for ServiceNow AI, API-based for external AI).
* **Comprehensive Monitoring Pipeline:** Includes collection of trace data, scoring by evaluation engines, aggregation into composite scores, and visualization with drill-down capabilities to session, trace, and span levels.
* **Role-based Workflow:** AI stewards manage scoring activation, metric configuration, and system inclusion; AI asset owners analyze results and adjust AI agent configurations accordingly.
* **Trend Analysis and Root Cause Identification:** Scores and trends enable detection of gradual performance degradation. Detailed drill-down helps pinpoint specific causes within AI interactions.
* **Integration with AI Control Tower Governance:** Monitoring complements inventory management and policy governance by providing operational visibility and helping manage AI sprawl through automatic discovery of untracked AI assets.

## Key Outcomes for ServiceNow Customers

* **Proactive Quality and Safety Assurance:** Continuous, automated evaluation prevents unnoticed AI system degradation and safety risks.
* **Customizable Evaluation Aligned to Business Needs:** Flexible metric templates allow organizations to emphasize the most relevant quality and safety dimensions.
* **Efficient Investigation and Remediation:** Drill-down from portfolio-level scores to individual interaction details supports rapid root cause analysis and targeted improvements.
* **Holistic AI System Management:** Integration with AI Control Tower's inventory and governance functions ensures a comprehensive approach to AI system oversight.
* **Support for Both ServiceNow and External AI:** Enables consistent monitoring practices across diverse AI environments within the enterprise.  
Learn how AI Control Tower automatically scores live AI interactions against quality and safety metrics using an OpenTelemetry-based data model of sessions, traces, and spans.

## Monitoring overview {#mon-ai-exploring-aict-evaluations__cf-exploring-parent-overview}

An AI system that performs well in testing can drift in production. Models degrade, data sources change, and tool integrations break. Without ongoing evaluation, these problems go undetected until users report them.

Monitoring in AI Control Tower addresses this by automatically scoring a sample of live AI interactions against configurable quality and safety metrics. Scores are aggregated into weighted measures and surfaced alongside trend
data so you can detect regressions early and investigate before users are affected.

* Get continuous visibility into AI system quality and safety through automated large language model (LLM)-based scoring.
* Align scoring to your organization's priorities by selecting which metrics are evaluated and how much each one contributes to the overall score.
* Assess both ServiceNow and external AI systems through a single evaluation framework.
* Catch gradual regressions, not just one-off failures, by tracking score trends over time.
* Identify the exact cause of a quality or safety issue by drilling from an aggregate score down to a specific span.

{#mon-ai-exploring-aict-evaluations__ul_what_you_can_do}  
Important:  
Evaluation scores are generated by AI models. Review AI-generated scores alongside session-level detail before taking action on individual cases.

## Monitoring users {#mon-ai-exploring-aict-evaluations__cf-exploring-parent-users}

{#mon-ai-exploring-aict-evaluations__table_users__entry__2}

| User | Description |
|-|-|
| AI steward | Activates evaluation scoring, configures metric templates and weights, and manages which systems are included or excluded from scoring. Reviews quality and safety scores across the AI portfolio, investigates low-scoring sessions, identifies trends and regressions, and coordinates with AI system owners to improve agent performance. |
| AI asset owner | Works with AI stewards to understand evaluation results for their specific AI systems and adjusts agent configuration, prompts, or tool access to improve scores. |
[Table 1. Users]

{#mon-ai-exploring-aict-evaluations__table_users}

## How monitoring works {#mon-ai-exploring-aict-evaluations__cf-exploring-parent-how-it-works}

The monitoring pipeline has four stages:

1. **Collection.** For ServiceNow AI, trace data is collected automatically. For external AI, trace data is sent to ServiceNow via API using a generated API key.
2. **Scoring.** The evaluation engine applies your configured metrics to each sampled session. Metrics are scored by LLM judges, which are custom models fine-tuned for specific evaluation criteria. For ServiceNow AI, the AI Skill Kit Auto Eval engine scores sessions. For external AI, sessions are scored by ServiceNow evaluation models.
3. **Aggregation.** Per-metric scores are combined into composite quality and safety scores using the weighted average formula you define in your metric template.
4. **Visualization.** Scores, trends, and session-level detail are available on the overview and through drill-down into individual sessions, traces, and spans.

{#mon-ai-exploring-aict-evaluations__cf-exploring-how-it-works-ol}

The evaluation metrics you select determine which metrics are scored for each session. The metric templates you configure determine which of those scored metrics contribute to your quality and safety scores. Metrics that are scored
but not included in a template are still visible when you investigate individual sessions, traces, and spans.

## Monitoring workflow {#mon-ai-exploring-aict-evaluations__cf-exploring-parent-workflow}

This infographic shows a sample end-to-end workflow of how different users work together to set up monitoring, analyze AI system quality and safety, and investigate issues.
Figure 1. Monitoring AI systems in AI Control Tower  
In this workflow:

1. The system administrator installs AI Control Tower which includes the Evaluations plugin. The Evaluations plugin enables the evaluation framework, scoring engine, and monitoring components.
2. The AI steward activates evaluation scoring for ServiceNow AI systems. For external AI systems, the AI steward generates an API key and configures the external system to send trace data or configures trace connectors to send trace data from a supported cloud platform.
3. The AI steward and AI asset owner enable evaluations for individual assets by updating asset records in the AI Control Tower inventory.
4. The AI steward configures evaluation metric templates by selecting quality and safety metrics and assigning weights to create a scoring formula.
5. The evaluation scoring engine automatically scores a sample of live AI interactions. For ServiceNow AI, the AI Skill Kit Auto Eval engine scores sessions. For external AI, sessions are scored by ServiceNow evaluation models.
6. The AI steward and AI asset owner review aggregated quality and safety scores, identify underperforming AI systems, and track trends over time.
7. The AI steward and AI Asset owner investigate low-scoring sessions by drilling into session detail, reviewing per-metric scores and judge reasoning, and tracing root causes through individual traces and spans.
8. The AI steward coordinates with the AI system owner to adjust the agent's configuration, prompts, or tool access based on investigation findings.
9. The AI system owner makes adjustments to agents as necessary.
10. The AI steward and AI Asset owner verify improvements externally at build time and continue verification in AI Control Tower at run time.
{#mon-ai-exploring-aict-evaluations__cf-exploring-parent-workflow-ol}

## ServiceNow AI vs. external AI {#mon-ai-exploring-aict-evaluations__cf-exploring-sn-vs-external}

Both categories use the same evaluation framework, but they differ in data collection and available metrics.
{#mon-ai-exploring-aict-evaluations__table_sn_vs_external__entry__3}

|   | ServiceNow AI | External AI |
|-|-|-|
| Data collection | Automatic | Via API key (ai_observe.ai_data_sender role) or trace connector |
| Scoring engine | AI Skill Kit Auto Eval | ServiceNow evaluation models |
| Default quality metrics | Overall task completeness, Tool calling correctness | Task completion, Answer relevancy |
| Safety metrics | Not available with ServiceNow AI systems | Secrets detection (additional safety metrics available) |
[Table 2. ServiceNow AI vs. external AI comparison]

{#mon-ai-exploring-aict-evaluations__table_sn_vs_external}

## Where monitoring fits in AI Control Tower {#mon-ai-exploring-aict-evaluations__cf-exploring-where-fits}

AI Control Tower provides three complementary governance areas:

* Inventory manages what AI systems exist and their lifecycle state.
* Govern enforces policies, approvals, and compliance requirements.
* Insights provides ongoing operational visibility into how those systems actually perform.
{#mon-ai-exploring-aict-evaluations__ul_kfl_4rr_v3c}

Insights depends on inventory, since only managed assets are evaluated. The relationship also runs in reverse: connecting an AI system to send trace data can surface other AI systems and agents that aren't yet tracked, adding them to
inventory automatically and helping you manage AI sprawl. To learn more about connecting your AI systems for evaluation and populating inventory, see [Configuring trace connections](https://www.servicenow.com/docs/ryWU9mayy2W0VEuV6UVRnA "Collect trace data from AI agents running on supported cloud platforms and monitoring services to generate evaluation metrics, populate AI inventory, and support security monitoring in AI Control Tower.").

## Monitoring benefits {#mon-ai-exploring-aict-evaluations__cf-exploring-parent-benefits}

{#mon-ai-exploring-aict-evaluations__table_benefits__entry__3}

| Benefit | Feature | Users |
|-|-|-|
| Automatically score live AI interactions without manual review using LLM-based judges fine-tuned for evaluation criteria. | [How evaluation scoring works](https://www.servicenow.com/docs/lPJ171BGrbCTJBIWy_AzRA "Understand how AI Control Tower calculates quality and safety scores so you can interpret results accurately and configure scoring to reflect your priorities.") | AI steward |
| Define what quality and safety mean for your organization by selecting metrics and assigning weights to create a custom scoring formula. | [Configure an evaluation metric template](https://www.servicenow.com/docs/~6Wl1cgX0wLnQujKladjaw "Adjust the scoring formula for a metric template to change which metrics contribute to quality or safety scores, or to change how much weight each metric carries.") | AI steward |
| Track quality and safety trends over time to catch gradual regressions, not just one-off failures. | [Reviewing quality and safety scores](https://www.servicenow.com/docs/rTANry4rG0MmFLF0_q9vjw "Determine whether your AI systems are meeting quality and safety targets by reviewing scores and evaluation trends.") | AI steward, AI asset owner |
| Drill from portfolio-level scores down to a specific span in a trace to pinpoint exactly where an AI system's behavior broke down. | [Investigating low-scoring sessions](https://www.servicenow.com/docs/Z~3bRlp5GJO~5hvq52k8Pg "Narrow a quality or safety issue from a broad score to a specific cause by reviewing evaluated sessions and session details.") | AI steward, AI asset owner |
| Assess the quality and safety performance of a specific AI system by reviewing its scores, metric breakdowns, trends, and evaluated sessions. | [Monitoring an AI system](https://www.servicenow.com/docs/mvqiNll5_pMdoOE25_J42A "Assess the quality and safety performance of a specific AI system by reviewing its scores, metric breakdowns, trends, and evaluated sessions.") | AI steward, AI asset owner |
[Table 3. Monitoring benefits]

{#mon-ai-exploring-aict-evaluations__table_benefits}

## What to explore next {#mon-ai-exploring-aict-evaluations__cf-exploring-parent-links}

To learn more about configuring and monitoring in AI Control Tower, see:

* [Configuring monitoring and evaluations in AI Control Tower](https://www.servicenow.com/docs/KNYWMWf8qo9BqdWvC2Ff7g "Set up monitoring in AI Control Tower by activating evaluations, adding metrics, and configuring metric templates.")
* [Reviewing AI system quality and safety scores](https://www.servicenow.com/docs/rTANry4rG0MmFLF0_q9vjw "Determine whether your AI systems are meeting quality and safety targets by reviewing scores and evaluation trends.")
* [Investigating low-scoring sessions](https://www.servicenow.com/docs/Z~3bRlp5GJO~5hvq52k8Pg "Narrow a quality or safety issue from a broad score to a specific cause by reviewing evaluated sessions and session details.")
{#mon-ai-exploring-aict-evaluations__ul_explore_next}