Exploring AI system monitoring in AI Control Tower

Learn how AI Control Tower automatically scores live AI interactions against quality and safety metrics using an OpenTelemetry-based data model of sessions, traces, and spans.

Monitoring overview

An AI system that performs well in testing can drift in production. Models degrade, data sources change, and tool integrations break. Without ongoing evaluation, these problems go undetected until users report them.

Monitoring in AI Control Tower addresses this by automatically scoring a sample of live AI interactions against configurable quality and safety metrics. Scores are aggregated into weighted measures and surfaced alongside trend data so you can detect regressions early and investigate before users are affected.

• Get continuous visibility into AI system quality and safety through automated large language model (LLM)-based scoring.
• Align scoring to your organization's priorities by selecting which metrics are evaluated and how much each one contributes to the overall score.
• Assess both ServiceNow and external AI systems through a single evaluation framework.
• Catch gradual regressions, not just one-off failures, by tracking score trends over time.
• Identify the exact cause of a quality or safety issue by drilling from an aggregate score down to a specific span.

Important:

Evaluation scores are generated by AI models. Review AI-generated scores alongside session-level detail before taking action on individual cases.

Monitoring users

Table 1. Users

User	Description
AI steward	Activates evaluation scoring, configures metric templates and weights, and manages which systems are included or excluded from scoring. Reviews quality and safety scores across the AI portfolio, investigates low-scoring sessions, identifies trends and regressions, and coordinates with AI system owners to improve agent performance.
AI asset owner	Works with AI stewards to understand evaluation results for their specific AI systems and adjusts agent configuration, prompts, or tool access to improve scores.

How monitoring works

The monitoring pipeline has four stages:

1. Collection. For ServiceNow AI, trace data is collected automatically. For external AI, trace data is sent to ServiceNow via API using a generated API key.
2. Scoring. The evaluation engine applies your configured metrics to each sampled session. Metrics are scored by LLM judges, which are custom models fine-tuned for specific evaluation criteria. For ServiceNow AI, the AI Skill Kit Auto Eval engine scores sessions. For external AI, sessions are scored by ServiceNow evaluation models.
3. Aggregation. Per-metric scores are combined into composite quality and safety scores using the weighted average formula you define in your metric template.
4. Visualization. Scores, trends, and session-level detail are available on the overview and through drill-down into individual sessions, traces, and spans.

The evaluation metrics you select determine which metrics are scored for each session. The metric templates you configure determine which of those scored metrics contribute to your quality and safety scores. Metrics that are scored but not included in a template are still visible when you investigate individual sessions, traces, and spans.

Monitoring workflow

This infographic shows a sample end-to-end workflow of how different users work together to set up monitoring, analyze AI system quality and safety, and investigate issues.

Figure 1. Monitoring AI systems in AI Control Tower

In this workflow:

1. The system administrator installs AI Control Tower which includes the Evaluations plugin. The Evaluations plugin enables the evaluation framework, scoring engine, and monitoring components.
2. The AI steward activates evaluation scoring for ServiceNow AI systems. For external AI systems, the AI steward generates an API key and configures the external system to send trace data or configures trace connectors to send trace data from a supported cloud platform.
3. The AI steward and AI asset owner enable evaluations for individual assets by updating asset records in the AI Control Tower inventory.
4. The AI steward configures evaluation metric templates by selecting quality and safety metrics and assigning weights to create a scoring formula.
5. The evaluation scoring engine automatically scores a sample of live AI interactions. For ServiceNow AI, the AI Skill Kit Auto Eval engine scores sessions. For external AI, sessions are scored by ServiceNow evaluation models.
6. The AI steward and AI asset owner review aggregated quality and safety scores, identify underperforming AI systems, and track trends over time.
7. The AI steward and AI Asset owner investigate low-scoring sessions by drilling into session detail, reviewing per-metric scores and judge reasoning, and tracing root causes through individual traces and spans.
8. The AI steward coordinates with the AI system owner to adjust the agent's configuration, prompts, or tool access based on investigation findings.
9. The AI system owner makes adjustments to agents as necessary.
10. The AI steward and AI Asset owner verify improvements externally at build time and continue verification in AI Control Tower at run time.

ServiceNow AI vs. external AI

Both categories use the same evaluation framework, but they differ in data collection and available metrics.

Table 2. ServiceNow AI vs. external AI comparison

	ServiceNow AI	External AI
Data collection	Automatic	Via API key (ai_observe.ai_data_sender role) or trace connector
Scoring engine	AI Skill Kit Auto Eval	ServiceNow evaluation models
Default quality metrics	Overall task completeness, Tool calling correctness	Task completion, Answer relevancy
Safety metrics	Not available with ServiceNow AI systems	Secrets detection (additional safety metrics available)

Where monitoring fits in AI Control Tower

AI Control Tower provides three complementary governance areas:

• Inventory manages what AI systems exist and their lifecycle state.
• Govern enforces policies, approvals, and compliance requirements.
• Insights provides ongoing operational visibility into how those systems actually perform.

Insights depends on inventory, since only managed assets are evaluated. The relationship also runs in reverse: connecting an AI system to send trace data can surface other AI systems and agents that aren't yet tracked, adding them to inventory automatically and helping you manage AI sprawl. To learn more about connecting your AI systems for evaluation and populating inventory, see Configuring trace connections.

Monitoring benefits

Table 3. Monitoring benefits

Benefit	Feature	Users
Automatically score live AI interactions without manual review using LLM-based judges fine-tuned for evaluation criteria.	How evaluation scoring works	AI steward
Define what quality and safety mean for your organization by selecting metrics and assigning weights to create a custom scoring formula.	Configure an evaluation metric template	AI steward
Track quality and safety trends over time to catch gradual regressions, not just one-off failures.	Reviewing quality and safety scores	AI steward, AI asset owner
Drill from portfolio-level scores down to a specific span in a trace to pinpoint exactly where an AI system's behavior broke down.	Investigating low-scoring sessions	AI steward, AI asset owner
Assess the quality and safety performance of a specific AI system by reviewing its scores, metric breakdowns, trends, and evaluated sessions.	Monitoring an AI system	AI steward, AI asset owner

What to explore next

To learn more about configuring and monitoring in AI Control Tower, see:

• Configuring monitoring and evaluations in AI Control Tower
• Reviewing AI system quality and safety scores
• Investigating low-scoring sessions

• Sessions, traces, and spans
  Understand the three levels of AI interaction data that AI Control Tower uses to structure, score, and display runtime behavior.
• How evaluation scoring works
  Understand how AI Control Tower calculates quality and safety scores so you can interpret results accurately and configure scoring to reflect your priorities.
• Evaluation scoring examples
  Worked scoring examples show how metric configuration, weights, and evaluation levels combine to produce quality and safety scores for different AI system types.
• How monitoring scores contribute to AI value
  Quality scores from monitoring can feed the value calculation. The value an AI system reports then reflects how well it performs, not just how often it is used.