Interested in a ServiceNow event built for developers? Registration for now[dev]26 is officially open!

Eliza
ServiceNow Employee

 

AI Control Tower Observability & Monitoring

Frequently Asked Questions

Quick Navigation

Overview — What is AICT Observability?
Getting Started — Prerequisites & setup steps
Evaluations — Metrics & evaluation framework
Traces — Sessions, traces, spans & data
Scoring & Dashboards — Results & visualization
Troubleshooting — Common issues & resolutions


Overview

What is the AI Control Tower?

The AI Control Tower is the centralized hub for the management of AI assets within your organization. It serves to provide visibility, control, and governance over AI systems, models, datasets, prompts, and MCP servers throughout their lifecycle.

Key capabilities include:

  • Discovery: Identify and document the AI systems that exist within your enterprise through three key methods – manual entry, automated discovery using Service Graph connectors, and identification of Shadow AI.
  • Governance & Compliance: Ensure that AI systems follow company policies and global regulations, focusing on privacy, data governance, and ethical AI.
  • Security, Access, & Privacy: Manages AI agent identities, access controls, and sensitive data to ensure security.
  • Observability & Monitoring: Runtime monitoring and performance evaluation across all your environments.
  • AI Strategy & Value: Tracks how AI improves efficiency, revenue, and cost savings.
  • Lifecycle Management: Set up and manage the complete lifecycle of AI assets, driving actions that must occur at each step of the process.

What does observability and monitoring mean in the context of AI Control Tower?

Observability enables you to understand runtime behaviors of AI systems through comprehensive data collection and analysis. We collect runtime telemetry from your AI systems and pass it through our evaluation engine, which scores performance against quality and safety metrics.

This foundation powers AICT features that:

  • Identify and remediate anomalies in real time
  • Quantify operational costs and value gains
  • Detect and act to remediate security issues
  • Verify compliance with established controls

Each of these processes help you to gain insights into actual system performance, with concerning elements surfaced and actioned upon.

When should I look to use the observability components of AI Control Tower?

Observability and monitoring are key components to understanding how your AI systems are performing over time. Including evaluations in your AI management processes help with:

Production risk mitigation

Detect quality degradation and performance drift in deployed AI systems before they impact end users or business operations.

Runtime validation

Validating that AI systems perform as expected under real workload conditions.

ROI quantification

Automated usage tracking combined with quality metrics enables value formula calculation: Usage × Time Value × Quality = Productivity Gain.

Compliance and auditability

Transparent audit logs with trace-level evidence support regulatory requirements and internal governance policies. LLM judge reasoning provides explainable scoring for compliance reviews.

Accelerated troubleshooting

Span-level execution visualizations pinpoint failure, enabling faster root cause analysis and remediation without the need for manually parsing logs.

How do I measure how well my AI Agents are performing in production environments?

AICT contains an evaluation framework which can be used to help you discern the performance of AI systems within both ServiceNow and external environments. Runtime traces from real user interactions with your AI systems are reviewed by our LLM-as-a-judge models that are specialized in evaluating the performance of your AI systems against 16 OOTB provided metrics that focus on both quality and safety. 

The results are displayed in an interactive dashboard that allows you to gain a detailed understanding of your AI systems performance. AI-generated insights stemming from the results are also displayed, given you a checklist of the most important actions to take to remediate any issues.

Eliza_0-1788371532659.png

 

What is drift, and how can AI Control Tower help to identify it?

Drift occurs when model performance degrades over time due to shifts in data distribution or operating conditions. Because drift definitions vary by organization and use case, there's no universal metric or visualization that works across all contexts.

AICT can help you address this complex and multifaceted issue through our observability framework: one can detect drift using the underlying quality & safety focused metrics and telemetry we expose. Detection can be automated via user-defined thresholds that trigger workflows, surfaced through our AI-driven top action items (which flag evaluation outcomes requiring attention), or discovered through manual review of results over time.

We provide the observability data, metrics, and automation capabilities that let you define, monitor, and act on drift according to your specific business and operational requirements.


Getting Started

Prerequisites

  • Instance on Zurich Patch 11 or Australia Patch 4+
  • A license that grants you access to AI Control Tower (learn more here)
  • At least one AI Agent marked as managed in AICT inventory
  • User role: sn_ai_observe.ai_observability_admin
  • Plugins: AI Native Experience for AI Control Tower

4-Step Setup Process

1

Populate AI Inventory

Use manual entry or automated discovery via Service Graph Connectors.

2

Set up Trace Collection

Configure the Trace Collector or the Traceloop SDK for your AI system provider.

3

Select Evaluation Metrics

Go to Rules and Templates > Evaluations. Choose metrics and configure sampling rates.

4

Adjust Metric Weighting (Optional)

Define the metric template that sets the makeup of the Quality and Safety scores. See scoring section for more.

5

Monitor Results

After configuring steps 1-4, evaluations will begin to be conducted on new traces produced. The results will automatically be populated in the Monitor dashboard within AICT.

Roles & Permissions

Role Definition
sn_ai_governance.ai_steward Access to all AICT features and data
sn_ai_observe.ai_observability_admin Modify evaluation settings (included in ai_steward role)
sn_ai_observe.ai_data_sender Proxy for external AI agents to authenticate and send trace data

Timeline to Visibility

ServiceNow AI Agents: Close to real time - there is a slight delay whilst with evaluation takes place.

External AI Agents: Latency is determined by the time it takes to intake the traces. For Trace Collectors, this is set by the collection frequency setting during configuration. The Traceloop SDK is faster as it pushes the traces to our evaluation engine automatically. 

Overall: Typically ≤5 min from trace ingestion to dashboard visibility


Evaluations

What AI systems can be evaluated within AICT?

AI Control Tower offers the ability to continuously monitor the quality, safety, and performance of ServiceNow and external AI agents at runtime. 


Supported Metrics

AI Control Tower evaluates both ServiceNow and external AI agents across quality, safety, and performance dimensions:

Category Applicable to Metric Name Description
Quality External AI Agents Task Completion Validates agent decision paths and output against the user's ask.
  External AI Agents Agent Goal Accuracy Ensures AI agents achieve their intended objectives.
  External AI Agents Agent Goal Completeness Measures whether an agent successfully accomplished all user goals.
  External AI Agents Agent Tool Trajectory Compare actual tool calls against expected reference tool calls.
  External AI Agents Context Relevance Validates that retrieved context is pertinent to the user query.
  External AI Agents Tool Error Detect errors or failures in agent tool execution to monitor agent tool performance.
  External AI Agents Answer Relevancy Verifies that responses address the query and remain on topic.
  External AI Agents Faithfulness Detects hallucinations and verifies facts to maintain accuracy and truthfulness.
  External AI Agents Answer Correctness Evaluates factual accuracy by comparing answers against ground truth.
  External AI Agents Answer Completeness Measures how completely responses use relevant context to address all relevant information.
  External AI Agents Conversation Quality Evaluates tone, clarity, flow, responsiveness, and transparency of a conversation.
  ServiceNow AI Agents Overall Task Completeness Determines whether an agentic workflow successfully completes its assigned task.
  ServiceNow AI Agents Tool choice Evaluates an AI agent's ability to select the most appropriate tool for each step while completing a task. It evaluates whether the agent follows an optimal task execution strategy by examining whether the chosen tool aligns with the intended action.
  ServiceNow AI Agents Tool calling Assesses whether an AI agent correctly constructs tool calls by validating the accuracy, completeness, and formatting of the inputs it provides. It assesses whether the agent selects appropriate parameters, maps them correctly to the expected values, and adheres to predefined formats. This metric helps determine if the AI agent can effectively interact with tools without errors or inconsistencies.
Safety External AI Agents Tone Evaluates the emotional tone and style of the response. Safe: joy, neutral, love. Harmful: fear, surprise, sadness, confusion, anger, annoyance.
  External AI Agents Profanity Detection Flags inappropriate language to maintain content quality standards.
  External AI Agents Secrets Detection Detects leaks of credentials, API keys, or other sensitive secrets.
  External AI Agents Sexism Detection Detects sexist or discriminatory content.
  External AI Agents Instruction Adherence Measures how well the LLM response follows given instructions.
  ServiceNow AI Agents Guardian Utilizes ServiceNow Guardian to detect and prevent safety-related issues.
Performance ServiceNow and External AI Agents Latency The length of time for a response to be received.
  ServiceNow and External AI Agents Token consumption The number of tokens consumed when performing an action. For ServiceNow AI Agents, assists are counted.

 

How Evaluation Works

7-phase process:

  1. Identification: The AI asset must be present in the AI Inventory. This can be done through manual input, automated discovery (through Service Graph connectors), or through Shadow AI detection.
  2. Configuration: Within the AICT inventory, select which AI systems you wish to undergo this active evaluation process by setting them to be “managed”. Then go to the Evaluation settings page (found within the Rules and Templates tab) and select which of the metrics you wish to include in the evaluation.
  3. Connection: Configure the Trace Collector or Traceloop SDK for your AI system provider to retrieve the runtime logs generated by your agentic systems. Traces from ServiceNow AI Agents are automatically gathered.
  4. Initiation & Analysis: The evaluation process occurs automatically. Traces will begin to be delivered into our evaluation framework for review.
  5. Review: The results are returned to your instance and displayed within the dashboard hosted in the tab labelled Monitor. Overall trends and insights are delivered on the main dashboard, with deeper analysis (such as LLM judge reasoning) provided on individual session and trace views. Results for each AI system are also hosted directly within their records.
  6. Action: After identifying the areas of improvement, update the how your AI system is configured – perhaps refresh a prompt, or add additional guardrails. AI-generated recommended tasks are also provided on the dashboard to direct your attention to the AI systems that need it the most.
  7. Iteration: Results stemming from further usage of AI systems are automatically re-evaluated, allowing you to identify if the performance of your AI systems benefited from the improvements made, or if they need additional tuning.

 

Eliza_1-1788371608835.png

LLM-as-a-Judge

LLM-as-a-judge uses specialized language models with engineer-curated prompts to autonomously evaluate AI system outputs. The judge receives interaction details and determines performance against each evaluation metric, producing both a score and reasoning for transparency.

Is there human in the loop component to evaluations?

Once configured, the evaluation process occurs in an entirely automated manner. The results are delivered directly to your instance, with insights automatically generated to prompt admins to review the items that require specific attention. Once the results have entered the platform, the process is entirely dependent on your organization's processes to include human-in-the-loop. We display results in a dashboard for an administrator to review manually, but one can also automate portions of this by having results that hit a specific threshold automatically trigger a remediation process.

Where is the ground truth sourced from for use in metrics?

We do not have access to the exact ground truth sources in real-time evaluations, so grounding is based on context data that is passed to the agent (and thus available in the prompt). For example, in a prompt for an AI Agent with a RAG pipeline, the structure is: "Answer the user's question given the following information: [context]. Question: [user query]". We use the information retrieved as a source of truth to compare the answer against.

How does the Personally Identifiable Information (PII) detection metric identify PII?

We utilize the Presidio model for PII detection. It identifies PII using the following techniques:

  • Named Entity Recognition
  • Regular Expressions
  • Rule-based logic
  • Checksums.

We do not have the option of configuring the list of PII that is to be detected today.

Which frameworks do we use to evaluate AI systems?

We have two primary providers for evaluations – ServiceNow's own proprietary evaluation framework and Traceloop.

ServiceNow's Autoeval
ServiceNow has built its own LLM-as-a-judge solution with custom prompts that handle ServiceNow data exceptionally well, using the same foundational method of evaluation used during design-time evaluations within AI Agent Studio. This method evaluates only AI Agents created and managed within ServiceNow.

Traceloop
Traceloop, acquired by ServiceNow in 2026 and fully integrated into AI Control Tower, provides an observability and evaluation framework that allows AICT to monitor, trace, and score the performance of external (non-ServiceNow) AI agents in production. You do not require a separate license for Traceloop – ServiceNow manages the connection for you.


Traces & Data

Hierarchy: Sessions, Traces & Spans

SESSION
One complete interaction (e.g., entire chat session) An employee asks an ITSM agent to create an incident for a VPN failure.
TRACE
Single turn (user question + agent response) The agent asks "Which VPN client are you using?" and the user answers.
SPAN
Single operation within a trace (LLM call, tool call, DB query) The agent calls the CMDB API to look up the user's VPN configuration.

 

What data is used during the evaluation?

We pass the judge LLMs the traces provided by the AI Agent provider. Traces capture the full call graph from a user request through an orchestrator, sub-agents, tool calls, and LLM invocations. These traces contain:

  • LLM details (model name, endpoint, prompt content, token counts, responses, latency, status codes, parameters)
  • Tool & Agent Actions (tool name, arguments, return values, execution time, success/failure status)
  • Agent Decision Points (planning steps, retrieval queries, memory reads/writes, tool selection logic, retries)
  • Structured Context (trace ID, span ID, session/conversation ID, user attributes, custom business attributes)

Data is stored in the following tables:

  • Sessions: sn_ai_observe_ai_session
  • Traces: sn_ai_observe_ai_trace
  • Spans: sn_ai_observe_ai_span

How is scoring performed across the three levels?

Different metrics are evaluated at different levels because each level captures different aspects of behavior.

Sessions have no dedicated metrics, but all metrics scored within that session get rolled up to provide 1 overall quality score and 1 overall safety score.

Trace-level metrics assess whether a single exchange performed as expected for that metric (e.g., whether an AI Agent produced a complete, accurate response).

Span-level metrics assess whether the AI system made good decisions at the individual step level (e.g., whether the correct tool was called with valid parameters).

When a metric is evaluated at both the span and trace level (e.g., Context Relevance), the trace-level evaluation takes precedence as it captures the quality of the combined output, not just the individual steps that produced it.

Are traces used elsewhere in the AI Control Tower?

Yes. We gather traces from internal and external AI systems for use in a broad range of features within AI Control Tower. Traces, which contain a wealth of telemetry data, help to power the following features:

  • AI Agent performance evaluations
  • AI Agent security evaluations
  • Token and cost monitoring
  • Calue calculations
  • Control compliance indicators

 

Trace Collection Methods

Trace Collectors (Recommended)

MID server-based polling. Supports AWS CloudWatch, Azure Application Insights, Gemini Enterprise Agent Platform, and Application Insights.

Traceloop SDK (For any framework where you have access to the codebase for the AI Agent)

OpenTelemetry-based library. Deploy on CrewAI, LangChain, LlamaIndex, or custom frameworks for push-based trace delivery.

 

How does trace collection work?

First step in the process is the collection of traces. This process deviates slightly depending on if you are using the Trace Collectors or the Traceloop SDK:

  1. Trace Collector: The MID server periodically polls the cloud provider to retrieve new trace data. This frequency is determined by the value you set for the Collection Frequency field when configuring a connection. The MID server then passes the traces into our evaluation framework for review.
  2. Traceloop SDK: The SDK will automatically push traces into our evaluation framework as they occur.

Once collected, the raw traces and the evaluation results are sent into your ServiceNow instance for use within AICT. They are stored in the same record within the sn_ai_observe_ai_trace table.

Raw traces are used to populate metadata in the AI inventory, and flow into the value monitoring and cost management features within AI Control Tower.

Evaluation results drive the security monitoring, value monitoring, cost management, and monitoring dashboards.

What do I need to set up trace collection?

If you are connecting using a Trace Collector, then you need the following:

  • Cloud provider credentials for the target provider (e.g. AWS/GCP/Azure)
  • An active and validated ServiceNow MID Server
  • Provider specific details, such as region for AWS connections
  • AI asset records for the AI systems you wish to monitor. They must be marked as managed. It is recommended that you use the Service Graph connectors to populate these asset records, as they can bring in the required metadata.

When configuring the connector, you will also need to define:

  • A collection frequency (i.e. how often the MID server will pull the traces in)
  • A unique and descriptive connection name. This will be used to identify different connections set up for the same provider.

Documentation addressing the configuration process can be found here.

If you are connecting a provider not noted above, then you need to connect using the SDK. This process requires:

  • Third-party AI Agent built using a provider not listed above e.g. CrewAI, LangChain. You must have access to this AI system's codebase.
  • A tracing instrumentation library. Traceloop is the only library officially supported by ServiceNow and thus is the preferred method.


Documentation addressing the configuration process can be found here.

I have multiple accounts for a certain provider – can I set up individual Trace Collectors?

Yes – when configuring the Trace Collector, you will specify a unique name. This name will be used to identify individual configurations from the same provider.

What exactly is the Traceloop SDK, and how do I use it?

The Traceloop SDK is the tracing instrumentation library used to pipe AI Agent trace data into ServiceNow. It is only recommended to be used when you are trying to collect telemetry from AI Agents built using frameworks not currently supported by Trace Collectors. You can find information on how to configure it here.

Do I need to specify the format of the traces?

No – once you have added the reference to the Traceloop SDK, the traces will automatically be formatted according to the OTLP structure required. The only attributes you may have to define manually are business specific attributes, such as where to gather the profile of the user who triggered the AI Agent. These can be specified in the association properties or within the resource attributes for the AI Agent.

Expected payload structure: Request (chunk of trace data) contains Resource Span (unit of processing work), which includes Resource (application/AI system info), Scope Span (logic unit), and Span (tracked actions tied together by trace ID).


Scoring & Dashboards

How are scores aggregated?

Session scores are the foundation for broader scores across AI Control Tower. At the system and portfolio level, scores are simple averages with no additional weighting.

AI system score

Averages quality scores across all evaluated sessions for that particular system. Each session counts equally regardless of duration or token usage. View scores by going directly to the AI system record page within AI Control Tower.

Portfolio score

Averages quality scores across all AI systems. Each system counts equally regardless of how many sessions it has. These are displayed within the Monitor dashboard in AICT.

What are the formats used for scoring?

All metrics use a scale where the higher the score, the better.

Binary (0% or 100%)

Produces a score of 0% (fail) or 100% (pass). Sexism Detection, Profanity Detection, and Tool Calling Correctness are binary. Because there are no partial scores, a single failure in a set of evaluations has an outsized effect on the average.

Percentage (0–100%)

Produces a score on a continuous scale from 0% to 100%. Answer Completeness and Context Relevance are examples of percentage metrics.

Tiered (Discrete values)

Produces specific score values rather than a full continuous scale. An example is that of the Task Completeness metric, with scores being marked as successful (100%), partial (50%), or unsuccessful (0%). Tone scores are tiered as Safe (100%), Neutral (70%), or Harmful (0%).

Can I adjust the metrics used in the score aggregation process?

Yes - you can adjust the scoring formula for a metric template to change which metrics contribute to quality or safety scores, or to rebalance how much weight each metric carries. To do so:

  1. Navigate to All > AI Control Tower > Settings > Rules and Templates > Evaluation metrics.
  2. Select the Metric templates sub-tab.
  3. Select the metric template that you want to edit.
  4. Update the metric template as needed. For example:
    • Adjust a metric's weight to increase or decrease its influence on the score
    • Add a metric by selecting + Add metric and assigning a weight
    • Remove a metric from the formula
    • Change the primary or secondary metric designation
  5. After any change, verify that all weights still sum to 100%.
  6. Verify the updated formula by expanding the "How is this score calculated?" section.

Note: Only metrics that are enabled for scoring on the AI evaluations page are available to add. The updated formula will now be applied to new evaluation sessions. Scores on the monitoring overview reflect the updated formula as new data is collected.

Eliza_2-1788371698546.png

 

How do I add more metrics into the scores?

You need to navigate to AI Control Tower > Rules and Templates > Evaluation. Within this view, you can select from the list of metrics which ones to include. Be sure to review this for both ServiceNow AI systems and external AI systems, as the list of available metrics differs.

Where do I view the results of the evaluation?

The results of evaluation can be found within the Monitor tab within the Insights section of the AI Control Tower. This dashboard provides an overview of your AI systems average quality and safety scores across the given time period (adjustable by the filter found in the top right of the dashboard). Top action items derived from the results are also displayed, to inform you of assets requiring your attention.

Eliza_3-1788371926514.png

 

The AI systems ranked by score table shows a list of the highest and lowest scoring AI systems. Click on these records to navigate directly to the asset record, which will display the recent results for this section across the actively monitored metrics.

Eliza_4-1788371959189.pngEliza_5-1788371980635.png

 


The Monitor agent activity graph showcases the weighted average for evaluated AI systems across the time period selected within the filter at the top-right corner of the page. Adjust the filter to display either Quality or Safety metrics to help you identify trends in the performance of your AI systems.

Eliza_6-1788372135866.png

 

If you click on the expand icon in the top right of most reports in the dashboard, you will be presented with a breakdown of how the score was calculated.

Eliza_7-1788372167574.png

How can I filter the results within the monitoring dashboard?

The results can be filtered using the options given within the dashboard to limit the results to a particular timeframe. These filters will update the data presented across all reports in the dashboard.

Eliza_8-1788372188349.png

Where can I find information about a specific AI system's performance?

If you are looking for results for an individual AI system, click on the AI system's name from within the dashboard or navigate to that AI system's record from the inventory.

Eliza_9-1788372231015.png

Where can I find information on specific sessions or traces?

The evaluated sessions section can be found within the tab group at the top of the page.

Eliza_10-1788372273207.png

 

This page grants you the ability to view trends in performance over time. The top graph, named "AI systems with evaluated sessions," can be set to display the AI systems with the largest score decline, or your top performing AI systems. Below you can find the list of evaluated sessions, sorted by the most recent. You have the option to filter and search for specific sessions.

Eliza_11-1788372317626.png

 

Session View: Clicking on the session name will present you with a highly detailed breakdown of the session scores alongside justifications of the scores from the LLM judges. The session overview also delivers insights into how a particular interaction with an AI system went. Key session details are listed at the top, including AI system, AI system type, duration, number of traces, input tokens, output tokens, and total runtime tokens.

Eliza_12-1788372358582.png

 

Traces View: Opening a Trace record displays granular interaction details for a particular user exchange. Traces within a session are ordered by execution and show high-level data: duration, latency, number of spans, input/output tokens, total runtime tokens, and metric scores with reasoning. Expanding a trace reveals the trace scores table (all metrics evaluated with scores and reasons), along with Input and Output details to clarify the back-and-forth with the AI system. The span map visualizes the steps the AI system took to complete the trace, with nesting representing span relationships. Clicking any span displays its details, scores with reasoning, and applicable inputs or outputs. This view is key for troubleshooting or conducting manual reviews of AI system performance.

Eliza_13-1788372376669.png

I have a low performing AI system – what now?

It depends on which metric is currently identified as requiring attention, but generally, we would advise that the owner of the AI system review the details presented in the dashboards to understand the scenarios in which it underperformed. To drive this action, it is recommended that you create AI tasks asking system owners to review and remediate the issue within AICT.

What does it mean if my dashboards don't show any results?

If no results are populated, then any of the following may be true:

  • The AI system has not been configured properly to send traces into ServiceNow
  • There are insufficient traces being produced to analyze
  • The session/trace/span data retrieved can't be scored with the metrics selected within the AI evaluation setup
  • The evaluation process is still being conducted

What is the Quality Score?

The quality score is the equation used to dictate the overall score used within the Quality and Safety sections in the evaluation dashboards. The equation is:

Overall score = (Primary metric × weight) + (Secondary metric × weight) + (All other metrics × weight)

You can adjust the weighting for each metric in the Metric Templates configuration screen, which you can access from Settings > Rules and Templates > Evaluation > Metric Templates.

Can I modify what appears on the dashboards?

Not today. You can limit which metrics are displayed by only activating those which interest you, and turning on/off evaluations for internal or external AI systems if desired. Future updates to AI Control Tower will include options for configuration.

How can I trigger actions if a metric performance falls below a specific threshold?

There is no direct in-product support for this today. You do however have the option of building a workflow within Workflow Studio to create what you need. To do so:

  1. Set the trigger to be an evaluation result from the sn_ai_observe_ai_trace table
  2. Within your workflow, create the steps you wish to be performed in order to review or remediate the issue. For example, you may wish to generate a task/incident or case record and assign it to a team to review.

Troubleshooting

How can I limit access for those reviewing the dashboards/session/trace data?

At present, we do not have granular role access to specific traces. Those with the sn_ai_governance.ai_steward role can see all the contents within AI Control Tower. An option for the interim is to create a custom ACL for the tables containing data you do not wish to be exposed.

Why was my trace rejected?

If the system is receiving errors while exporting traces, it may be encountering the following issues:

Invalid format

Must be in the OpenTelemetry Protocol (OTLP), binary protobuf, or JSON format.

Missing API key

The payload must include the x-snc-observability-token header.

Unauthorized

The API token must be valid on the instance, and the associated user must have the sn_ai_observe.ai_data_sender role.

 

Why wasn't my trace evaluated?

Sampling rate too low

Check that sampling is greater than 0%. Default is 5%; increase for more coverage.

Asset discovery in progress

First trace triggers discovery. Evaluation starts on the next trace after discovery completes.

Missing required fields (Traceloop SDK)

Ensure trace ID, span ID, and asset info are included in all spans.

 

Why am I not seeing a trace straight away with scores?

There is a mix of scheduled jobs running periodically, and our evaluation framework needs time to import a trace and then score it before the metrics can be picked up. This process can take several minutes to complete.

What is the impact of having multiple traces (with different trace IDs) in the same session, but no session ID is set?

Since the trace ID is used as the session ID when session ID is not present, the traces will be considered different sessions. For this reason, it is best to set the same session ID across all traces that are in the same session.

Why is my API key getting rejected?

A key may be inactivated, so please go to the instance and verify the key is active, and the value itself is correct.

 

General FAQ

How many assists are used during the evaluation process? The assist consumption rate is 500 assists per month, per AI asset marked as managed.

Is this compatible with GCC/FedRAMP? Not today.

Does this support regional deployments? Yes

Can I see data from other ServiceNow instances? Not today – it is limited to evaluating ServiceNow AI systems within the current instance, and external providers only.

Version history
Last update:
an hour ago
Updated by:
Contributors