Overview
Agentic evaluations provide a systematic approach to testing and monitoring the performance of your AI agents and agentic workflows. This feature allows you to evaluate your agents against custom datasets, measure key performance metrics, and compare results across different benchmarks. By running automated evaluations, you can ensure your AI agents consistently meet quality standards and identify areas for improvement.
This guide is part 3 in a series of articles describing how to enable Agentic workflows in your instance. If you would like to navigate to the other articles, please follow the links below. See here for the Evaluations guide using legacy AI Agent Studio.
Navigate to Automated Evaluation
- Access the AI Agent Studio from your ServiceNow instance
- Select an agentic workflow or AI Agent in the Agentic Solution page
- Click the "Add Scenarios" button to setup the evaluation process
Configure Scenarios
- Choose the table from which to run evaluations (for example, "Incidents")
- Enter the maximum number of records to include in the evaluation
- Add any filters to narrow the dataset scope
Important Note
The agentic evaluation consumes 1 assist per record processed during evaluation.
4. Enter your objective with dynamic field references (for example, "Investigate {{incident.number}}")
5. Use the pill picker on the right side to select the correct table field
6. Click Add scenarios to return to the main build page.
Review and Launch Evaluation
- On the top right of the agentic workflow or AI Agent build page, click "Run Test" and then select Evaluation.
2. Verify that the workflow, dataset, and objectives are as intended.
3. Select additional metrics to calculate if needed.
4. Click "Run evaluation" when ready to begin.
Optional: For custom agents, it is possible to run optimizations.
Processing Time Note
The execution log generation process may take some time to complete depending on the dataset size.
Evaluation Duration Note
The evaluation process may take a considerable amount of time depending on the number of records.
Review Evaluation Results
Analyze Performance Data
- View high-level performance metrics
- Switch to the Individual Scenarios tab for granular analysis of individual evaluations
- Review specific execution details and metric scores for each evaluated record
Access Documentation
For comprehensive information on agentic evaluations, visit the ServiceNow documentation at: https://www.servicenow.com/docs/r/intelligent-experiences/execute-aia-eval.html
AI Agents Troubleshooting
Verify Prerequisites and Permissions
- Confirm that all configuration steps have been completed in sequence
- Verify that you have the appropriate entitlements to use Otto AI Agents
- Check that AI Search has been enabled in your instance
- Ensure the Otto Panel is turned on and accessible
Update Required Components
- Navigate to the store apps section and verify all ServiceNow Otto apps are updated to the latest version
- Sync the plugins manager page to retrieve the most recent plugin versions
- Repair plugins after updating to ensure changes and fixes take effect properly
Check System Properties and Roles
- Verify that the system property
sn_ais_assist.dpr_ingestion_completedis set to true - Confirm your user account has the necessary roles for agent access
- Ensure the agent configuration allows the expected roles for user access and data access (ACLs)
Additional Troubleshooting Steps
- Log out of your ServiceNow instance and log back in after making configuration changes
- Clear browser cache if interface elements are not displaying correctly. For the redesigned AI Agent Studio, navigate to /cache_aiux.do.
- Contact Now Support by logging a case if issues persist after completing all troubleshooting steps
- You can view detailed logs in the sn_aia_execution_plan table.
- If you need to return to the legacy AI Agent Studio, you can do so by navigating to Settings > View legacy AI Agent Studio