Overview
Agentic evaluations provide a systematic approach to testing and monitoring the performance of your AI agents and agentic workflows. This feature allows you to evaluate your agents against custom datasets, measure key performance metrics, and compare results across different benchmarks. By running automated evaluations, you can ensure your AI agents consistently meet quality standards and identify areas for improvement.
Once evaluations are complete, you can analyze results through both overview dashboards and detailed breakdowns. The platform enables you to export reports, clone evaluations for comparative analysis, and track improvements across iterations. This structured evaluation framework ensures your AI agents deliver reliable and consistent outcomes in production environments.
Navigate to Automated Evaluation
- Access the AI Agent Studio from your ServiceNow instance
- Select the Testing section from the navigation menu
- Click the "Start automated evaluation" button to begin the evaluation setup process
Configure Evaluation Settings
Name and Select Workflow
- Enter a descriptive name for your evaluation in the name field
- Add a description that explains the purpose of this evaluation
- Use the drop-down search field to select the agentic workflow you want to evaluate
- Click the Continue button in the lower-right corner to proceed to the next step
Select Evaluation Metrics
- Review the available evaluation metrics on the "evaluation metrics" page
- Select the specific metrics you want to measure during the evaluation
- Click Continue to move to the dataset configuration
Create and Configure Dataset
Set Up Dataset Parameters
- Enter a name for your new dataset
- Provide a clear description of the dataset purpose
- Select "By running agentic workflow and using the generated execution logs" as the dataset creation method
- Choose the table from which to run evaluations (for example, "Incidents")
- Enter the maximum number of records to include in the evaluation
- Add any desired filters to narrow the dataset scope
Important Note
The agentic evaluation consumes 1 assist per record processed during evaluation.
Add Workflow Instructions
- Navigate to the "Add instructions" section
- Enter your instruction using dynamic field references (for example, "Investigate {{incident.number}}")
- Use the pill picker on the right side to select the correct table field
- Click Continue to proceed to the review stage
Review and Launch Evaluation
Review Configuration Summary
- Examine all configuration details on the "Review summary" page
- Verify that the workflow, metrics, dataset, and instructions are correctly configured
- Click "Start evaluation" in the lower-right corner when ready to begin
Processing Time Note
The execution log generation process may take some time to complete depending on the dataset size.
Monitor Evaluation Progress
- Wait for the execution log generation to complete
- Verify that the "Start Evaluation" button becomes available
- Click the button to initiate the evaluation run
Evaluation Duration Note
The evaluation process may take a considerable amount of time depending on the number of records.
Review Evaluation Results
Analyze Performance Data
- Access the Overview tab to view high-level performance metrics
- Switch to the Detailed Results tab for granular analysis of individual evaluations
- Review specific execution details and metric scores for each evaluated record
Export and Share Results
- Use the clone functionality to duplicate evaluations for comparative testing
- Click the export option to download results as a CSV report
- Share reports with stakeholders for performance review and decision-making
Access Documentation
For comprehensive information on agentic evaluations, visit the ServiceNow documentation at: https://www.servicenow.com/docs/r/intelligent-experiences/execute-aia-eval.html
Troubleshooting
Verify Prerequisites and Permissions
- Confirm that all configuration steps have been completed in sequence
- Verify that you have the appropriate entitlements to use Now Assist AI Agents
- Check that AI Search has been enabled in your instance
- Ensure the Now Assist Panel is turned on and accessible
Update Required Components
- Navigate to the store apps section and verify all apps are updated to the latest version
- Sync the plugins manager page to retrieve the most recent plugin versions
- Repair plugins after updating to ensure changes and fixes take effect properly
Check System Properties and Roles
- Verify that the system property
sn_ais_assist.dpr_ingestion_completedis set to true - Confirm your user account has the necessary roles for agent access
- Ensure the agent configuration allows the expected roles for user access and data access
Additional Troubleshooting Steps
- Log out of your ServiceNow instance and log back in after making configuration changes
- Clear browser cache if interface elements are not displaying correctly
- Contact Now Support by logging a case if issues persist after completing all troubleshooting steps
