Agentic evaluation run results

Learn about agentic evaluation runs and the meaning behind different evaluation scores from the agentic evaluation results page.

Agentic evaluations overview

Agentic evaluations measure how well AI agents and agentic workflows are accomplishing their objectives. A Now LLM Service model judges the AI agent or agentic workflow based on the execution logs. The results page of an evaluation run shows multiple metrics and scores measuring task completeness and tool use.

If you run an overall task completion evaluation, the results page shows recommended actions for the AI agent or agentic workflow. Recommended actions give you suggestions for deployment or improvement to help ensure that the agentic workflows that you deploy are performing up to your standards.

For more information on AI agent usage and other analytics, you can review the AI Agent Analytics dashboard in the AI Agent Studio.

Evaluation results overview

For each evaluation method that you execute, the results page displays an overall score for the agentic workflow with a percentage of successful record evaluations and a label of Excellent, Good, Moderate, or Poor. You can change the metric thresholds for each label by selecting Customize metric thresholds.

Tableau 1. Overall task completeness evaluation run results

Label	Description	Recommended action	Default threshold
Excellent	Tasks were consistently performed at a high standard. The agentic workflow is working well.	Proceed with confidence	90%–100%
Good	Most tasks were performed successfully, but some performance inconsistencies suggest areas for improvement.	Deploy with caution	70%–89%
Moderate	A significant number of tasks weren’t fully completed. Performance is below the desired level.	Investigate the root causes of poor task completion	50%–69%
Poor	The agentic workflow is consistently failing to complete tasks adequately. Major issues are present.	Do not deploy	0%–49%

Individual record metric scores

Evaluations are run against the log tables of agentic workflow executions. Each record is individually scored for each evaluation plan that you run. Individual record evaluations are scored according to the following metrics.

Tableau 2. Overall task completeness record metric scoresThe overall task completeness metric assesses whether an AI agent successfully completes its assigned task. It evaluates the execution logs of the agent, ensuring that all required steps were taken and the task was logically and effectively completed.

Number	Score	Description
3	Successful	The main task was fully completed. All subtasks were resolved, and the steps followed a logical sequence with no critical errors.
2	Partially successful	The task was partially completed. Some subtasks remain unresolved or inefficiencies affected the process.
1	Unsuccessful	The task wasn’t completed. Critical subtasks were abandoned or unresolved or the execution failed entirely.

Tableau 3. Tool performance record metric scoresThe tool performance evaluation metric assesses an AI agent’s ability to select the most appropriate tool for each step while completing a task.

Number	Score	Description
1	True	The right tool was chosen for the action in the plan.
0	False	The right tool wasn’t chosen.

Tableau 4. Tool calling records metric scoresThe tool calling evaluation metric assesses whether an AI agent correctly constructs tool calls by validating the accuracy, completeness, and formatting of the inputs it provides.

Number	Score	Description
1	True	Input key completeness, input value correctness, and input format correctness are all successful. Input key completeness: 1 - True – All required parameters are present with exact name matches, and no unexpected parameters are included. Input value correctness: 1 - True – Tool input values are correctly mapped. Input format correctness: 1 - True – Tool inputs are in the correct format.
0	False	One or more of input key completeness, input value completeness, or input format completeness wasn’t successful. Input key completeness: 0 - False – A mandatory parameter is either missing, its name doesn’t match exactly, or an unexpected parameter was found. Input value correctness: 0 - False – Tool input values are not correctly mapped. Input format correctness: 0 - False – Tool inputs are not in the correct format.

Remarque :

The values of the sub-metrics are aggregated using an AND operator. If any one value is 0, then the entire tool calling records metric score will be 0.