Enterprise AI agents usually look brilliant in a demo. When they meet production, it’s often a different story: thousands of incidents per week, a different knowledge base for every customer, and people who need the fix to work at 2 a.m.
That difference between an AI agent that impresses and an AI agent people trust with their work is where enterprise AI can succeed or stall. We addressed that at ServiceNow not with one breakthrough model or one clever prompt, but with a discipline evolved from computer science: hill climbing.
The method works like this: Run a controlled experiment, measure the results honestly, and keep the change only if it improves the AI agent, alone or in combination with other changes. Then repeat, one small step at a time.
Like a human climbing a hill, the ascent is rarely linear. Some steps slide backward. Some segments are flat. And near the end, progress can come faster because the early footholds made the final push possible.
Shipping enterprise AI agents is the same. You don't ship an enterprise agent and just hope it gets better. You improve it through repeated, measured steps, the same way a climber makes progress one step at a time.
We built our AI specialists, autonomous workers that can take action with appropriate governance but without human intervention, using that approach. We started with our L1 Service Desk AI Specialist, which resolves IT incidents end to end. Here's what the climb looked like, dips included.
That capability depends on two feedback loops. The first loop runs on judgment and doesn't wait for results. An automated reviewer, built to grade work the way an experienced engineer would, reads the AI agent's output as soon as it's produced, evaluates it against past incidents, and decides whether the answer is likely to be correct, complete, and safe. Its verdict is an informed estimate, and it arrives quickly.
The second loop runs on outcomes. It’s slow because the truth arrives later: tickets that remained closed versus those that were reopened. This is the true scorecard, but it takes time.
The two loops provide checks: The faster loop enables experimentation, and the slower loop tells us whether the former was right. Without the first, feedback arrives too late. Without the second, teams chase scores that look good in theory while, in reality, tickets are reopened.
The speed of learning ties the two together, and one number tracks it: mean time to learn, or how long it takes from shipping a change to knowing, with evidence, whether it worked.
Whereas mean time to resolution shows how fast a problem gets fixed, mean time to learn shows how fast an AI agent improves. When that cycle is short, teams run more experiments and settle debates with data. When it’s long, teams trade anecdotes while the AI agent stays the same.
A fast first loop is only as valuable as its agreement with the second. So real-world results flow back continuously into both the incident library and the automated reviewer, which keeps the quick estimate honest. Evaluating enterprise AI agents this way can be a product in its own right, informing which changes reach customers, but getting it right is difficult for three reasons:
- Every customer is different. Knowledge, case mix, the ways users file incidents, and closure habits vary widely. A win in one instance may not carry over to the next.
- A wrong answer costs more than no answer. The alternative is a clean handoff. When an AI agent can't provide a reliable answer, it passes the case to a human. An AI agent that hands off the right cases at the right times earns trust.
- Truth arrives later. An AI agent's work is proven only when the outcome holds, which may not be clear until days later.
We recommend the five-step process we followed for improving AI agents.
We improved two key metrics side by side: deflection rate (the share of work the AI agent resolves without human involvement) and resolution quality (whether the AI agent got it right). Either metric alone is easy to game. Push the AI agent to handle everything, and quality collapses. Let it take only the obvious cases, and quality appears high while nobody’s workload gets lighter.
As a trade-off, we tracked the two together rather than as a single score. Picture a chart with one axis showing how much work the AI agent takes on and the other showing its accuracy; data scientists call this a precision-recall curve. Each way of tuning the agent lands at a different point on the curve.
A significant improvement moves the whole curve up and to the right so the AI agent can simultaneously take on more work and get more of it right. Where to operate on that curve is a business decision, not a technical one, for each customer.
The quickest way to build a test for an AI agent is to write tasks that are easy to grade. That’s also the quickest way to build a useless test. Instead, we created our test sets from real IT incidents, drawn from customers’ systems, that had already been closed.
We sampled them so every customer and problem type was fairly represented, including cases the AI agent should pass to a human. Each customer’s data stayed separate: The agent was tested on a customer’s incidents using only that customer’s knowledge, and each customer was scored on its own.
The correct answers came from records the business already produced: Resolution notes explain what fixed the incident, and whether the ticket stayed closed reveals if the fix worked. What counts as correct is what really happened, not what merely sounds right.
The AI agent writes open-ended answers so another AI model can grade them as a judge. A flawed judge doesn’t just add noise; it steers the whole effort the wrong way.
We treated the judge as a project of its own. It checks specific facts rather than giving vague ratings, and its grades were tested against subject matter experts. We chose its model and thinking time based on testing and had it grade each answer several times until its scores held steady. That consistency told us how big a change had to be to count as real.
No single expert knows everything, so disagreements were settled on evidence, and sometimes the judge caught what an expert missed. Throughout, we read the full record of each case and studied the mistakes. Scores are summaries; the records are the truth.
Every experiment went into a shared journal with a prediction and a human owner. Each round tested one idea and advanced only if it improved results on both familiar and unseen cases. Changes that fell short were saved, since some help only in combination. The journal forced a prediction before every change, made dead ends as visible as wins, and gave leaders a portfolio they could steer.
Two or three rounds without progress are a signal, not a setback: The easy wins are gone, and the AI agent has done what it can. We then studied the remaining failures one by one, which pointed to five fixes:
- Better search, when the answer exists but isn’t found
- Richer context about the user and their systems
- New tools and skills for actions the AI agent couldn’t take
- Memory, so recurring problems aren’t rediscovered each time
- A stronger model or more thinking time, weighed against cost and speed
Some failures had no fix in the AI agent because the answer was never documented in an IT ticket. Those failures went to the humans who own that knowledge. When no single change helped, we tried the combinations saved in the journal.
The curve in the chart illustrates our path to general availability, the point at which the product was released to all customers. The journey included a weak start, early setbacks, and long stretches without progress, as well as gains that built on one another as the work matured.
It isn’t a smooth staircase, and it doesn’t need to be. That’s the most honest thing we can tell you about building AI agents for the enterprise. We learned important lessons along the way:
- Setbacks are part of the path. One early experiment pushed quality below the starting point, which was already modest. Our testing caught it within a single round, we undid the change, and the next experiment improved both metrics.
- Plateaus are normal. Long, flat stretches mean the easy wins are gone. That’s the moment to sort the remaining failures by their causes.
- The two metrics can move in opposite directions. One experiment raised quality while the share of work the AI agent handled on its own fell sharply. Looked at individually, either number would have told the wrong story.
- Final versions can slip too. The version just before release dropped sharply. Because every version goes through the same tests, we identified the issue, fixed it, and shipped the strongest version.
- Gains compound. The biggest improvement came late, once better reasoning was added to an already accurate AI agent. The discipline built early is what made late gains possible.
If you’re building or buying enterprise AI agents, these are the standards we hold ourselves to.
A method that lives inside one team is a heroic effort; a method built into the ServiceNow AI Platform is a standard. We’re turning this playbook into shared infrastructure: a common testing service for AI specialists, a single format for recording what AI agents do, and an automated loop that runs the cycle of testing and improvement on its own.
Test results also route each task to the models that handle them best. And responsible AI is built in so each grade can be traced and audited.
Enterprise AI agents don’t become trustworthy at launch. They earn trust one measured step at a time, with an honest judge, data the AI agent has never seen, and a record of every dead end. This results in service desks that stop fighting fires and start fixing the systems behind them—and people who get back to doing the work that only humans can.
Find out how ServiceNow can help scale your team with domain-specific AI specialists.
