In Article 2, - Nandhakumar and I argued that agentic AI projects fail on data, not models and cited Gartner’s projection that over 40% of agentic AI projects will be scrapped by the end of 2027, citing escalating costs and unclear business value. This article is the proof. It’s a real engagement where an agentic workflow was heading toward a “No-Go” decision, and the thing that turned it around wasn’t a better model or a smarter prompt. It was data discipline. I worked this engagement alongside Richa, an Engineer on the delivery team, and I asked her to tell it from the ground.
Ragu Ramakrishnan, PM, AI Data Kit | Richa Kumari is an Engineer at ServiceNow who worked the go-live support engagement for this agentic workflow — from the first failing evaluation to production sign-off.
The Setup: A Four-Agent Workflow, 29 Days to Go-Live
Let me set the scene, because the stakes matter. A telecom operator was deploying its first serious agentic workflow — a four-agent sequential pipeline for AC power-loss alert triage on their cellular network. When a cell site loses AC power, the workflow validates whether the alert is in scope, discovers related incidents, enriches the picture with power telemetry, and writes a consolidated summary into the NOC engineer’s work notes. The goal: cut mean-time-to-resolution by putting site status, telemetry, and incident history in one place, automatically.
It runs autonomously. No approval gate during execution. The NOC engineer sees only the final work note. That autonomy is exactly what makes it valuable — and exactly what makes evaluation non-negotiable. If the agent silently writes an incomplete or misleading summary, nobody catches it in the moment.
Go-live was 29 days out. And the evaluation was not cooperating.
The Moment It Looked Like a No-Go
The evaluation score wouldn’t hold still. One run gave me 93% — a signal that said the workflow was ready. The next run on the same data: 70% — a signal that said it wasn’t. I had no way to know which one reflected reality, which meant I had no reliable measurement at all.
And while I was trying to figure that out, other problems were surfacing. The agent was looping on tool calls in edge cases. The test button was failing intermittently. DEV and SIT were configured differently. Some of these looked like they might be tied to why the evaluation was unreliable. Some looked fixable on their own. But I couldn’t be sure whether fixing them would actually stabilize the score.
That’s the position I was in heading into the go/no-go conversation: an evaluation I couldn’t trust, and genuine uncertainty about whether we could ship. I had no defensible answer either way.
Why Agentic Evaluation Is So Easy to Get Wrong
Here’s the trap the team walked into — and it’s one almost every agentic project hits.
When your evaluation score swings from 93% to 70% on the same dataset, your instinct is to blame the workflow. But often the workflow is fine. The instability is coming from somewhere else: a test dataset too small or too uniform to produce stable scores, environments configured differently so results aren’t comparable, or LLM non-determinism amplified by a thin evaluation set.
This isn’t a fringe problem. The τ-bench study from Sierra and UC Berkeley found that agents averaging over 60% single-run success can drop below 25% consistency when the same task is run repeatedly. Agents are non-deterministic by nature; a thin dataset just amplifies the swing until the number stops meaning anything.
You cannot make a Go/No-Go decision on a number that won’t hold still. And you cannot fix what you cannot reliably measure. Before you can trust the workflow, you have to trust the evaluation. And before you can trust the evaluation, you have to trust the data underneath it.
That’s the order of operations most teams get backwards. They tune the agent when they should be fixing the dataset.
Building a Dataset That Actually Represents Production
The fix started with the data. Not the agent — the data.
A real production environment doesn’t just contain happy-path alerts. It contains alerts against out-of-scope equipment, in-scope alerts with no matching incident, and alerts where the site information is simply missing. If your evaluation dataset only contains clean, correlated records, your agent passes evaluation and then fails in production on the messy reality you never tested.
Using AI Data Kit’s Multi-table Generator, the team built a 100-record synthetic dataset engineered to mirror production distribution across four deliberate scenarios:
|
Scenario |
Share |
What it tests |
|
Happy path |
63% |
Alert correlates to an incident, all fields populated |
|
In-scope, no incident |
10% |
Agent handles an unmatched alert correctly |
|
Missing site information |
7% |
Agent handles incomplete configuration data |
|
Out-of-scope alert |
20% |
Agent correctly filters and excludes |
The multi-table part matters. This isn’t one flat table of alerts. It’s alerts and incidents and their correlations — related records with referential integrity, generated together, mirroring how the data actually connects in production. That’s what lets you test whether the agent correlates correctly, not just whether it processes a single record.
What Building That Dataset Was Actually Like
We started by defining the telecom network operations scenario in the generator — describing the business context so it would create contextually relevant data. Then we selected the em_alert and incident tables. The generator auto-generated all the column descriptions, which worked well and saved setup time.
Setting the scenario distribution came next: 63% happy path, 10% in-scope with no incident, 7% missing site, 20% out-of-scope. This distribution covered all the production scenarios we needed to validate, not just happy paths — providing the data quality the earlier evaluation had lacked.
What was fiddly: the Configuration Item field came back empty after insertion. The scenario distribution didn’t split exactly as specified on the first pass. Referential integrity across the alert-incident relationship required careful sequencing. Each issue had a straightforward workaround — and each became feedback for the product roadmap, improvements to make the tool more self-sufficient for future teams.
The Turnaround: A Threshold You Can Defend
Once the dataset represented reality, two things became possible.
First, the team could set a defensible threshold. Not an arbitrary “let’s aim for 90%.” A reasoned one. There’s no universal pass mark for agentic evaluation — credible practitioners are explicit that the threshold has to be calibrated to the use case and its risk tolerance. A high-stakes financial workflow might demand 90%; a low-stakes internal tool might accept 75%. So the right bar depends entirely on the workflow’s risk profile. This workflow is read-only and a human owns the final decision — which argues for a moderate bar. But it runs autonomously with no approval gate, its output actively accelerates triage decisions, and some failure modes are silent. That argues for a higher one. The team landed on 85% as the initial production target — deliberately above a moderate baseline, achievable given LLM non-determinism, and tightenable to 90%+ after 30 days of real telemetry.
Second — and this is the part that closed the Go/No-Go — the team could validate that threshold by running the evaluation repeatedly against the stable synthetic dataset:
|
Eval run |
Score |
Rating band |
|
Run 1 |
90% |
Excellent (lower bound) |
|
Run 2 |
91% |
Excellent |
|
Run 3 |
92% |
Excellent |
|
Run 4 |
93% |
Excellent |
Four runs. A three-point band. Every run clearing the 85% threshold by at least five points. That’s the difference between “the score sometimes says 70%” and “the score sits consistently in the 90–93% band across independent runs — a signal we could stand behind.” The instability was never the workflow — it was the evaluation setup. Fix the data, and the real performance shows through.
That’s what turns a nervous No-Go into a confident Go.
What Changed for the Team
Once we had four evaluation runs consistently returning 90–93%, everything shifted. I walked into the go/no-go conversation with proof instead of doubt. Same workflow, same data, four independent runs — all showing the same story. The customer went from skeptical to confident. That consistency was what they needed to commit.
For me, it changed how I’d approach every agentic engagement going forward. I learned where the real lever is: not smarter prompts or better models, but investing in dataset quality upfront. The team walked away with a reusable threshold framework, eight workflow components that became a foundation for this customer’s future agentic use cases, and five product improvements fed back to the AI Data Kit roadmap.
If you’re starting an agentic go-live and your eval numbers won’t hold still, don’t reach for the model knobs first. Build the dataset that represents what production will actually throw at your workflow. That’s where the credibility lives.
The Pattern Worth Remembering
Strip away the telecom specifics and the shape of this story is universal.
An agentic workflow looked risky not because it was bad, but because it couldn’t be measured reliably. The evaluation was unstable because the test data was thin and the environments didn’t match. The fix wasn’t more prompt engineering — it was a synthetic dataset engineered to represent production reality, and a threshold reasoned from the workflow’s actual risk profile.
Data discipline is what moved this from “we’re not sure we can ship” to “we can prove we’re ready.” Not the model. The data.
If you’re staring down a Go/No-Go on an agentic workflow and your numbers won’t hold still — don’t tune the agent yet. Look at your data first.
Could This Be Your Story?
Every article in this series is built around a real practitioner and a real problem. If you’re building or evaluating agentic workflows on ServiceNow and hitting the data wall, reach out to Ragu Ramakrishnan on the ServiceNow Community. Let’s tell your story next.
Richa Kumari is an Engineer at ServiceNow.
Ragu Ramakrishnan is Product Manager, AI Data Kit & Evals at ServiceNow.
Other Sources
Gartner, “Gartner Predicts Over 40% of Agentic AI Projects Will Be Canceled by End of 2027” (press release, June 2025).
Yao et al., “τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains” (Sierra / UC Berkeley, 2024) — arxiv.org/abs/2406.12045
