AI Data Mastery Series | Article 2 Agentic AI Will Fail Without Data Discipline
Co-authored by Ragu Ramakrishnan and Nandhakumar Apparsamy
In our first article, Yogesh Shinde showed how Solution Consultants use Now Assist Data Kit's Synthetic Generator to go from blank instance to demo-ready in under 30 minutes. This article goes deeper - into the agentic development lifecycle itself. I spoke with Nandhakumar, an Agentic Developer at Work4Flow, one of ServiceNow's top Agentic AI partners, to understand what it actually takes to ship agents that hold up in production. - Ragu Ramakrishnan, PM, Now Assist Data Kit
Nandhakumar builds, ships, and governs agentic solutions at Work4Flow — from Now Assist skills to multi-agent workflows and cross-platform integrations on ServiceNow.
The Number That Should Stop You
Gartner predicts over 40% of agentic AI projects will be canceled by end of the end of 2027 - due to escalating costs, unclear business value, or inadequate risk controls.
Not paused. Not pivoted. Canceled.
"Most agentic AI projects right now are early-stage experiments driven by hype and often misapplied," says Anushree Verma, Senior Director Analyst at Gartner. "This blinds organizations to the real cost and complexity of deploying AI agents at scale."
From where I sit - and from what Nandhakumar sees daily at Work4Flow — the cost and complexity Gartner is pointing at isn't the agent framework, the orchestration layer, or the model. It's the data.
This Isn't Software. Stop Building It Like Software.
Coming from software engineering, I assumed the hardest part of building agents would be selecting the right model or writing better prompts. That turned out to be the easier part. The real challenge is giving the agent the right context and enough representative data to make reliable decisions across different scenarios. Unlike traditional software, you can't expect the same input to always produce the same reasoning path. That changes how you build, test, and improve the system. I quickly realized that agent quality depends far more on data quality and evaluation than prompt engineering. prompt engineering does. Once you accept that shift, you stop optimizing prompts and start improving the data that drives every decision.
Where the Time Goes
In traditional software, you write code and run tests. Testing is a small fraction of the work.
Agentic development doesn't work that way. The same input can produce different reasoning paths, different tool calls, different outputs - on every single run. There's no fixed correct answer to assert against. You can't write the tests once and ship.
The time breakdown for a cold-start agentic project looks like this: roughly 60% on data - curation, synthetic generation, labeling, ground truth. Around 25% on evaluation setup and analysis. About 15% on instruction tuning, which most developers assume will be the hard part.
That 60% figure isn't a guess. Across multiple independent surveys of ML practitioners, data preparation consistently dominates - Anaconda's State of Data Science surveys put it at 39-45%; CrowdFlower's research goes as high as 73%. Agentic development inherits that cost, and adds non-determinism on top of it.
What Day One Actually Looks Like
When I start a new agentic use case at Work4Flow, I don't begin with prompts — I begin with data. I first identify the knowledge articles, catalog items, workflows, APIs, or enterprise records the agent will depend on. Then I ask two questions: what does success look like, and where is the agent most likely to fail? Those answers usually expose gaps in documentation, inconsistent data, or missing edge cases long before development begins. In practice, poor data leads to incorrect tool selection, incomplete responses, or failures during multi-step workflows, even when the model itself performs well. Instead of tuning prompts immediately, we focus on improving the underlying data and building representative evaluation scenarios before expanding the agent's capabilities.
You're Evaluating Trajectories, Not Answers
Most developers think evaluation means checking if the agent got the right answer. For a single skill, that's roughly true. For an agent, it's much harder.
Did it call the right tools, in the right order? Did it recover from an error? Did it stay within guardrails on an ambiguous input? Two runs can reach the same answer via completely different paths — only one of which is acceptable in production.
This is why your evaluation dataset has to be designed, not just collected. Happy paths, edge cases, adversarial inputs, multi-turn context - all of it needs to be deliberately engineered into your golden dataset. A thin dataset gives you false confidence. The agent passes your eval and fails in production on exactly the scenarios you never thought to represent.
The Brookings Institution, convening 40+ experts on agentic AI governance in 2025, put it plainly: "We cannot govern what we cannot measure."
How Nandhakumar Handles It
I treat evaluation as an integral part of development rather than something to do before release. I start by building a small but representative evaluation dataset that covers common requests, edge cases, and failure scenarios. For every example, I define not only the expected response but also whether the agent selected the right tools, used the correct enterprise context, and followed the intended workflow. I run a small baseline, analyze where the agent struggles, improve either the data or the instructions, and repeat the process. In my experience, most improvements come from refining datasets rather than changing models. My biggest advice to new agentic developers is simple: invest in high-quality evaluation data early, because you can't improve behavior you don't consistently measure.
The Iteration Loop Is the Job
Agentic development isn't a pipeline with a finish line. It's a loop.
You define the agent's behavior. You run a 10–20 sample baseline to see the floor before investing in data. You curate, generate, label. You run a full evaluation. You analyze what broke — instruction gap? Missing data? Bad labels? You fix one thing, and you run again.
Most teams need three to five full cycles before hitting reasonable production metrics. If you're there in one cycle, your eval isn't sensitive enough.
The implication: data isn't a prerequisite you complete before development starts. It's a continuous discipline that runs alongside every iteration.
What Tools Change
Today, a surprising amount of time goes into moving datasets between different tools, maintaining versions, and keeping labels consistent as requirements evolve. Having synthetic data generation, labeling, dataset management, and evaluation in one place — like Now Assist Data Kit — would make those iterations much faster. Instead of managing spreadsheets and disconnected workflows, I we could generate realistic edge cases, organize evaluation datasets, and measure improvements from a single workspace. That lets the team spend more time improving agents instead of managing data.
Where to Start
It doesn't start with picking a framework or designing an orchestration layer.
Spend the first week on instruction clarity — write down what success looks like for five to ten examples. Then run a 20-sample baseline. You'll learn more from the failure patterns than from any amount of planning.
If you have real data, even messy, start there and augment edge cases synthetically. If you're greenfield, synthetic generation seeded with a few hand-crafted examples gets you to 100–200 labeled records in hours, not days.
Label systematically. Labeling consistency is what makes your evaluation meaningful.
And plan for the loop. Gartner predicts at least 15% of day-to-day work decisions will be made autonomously through agentic AI by 2028, up from virtually zero today. The developers who get there won't have had the best model. They'll have had the best data.
Could This Be Your Story?
Every article in this series is built around a real practitioner and a real problem. If you're building agentic solutions on ServiceNow and hitting the data wall — reach out to Ragu Ramakrishnan on the ServiceNow Community. Let's tell your story next.
Nandhakumar Apparsamy is an Agentic Developer at Work4Flow, a Top Agentic AI Partner for ServiceNow.
Ragu Ramakrishnan is Product Manager, Now Assist Data Kit at ServiceNow.
