When you spend enough time working at the intersection of AI research and enterprise operations, one thing becomes clear: The tools we use to evaluate AI agents weren't built with the enterprise in mind. They were built for a different era, one of isolated, single-step tasks performed against clean, controlled datasets.
That's not what enterprise work looks like, and it's not what enterprise AI needs to handle.
Enterprise AI demands long-horizon planning capabilities across multiple domains and tools. It also requires persistent state management in interconnected systems, policy and compliance adherence, and reliable error recovery.
That difference matters in practice. When organizations make deployment decisions based on benchmark scores that don't reflect their operational environments, they're not making informed decisions; they're making optimistic ones.
The risk is that AI models could appear capable in benchmarks but fail catastrophically in reality. That's what motivated us to build NOWAI-Bench, with EnterpriseOps-Gym as its first component.
Here’s some important context regarding the findings below: NOWAI-Bench evaluates general-purpose AI models without any platform support—raw capability, without orchestration, guardrails, or workflow intelligence. The results reflect the floor of what's possible, not the ceiling. They’re not a reflection of what a purpose-built enterprise AI platform, including ServiceNow’s offering, can deliver.
We believe NOWAI-Bench: EnterpriseOps-Gym is the most comprehensive benchmark to date across enterprise business workflows spanning IT service management (ITSM), customer service management (CSM), and HR. It’s designed specifically to evaluate AI agents against the complexity that exists in production environments.
What makes enterprise workflows genuinely challenging is that they're interconnected. A single IT service request doesn't live in isolation. It may trigger a chain of actions across HR operations, asset management, and customer service.
Data is distributed across dozens—sometimes hundreds—of interdependent tables. Business logic is layered, sequential, and deeply context dependent. Most benchmarks sidestep this entirely. NOWAI-Bench was built to confront it directly.
The benchmark evaluates AI agents across 1,150 real-world enterprise tasks, incorporating 512 functional tools and 164 interconnected database tables. It creates an evaluation environment that reflects how enterprise platforms operate with cross-departmental dependencies, multistep task execution, and contextual reasoning requirements that can't be reduced to a single prompt and response.
The goal was to give enterprises something that’s been conspicuously missing: a reliable, standardized way to assess whether an AI agent is ready for its environment before it's deployed. The tool is grounded not in controlled demonstrations, but in measurable performance against realistic operational complexity.
We evaluated a range of AI models (leading proprietary and open-source options alike) across the full EnterpriseOps-Gym benchmark. What we found confirmed what we had suspected from working in this space: Enterprise workflows are significantly more challenging than existing evaluations suggest, and they expose failure modes that general-purpose benchmarks simply don't reveal.
All findings below reflect raw model performance without platform orchestration or guardrails.
When agents were provided with expert-generated task plans, performance improved by 15% to 35% on the most complicated enterprise domains. Notably, tool selection and execution remained stable even under adversarial conditions.
This tells us that models don't fail at the point of action. They fail at knowing what to do across a constrained, multistep workflow when no one has mapped it out for them. The bottleneck is planning, not execution.
We designed 30 tasks that a well-calibrated enterprise AI agent should simply refuse—requests involving policy violations, missing permissions, or unavailable resources. The highest-performing model correctly identified these as infeasible only about half the time. The failures weren't benign: They frequently resulted in unintended system changes.
In enterprise environments, where policy compliance and data integrity are nonnegotiable, an AI agent that can't reliably say no isn't production ready. This is precisely why purpose-built enterprise AI platforms invest heavily in guardrails and safety layers.
The ways agents broke down across ITSM, CSM, and HR workflows weren't random; they were structurally predictable. We saw referential integrity violations in HR, service-level agreement mismanagement in ITSM, and entitlement verification failures in CSM.
These failures trace directly to the business rules that enterprise orchestration platforms are explicitly built to enforce. They're not gaps in model intelligence alone; they're gaps that platform-level context and governance exist to close.
NOWAI-Bench measures raw model capability, what AI agents can do before any platform support is applied. ServiceNow AI Agents, including Now Assist, combine model intelligence with the workflow orchestration of the ServiceNow AI Platform. This entails domain-specific guardrails built from billions of enterprise workflow executions, as well as human-in-the-loop escalations for high-stakes decisions.
In other words, the benchmark quantifies the gap that platforms are built to close. The finding that expert planning improves agent performance by up to 35% directly validates why that platform layer matters.
ServiceNow’s semantic layer that powers the company’s own generative AI tools spans workflow intelligence, knowledge graphs, asset graphs, and access controls. It provides exactly the structured context that raw models lack. It’s the prerequisite for making enterprise AI reliable at scale.
NOWAI-Bench is also designed to benefit the broader AI ecosystem. By establishing a rigorous, enterprise-grounded evaluation standard, it gives model developers a concrete research roadmap and gives enterprise buyers an objective basis for evaluating AI agent capabilities. ServiceNow is using these same insights to guide its own approach to model evaluation and deployment.
As part of ServiceNow AI Research's broader enterprise AI evaluation initiative, NOWAI-Bench will scale from hundreds of workflow scenarios to thousands of compositional, cross-domain tasks. Multi-agent coordination and voice/multimodal evaluation, including speech-to-action workflows and document understanding, are also on the roadmap as enterprise AI moves well beyond text in, text out.
Core benchmark tasks and the evaluation environment will be released publicly to support reproducibility and community contribution, and enterprise-grounded datasets drawn from real-world workflow patterns will be available through controlled research partnerships. The goal is straightforward: to give enterprises a trusted, transparent way to evaluate AI agents against the operational realities that actually matter.
NOWAI-Bench is the result of deep collaboration across ServiceNow. We want to thank the ServiceNow applied AI team (Shiva Malay, Shravan Nayak, Jishnu Nair, Sagar Davasam, Aman Tiwari, Sridhar Nemala, Srinivas Sunkara, and Sai Rajeswar) for the scientific rigor, benchmark design, and evaluation infrastructure that spearheaded this work.
Equal credit goes to the AI Foundations product and engineering teams (Ravi Krishnamurthy, Ganpathy Krishnan, Joyce Li, Raahul Srinivasan, and Nitin Aggarwal), whose real-world perspective on enterprise workflows, model lifecycle management, and responsible AI deployment helped ground this benchmark in operational realities that matter most to our customers.
This is just the beginning. We look forward to working with the research community to push enterprise AI evaluation forward together.
Find out more about ServiceNow AI Research.
References
