How ServiceNow protects high-risk AI systems

AI is transforming every industry and reshaping critical functions such as human resources. As AI's influence grows, so do the stakes.

When AI systems fail, the consequences aren’t theoretical. They’re real, personal, and often irreversible. That’s why ServiceNow takes a fundamentally different approach to testing AI—especially when the risk of harm is high.

Our high-risk AI testing strategy is designed for trust, not just for compliance. It’s built to catch what others miss and to adapt as technology—and regulation—evolves.

Defining high-risk AI

At ServiceNow, we define high-risk AI systems as those with the potential to cause significant legal, ethical, or reputational harm. This includes use cases flagged in regulatory frameworks such as the EU AI Act: biometric surveillance or public benefits decisions.

It also includes internal assessments—cases where AI outcomes may threaten safety, fairness, or data privacy. These systems are held to a higher bar, and for good reason.

Testing for real-world impact

Traditional AI testing often focuses on average accuracy. But even a system with 95% accuracy can fail in critical, life-changing ways, such as misdiagnosing a serious illness or wrongly denying a loan application.

ServiceNow’s approach focuses on three questions:

• What could go wrong? We identify rare but high-impact failure scenarios, such as excluding a qualified job candidate or leaking personal data.
• How bad could it be? We measure not just whether errors occur, but also how harmful they are.
• How likely is it? We estimate the real-world frequency of these edge cases, because even rare events deserve attention.

Built-in compliance with global standards

Our process does more than meet technical goals. It aligns with top global frameworks, including the EU AI Act and the NIST AI Risk Management Framework.

Here’s how we stay in sync:

This results in traceable, transparent, and review-ready AI systems from development through deployment.

How the testing process works

Once a system is labeled high risk, it goes through a multistep testing lifecycle:

1. Risk discovery and scenario mapping: We identify how the system could fail—whether due to adversarial prompts, misuse, integration issues, or model drift.
2. Targeted testing under stress: We test for both common use and edge cases:
   1. Input manipulation
   2. Subgroup-level accuracy
   3. Failures in safety features such as hallucination defense or data protection
3. Consequence-first error analysis: We evaluate errors based on their impact, not just frequency—prioritizing those with the most serious consequences.
4. Continuous remediation and review: When failures are found, they’re fixed, retested, and resubmitted for approval. Review cycles are structured and ongoing—not one-and-done.

What sets our testing apart

Many companies stop testing once a model ships. ServiceNow doesn’t. Our AI testing process runs through the entire lifecycle.

Re-review is triggered when:

• Models are updated or retrained
• Use cases expand to new geographies or user groups
• Regulations shift or new risks emerge

This ensures AI systems remain within risk tolerance as they evolve—not just on day 1.

Additionally, ownership is shared. Risk is tracked by engineering, as well as across product and compliance teams, with clear audit trails of every review.

Trustworthy AI, built to last

We test for what really matters: whether an AI system performs well on average and whether it fails at critical junctures. Our methodology includes:

• Consequence-weighted error metrics
• Subgroup-level analysis
• Retrieval-augmented generation (RAG)-specific tests for hallucinations and data exposure
• Adversarial prompt testing to defend against misuse

AI regulations are tightening and expectations are rising. Businesses that deploy high-risk AI systems can’t afford to test lightly—and users won’t accept systems that fail silently.

By building safety into every stage of development, ServiceNow delivers more than compliant AI. We deliver trustworthy AI.

Find out more about ServiceNow’s approach to responsible AI deployment.