From Build to Tested in One Conversation: Introducing ATF Test Agent

[Oluyinkaoginni]

If you have ever finished building something in Build Agent and then had to stop, switch tools, and write the tests for it separately, you know the friction I am talking about. During a recent Platform Academy session on ATF Test Agent, my colleague Ravi Mantrala, our inbound product manager for the product, and I walked through what changes with the Australia release: testing stops being a separate errand and becomes part of the same conversation where you build.

The testing bottleneck we all know too well

This session picked up where our March 31 Agentic ATF session left off, which introduced ATF Code Coverage (line-by-line visibility into what tests actually exercise) and the ATF Test Troubleshooting Agent (root-cause analysis compressed into a short, actionable report). Both are generally available today; the recording and blog post from that session are linked in the resources at the end of this one.

Poll 1 showed most of the room has not made the jump to Now Assist for Test Generation: 61% had not looked at it yet, 33% had seen a demo but not used it, and only a couple had used it hands-on. Of that small group, poll 2 was candid: 70% said it addressed 10% or less of their use cases. That is the gap the Test Agent closes. Test generation used to be a one-shot prompt, ask, and if it was not right, start over. The Test Agent understands intent across a conversation instead.

The underlying pain points are ones every testing-heavy team recognizes. Authoring a single, carefully assembled ATF test takes roughly 20 minutes. Troubleshooting one failing test typically eats 2 to 3 hours. Multiply either number by a few hundred tests, and coverage and quality start trading off against velocity, because most teams cannot justify the headcount to keep pace.

What changed with Australia: three pillars, one conversation

The ATF Test Agent lives inside Build Agent, in the App Engine Studio conversation you already use to build. It closes the loop the Troubleshooting Agent started: authoring and orchestrated execution now land directly in the agent alongside the autonomous troubleshooting that already shipped.

Authoring generates comprehensive tests from a single natural-language prompt. Because the agent shares Build Agent's full application context, it reads the code and intended logic and produces tests across every classification path and business rule, without you reading the codebase yourself first. Generated tests land in the standard sys_atf_test table, so existing ownership, naming, and review workflows carry over unchanged.

Execution runs directly from the agent through Cloud Runner, no separate runner setup, no module navigation. You get structured pass and fail results in the same conversation, with failing tests already identified.

Troubleshooting and maintenance keeps the loop closed. When a test fails, the agent reads the logs, identifies root cause, applies a fix at the right level (sometimes the test, sometimes the application code), and reruns until the suite is clean. Human review remains the gate before production reliance; the agent does the heavy lifting, but your team stays in the judgment seat.

The Test Agent requires a Creator Pro Plus subscription, with App Engine Studio and Build Agent activated. It's generally available from the Australia release, and the unified panel we demoed shipped this month with Australia Patch 2, or backported to Zurich Patch 9. Poll 3 showed where the room stands on that entitlement: 66% are still evaluating which access path makes sense, 19% are exploring on a personal developer instance, 11% already have full Creator Pro Plus entitlement in production, and a smaller group is on a Build Agent trial.

As I said on the session, the Test Agent is not there to replace your team's judgment about what a successful test looks like. It is there to relieve tension.

"The ATF Test Agent doesn't replace anything you already have. It unifies the pieces into a single flow inside the conversation you're already using."

Inside the demo: a coffee app, one prompt, ten tests

Ravi ran the live demo on a small custom-scoped app he built with Build Agent: a coffee ordering catalog item, with a business rule that makes an employee's first black coffee of the day free and charges $3 after that. Simple on the surface, but with the same building blocks most of us work with daily: catalog items, background business rules, approvals. This is not limited to custom scopes; it works in global or store scopes too, and on existing applications already on your instance, as long as the underlying metadata is understood by Fluent, ServiceNow's proprietary backend language.

With one prompt, "write me ATF tests for all of the feature permutations for this app," the agent generated ten tests covering the pricing logic for each coffee type, the free-first-order rule, and the order email notification, all saved in the standard ATF module like any test you would build by hand. A follow-up prompt executed all ten through Cloud Runner in the same conversation, and when one test failed, the agent triaged it, identified the cause, and fixed it, without a context switch. One caveat: today the agent does not automatically keep tests current as an application evolves, you have to ask it to update affected tests after a change, though Ravi confirmed smarter, incremental updates are on the near-term roadmap.

Poll 4 asked which pillar would unlock the most value first, and the answer was decisive: 67% chose authoring, getting tests written without burning developer hours, well ahead of troubleshooting at 23% and execution at 10%. That tracks with what we hear in customer conversations: the upfront cost of writing a comprehensive test suite is still the biggest lever teams want pulled.

Questions worth answering before you pilot

On the cost of Cloud Runner. The Test Generator on Cloud Runner app is a free store app that includes two Cloud Runner lanes at no charge; parallel execution across more lanes requires purchasing additional capacity.

On data protection. An attendee asked about alternatives to Cloud Runner for protecting their data. For cloud customers, Cloud Runner is ServiceNow's own infrastructure, not a third party, so test data is held to the same trust model as the rest of your instance. I shared a knowledge base article on Cloud Runner data handling in the session chat for anyone who wants it in writing.

On testing agentic workflows. One attendee asked whether the Test Agent can test agentic workflows, such as Now Assist skills or AI agents. It's built for apps and custom flows, not for validating an agent's own decision-making. Sharon (our awesome Platform Academy Host) drew a useful distinction: Agent Evaluations is the right tool while developing and validating whether an agentic workflow behaves as expected; once live, ATF is well suited to checking the expected, consistent outcomes it produces on your records over time.

On moving ATF update sets to production. My guidance has not changed: move tests to production for storage only, not execution, since running tests live risks side effects like real notifications firing. Deactivate tests before promoting them, and reactivate only after cloning down to sub-production.

What comes next

Poll 5 asked what attendees planned to do next: 33% will wait and revisit after a future release, 28% will socialize it with QA and dev leads first, 25% will pilot it on a bounded-scope application, and 14% are ready to activate the trial now. If you're in the pilot or evaluate camp: confirm your Creator Pro Plus entitlement and that App Engine Studio and Build Agent are active, pick one bounded-scope application for your first end-to-end cycle, schedule a human review pass on the generated tests, then roll out to broader teams. Starting small builds internal trust quickly without outpacing your QA review capacity.

Testing will probably never be the part of the job people wake up excited to do. What the ATF Test Agent changes is how much of that work has to happen outside the flow you are already in.

"With natural language, you're only limited by your imagination, so let your imagination drive you to where the future is."

Watch the Full Session

The complete Platform Academy session, including the full live demo on Build Agent and the Q&A, is available on YouTube. If you're weighing whether to pilot ATF Test Agent on your own instance, watching the demo first will save you time.

Watch on YouTube

Have questions about your pilot, or a testing challenge you want to work through? Drop it in the comments below. I read every one.