Grading AI Coding Agents on ServiceNow, Part 1: ATF Tests That Fail Before the Fix

Sreeram Nair · 10m ago

The grading method is only as good as its tests, so I started this as a series and this series starts there: how to turn a closed story into an ATF test that fails before the fix and passes after it.

In Before You Pick a Coding Agent for ServiceNow, Run Your Own Mini-Benchmark, I suggested adapting the method from Benchmarking Coding Agents on the Now Platform Codebase to platform work. The Touchstone team kindly reviewed it and pointed out three ways it could break: tests that depend on details the task never gives, fixes that leak through instance history, and samples too small to rank agents.

This part deals with the first two. The plan for the rest of the series:

Part 1 (this post): writing tests that fail first, checked against the task text, with the answer sealed away.
Part 2: running agents fairly, with a resettable starting point, three attempts and paired comparisons.
Part 3: scoring cost, consistency and platform hygiene, and what the results mean.

Why the test must fail first

A test that already passes on the starting state measures nothing, because any agent will pass it, including one that changed nothing at all.

Touchstone only accepts a task when its tests fail on the code before the fix and pass with the merged change. The same rule works for ATF. Before an agent sees a task, you need two runs on record:

Starting state: the test fails, for the reason the story describes.
Your team's real fix applied: the test passes.

If you can't get both, the task isn't ready. Either the test isn't checking the behaviour the story asked for, or the starting state already contains part of the fix.

Choosing stories

Good tasks describe a behaviour a user can observe, so a test can check it without knowing how it was built.

Good candidates	Skip these
A field becomes mandatory, read-only or hidden under a condition	Pure refactors with no visible change
A business rule sets or calculates a value	Performance work, unless you can measure it reliably
An ACL lets one group see or edit something another can't	Stories that depend on an external system
A record producer or flow creates a record with the right values	UI layout and styling changes
A defect with clear steps to reproduce	Stories whose description was written after the fix

Aim for a mix of work types. If every task is a script fix, you'll only learn which agent is best at script fixes.

The last row of the skip list matters most. Use only the text written before the fix: the description and steps to reproduce. Leave out work notes, acceptance criteria added later, and anything that hints at the solution.

Worked example: from story to test

The test should check what the story asked for in the user's terms, not how your team happened to build it. Here's an illustrative story (not from a real backlog) to show the steps.

The story, as written before the fix:

As a finance approver, I need the Budget field on a budget request to be locked once the request is Approved, so it can't be changed after sign-off.

The ATF test, step by step:

Impersonate a user with the requester role, not an admin.
Open an Existing Record: a budget request in the Approved state, created by the test's setup steps.
Field State Validation: Budget is read-only.
Open an Existing Record: a request in the Draft state.
Field State Validation: Budget is editable, so the fix hasn't simply locked the field everywhere.

Step 5 matters as much as step 3. A test that only checks the Approved case would pass an agent that made Budget read-only for everyone.

Decide whether the form is enough. "Can't be changed" could mean on the form or anywhere. A UI policy passes a form-only test, but someone could still change the value through a list edit, an import or the API. If the story means the data must be protected, add a server-side check, for example a Run Server Side Script step that tries the update with GlideRecordSecure and expects it to fail. Then a UI-only fix fails, which is what you want.

Prove it: fail, then pass

Run the test twice and keep both results: once on the starting state, once with your team's actual fix applied.

Build the starting state. Use the app as it was before the fix: the previous app version from source control, or the update sets before the fixing one. Don't clone production, for the reasons in the sealing section below.
Run the test. It should fail, and fail at step 3 for the right reason. If it fails at step 1 because the test user or data is wrong, fix the test first.
Apply the real fix and run it again. It should pass every step.
Run it a few more times. If the result changes between runs without any change to the instance, the test is flaky. Fix the timing or data setup, or drop the task. A flaky test makes every agent look randomly better or worse.

Record both runs alongside the task. They're your evidence that the task is fair, and the first thing anyone reviewing your results will ask for.

Check the test against the task

Every name and value the test depends on must appear in the task text or already exist on the starting instance. Otherwise no agent can pass, however good it is.

The Touchstone team called this the most common way a task turns out to be unsolvable, and they check every task this way before a release. In practice, list everything the test touches and tick each one off:

What the test uses	Example	Where the agent can find it
Tables	Budget request	Exists on the starting instance
Fields	Budget, State	Exists on the starting instance
Values	Approved, Draft	Existing choice values
Roles	Requester role	Exists on the starting instance
New names the fix created	None in this example	Must be in the task text

The last row is where tasks break. If your team's fix added a new field called u_locked_reason and the test checks it, an agent can't guess that name. Either rewrite the test to check behaviour without the name, or add the name to the task text as a requirement.

Seal the answer

An instance remembers a lot about its own history, so an agent with broad access can find the fix instead of writing it.

The Touchstone team seals each task in a snapshot with no history and runs the agent in a sandbox. On the platform, these are the places the answer can leak from:

Where the fix can leak	Why	How to close it
Update history and versions	A cloned instance keeps the fix's record versions	Build the starting state from source or old update sets, not a clone
Source control history	The fix's commit is in the repository log	Give the agent a fresh repository without that history
The story and its work notes	Notes and comments often describe the solution	Pass the task text in the prompt; deny the agent's account access to the story
Your ATF tests	The test sits on the instance and shows what passes	Load tests only after the agent finishes, or deny the agent read access to them
Audit and journal records	They can show old values and who changed them	Start from a fresh instance with no history of the fix

The ATF row is easy to miss. If the agent can read the test, it can write code to satisfy the test rather than the story. Hold the tests back until grading, and revert any changes the agent makes to them before you run them.

Checklist and what's next

A task is ready for an agent when every box below is ticked.

☐ The task text was written before the fix, with no hints at the solution
☐ The test checks behaviour, including a case where nothing should change
☐ The test fails on the starting state, for the right reason
☐ The test passes with the real fix, every time
☐ Every name and value the test uses is in the task text or on the starting instance
☐ The starting state has no history of the fix
☐ The agent can't read the story, its work notes or the tests

Part 2 will cover running the agents: a resettable starting point, three attempts per task, and why comparing results task by task beats comparing two overall percentages.

If you try this, I'd love to hear how it goes. Which kinds of stories were hardest to turn into fail-first tests? I suspect flows and UI Builder changes will be trickier than business rules and ACLs, and I'd welcome being proved wrong.