Before You Pick a Coding Agent for ServiceNow, Run Your Own Mini-Benchmark

[Sreeram Nair]

The best way to choose a coding agent for ServiceNow work is to test it on your own backlog, and you can do that in a couple of weeks with tools you already have.

Why I wrote this

The recent Benchmarking Coding Agents on the Now Platform Codebase post is one of the most honest benchmark write-ups I've read. It reported cost next to solve rate, ran three attempts per task, and showed that seven of the top cells were effectively tied. It also found that the cheapest-per-token model was not always the cheapest per task, and that the repository's own agent instructions made runs slower without making them better.

There's one gap for most of us, though. Those 200 tasks came from ServiceNow's internal Java, TypeScript, Python and Go repositories. Most customers and partners don't write that kind of code. We write scoped apps, script includes, business rules, client scripts, flows and UI Builder pages. This post is my attempt to translate the method into something a platform team can run on its own instance.

Why platform work needs its own test

An agent that's good at fixing a Java bug isn't automatically good at building on the Now Platform, because the failure modes are different.

In a normal repository, a wrong answer usually means failing tests or a broken build. On the platform, an agent can produce code that runs fine and is still wrong for reasons a test won't catch:

• Scope. Records created in Global instead of your app's scope, or cross-scope calls that need access settings nobody approved.
• Server vs client. GlideRecord in a client script, synchronous GlideAjax, or logic in the wrong layer.
• Metadata, not files. A fix that needs a dictionary change, an ACL or a UI policy, not just script. Agents that only edit script files miss these.
• Update set hygiene. Changes that don't get captured, or that drag in unrelated records.
• Upgrade safety. Modifying OOB records instead of extending them, which creates skipped records at the next upgrade.

None of these show up in a simple pass/fail score, so your benchmark needs to look for them on purpose.

Step 1: Build tasks from your own closed work

Your best tasks are stories and defects your team has already closed, because you know the right answer and the agent can't have seen it.

Touchstone mined merged pull requests and quoted the original ticket text verbatim. The platform equivalent is your closed stories and defects in Agile Development or your backlog tool, paired with the update set or app version that fixed them.

1. Pick 20 to 40 closed items from the last few months. Fewer than 20 and the noise will swamp any difference between agents.
2. Use only the text written before the fix: the description and the steps to reproduce. Leave out work notes, acceptance criteria written later, and anything that hints at the solution.
3. Mix the work types: script fixes, new business rules, ACL changes, UI policies, a flow change, a UI Builder tweak. If every task is a script fix, you'll only learn which agent is best at script fixes.
4. For each task, record the app version or update set state from before the fix. That's the starting point every agent gets.
5. Retire a task once it's been used and discussed publicly. Don't edit it to make it work.

If you're a partner, a sanitised set of tasks from a customer's backlog (with permission) is worth more than any public benchmark, because it's the work you actually get paid to do.

Step 2: Grade with ATF, not opinions

A task should pass only when an Automated Test Framework test you wrote before the run passes. No LLM judge, and no "looks right to me".

This was the part of Touchstone I liked most: the tests came from the original engineers, and agent edits to the tests were reverted before grading. You can copy that directly.

• Write the ATF test against the known fix. Confirm it fails on the starting state and passes with your team's actual fix. If it doesn't fail first, it's not testing anything.
• Hold the tests back. Don't give the agent the test, and roll back any change it makes to test records before you grade.
• Test behaviour, not implementation. Assert that the field is mandatory or that the record routes to the right group, not that a specific script exists. Agents will find different valid solutions.
• Add a hygiene check. After the ATF result, run a quick review for the platform issues in the section above: right scope, nothing modified OOB, everything captured. Score it separately so a passing but messy fix is visible.

That second score matters. In my experience, the difference between a useful agent and a risky one on the platform is rarely whether the test passed. It's what else changed.

Step 3: Run it fairly

Give every agent the same starting point, the same time limit and at least three attempts per task, then compare them task by task.

• Use a sub-production instance you can reset. Clone or roll back to the starting state before every attempt. An agent that inherits the previous agent's half-finished work isn't being tested fairly.
• Work from source where you can. The ServiceNow SDK lets you develop scoped apps as source in a local project, which most coding agents handle far better than clicking through forms. Note which tasks needed instance-side metadata changes the agent couldn't reach.
• Run three attempts per task. Touchstone found the three attempts agreed only 79% of the time across all cells. A single run will mislead you.
• Compare pairs on the same task. "Agent A solved task 12 and Agent B didn't" is more useful than two overall percentages. With 30 tasks, a gap of a few points is noise, so don't declare a winner on it.
• Fix the clock. Give every agent the same time limit, count timeouts as failures, and report how many there were.
• Run headless if that's how you'll use it. If the agent will work from a ticket in a pipeline, test it that way. If a developer will sit with it, test that instead. They're different jobs.

 

Step 4: Measure what you'll actually pay for

Track cost per solved task, how many model calls each attempt made, and how consistent the agent is. Solve rate alone won't tell you which agent to buy.

Measure Why it matters How to get it

Solve rate (ATF pass)	The headline number, but only meaningful with intervals	ATF results per attempt
Cost per attempt	Touchstone found per-attempt cost differed 5x between two models with similar token prices	Token usage from the agent's logs, at your contract rates
Model calls per attempt	The best predictor of cost in Touchstone, more than token price	Agent session logs
Solved on all 3 attempts	What you can rely on without a human re-running it	Count of tasks passing every attempt
Hygiene score	Catches passing fixes that would hurt you later	Your checklist from Step 2
Wall-clock time	Matters for developer experience and pipeline throughput	Session start and end times

The consistency column is the one I'd watch most closely. An agent that solves 60% of tasks every time is easier to build a process around than one that solves 65% but different tasks on each run.

Three findings worth testing on your own backlog

Touchstone produced three results that run against common assumptions. Each is cheap to check on your own tasks before you set team standards.

Your instructions file might be costing you money. Touchstone found that giving agents the repository's own CLAUDE.md and AGENTS.md files didn't change the solve rate but raised cost by up to 58% for one model. The files were written for people working interactively, so they told the agent to run long test cycles. For platform teams, it's worth having two versions: a full one for developers pairing with an agent, and a short one for headless runs that says which scope to use, what not to touch, and how to run only the relevant ATF suite.

Higher reasoning effort didn't pay. Raising effort above the vendor default cost 10% to 160% more and didn't reliably improve results. Start with defaults and only raise effort if your own paired comparison shows a gain.

The harness matters for some models and not others. For seven of the ten models tested under more than one harness, the harness made no measurable difference to solve rate, but cost moved a lot. Check harness settings too. Touchstone lost attempts to a workspace snapshot feature and to permission prompts that nobody answered in headless mode. Those are configuration problems, not model problems, and they'll show up in your results if you don't look for them.

A starter results sheet

One row per attempt is all you need, and you can paste this straight into a spreadsheet.

task_id,work_type,agent,harness,attempt,atf_pass,hygiene_issues,model_calls,cost_usd,minutes,timed_out
STRY0012345,business_rule,Agent A,Harness X,1,TRUE,0,18,0.42,6.5,FALSE
STRY0012345,business_rule,Agent A,Harness X,2,TRUE,1,22,0.51,7.1,FALSE
DEF0004567,acl,Agent B,Harness Y,1,FALSE,0,9,0.12,3.0,FALSE

The rows above are placeholders to show the format, not real results.

Over to you

You don't need 200 tasks to learn something useful. Thirty tasks, three attempts each and ATF as the grader will tell you more about your own work than any public leaderboard.

If you run something like this, I'd love to compare notes. Which work types did agents handle well on your instance, and where did they struggle? I'm especially curious about ACLs, flows and UI Builder, where I suspect results will differ most from code-only benchmarks. Share in the comments, even if your numbers are small.

• AI
• Automated Test Framework
• Scoped Applications
• Best Practices