New ATF Failure Insight Feature in Zurich Shows You Exactly Why Your Tests Are Failing

[Earl Duque]

If you're a developer or quality engineer who maintains Automated Test Framework (ATF) suites, you've been there: a regression test that's been passing for months suddenly fails. Your first step is to start tracing through steps and configuration, trying to figure out what tiny environmental or metadata change threw a wrench into your previously successful test. It can be a massive time sink!

The good news is that the ServiceNow Zurich release introduces a feature specifically designed to solve this problem: ATF Failure Insight.

What Does Failure Insight Do?

Failure Insight is a tool to help you maintain your ATF tests by giving you a clear window into the metadata changes that happened between your last successful run and the current failed one.

Instead of guessing, you get a direct answer. It essentially gives you a "diff" for your test's environment and dependencies.

Key Features and How It Saves You Time

1. Metadata Comparison: The core feature is the ability to see side-by-side file comparison of the metadata. For example, if a test fails on a "Set Field Values" step, the insight may show you that the impersonated user's role was changed or that an underlying field value in the form was updated. The tool tells you what changed, allowing you to focus on the fix.

2. Two-Click Debugging: From a failed test result, you simply click the link "Find changes since the last successful run" to launch the insight tool. It’s that fast.

3. Built-in Workflow for Resolution: Found the issue but you're an admin and not the developer who needs to fix the test? No problem. Failure Insight includes a "Create Task" button. By default, this will create an Incident task (though this is configurable to a Problem or any other task-derived table) and pre-populate it with key information about the failure, allowing you to easily assign the resolution to the correct team member.

Demo Deep Dive

During the stream, we saw two great examples of how this works:

• Case 1: Impersonation Failure. The test was failing on a step that sets field values. The Failure Insight showed that the user being impersonated was changed from a user with the correct roles (the ATF user) to one without them (Able Tutor), causing the form access to fail. The true cause of the failure was quickly identified, even though the error message pointed to a later step.

• Case 2: Field Value Change. A test was set to validate a field value. The insight clearly showed that the field value for "Urgency" had been changed from "2-Medium" to "3-Low" since the last successful run, making the test's expected outcome incorrect.

A Look Ahead: Full Root Cause Analysis

While Failure Insight is already a huge time-saver, the team gave a preview of what's next: an upcoming agent that will provide a complete Root Cause Analysis (RCA). This agent will look beyond just metadata, analyzing server logs, client logs, and snapshot changes to give you the precise reason for the failure and a list of suggested steps to resolve the issue.

This is an evolution in test maintenance, moving from simple failure reporting to proactive resolution guidance.