AI-Assisted ServiceNow Investigation - Using Copilot as a Virtual Analyst With Human Hands

andersonk17474 · 2 hours ago

Note: Portions of this document were authored using AI

I find myself leaning more and more into MS Copilot, trying to push just how semi-autonomous I can get this simple chat interface through a web page to behave. In my latest experiment, on a somewhat groggy Friday morning I experimented with an approach where AI acted as the primary analyst while a human operator served only as the execution mechanism.

The setup was as follows. I receive a notice of a bug in a catalog item process. Nothing serious, but customer facing issue around a notification needing a quick resolution. I have been using the AI tooling like MS Copilot extensively as of late, having it perform significant data analysis. My thought is, can it find the root of this bug with minimal input from me? I hand it the ticket, give it some platform background documents I have assembled, arm it with the idea of its only interface being a background script runner in sub-prod, and tell it to investigate the issue.

The AI had no direct access to the ServiceNow instance, and worked through this loop:

- Problem Statement

- AI Generates Investigation Script

- Human Reviews and Executes Script

- Human Returns Output back to AI Session

- AI Analyzes Results of Script Execution

- AI Generates Next Investigation Step

This loop had the AI performing as a remote ServiceNow developer, investigating the issue through scripted data exports and hypothesis testing based on the data sets returned. The human role for this experiment was intentionally minimized to the following actions:

- Run Script

- Copy Output

- Paste Output

- Repeat

One caveat to this process was that on some occasions the human needed to provide visual confirmation for UI elements this approach could not fully investigate. The AI was instructed to handle the research in step-by-step manner and never present multiple processing steps at once. Every AI interaction was focused on a singular activity per interaction, which includes phases such as:

Investigation Strategy -> Hypothesis Generation -> Root Causes Analysis -> Process Documentation -> Design Recommendations

The first activity was inventory. The AI first authored scripts that started from the target catalog item and exposed catalog item form contents, related flow, and the flow components. From there the AI begin by using the CartJS API and the ideas in the RITM Printer (https://www.servicenow.com/community/service-catalog-forum/catalog-item-testing-ritm-printer/m-p/359...) to replay existing requests for the target catalog item to observe current behavior. The AI analyzed approvals, tasks, related events, and the notifications entirely using small investigation scripts run against the records produced by the submission of the catalog item to fully understand the business process behavior, and how it deviated from the expectation documented in the bug report that started this whole process off.

This analysis eventually revealed a related business rule associated with the reported issue. The AI proposed several hypothesis to test against the business rule, and then crafted test scripts that were run successively to validate the assumption and identify where the issue resided with the notification issue. Several approaches were identified, a few of the hypothesis were deemed inaccurate and thrown out. The testing revealed that the bug did not reside in the business rule, but a little higher up-stream in the flow that was creating elements the business rule fired from. At this point, some human intervention was required, as the AI could not "see" the flow in full as I could through the designer interface. I provided a copy-paste of the flow contents from the designer UI, along with some configuration details observed in a few of the steps that scripting could not dig into, and the AI proposed a new hypothesis of the root cause, which we proceeded to test using the RITM replay technique. The issue was ultimately identified and a minor code fix in the flow if-else conditions resolved the issue. While this bug fix was not technically challenging, and the root cause possibly could have been identified by a human developer in less time, the concept of having the AI drive the analysis, and successfully find the root cause is the more interesting aspect of this experiment.

The most interesting aspect of the exercise for me, was that the AI never had direct platform access. Every conclusion that was proposed by the AI derived from:

- Run Script

- Copy Output

- Analyze

With that simple process the AI successfully:

- Identified existing implementation artifacts
- Reconstructed process architecture
- Generated test transactions
- Executed controlled experiments
- Eliminated incorrect hypotheses
- Isolated the likely root cause

I served as the hands for the AI, taking the generated investigation scripts, reviewing them for intent, running the script against the environment, then copying the results back to the AI for analysis. Minimal thinking for the bulk of this process on my part outside of a safety check on the scripts that I was executing on the AI's behalf, and some minor visual confirmation where the AI scripts could not see into easily. Each script answered one question. Example concepts that the scripts were validating:

- Does a notification exist?

- Does an event exist?

- Was the event generated?

- Did the task get created?

- Did the condition evaluate true?

The investigation only reached the correct conclusion after executing a live request and observing actual system behavior. The strongest contribution from the AI was not code generation. Instead it was continuously identifying the most valuable next question, which in turn led to each answer influencing the next script.

Looking back on the exercise, the most valuable contribution from the AI was not script generation. Most of the scripts were relatively small and straightforward. The more important capability was maintaining a disciplined investigative approach. Rather than jumping between theories, the AI consistently focused on identifying the next piece of evidence needed to eliminate uncertainty and then authored the smallest script necessary to obtain that evidence.

As ServiceNow continues expanding support for tools like CLI, SDK, Fluent, and Source Control, more platform artifacts will become machine readable. As that happens, the boundary between AI Analyst and AI Developer will continue shrinking. For the time being, the process outlined above proved remarkably effective at performing an investigation into ServiceNow process and helping identify the source of the behavior gap in the process. I look forward to attempting this model of AI driven analysis on additional issues as they arise to see where the limitations are and how this may help augment some of our tier 1 and tier 2 triage on incoming issues.