"49 Tickets by Morning": How We Solved the Mystery of Discovery Overwriting Our Maintenance Status

Selva Arun · ‎06-27-2025

Tags: Discovery, CMDB, Install Status, Operational Status, Change Management, WLC, Best Practice

Hello fellow ServiceNow practitioners,

Have you ever come into work to find your queue flooded with unexpected tickets? We did. I want to share a real-world story about how our NOC team received 49 automated monitoring tickets overnight, and how tracing them back led to a critical discovery about ServiceNow, Wireless LAN Controllers, and our Change Management process.

This is the story of the "flipping Install Status" and how we finally solved it.

The Problem: A Flood of Unexpected Alerts

It started on a Friday morning. I was contacted by our Network Operations Center (NOC). Their monitoring system, which relies on the CMDB, had generated 49 tickets for devices that were supposed to be in a maintenance window.

The Trigger: In every case, a device that was manually set to "In Maintenance" in the CMDB was automatically changed back to "Installed." This change removed the maintenance flag from our monitoring tools, causing them to generate false-positive alerts when the devices were being rebooted.
The Culprit: The audit history for every single CI pointed to the same user: SNC MIDServer. The updates all happened around 1:32 AM.

The business impact was clear: wasted time for our NOC, alert fatigue, and a loss of trust in our maintenance process. I was tasked with finding out why Discovery was "misbehaving" and overriding our approved changes.

The Investigation: Chasing Ghosts in the Logs

My first instinct was to find the discovery log for one of the affected CIs, a Wireless Access Point (WAP). I was sure I'd find a simple, direct scan that caused the update.

Discovery Status History: I filtered for the WAP's own IP address for that day. Result: Nothing.
ECC Queue: I searched for any input records from the WAP's IP address on that date. Result: Nothing.

This was the core of the mystery. The audit log proved the MID Server updated the CI, but the discovery logs showed it was never directly scanned. How was this possible?

The "Aha!" Moment: It's Not the Device, It's the Controller!

The breakthrough came when we remembered the CI's class: Wireless Access Point.

In an enterprise environment, individual WAPs are managed devices. They report their status to a central Wireless LAN Controller (WLC). ServiceNow Discovery's out-of-the-box patterns are smart enough to use this architecture for efficiency.

The actual discovery process was:

A nightly discovery schedule targeted the IP address of the WLC, not the individual APs.
The powerful "IP Switch" pattern ran against this controller.
This pattern queried the WLC for its full inventory of managed devices.
The WLC then provided a large list of all its active APs, including our 49 affected devices.
The pattern looped through this list and updated each individual WAP CI in the CMDB based on the data it received from the controller.

We confirmed this by examining the raw discovery payload from the WLC's scan. It showed the controller as the primary device but then listed all our "problem" CIs as related items to be updated in the same transaction.

The mystery was solved: The updates were triggered by the successful scan of the WLC, which then updated all its child CIs. The entire transaction was logged against the WLC's IP, not the APs' IPs.

The Root Cause: Using the Wrong Field for Maintenance

This investigation proved the real issue was a process-to-platform misalignment.

The question was no longer "Why is Discovery breaking?" but "Why does Discovery feel the need to change the status at all?"

The answer lies in the official ServiceNow documentation for the two main status fields:

Install Status (install_status): This field is designed to track the physical asset lifecycle. It answers the question, "Is the device online and reachable?" Discovery's core function is to keep this field accurate based on what it sees in the real world. If it can talk to a device (or its controller), that device is considered "Installed."
Operational Status (operational_status): This field is designed to track the business or service lifecycle. It answers the question, "Is this device in use for production?" The out-of-the-box Discovery patterns do not modify this field.

Our organization was using the Install Status field to track a business process (maintenance), creating a direct conflict with Discovery's fundamental design.

The Solution: Aligning Our Process with the Platform

The permanent solution is to adapt our processes to the platform's intended design.

1. The Best Practice: Use Operational Status

Moving forward, our Change Management process will be updated to use the Operational Status field to track maintenance states.

When a CI goes into maintenance, the Operational Status will be set to In Maintenance.
Discovery can continue its nightly scans and will correctly set the Install Status to Installed (because the AP is online), but it will not touch our manually set Operational Status.
Both states will now coexist accurately in the CMDB, our monitoring tools will get the right data, and the false-positive tickets will stop.

2. The Short-Term Fix

To give our teams time to adapt and prevent any further incorrect alerts, we will be temporarily excluding our wireless controllers from the discovery schedules for the next couple of days. This will stop the immediate problem while we communicate the permanent process change to all teams. Once the new process is understood, we will re-enable the scans.

Key Takeaways for the Community

If a CI is updated by Discovery but you can't find logs for its IP, look for a controller (WLC, vCenter, Storage Array, etc.). The log will be under the controller's IP.
The Install Status field is owned by Discovery and reflects the physical state. Don't use it for manual process tracking.
The Operational Status field is owned by you and your business processes. It is the correct place for states like "In Maintenance," "Testing," or "Awaiting Repair."
Don't customize core Discovery patterns. It's always better and more sustainable to align your processes with the platform's out-of-the-box design.

I hope this detailed breakdown of our investigation and resolution helps anyone else who gets that dreaded call from the NOC about a flood of false-positive tickets!

If you believe the solution provided has adequately addressed your query, could you please **mark it as 'Helpful'**. This will help other community members who might have the same question find the answer more easily.

Thank you for your consideration and Happy discovering!

Selva Arun
ServiceNow Rising Star 2024

Sangeeta Das · ‎06-27-2025

This is such a helpful articl about Discovery and overall maintenance of CMDB. Thank you so much sharing such examples.

In our org we have a use case where for years business has been using Operational Status OR Install Status for different CI classes. On one occasion when we wanted to import import Cisco audio - video device data into ServiceNow, ( via manual import, not Discovery) it was zeroed on

using IP Phone CI class.
Based on that, when we started to peform rest of the data modelling, we found that IP Phone CI Table does not have OOB Hardware Status/ Operational Status, but it has Install Status
Plus, we also wanted simultenuous asset to be created and updated based on CIs.
To minimse customization and leverage OOB sync between Asset and CI Install Status field mapping it we are thinking of utlising Install Status field.
- Thankfully because these devices are not using Discovery method perhaps we are saved from the issue that you have encountered. ServiceNow also recommended us to enable CDSM lifecyle stage and status, but that is again a big project in itself.

Excellent explanation how process gaps can create/impair CMDB data accuracy.

Selva Arun · ‎06-27-2025

@Sangeeta Das

Thank you so much for your kind words and for sharing your experience.

Regards,

Selva Arun