MID Server post-upgrade recovery automation – supported SRE patterns and lessons learned?

[Jeff Boltz1]

Hi Community,

We are looking for experience and guidance from others operating ServiceNow MID Servers through a frequent platform patching cycle.

Our goal is to make MID Server upgrades and post-upgrade recovery as reliable and automated as reasonably possible, while avoiding automation that simply masks an underlying problem or performs unsupported recovery actions.

Situation

During a recent production platform patch, multiple MID Servers required remediation following the upgrade.

The conditions we observed included:

• MID Server encryption keys did not match / were no longer valid (UpdatePublicKey)

• File Discovery could not decrypt the file discovery whitelist after synchronization

• TrustStoreMigrationFailure during upgrade on our RHEL MID Servers

The affected MID Servers were subsequently remediated and returned to service. Post-recovery validation confirmed the MID estate was back UP and passing validation.

We do not have an active production outage. We're now looking at this as an SRE / continuous-improvement problem.

Why Automate?

We patch ServiceNow regularly. Having engineers monitor MID Server upgrades and manually intervene when recoverable conditions occur introduces recurring operational effort and cost.

Our preferred operating model would be:

Detect -> Classify -> Safely Remediate -> Validate Recovery -> Escalate if Unsuccessful

Importantly, we do NOT want to simply automate a manual rebuild/restore procedure.

We found KB0713557 particularly interesting because ServiceNow describes the manual MID Server restore/upgrade procedure as a last resort and recommends identifying the underlying cause rather than potentially repeating the same recovery on every upgrade.

That is exactly what we're trying to avoid.

Questions for the Community

1. Has anyone automated MID Server post-upgrade health detection and recovery?

2. Has anyone encountered recurring UpdatePublicKey / encryption-key mismatch conditions following MID Server upgrades? If so, did you identify the underlying cause rather than simply invalidate/revalidate?

3. Has anyone automated recovery from an encryption-key mismatch, or intentionally decided that this should remain human-gated?

4. Has anyone encountered "Could not decrypt file discovery whitelist after sync" as part of the same upgrade/recovery sequence?

5. For those who have encountered TrustStoreMigrationFailure during upgrades, were you able to eliminate the underlying cause or safely automate recovery?

6. What signals do you use to determine that a MID Server has genuinely recovered?

For example, we're interested in the distinction between:

• Service/process running

• MID reporting UP

• MID validated

• ECC processing restored

• No unresolved MID Server issues

• Functional workloads successfully executing

7. Has anyone built guardrails around automated MID recovery, such as limited recovery attempts followed by escalation rather than an indefinite remediation loop?

8. Are there MID Server recovery operations that experience has taught you should always remain human-gated?

9. Are you using Flow Designer, scheduled/background processing, external orchestration, MID lifecycle events, or another mechanism to orchestrate this?

What We're Trying to Build

We're considering a MID Server SRE pattern along these lines:

Detect failure

        ↓

Correlate with upgrade lifecycle

        ↓

Classify known condition

        ↓

Supported/safe to automate?

     ↙             ↘

   Yes              No

    ↓                ↓

Remediate          Escalate

    ↓

Validate health

    ↓

Recovered?

  ↙       ↘

Yes        No

 ↓          ↓

Close      Escalate

The critical design principle for us is that completing a remediation action should NOT itself constitute success. Recovery should be independently validated.

We have also opened ServiceNow Support case CS9859892 to get the official/supportability perspective. Here, we're especially interested in practical experience from other customers and practitioners operating MID Servers at scale.

Would appreciate hearing what has worked, what has not, and particularly any approaches you've found effective for reducing manual intervention without hiding genuine MID Server problems.

Thanks!