Interested in a ServiceNow event built for developers? Registration for now[dev]26 is officially open!

Monthly Patching and Weekly Security Patching

Chuck Nusbaum
Tera Expert

3 weeks ago

The new weekly security patching is throwing me for a bit of a loop with respect to monthly patching. Here's the scenario.

 

At the beginning of August, we decided we wanted to go to Zurich Patch 11m to fix a known issue. We upgraded non-prod instances to 11m and scheduled prod to go to 11m as well at the end of the month.

 

In the middle of the month, weekly security patches started being deployed, and 11m was determined to be a downgrade from Patch 10 Hotfix 4 w33, so our prod update to 11m got canceled.

 

Today, we got notification of the October patching target: Zurich Patch 10 Hotfix 4b.

 

I'd like to get our instances on the same patch level again...but I suspect that if we schedule to go to Patch 12, then weekly security patches are going to determine it as a downgrade and it will be canceled again.

 

Shall we just give that a try or does anyone have a different/better approach?

  • 1,389 Views
7 REPLIES 7

AutomatePro
Tera Contributor

3 weeks ago

Yes — scheduling straight to Patch 12 and hoping it clears is a gamble, and it's likely to fail the same way for the same reason. Here's why, and what I'd do instead.

 

ServiceNow's downgrade protection doesn't compare patch numbers; I suspect it is comparing the actual build/release timestamp of your current instance state against the build timestamp of the target. Weekly security patches are cumulative and ship every week. Named monthly patches (Patch 11, 12...) are built off an earlier branch point and take longer to certify. So it's entirely possible for "Patch 10 HF4 plus several weekly security patches" to have a later effective build date than "Patch 11," even though 11 is a higher number than 10. That's exactly what happened to your prod change.

 

This isn't likely to be a one-time issue and I'm sure you are not alone in experiencing this. Weekly patches will keep advancing every week regardless of what you target. If it took only until mid-August for 11m to be overtaken, there's a decent chance Patch 12 gets overtaken too before or during your scheduled window, especially if it lands weeks out.

 

What I'd do instead of just retrying with Patch 12

  1. Don't guess the patch number, check the build date first. Before scheduling anything, pull the current build date for prod from the Instance Dashboard in Now Support, and compare it against the target patch's release/build date. If the target's build predates your current one, it'll be blocked regardless of the number, so you can rule it out before burning a change window.

  2. Follow ServiceNow's own recommended target rather than picking a number yourself. The October target they've given you isn't "Patch 12," it's Patch 10 Hotfix 4b, i.e. ServiceNow itself is staying inside the Patch 10 hotfix train rather than jumping to a new named Patch, almost certainly for this exact reason. That's a strong signal the safer route to re-alignment is to bring prod to whatever the currently recommended compatible target is, rather than trying to leapfrog to 11 or 12 independently.

  3. Reconcile non-prod to match, rather than pushing prod ahead to catch non-prod. Since non-prod is sitting on 11m and that's now behind prod's effective build, the cleanest realignment might be pulling non-prod back onto the Patch 10 hotfix line (e.g. HF4b once available) with a clone back, rather than trying to force prod up to 11/12. That gets both environments onto the same actively maintained line instead of chasing a moving target.

  4. Raise it with your Now Support account team explicitly. This interaction between the new weekly security patch cadence and named monthly patches is a known friction point right now, worth asking them directly: what they recommend i.e. what target will actually be accepted as a same or later build for prod, given current weekly patch state?. They can see the build math from their side and may steer you to a target you can't easily see is safe.

I wouldn't schedule Patch 12 on spec. Verify the build date comparison (or get ServiceNow to confirm it) before you commit a change window, otherwise you're likely burning another cycle on the same failure mode.

 

Hope this helps.

Chuck Nusbaum
Tera Expert
In response to AutomatePro

3 weeks ago

Thank you for the reply!

 

I'm glad to know I'm not the only one finding the interplay between weekly security patches and monthly patches to be a source of friction...but I'll be darned if I could find anyone else posting or asking about it!

 

I contacted our account team, and they told us to open a Now Support case. I opened a Now Support case and was pretty much told the same information I already know. Hence why I'm here!

 

I understand your point about build/release timestamps. Cloning over our Dev instance could be a challenge given some of the development projects we currently have underway, but rolling back to the same patch level as Prod makes sense. I'll talk with our team about it.

 

Really appreciate your insight--thank you!

Kristin J
Mega Sage

3 weeks ago

Hello there, I've been digging around about this myself. It seems they are now distinguishing between quarterly patching (one full patch, 2 security patches) vs. weekly security hotfixes. I found a KB article (ServiceNow Patching Program FAQs) that seems to directly address what you're dealing with, specifically with a question "Can I patch from a security patch to the next patch version".

A related link in that article is Weekly Security Hot Fixes - Support and Troubleshooting, which delivers (in my opinion) the most convoluted attempt to explain the new naming convention and timing that I have ever seen. But the attempt is there! I think it supports what AutomatePro was saying about the naming/numbering not aligning with the build date.

Essentially I think they're saying that by the end of August, everyone who is hosted will be on a weekly schedule for subprods to go to HotfixX on Saturday, and Prod following on Sunday. I'm still not sure how "easy" it will be to figure out timing for validating a larger patch or upgrade in subprod and get it signed off before the patch deprecates for Prod, but I guess I will need to ride this one out and see how it goes.