Monthly Patching and Weekly Security Patching

[Chuck Nusbaum]

The new weekly security patching is throwing me for a bit of a loop with respect to monthly patching. Here's the scenario.

At the beginning of August, we decided we wanted to go to Zurich Patch 11m to fix a known issue. We upgraded non-prod instances to 11m and scheduled prod to go to 11m as well at the end of the month.

In the middle of the month, weekly security patches started being deployed, and 11m was determined to be a downgrade from Patch 10 Hotfix 4 w33, so our prod update to 11m got canceled.

Today, we got notification of the October patching target: Zurich Patch 10 Hotfix 4b.

I'd like to get our instances on the same patch level again...but I suspect that if we schedule to go to Patch 12, then weekly security patches are going to determine it as a downgrade and it will be canceled again.

Shall we just give that a try or does anyone have a different/better approach?

[AutomatePro]

These are super helpful links Kristin thank you.

There was a lot of discussion about this on a webinar we hosted with NowBen yesterday.  Some platform owners were not aware of the weekly security patches and others are struggling with how to validate them at the weekend when there is no support team in place to deal with issues.

Some of our customers are scheduling full automated regression tests to run after the patch is installed on a sub-prod instance, which at least gives some time to review the test results and request to pause the prod install (which apparently you can do with approval from ServiceNow).  They can use the before and after test results comparison report and screenshots to provide the evidence to ServiceNow that the patch has broken their instance, which substantiates the request to delay the prod install.

[Chuck Nusbaum]

Thanks, y'all!

Kristin, I've read through those two KBs so many times my head is spinning. 🙂 I find them quite confusing as well. I'm sure they continue to be updated with additional details as ServiceNow is figuring things out (I just subscribed to keep more on top of them).

Oddly, I hadn't seen our subprods on Z11m get any weekly security hotfixes all of August, unlike Prod on ZP10HF4a which received weekly security hotfixes every week. (As an aside, weekly security hotfixes and patch hotfixes...it's even more confusion with what things are being called.)

We're going to see what happens this weekend when our Dev instance should go to ZP12 and, presumably, Prod will get another weekly security hotfix. Will our Prod CHG to go to ZP12 later this month get canceled again? Tune in next week for the next installment! 😉

[Chuck Nusbaum]

As you expected, @AutomatePro, the CHG we scheduled to update Prod to ZP12 was canceled for being a downgrade from the weekly security hotfix.

Our plan at this point is, on Monday, 9/21, to schedule a CHG to update Prod to the latest weekly security hotfix version of Zurich Patch 12 on Friday, 9/25. That CHG should not get canceled because the weekly security hotfix won't have applied yet. We'll see if that works!

[Rebecca3]

Chuck and others were you able to actually schedule an upgrade to a version containing all of Patch 12 fixes? Or even apply a version in sub-prod instances?

I've had a lot of issues with the various versions for the same reason - weekly security fixes are deemed 'newer' versions of the broader Patch 12 and changes got auto-cancelled.

For September monthly patching, I planned to patch to from Zurich Patch 10 to Zurich Patch 12 to address several issues. I changed the target patch version from Zurich Patch 10 Hot Fix 4b to Zurich Patch 12 W35 for all of the monthly change tickets that were created in August.

Our DEV instance upgrade completed as expected but when the first batch of weekly security patch changes for Zurich Patch 10 Hot Fix 4a W36 were applied on Sept 5 & 6, the remaining change tickets for ZP12 were autocancelled as Zurich Patch 10 Hot Fix 4a W36 was deemed a higher patch version than Zurich Patch 12 W35.

Then when the second batch of weekly security patch changes were created for Sept 12 and 13, I adjusted the Target Patch version to Zurich Patch 12m W37 as there was no option to apply a non-M version of ZP12. I assumed this would include the changes included in Patch 12; however this is not the case. The M version didn't contain the same core/baseline fixes that Patch 12 does.

I ended up having to rollback the patch for two instances from Zurich Patch 12m W37 to Zurich Patch 10 Hot Fix 4a W36, and was advised by NowSupport that I could go to Zurich Patch 12 W38 (non-M version), and as per release notes, this should have includes the previous non M-versions of the W33-37 versions of Patch 12 but that was not the case. ZP12 W35 that was applied to our dev instance had over 10,000 changes applied but ZP12 W38 that was applied to our test environments only contained 950 changes, and we are experiencing issues that should have been fixed with Patch 12, and also have an issue with Flow Designer (unsure if this is due to the patch rollback, or ZP12 W38 itself).

We are now in a situation were our DEV environment is on ZP12W35 that has no issues, our QA and TEST environments are on ZP12W38 that has several issues, and our PROD instance is still on ZP10 as I am still trying to work through this issues with NowSupport.

I have also read through the KBs multiple times and am still struggling to understand how to keep up with and maintain patching.

We also are concerned with how to validate weekly security patches at the weekend when there is no support team in place to deal with issues that arise. And even if we did have a support team in place on the weekend, there is at most 1 day between sub-prod and prod being patched which isn't sufficient time to validate

[Chuck Nusbaum]

Hi Rebecca,

Oh my--hearing about the issues you're having makes me nervous. We have our DEV instance on ZP12w35 and have scheduled PROD to go to ZP12w38 (from ZP10HF4aw38) on Friday, before the weekend when another security hotfix would be applied and potentially cause our P12 update to be canceled.

At this point, we're trying to get PROD and DEV both on P12 to align them...but I'd never have thought that w35 and w38 would have so many differences.

As an aside, I've been ignoring the "m" patches because that's just another dimension of this multi-dimensional patch/hotfix/security hotfix dimensional mess.

I just got a notification today that the Weekly Security Hot Fixes KB article (3146385) was updated...I'll have to take another look at it.

I agree with you regarding validating the security patches...since they aren't applied to any subprod instance ahead of time (and sometimes not at all...like why is DEV still on w35?), there's not even a chance to see if something is impacted before everyone's back online on Monday. We also haven't observed any consistency about the scheduling of the security hotfixes...sometimes they're Saturday, sometimes they're Sunday.

I'll post an update if we're able to get PROD patched to P12 this Friday or not.