Upgrading and Cloning Instances in Parallel

Tim Grindlay · ‎09-15-2022

Hi All,

We've been working with ServiceNow since Geneva back in 2016 and have run 90% of upgrades and patching without the support of a Service partner and over the years we've worked on cutting down the timeframe to patch and upgrade and generally we work to a 2 - 3 week timeframe from upgrading development to upgrading production.

We have 3 instance environments DEV, UAT and Production and our DEV/UAT portion of the upgrade plan will look something like this:

Day 1

(Morning / Afternoon) Pre-clone activities, comms etc.
(5pm) Lock out testers and developers. (Upgrade administrator access only)
(5pm) Backup all update sets in development.

Day 2

(3am) : Scheduled clone for both DEV and UAT.
(5am) : Scheduled upgrade both DEV and UAT.
(8am) : Post clone / upgrade smoke tests
(9am) : Process skipped updates

Once the skipped updates have been processed and we're happy the instances are in a good state, we'll open the environments to testers and developers.

We're shifting this activity to our Service Partner so our core team can focus on other work, but they're of the opinion that the environments, clones, and upgrades should be done on different days as there is a risk in cloning and upgrading both environments on the same day which means this portion of the process gets extended by 3 or 4 days.

What risk is there in doing the clones and upgrades on the same day?

Eric W · ‎09-16-2022

If you've been able to follow a similar cadence since Geneva for 90% of your upgrades, you've got a pretty good sense of where things can go wrong with your upgrade.

It also suggests to me that your instances are small enough to be cloned and upgraded comparatively quickly, and that you've managed to keep customization to a minimum, making the skip list process go fairly smoothly.

As instances get larger, and customizations (and integrations) get more complex, then there are more opportunities to fail -- both in the clone and the upgrade. If your organization has the ability to absorb those atypical events when they happen, I would personally recommend against *planning* to fail by pre-emptively moving clones and upgrades to different days. However, it is important to be able to be resilient in the case one of the actions do fail, and to learn and adapt for "next time."

When an instance clone by itself takes between 1-3 days, and it fails on the third day requiring a restart that takes another 3 days, and this has happened on each of your last 2 clones, then definitely plan more time next time. The same consideration applies for the upgrades.

View solution in original post

Musab Rasheed · ‎09-21-2022

Hello Tim,

Sorry for the confusion, Clone never gets failed but I have seen it used to take 12 13 hours sometimes, I didn't review my comment and left like that and I actually meant about clone time getting extended rather than clone getting failed.

Anyways, as you are saying generally clone gets completed in 2 hours so I suggest you keep gap of at least 8 hours between clone and upgrade and finish both in a day rather than in two days.

In a nutshell clone and upgrade timings depend on size of instance , if instance size is less you can wrap both in a day.

Best Regards

Please hit like and mark my response as correct if that helps
Regards,
Musab

Tim Grindlay · ‎09-22-2022

Thanks for the response Musab!

Terri Kouba · ‎09-16-2022

Hi Tim,

At the University of California, Berkeley, in one of our environments, we also have 3 instances: dev, test, prod. We clone before every quarterly patch and every annual upgrade. We do patches in 1.5 weeks. We allocate 4 weeks for the annual upgrades because we do deeper testing than we do on a patch and we allow 1.5 weeks for customer UAT, if needed.

Schedule for quarterly patches:

Friday we run the automated tests in dev and test.
Monday, during working hours, we clone ServiceNow dev and test on the same day, an hour or so apart. We do the post-clone steps and run the automated tests in dev and test.
Tuesday, during working hours, we patch dev and do the post-patch steps. Run the automated tests, do manually testing, targeted validation.
Thursday, during working hours, we patch test and do the post-patch steps. Run automated tests, do lite re-testing
the next Tuesday, during non-working hours in the evening, we patch production. We do the post-patch steps and lite, non-invasive smoke testing.

We also don't like the idea of doing the clone and patch on the same day, due to the risks noted by your partner. We don't lock users out of dev/test/prod during a patch or upgrade. We inform them when the clone happens but we don't lock them out.

We don't patch/upgrade dev and test at the same time for the reason Musab mentioned: If we find something wrong in dev, we want to validate the functionality against test. We don't want to validate the functionality in production because we may be required to submit a service request or close an incident, and we never want to do that type of invasive validation in production.

If you have any questions, don't hesitate to ask.

Thanks,

Terri

Tim Grindlay · ‎09-18-2022

Hi Terri,

Love that you're doing the work during office hours, the automated testing, and it sounds like you have the patch and upgrade process dialled in! The reason we lock developers / testers out of the environments is so that we don't have anyone committing changes that will get lost after the clone because we're backing up the update sets immediately after we lock them out.

How complex / customised is you're environment? I understand your point about invasive validation in production but even outside of the patching upgrade process there will be occasions we need to do it because the environments will never be 100% the same, so in terms of risk I suppose it comes down to the likelihood and impact of something occurring, which I think in our case is pretty low.

Thanks for your reply!

Terri Kouba · ‎09-21-2022

Hi Tim,

You asked: How complex / customized is our environment?
Well, it depends on who you ask. ServiceNow says we have far too many customizations and our customers say we don't have enough! 😉

The RITM/Task process is very customized, as we have SOAP calls out to an external billing system integrated in the process.

We use Project Management and customized it slightly, adding a few fields and unique email notifications.

We have over 20 custom tables but those aren't usually impacted by the upgrade/patches.

We don't lock our developers out. They are involved enough with the process that they are aware of the schedule. The person doing the patch/upgrade exports open update sets from dev before cloning.

Thanks,

Terri