Upgrading and Cloning Instances in Parallel

Tim Grindlay · ‎09-15-2022

Hi All,

We've been working with ServiceNow since Geneva back in 2016 and have run 90% of upgrades and patching without the support of a Service partner and over the years we've worked on cutting down the timeframe to patch and upgrade and generally we work to a 2 - 3 week timeframe from upgrading development to upgrading production.

We have 3 instance environments DEV, UAT and Production and our DEV/UAT portion of the upgrade plan will look something like this:

Day 1

(Morning / Afternoon) Pre-clone activities, comms etc.
(5pm) Lock out testers and developers. (Upgrade administrator access only)
(5pm) Backup all update sets in development.

Day 2

(3am) : Scheduled clone for both DEV and UAT.
(5am) : Scheduled upgrade both DEV and UAT.
(8am) : Post clone / upgrade smoke tests
(9am) : Process skipped updates

Once the skipped updates have been processed and we're happy the instances are in a good state, we'll open the environments to testers and developers.

We're shifting this activity to our Service Partner so our core team can focus on other work, but they're of the opinion that the environments, clones, and upgrades should be done on different days as there is a risk in cloning and upgrading both environments on the same day which means this portion of the process gets extended by 3 or 4 days.

What risk is there in doing the clones and upgrades on the same day?

Eric W · ‎09-16-2022

If you've been able to follow a similar cadence since Geneva for 90% of your upgrades, you've got a pretty good sense of where things can go wrong with your upgrade.

It also suggests to me that your instances are small enough to be cloned and upgraded comparatively quickly, and that you've managed to keep customization to a minimum, making the skip list process go fairly smoothly.

As instances get larger, and customizations (and integrations) get more complex, then there are more opportunities to fail -- both in the clone and the upgrade. If your organization has the ability to absorb those atypical events when they happen, I would personally recommend against *planning* to fail by pre-emptively moving clones and upgrades to different days. However, it is important to be able to be resilient in the case one of the actions do fail, and to learn and adapt for "next time."

When an instance clone by itself takes between 1-3 days, and it fails on the third day requiring a restart that takes another 3 days, and this has happened on each of your last 2 clones, then definitely plan more time next time. The same consideration applies for the upgrades.

View solution in original post

Mark Manders · ‎09-15-2022

If you encounter an issue on your DEV, you only have your PROD to check how it was before (because your test will probably have the same issue). And that's always a risk. I am also used to do it one after another, just for that reason, but UAT will be done at the end of day 2 (so only 1 day extension). That second day is to verify DEV, check if everything is still OK and then do TEST and by end of day 3 both instances are released for testing processes and such.

So an extension of 3-4 days I can understand for the first time (because they don't know your instance as well as you do, but it shouldn't be standard. It's just an extra security measure (worst case when doing it simultaneously is you have to request a backup). In my opinion it would be one day extra (and I'm used to having a couple of days between the OK on non-PROD and upgrading to PROD, so it's not extending the entire upgrade process.

If my answer helped you in any way, please then mark it as helpful. If it resolved the issue, please mark it as correct. This way others will find it in the solved queue and helps them on similar queries.

Mark

Please mark any helpful or correct solutions as such. That helps others find their solutions.
Mark

Tim Grindlay · ‎09-18-2022

Hi Mark,

So if I read what you're saying correctly, you're suggesting that the DEV and UAT instances still be done a day apart in case there is a production issue but the upgrade could still be done directly after the clone?

When we've had issues on a sub-prod environment after a clone/upgrade we've never had to request a backup restore or roll back a clone - but we have scheduled another clone / upgrade and we've also requested a temporary instance to retrieve lost updates. Neither of the issues could have been prevented by doing the instances on separate days which is why I'm posting here looking scenarios that would give cause to have them done on different days.

Thanks for the reply!

Musab Rasheed · ‎09-15-2022

Hello,

They are right because time between clone and upgrade is very less and sometimes if clone fails and upgrade takes place then it's of no use as instance doesn't have latest prod copy.

Generally clone takes at least 5 to 6 hours and in some cases I have observed it takes 12 to 13 hours but it depends on your instance size as well.

So keeping in mind all these I suggest you don't keep cloning and upgrading on same day.

What we follow is generally we do cloning and upgrading on weekends so that both happened during off hours which will neither impact customers nor impact developers. so I Insist you follow the same.

Please hit like and mark my response as correct if that helps

Regards

Please hit like and mark my response as correct if that helps
Regards,
Musab

Tim Grindlay · ‎09-18-2022

Hi Musab,

Clone failure - that's an interesting one, we've never had that happen. I suppose the argument in this case is that because the clone failed - the whole process has failed so regardless of whether the upgrade happened or not, you're still back at square one. Arguably you're one step ahead because you have some information like how long the upgrade took and you'll know how many skipped updates there are to process. Also, our clones usually take just under 2 hours.

Thanks for the reply!