Upgrading your production instance requires a lot of planning, testing, and follow-through. The upgrade should be tested thoroughly in sub-production instances before proceeding with production upgrade.
Check out my guide for validating upgrades.
But then still, there are surprises! You might experience an issue on your production upgrade which was not encountered in sub-production upgrade testing. However, many of these can easily be avoided. This post serves as a quick guide to upgrading your ServiceNow instance. It includes 10 upgrade tips (dos and don'ts, if you will) to help you go from our London release to our Madrid release.
Top 10 instance upgrade tips - do's & don'ts
Below is a run-down of the top 10 instance upgrade do's & don't list that everyone needs to ensure they have gone through prior to upgrading.
-
Review ServiceNow Upgrade Resources
-
"Full clone" for sub-production testing
-
Validate schema changes during upgrade
-
Validate user experience in lower environments during upgrade
-
Create a Plugin activation and update sets checklist
-
Test with Automated Test Framework
-
Validate and compare instance upgrades
-
Check when the Upgrade is scheduled to run in HI
-
Check the Timezone of your account on HI
-
Review the features impacted during an upgrade
1. Do Review ServiceNow Upgrade Resources
The product documentation is your go-to guide to instance upgrades!
ServiceNow product docs, release notes and knowledge articles comprehensively cover all the steps you need to follow to ensure a smooth instance upgrade experience. These should be reviewed well before in time to ensure you are aware of what new features/fixes will come as part of the package of upgrading the instance. The resources listed below should be your go-to resource when you are planning to upgrade:
- Upgrade your instance
- How to upgrade
- Upgrade planning checklists (per release)
- Upgrade and patching resources
- Instance upgrades - Customer Success
- Upgrade FAQ (Frequent queries received as tickets to tech support)
2. Do a "Full clone" for sub-production testing
I am sure everyone is aware that upgrades should always be tested in sub-production instances before upgrading your production instance. However, the catch here is to ensure it is a full clone.
A full clone is one in which you make sure that all the clone options are unchecked except 'Preserve theme' as shown below:
Also, you need to make sure that while doing the upgrade testing in lower environments, test the upgrade in instances that are closest in infrastructure setup as production. So if you have gateways (shards) configured in production, test your upgrade on a lower instance that also has gateways (shards) configured.
3. Do validate schema changes during upgrade
Any schema change made during an alter is recorded in the sys_schema_change table. This table lists all the schema changes made on the instance. Filter the results on this table based on 'Created' between the time of the upgrade as shown below where the instance was upgraded between:
Check the 'State' field of these records to make sure they are in the 'Complete' state and gives an error message. If you see any as schema change errors, contact ServiceNow Customer Support to validate the issue. Best to do this in your lower environment prior to the production upgrade to make sure you are not up for surprises.
4. Validate user experience in lower environments during upgrade
When your sub-production instance is upgrading, log into the instance and perform a few basic test use cases like accessing dashboards, reviewing knowledge articles etc to ensure that you do not see any impact on user experience during the upgrade. If there is, feel free to reach out to ServiceNow Customer Support to understand if it is expected behaviour or not.
5. Do create a Plugin activation and update sets checklist
The most common activity post-upgrade is to commit update sets which migrate your customizations from your development to production instance. This activity should be thoroughly tested in a recently cloned and upgraded sub-production instance (not your development instance) to ensure the update sets commit activity does not run into errors.
Many times, these customizations are on top of some plugins activated in development instance. When you migrate these update sets to production, prior to committing, make sure the relevant dependent plugins are committed. If these plugins are not activated prior to committing of update sets, it can lead to unexpected results. So to avoid this issue from happening, create a plugin activation and update set checklist during your testing phase.
Also, some plugins require a subscription in production. Make sure you have the associated subscription beforehand as subscription requires a few business days to process through.
6. Don't get frustrated with instance upgrade testing
Automated Test Framework (ATF) to the rescue!
Test, test and retest! Upgrade testing can be time-consuming and resource intensive but ServiceNow's ATF framework can help ease out that frustration. In Madrid, you can copy and customize ServiceNow-provided quick start tests to validate that your instance still works after you make any configuration changes such as apply an upgrade or develop an application.
Use this feature to ease your way through upgrade testing and have reusable test cases to use over and over again with each new upgrade.
7. Do validate and compare instance upgrades
ServiceNow's "Upgrade History" module is your go-to place to validate an instance upgrade. It will list all the changes applied or skipped by the upgrade. Review these skipped changes to ensure that you are not missing on any new enhancement - may be a chance to revert your customization in favor of improved platform in-built feature.
As they say, the lesser the customizations, the lesser the hassle to validate each upgrade.
If you notice any customization that was updated by the upgrade when it shouldn't have or the number of updates applied across two instances does not match, review these using the troubleshooting tips mentioned here to confirm if it was a valid update or not.
For more information see How to validate and compare upgrades.
8. Do double-check when the Upgrade is scheduled to run in HI
"I have resources waiting on an upgrade to complete that hasn't started! Help! - Part 1"
On the day of the upgrade, you'll have done all your pre-upgrade activities and have resources on standby for post upgrade activities and then you notice that at the time of the scheduled start of the upgrade, the upgrade does not kick in. The upgrade change request in HI has not moved to 'Implement' state. But why?
The most common reason for this is the 'Next action' field on the Upgrade job. The timestamp here will determine when the upgrade will kick off on the instance. This job is an hourly job so in the worst case, the upgrade will kick in within an hours time from the scheduled start time of the upgrade. But to avoid this time lost, remember to always schedule your change request for Upgrade around the time the hourly upgrade job will run.
The hourly job is defined here in the instance System Scheduler-> Scheduled Jobs -> Search for job called Upgrade:
In the above instance, the upgrade runs every hour at around the 37th min. If the upgrade is scheduled for, say 2019-04-22 10:00:00 in HI, the upgrade won't be picked by the instance till 10:37 as that is when the job runs. The best time, in this case, would be to schedule the upgrade at 2019-04-22 10:30:00.
9. Do check the Timezone of your account on HI
"I have resources waiting on an upgrade to complete that hasn't started! Help! - Part 2"
It can also be that the scheduled time on the upgrade in HI was not set properly. It has happened that sometimes, the user setting up the change request for an upgrade in HI, sets up the upgrade start time based on their instance timezone but their timezone in HI is different. Always ensure that your timezone in HI matches your timezone in the instance. So if you are planning to upgrade the instance on say, 2019-04-22 10:30:00 PDT in the instance, make sure you set your timezone as PDT in HI before submitting the change request.
Timezones can make a big difference!
10. Do review the features impacted during an upgrade
Make sure you have an understanding of what features are impacted during upgrades to avoid any surprises!
One, in particular, to know beforehand is that during an upgrade, only upgrade safe jobs run. What that means is jobs in the sys_trigger table that have the "Upgrade Safe" flag set as true will be the only jobs that run during the upgrade.
All upgrade unsafe jobs continue execution as soon as the upgrade completes.
Note:
- Do not change the upgrade_safe flag on any record to upgrade_safe=true.
- All custom sys_triggers must be marked upgrade_safe=false to avoid any unexpected results during upgrades.
---
Each ServiceNow release offers new and interesting features, applications, and functionality to enhance your work experience. To take advantage of these benefits, a safe and efficient upgrade can make it easy to move from London to Madrid! I hope these 10 Instance Upgrade tips have been helpful.
Also, check out my guide for validating and comparing upgrades.
If you have upgrade questions feel free to drop them below!
4 Comments
