ServiceNow Version Upgrade – Methodology, Best Practices &Tips

[Eitan Salita]

ServiceNow Version Upgrade – Methodology, Best Practices & Tips

Introduction

From time-to-time ServiceNow administrators, platform owners and developers are required to perform platform upgrades and they are searching for assistance, best practices and tips to perform their upgrade in the best way, as well as avoiding any issues such as post upgrade malfunctions, data/code loss, privileges issues etc.   

In addition to the ServiceNow official documentation (here and here), below are additional methodologies, guidelines and best practices, based on my own upgrade experience.

Why?

ServiceNow releases a new version approximately every 6 months.   

It’s not a question of if to upgrade, but when or even how often upgrades should be performed. If your latest platform version is N-2 (two versions prior to the latest version) you won’t be eligible for support.

Additionally, by keeping your ServiceNow environment up to date, you will:

• Avoid the issues inherent when running on an unsupported version
• Reduce the risk of encountering issues that have already been resolved on the latest version 
• Leverage the latest features and functionality
• Maximize your investment (licenses) for the platform
• Enhance platform levels of security, availability, stability and performance

What?

The main idea is to "obtain and address" all potential issues in the sub-production environments, thereby reducing the possibility of encountering issues during Go-Live in the Production environment. The clone and upgrade should start from the lowest instance and gradually move up to  the higher instances.

Required Steps:

Sub-production (DEV) - lowest environment

1. Clone Production instance over DEV instance
   
2. Upgrade to the chosen version.
3. Address all skipped records.
4. Pack fixes in the Update set.
5. Perform sanity and regression tests in sub-production (DEV) instance.
6. Address all issues in sub-production (DEV) instance.
7. Pack fixes within the Update set

Perform the same steps for additional higher environments (PreProd / Staging / UAT), if available.

Sub-production (PreProd)

1. Clone Production instance over PreProd instance
2. Upgrade to the chosen version.
3. Deploy fixes from steps (4) and (7) above by committing the relevant Update sets.
4. Perform sanity and regression tests.
5. Address any issues found.
6. Pack fixes within the unique Update set.

Production 

1. Upgrade to the chosen version
2. Deploy fixes from (3) and (6) above, by committing the relevant Update sets.
3. Perform sanity and regression tests in Live Bridge. Any user can participate and raise urgent issues, questions and clarifications as needed.
4. Address new issues, if any.

How?

Upgrading is a set of action items, steps and factors to consider and perform. Therefore, upgrading should be treated as a project on its own.

Project related aspects

Use all relevant project management best practices and tools (ServiceNow PPM / MS Project / Monday / Jira / MS Excel or any other PM tool you use to manage your projects).  

Ensure the topics below are covered:

• Identify and allocate resources - PM, QA and Business owners / stakeholders, consider staffing extra IT support (hyper care support) for the Post GoLive phase.

• Establish project timelines, milestones and key milestones.
• Assess risks for customized business areas in your instance.

Provide frequent and timely communication to all users.

Successful upgrade - tips and recommendations:

• 1^st key to a successful upgrade - frequent communication:

• Prior to upgrading, inform end users of what to expect and why they should look forward to upgrading. Whether the changes will affect them directly or indirectly, awareness is key.
• During the upgrade, let them know about any expected downtime or other disruptions to their day-to-day operations, and when they can expect to resume business as usual.
• Provide and communicate workarounds for any essential tasks.
  • 2^nd key to a successful upgrade - sanity and regression tests:
    • Divide platform into business processes / areas.
    • Allocate a business owner for each business process / area. Use template in appendix A.
    • Guide business owners to create tests scenarios.
    • Review test scenarios with business owners and verify all critical processes are covered.
  • 3^rd key to a successful upgrade - when establishing timelines, leave a buffer for unexpected behavior, especially in customized areas. Use the template in appendix B.
  • 4^th key - schedule upgrades for slow traffic times to minimize user impact.
  • 5th key  - communicating the completion of the successful upgrade with all the benefits.

Technical related aspects

General

• Schedule upgrade - can be done via Hi Portal. Official instruction can be found here. Note that in case of end of support if not scheduled by customer – the upgrade will be scheduled by ServiceNow automatically. Automatic change request record will be created, notification sent to the customer, upgrade date and version can be changed by the customer by updating the change request record.
• Schedule clone – can be done via ServiceNow instance. Official instruction can be found here .
• To perform tests that represent and simulate real situations, ensure that you have at least one sub - production environment that is identical to the Production (plugins, API integrations, etc.) environment.
• Review what's new in the official documentation release notes.

Sub – production environments

• Pre-cloning tips:
  • Communicate to all relevant personnel (particularly ServiceNow developers) to stop developments and customizations in DEV instance until sanity and regression tests are accomplished and all issued are addressed.
  • Preserve unique data in the sub-production instance to avoid overwriting data.
  • Use the Data Clone Preserve table with all relevant tables to be preserved to prevent overriding – Users, REST API credentials tables, System properties, etc.
  • Create an external backup for critical tables (MID servers, Users, etc.)
  • Back up all your open Update sets in DEV:
    • Option A:  Retrieve all uncommitted Update sets to Production instance before cloning, clone will bring them back to DEV.
    • Option B: Save as XML files and save them externally.

• After Cloning:
  • Verify MID Server configurations were upgraded successfully and not overwritten. (In some cases, despite preserving MID definitions, the MID definitions are overwritten).

• Pre-Upgrade tips:
  • Perform a Preview upgrade to simulate expected skipped records (new functionality, beginning with the Rome version) and review them.

• Post-Upgrade tips:
  • Review the list of skipped items. Based on the priority of the item. The system automatically assigns the priority, ranging from P1-P5.
    • P1: (Highest priority): xml content. Examples: UI Page, Macro.
    • P2: Script or script_plain. Examples: UI Actions, Script includes, ACL, Notification, Client Script.
    • P3: HTML content.
    • P4: sys_ui_form_section, sys_ui_related_list or sys_choice_set. Examples: Related lists, Variable set, Order guide, Record producers, UI policies, Catalog items.
    • P5 (Lowest priority): other. Examples: Widgets, Menu items, UI View, List layout, Form Layout
  • Address Skipped records
    • Review records with a 1-2 Priority. Discuss with business process stakeholders if necessary and then decide on an approach for each process.
    • Group records by business process, review them with business owners and record the resolution in each record (retain the customization or discard the customization and update the record to match the base system for this upgrade).
    • Open the Update set to pack Skipped records solutions
    • Address Skipped records with a 1-2 Priority. (When opening a skipped item, you can select Resolve Conflict to compare the code. Once you compare the code, and only minor changes exist, and the change can be merged with the OOB code, you can merge the code and select Save Merge.)
    • Review records with a 3-5 Priority to verify that there are no critical records to be addressed. If you have time and resources – address them as well.
    • Verify all changes were packed properly within the Update set.

• Sanity and regression tests 

• • Coordinate and begin sanity tests only after all skipped records have been addressed (reason: addressing Skipped records can affect platform functionality).
  • Tests scenarios:
    • Option A: Use ATF application (more effective way). Official documentation can be found here.
    • Option B: Use the template from appendix C.

Production environment

• Before Upgrade tips:
  • Schedule and define a Live bridge during the upgrade. Anyone can participate, raising urgent issues, questions and clarifications as needed.
  • Communicate to all users about potential downtime.

• After Upgrade tips:
  • Communicate with all users on the Live bridge
  • Perform sanity and regression testing
  • Communicate project closure and Hypercare support period
  • Schedule a Retrospective (Lessons learned) meeting

Appendix A, B & C - attached.

Good luck,

Eitan Salita

Please mark this article, if helpful.

[Kurban Nepesov]

Again many thanks for composing and sharing those!
Some of the points from this article are to be considered for our next upgrade 🙂

[Eitan Salita]

Thank you Kurban, I am glad this article is helpful for you!

you are welcome to share your experience after you accomplish the upgrade.

[Suzanne Coffman]

This is our irst upgrade, and I am having trouble finding anything that says whether to expect downtime or not.

You state: "Communicate to all users about potential downtime."

Is there always downtime?  How do I estimate the duration?  Can users continue to use the system unless it actually goes down?

[Martin Ivanov]

@Suzanne Coffman, in 99% ot the cases there is no downtime. The upgrade status is displayed on the Upgrade Monitor page - you see at what stage it currently is and what is happening behind the scenes. Users can continue using the system. But, your first task must be to review the skipped updates and take decision on these. It is recommended to review all Priority 1 and 2 skipped records, but take a look on the others too. If there is a conflict with any of your customized components that prevents its normal operation, this may lead to some downtime, but not of the whole system. Generally speaking, expect a normal behaviour of your instance and do not expect a single moment in which your users cannot access it. Unless you are one of these <1% of the cases.

[Eitan Salita]

Hi @Suzanne Coffman,

On top of @Martin Ivanov  mentioned above, as mentioned in the article:

"The main idea is to "obtain and address" all potential issues in the sub-production environments, thereby reducing the possibility of encountering issues during Go-Live in the Production environment

to verify if and how long there will be down time".

In other words - you will be able to perform down time assessment for your specific environment - during sub productions upgrade.

The higher the sanity tests quality will be, the less is likelihood for a down time or other failures.

In addition, with respect to the above, since there's always some probability for "surprises" - for Production instance upgrade - I would recommend:

1) Communication to all users about potential downtime, that might occur during the upgrade window until all tests are accomplished.

2) During Live bridge - encourage users to raise any issue, including down time and address them accordingly.

[Prashanth Velam]

@Eitan Salita This is a very good article. Just a quick question about the Overall Platform Upgrade vs the individual store components upgrades.

Can we upgrade the individual GRC plugins (Risk Management, Vendor Risk Management Plugins etc.) to latest versions at the same time as the overall Platform Upgrade? or Do they have to be done independently?

Thanks! 

[Eitan Salita]

Hello @Prashanth Velam,

Appreciate your feedback.

All platform elements including plugins will be automatically upgraded at the same time the upgrade is performed, therefore you do not need to upgrade any plugin separately.

In your specific case, you can verify GRC plugins were successfully upgraded right after sub-production instance(s) was upgraded and then if tests successful, perform the production instance upgrade.

Good luck,

Eitan