Reducing the burden of home pages

GTSPerformance · ‎02-15-2021

Overview

This guide is written by the ServiceNow Technical Support Performance team. We are a global group of experts that help our customers with performance issues. If you have questions about the content of this article we will try to answer them here. However, if you have urgent questions or specific issues, please see the list of resources on our profile page: ServiceNowPerformanceGTS

Impact of Homepages

ServiceNow support teams often review the performance of various customer instances. One item which comes up time and again are poorly performing and/or extremely expensive home page loads (/home.do transactions). Home pages aren't generally seen as being central to an organisations key business indicators or objectives so there is a tendency to ignore their cost - the reality, however, is that poorly performing home pages can cause major impediment to the success of a ServiceNow instance:

In almost all cases they will lead to a frustrated user base
In extreme examples they can be a major contributor to a service affecting event (i.e. an instance outage)

Here we can generalise a little however the impact of home page loads is primarily down to two factors:

Home page loads are FREQUENT - by default a /home.to transaction is triggered whenever a user logs into the classic user interface (UI), is redirected back to their home page from some other part of the platform, or their home page refreshes (should this functionality be enabled)
Home page loads can be SLOW - users tend to put all kinds of content on their home page with little thought to performance

Many customers are shocked when they are shows the true impact home pages are having on their instance. For example, if transactions are aggregated by server side response time, it's not uncommon to see /home.do in the top 5 most expensive transactions. A recent example of this from one of our larger customers is shown below:

This is taken from a single 4 hour period - during this period:

There were 18850 individual /home.do transactions (an average of ~1.3 home page loads per second)
The average server side response time of a home page load was nearly 7 seconds
The instance spent (and users waited for) almost 36 hours on home page loads

Remember - this is from just 4 hours in the life of an instance! If the above is extrapolated out over a working week, month, or even a year, the impact to the productivity of end users alone is hugely significant. Another factor is that the majority of home page loads take place when an instance is already busy (when users log in first thing in the morning are when they return from lunch). This amplifies their impact across a relatively short period of time when the instance could really use the resources home pages are consuming to deliver an optimal user experience.

Addressing issues with home pages is one of the best (and potentially easiest) things a ServiceNow administrator can do to maximise performance of their instance. There are a few simple steps which can help with this which are discussed below.

Gaining Insights Into Home Page Use

Demonstrating that home pages are of significant expense for an instance only solves the first piece of the puzzle - in general this prompts a number of follow on questions - for example:

Why are there so many home page loads?
Which users have slow home pages?
Where should efforts on addressing issues focus?

At this point things become more complex - details of every /home.do transaction will be held in transaction logs (i.e. the syslog_transaction table) however the sheer volume of data in these logs can be overwhelming - a mechanism to build aggregate metrics is required. An example background script to help with this is attached to this article (home_page_loads.txt) - the results of this script executed against another large customers instance are shown below:

*** Script: ================================================================
*** Script: HOME PAGE REFRESH
*** Script: ================================================================
*** Script:
*** Script: Available choices from sys_properties (glide.home.refresh_intervals): 1800,3600
*** Script: Default from sys_user_preference (home_refresh): off
*** Script: Count of users with explicit setting grouped by value:
*** Script:
*** Script: VALUE NUMBER OF USERS
*** Script: ----- ---------------
*** Script: 1800 163
*** Script: 3600 9
*** Script: off 8
*** Script:
*** Script: ================================================================
*** Script: USERS WITH MOST EXPENSIVE HOME PAGES
*** Script: ================================================================
*** Script:
*** Script: Total home page load transactions from 2020-09-11 10:02:23 to 2020-09-11 20:41:53: 18886
*** Script: Average server side processing per home page load transaction (ms): 2465.62
*** Script:
*** Script: USER COUNT TOTAL TIME(ms) AVERAGE TIME(ms) MAXIMUM TIME(ms) REFRESH INTERVAL(s)
*** Script: -------------------- ----- -------------- ---------------- ---------------- -------------------
*** Script: user1 1237 2431466 1965.62 7375 Default (off)
*** Script: user2 635 964493 1518.89 5982 Default (off)
*** Script: user3 45 892217 19827.04 30870 1800
*** Script: user4 33 881056 26698.67 31013 1800
*** Script: user5 257 742709 2889.92 8446 1800
*** Script: user6 38 715156 18819.89 32863 Default (off)
*** Script: user7 33 589152 17853.09 21706 Default (off)
*** Script: user8 18 561337 31185.39 34797 Default (off)
*** Script: user9 18 523394 29077.44 31402 1800
*** Script: user10 31 489157 15779.26 21613 Default (off)
*** Script: user11 17 410417 24142.18 31146 Default (off)
*** Script: user12 12 369340 30778.33 31795 Default (off)
*** Script: user13 35 332268 9493.37 14789 Default (off)
*** Script: user14 30 286477 9549.23 12706 Default (off)
*** Script: user15 127 281383 2215.61 7542 Default (off)
*** Script :

Lets step through the about output - first of all the top section (home page refresh):

Initially the values of system properties controlling home page auto refresh are shown - in this instance:
- Auto refresh is disabled by default (this is good)
- Users can choose auto refresh intervals of 1800s (30 minutes) or 3600s (1 hour) - this is a significant improvement on the out of box defaults where users can choose auto refresh intervals of 5 minutes upwards
Next the number of users who have selected a specific auto refresh interval are shown - for example 163 users have selected to refresh their home page every 1800s. Note that users who have not selected a specific auto refresh interval will use the instance default (in this case off)

This section immediately identifies if there are potential improvements which can be made to auto refresh configuration - for example:

Can users select a refresh interval of 5 minutes - is this really necessary?
If auto refresh enabled by default - should this functionality be disabled?
Are there are a large number of users who have chosen a very frequent auto refresh interval - should these users be cleaned up?

Now lets step through the second section (users with most expensive home pages):

First of all we can see the time period represented by the output - in this case this is approximately a 10 hour window on 11th September 2020 (note that the script tries to run across 24 hours but aborts after a certain period of run time - see the script itself for more information)
Across this period we can see that details of 18886 /home.do transactions were found in transaction logs and the average server side response time for these transactions was ~2.5s
Next the script provides a break down of home page loads by user - by default users are ordered by 'total time' (i.e. the cumulative amount of time that user spent executing /home.do transactions over the window of interest) - essentially these are the users who added most 'impact' to the instance via loading of their home page. For each user this is further broken down showing:
- The number of home page loads triggered by the user during the window the script examined
- The total time the user spent executing/waiting on home page loads during the same window
- The average server side response time of the users /home.do transactions during the window
- The maximum server side time taken by a single home page load for the user during the window
- The auto refresh setting for that user

The script has suddenly provided huge amounts of additional detail:

In this instance auto refresh settings look reasonable so there is probably nothing further to do in this area
'user1' was 'most expensive' in terms of home page loads - their home page performs well (average of ~2s server side response time) but was loaded 1237 times in the 10 hour window - why was it loaded so often?
- This means that this user loaded their home page every 30 seconds for 10 hours straight. They might have a auto-fresh plugin on their browser.
- Another possible common cause is that these transactions are a result of an automated monitoring solution periodically checking the instance is available
- - Using home page loads for this, however, is generally a bad idea due to their expense - it is far better to test loading a very quick list or some static page such as /welcome.do
Other users ('user4', 'user8', 'user9', and 'user12') have extremely poorly performing home pages - all average over 25s server side response time per home page load
- Is further tuning of these users home pages required?

From this point it becomes very easy to determine areas which need attention and how these should be prioritised for maximum gain.

Reducing the Volume of Home Page Loads

This may be somewhat of a blanket statement but here goes - a huge proportion of home page loads are completely unnecessary and add no value whatsoever to end users. Lets explain this with a couple of examples:

A user interacts with a ServiceNow instance to complete specific tasks - during each interaction they have no intention of visiting their home page however every time they log into the classic UI the platform forces them to sit through a (potentially slow) home page load before they can navigate away and achieve something useful
A user configures their home page to automatically refresh every 5 minutes (after all more is better), logs in to the classic UI, allows their home page to load, and opens a second browser tab to start work. They remain logged in to and working within the ServiceNow instance for the next four hours. During this time their home page (still loaded in their initial browser tab) will trigger nearly 50 home page loads for absolutely no reason

In both scenarios this wastes resource within the instance. Possibly worse, however, is the impact on each end users experience:

The first user becomes frustrated whenever they log into the instance
Whilst working with the instance the second user feels like ServiceNow 'hangs' every 5 minutes (due to the concept of session synchronisation and new transactions triggered by the user getting 'queued' behind a running /home.do transaction)

ServiceNow starts to be seen as a platform which is frustrating, unreliable, and tiring to work with.

Reducing the volume of home page loads is probably the single best way in which to limit the impact of home pages on a ServiceNow instance. Both of the above scenarios can be avoided with via a few simple changes:

Limit refresh rates which users are able to select

As mentioned previously an out of box ServiceNow instance allows users to choose to automatically refresh their home page every 5 minutes - in most cases this is completely unnecessary
Most users don't derive any benefit from an automatically refreshing home page and for those that do refreshing every 15 minutes is generally more than sufficient
To enforce this consider removing the 300 second value from the 'glide.home.refresh_intervals' system property
From that point onwards users will only be able to select a refresh rate of 15 minutes (or larger)
Users who have already selected 5 minutes will remain on 5 minutes due to an existing/historic 'home_refresh' user preference so you might also choose to 'move' these users to the new fastest refresh rate - this can be achieved with a script such as the following (obviously lowThreshold should be changed as necessary):

makeChanges();
function makeChanges() {
   var lowThreshold = 900;
   var gr = new GlideRecord('sys_user_preference');
   gr.addQuery('name', 'home_refresh');
   gr.addQuery('system', '0');
   gr.setWorkflow(false); gr.query();
   while (gr.next()) {
      if (parseInt(gr.value) < parseInt(lowThreshold)) {
         gr.value = lowThreshold.toString();
         gr.update();
      }
   }
}

If there are existing users who absolutely require a 5 minute refresh rate (for example their home page is displayed on a large screen) this behaviour can be maintained by an administrator setting their home_refresh user preference set to 300 even if this is no longer a value which users can select

Ensure that the default value for auto refresh is set to 'off'

To enforce this set the corresponding system level user preference accordingly, i.e. within the sys_user_preference table create/update the following record:

Name: home_refresh
System: True
User: Empty
Type: String
Value: off

This ensures that any newly provisioned users will at least start with a home page which will not automatically refresh
Obviously the user may chose to change this later in their life however, if available refresh rates have already been limited (as above), the strain they can place on the instance is effectively capped

Implement a light weight splash screen which is loaded on login to the classic UI instead of a users home page

Many customers have chosen to alter platform behaviour in that, when a user logs in, a light weight/static page is displayed instead of the users home page
This page is extremely fast to load and places minimal strain on the instance
In addition the page will not auto refresh so will not be a generator of 'superfluous' transactions within the instance should the user leave this loaded and switch to an alternate browser tab to interact with the instance
To maximise user experience the page is generally configured to display a hyperlink which, if clicked, loads the users home page should they want to visit it. Once loaded the home page functions entirely as normal (including automatically refreshing as configured)
KB0712404 discusses this concept in detail and describes how this functionality can be made extremely configurable - for example the splash page functionality can be rolled out to only a subset of users based on a user preference, role and so on (basically any check you can think of which can be made against a user)
In some instances introducing a light weight splash page has been shown to reduce the total volume of home page loads by 50% or more so, even though it needs a little investment in terms of development time, can be hugely significant!

Consider disabling auto refresh functionality all together

This is somewhat draconian but should be mentioned as it is a possible option - to disable all home page refreshes instance wide set the 'glide.home.refresh_disabled' system property to true
Note, however, that if you have users who rely on auto refreshing home pages you should be ready to deal with some complaints!

Improving the Performance of Individual Home Pages

The background script discussed earlier in this article allows users with persistently slow home pages to be identified - if this script were used regularly over the course of a few days or weeks it should become clear which set of users have the toxic combination of loading their home pages regularly and their home pages being persistently slow. Efforts can then be focused on improving the performance of these specific home pages.

In general this approach is not recommended - fixing individual home pages can be:

Hugely expensive in terms of troubleshooting time
Of limited benefit in terms of reducing overall resource consumption by an instance
An ultimately frustrating exercise - fix a single users home page today then as soon as they log in tomorrow they customise it (badly) once more so things are back to square one

If you do decide to pursue this, however, the ServiceNow platform has one specific tool which can help you - 'Debug Homepage Render'. To use this tool:

Log into the classic UI as a user with admin role
Enable 'Debug Homepage Render' functionality (System Diagnostics -> Session Debug -> Debug Homepage Render)
Impersonate the user who has a poorly performing home page and allow the home page to load

Once the home page load has completed the tool will add details of load time to each widget on the page and will also display a list of all widgets and their load time at the bottom of the page - for example:

In this case:

The home page contains 3 widgets
Two of the three widgets perform well and load in under 0.5s
The third widget (GFRS Open Incidents), however, takes over 30 seconds to load and is therefore likely to be responsible for all of the performance issues this home page experiences

Widgets which are identified as being performance concerns should either be tuned (such that they load faster) or removed from the home page (so that they do not load at all).

Closing Comments

Home pages are some what of a necessary evil within ServiceNow instances where the classic UI is heavily used. As described by this article, however, there are some relatively simple steps which administrators can take to keep their impact under control and prevent them from becoming an excessive burden in instance performance.

Some Necessary Disclaimers

This article introduces users to new scripts and potentially new customisations which can be made to a ServiceNow instance. Note that in all cases:

Scripts and/or example code are provided as examples only for use at your own risk
Any scripts or modifications should be fully tested in a sub production instance before being applied to production
Modifications to system properties or committing update sets will trigger a cache flush across all of an instances application nodes which can lead to a short period of performance degradation - as a result such changes should only be made outside of periods of instance peak use
If further assistance is required around any aspect of this article please refer to ServiceNow documentation or open a case with ServiceNow customer support. Note that assisting with modification of custom code (such as shown in this article) is beyond the scope of assistance which ServiceNow customer support can ordinarily provide