MID Server pre-upgrade check
Summarize
Summary of MID Server pre-upgrade check
Before upgrading a MID Server, the system performs pre-upgrade validation tests to identify potential issues that could cause upgrade failures or MID Server outages. An AutoUpgrade monitor within each MID Server compares its version to the instance version and triggers these checks if an upgrade is needed. If blocking issues are found, the upgrade is halted and errors are logged, with the monitor retrying every hour until resolved. This process applies to both instance-triggered and manual MID Server upgrades.
Show less
Pre-upgrade Tests
The pre-upgrade validation includes two types of tests:
- Mandatory tests: All must pass to proceed with the upgrade. Failures are logged as active issues in the MID Server Issue [eccagentissue] table and block the upgrade.
- Alert tests: Failures generate warnings but do not stop the upgrade process.
Mandatory tests check for:
- Minimum 2 GB free disk space on MID Server host
- Access to the installation download site (install.service-now.com)
- Successful digital signature verification of a sample downloaded file
- Permission to extract, copy, read, and delete files in upgrade temp folders
- Host system must not be 32-bit
- On Linux, glibC version must be at least 2.17
Alert tests on Windows MID Servers include:
- Presence of upgrade-blocker services running on the host
- Status of Application Experience service on the host
- Warnings if the MID Server runs as LocalSystem or an administrator user (not recommended for security)
Common Blocking Issues and Warnings
Blocking errors that prevent upgrades are logged in the eccagentissue table and in the agent logs and include:
- Insufficient disk space (less than 2 GB free)
- Failure to download update packages due to network or permissions issues
- Digital signature verification failures
- Permission issues extracting or writing upgrade files
- Unsupported host architecture (32-bit) or outdated Linux glibC version
Non-blocking warnings include issues with Windows service account configuration and PowerShell availability, which do not prevent upgrades but may indicate environment misconfigurations.
Disabling Pre-upgrade Checks
By default, the pre-upgrade tests run automatically (controlled by the mid.upgrade.runprecheck parameter). Customers can disable these checks for individual MID Servers by setting this parameter to false in the MID Server’s config.xml, or globally for all MID Servers by creating the property in the MID Server Property [eccagentproperty] table with a false value.
Practical Guidance for ServiceNow Customers
- Ensure your MID Server host meets the minimum system requirements—especially disk space, architecture, and library versions—to avoid upgrade blocks.
- Verify network connectivity and permissions to access ServiceNow's install site and perform file operations during upgrade.
- Address any mandatory test failures promptly as they block upgrade progress.
- Review alert warnings to improve MID Server stability and security, such as running under least-privileged accounts.
- Keep the Application Experience service enabled on Windows hosts to support successful auto-upgrades.
- Consider disabling pre-upgrade checks only in controlled scenarios after evaluating risks.
Prior to an upgrade, the MID Server runs tests to identify issues that could cause the upgrade process to fail or result in a MID Server outage.
Each MID Server contains an AutoUpgrade monitor that compares the MID Server version with that of the instance to determine if the MID Server needs to upgrade. If the AutoUpgrade monitor discovers that the MID Server version is out of date, the monitor runs pre-upgrade validation tests for that MID Server. If an issue is detected, a message is logged to the MID Server Issue [ecc_agent_issue] table, and the upgrade is blocked. The AutoUpgrade monitor continues to run every hour, until all the tests pass. If there are no blocking issues, the MID Server downloads the appropriate upgrade package and begins the upgrade process. The MID Server pre-upgrade check also runs for the upgrades triggered by the instance when the instance was upgraded, as well as for manual upgrades.
Pre-upgrade tests
The pre-upgrade runs two sets of tests: Mandatory testsand Alert tests. Every mandatory test is required to pass to start MID Server upgrade. If a mandatory test was not passed, the test is recorded in the agent log and a MID Server active issue is created in the MID Server issue [ecc_agent_issue] table. These errors are published before the actual MID Server upgrade occurs and must be resolved before the upgrade can continue. MID Server stops the Upgrade process and changes the status based on MID Server states. If any alert test was not passed, a message is added in the agent log and MID Server issue table, but the pre-upgrade check continues and MID Upgrade doesn’t stop.
- At least 2 GB of free disk space
- Access to the download site at install.service-now.com
- Verify the digital signature of a sample downloaded file
- Permission to execute, which involves the following:
- Extract a Zip archiver to the upgrade temp folder: agent/work/upgrade_temp
- Copy files from the upgrade temp folder to the agent folder
- Read a text file and validate the content
- Delete the pre-upgrade check contents
- MID Server host machine is not a 32 bit system
- On Linux MID Servers, the glibC version is at least 2.17
- Any MID Server Upgrade blocker services is running on the host machine
- Status of Application Experience on the host machine
- For security best practice, run the MID Server as a non-administrator user. If the current Log On As user account for the MID Server is LocalSystem or is a user that is part of the Administrators group, the upgrade does not fail, but this configuration is not recommended.
Errors that block the upgrade
- Not enough free disk space. The system reports <n> bytes free
- This message is displayed when less than 2 GB of free disk space is detected on the MID Server host. This error is also written to the MID Server agent log.
- Unable to download updates from the install server
- This message indicates that either the MID Server host does not have permission to download the installation package from install.service-now.com, or network problems prevent connection. This error is also written to the MID Server agent log.
- Signature verification failed:<message>
- This message indicates that a Verification Exception has occurred when pre-upgrade check tries to verify the digital signature of the downloaded check file.
- Unable to verify digital signature:<message>
- This message indicates that the verification signature process was interrupted by an exception.
- Unable to extract contents of pre upgrade check zip
- This message indicates that the service account on the MID Server host does not have permission to extract the pre-upgrade ZIP archive to the temporary folder.
- Unable to create folder <upgrade check file path>
- This message indicates that the MID Server service account does not have permission to create the upgradeCheck folder for the pre-upgrade checking files in the agent/package path.
- Unable to verify file permissions: <message>
- This message indicates an exception has occurred when checking file permissions, such as a file that does not exist or access failure.
- The MID Server Windows service runs as LocalSystem or a user that is part of the Administrators group.
- This message warns that the Windows service is running with elevated privileges, which increases the risk of unintended access or actions.
- Upgrade Failure: The host machine does not meet the minimum system requirements to upgrade the MID server.
- MID upgrade is not supported on 32-bit host. Please refer to KB0863694 for more information.
- Upgrade Failure: Host does not meet the minimum system requirements to upgrade the MID server. Please refer to MID Server product documentation on GlibC library for more information
- This error can only occur for Linux MID Servers. Refer to Java version support for more information.
Non-blocking warnings
- Unable to parse $logOnAsUser
- This message warns that the Log On As User value for the Windows service is not in
either of these expected formats:
- user@domain.company.com
- domain\user
- Unable to look up Log On As user's groups
When the instance attempts to look up the logged on user's group memberships, it executes the net user <username> command. The instance expects a certain output structure by the Windows service from this command and issues this warning if the expected output does not match the actual output.
These PowerShell warnings are written only to the MID Server agent log. Because PowerShell is not required to use a MID Server, these configuration issues do not prevent a Windows MID Server from upgrading. However, these warnings might indicate issues in your environment that require attention.
- Skipping PowerShell upgrade checks since PowerShell is not usable
- PowerShell 3.0 (at a minimum) is not installed or powershell.exe is not available to the MID Server service user.
- Skipping Powershell test: Target folder has not been initialized
- Skipping Powershell test: Unable to detect the service name
- Service CiscoAMP running on the MID Server host machine is identified as a MID Server upgrade blocker
- For more information, refer to KB0870329.
- The Application Experience is disabled on the MID Server host, which can cause the auto-upgrade process to fail and the MID Server to go down. Enable the service on the host to ensure successful auto-upgrade
- For more information, refer to KB0597552.
- Continuing with upgrade, but the following issue was encountered during upgradeCheck: <exception message>
- This message indicates that there was an issue running the PowerShell portion of the pre-upgrade tests.
Disabling the pre-upgrade check
A MID Server configuration parameter called mid.upgrade.run_precheck is set to true by default, which allows the automatic pre-upgrade test to run. To disable these tests for a single MID Server, add this parameter to that MID Server's config.xml file and set it to false. To disable these tests for all MID Servers, add a new record to the MID Server Property [ecc_agent_property] table called mid.upgrade.run_precheck. Set the value of this property to false and leave the MID Server field blank.