Compare Tool
Welcome to Compare Tool - created and maintained by Tom Hauri.
Overview
Compare Tool provides an easy way to compare records on local or remote instances and before and after a deployment/upgrade activity. The tool is highly configurable so that you can adapt it to your needs.
The main features are:
- Comparing a single record showing all field differences
- Comparing a list of records
- Comparing the count of records of specific tables
- Taking snapshots to compare the configuration before and after a deployment / upgrade activity
- Combining multiple Compare configurations to be run simultaneously.
The tool has been created a long time ago (Jakarta) and I updated it for the community article Cross Instance Compare Tool (a Global Scope to Scope App journey) again last year. Based on feedback and customer demand I have enhanced the tool with specific features helping operations teams staying on top of their instances when deploying huge applications or preparing for the cloning of instances.
Disclaimer
Compare Tool is NOT an officially supported ServiceNow product. Compare Tool does NOT come with any kind of warranty. If you use it, you own it!
System Requirements
- Utah or later
- Access to https://github.com/haurit/now-compare_tool
License
Copyright 2023 by Tom Hauri
Licensed under the Apache License, Version 2.0 (the "License")
You may not use the Compare Tool except in compliance with the License.
You may obtain a copy of the License at: https://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Release Notes
1.0.0 - 01.2024
- Initial release
- 1.0.1 Added second option to generate CSV
- 1.0.2 Fixed an error on type Local One
1.0.3 - 03.2024
- Added error message if Global Tools are missing
- Added error message details if remote request fails
- Fixed the field selection on Key (Fields) and Output (Fields) lists
- Corrected some demo data to avoid SysIds in the keys and fields list
1.0.4 - 04.2024
- Added Name field which is automatically populated and shown in the progress bar and results page
- Added Decription field
- Adapted forms and demo data to use the new fields
- Fixed potential issue with "this" confusion
1.0.5 - 01.2025
- Clean hidden attachments on delete
- Improved table display (fix title on top and other cosmetics)
- Improved clarity in results (show local or remote instance names in title)
- Added timer and cache to improve performance
- Corrected right table in case changed by Local to Local (One)
- Corrected link to initial version history
Problem Statement
Administrators often need to know the exact state or difference between instances or before and after a deployment or upgrade activity.
Here are some potential use-cases where the Compare Tool can help:
- Compare a Script Include between the local and the remote instance
- Compare the installed applications or plugins between two remote instances
- Compare the data (count) between two instances
- Monitor changes to tables that are extended from Application File [sys_metadata] (e.g. are included in Update Sets and Applications) but contain instance specific configurations (like Connections, REST and SOAP Messages) to avoid downtime after a deployment. With the versioning feature you can roll-back specific objects with the baseline "Revert to this version" feature.
- Monitor changes to tables (or data in those tables) that are not extended from Application File [sys_metadata] by taking snapshots of the objects. With the link to the objects in the snapshot you can roll-back specific objects by downloading the XML and loading it to the instance via the Import XML feature.
- Monitor changes to the table structure to identify added tables (or tables that did not contain data before and do contain data now) so you can include them in your Clone Profile Excludes or Data Classifications to ensure that data privacy requirements are met when cloning instances to sub-production.
Prerequisites
The Compare Tool makes usage of the Global Tools - YAGUSIS application. Ensure that you have successfully installed the latest version of the application before running a compare.
Installation
- Create an account on GitHub - if not done already.
- Create a personal access token for your GitHub account.
- Add credentials to access GitHub - use "Basic Auth".
- Fork the repository https://github.com/haurit/now-compare_tool.
- Go to Studio and import the Compare Tool application from source control.
You can also download an update set for the Compare Tool including Global Tools from the ServiceNow Share repository.
Usage and use cases
As defined in the Problem Statement section there are multiple use-cases where the Compare Tool can make your life easier.
Usage
The Compare Tool comes with its own application menu that is available for administrators only. Navigate to "Compare Tool --> Compare Configuration" to see the currently configured compare operations.
Configuring Remote Instances
If you want to compare objects from the local to remote or from remote to remote instance you need to configure the Connection & Credential Alias first. You can either navigate to the Connection & Credential Alias menu in the Navigator or open the default Alias records from the Compare Configuration records related list.
Please ensure that the credentials used to connect to the remote instance have appropriate rights to read the Table API and Stats API for the tables you want to compare. Also ensure that the date format is the same as you have for your local or other remote system to be able to compare dates correctly.
Use Case: Compare one record locally or remotely
Select this option to compare a single record using the baseline DiffHelper ScriptInclude. You have the freedom to select any record (even from different tables), but the results mostly make sense when you compare records from the same table.
When comparing a single record between different instances you can choose how the records are selected:
- SysId --> Using the SysId of the Record (Left) to identify the record on the Remote Instance (Right)
- Filter Condition --> Using a Filter Condition to identify the record on the Remote Instance (Right)
- Key Fields --> Using the one or multiple fields of the Record (Left) to identify the record on the Remote Instance (Right)
It is important that only one record is found on the remote instance, otherwise an error will be displayed.
Use Case: Compare a list of records
Select this option to compare a list of records using either the local instance or a remote instance (left) and compare the records that fulfill the same Filter Condition on a remote instance (right). Some lists can contain a huge amount of data (Script Includes including the "script" field). Please limit the number of fields or records to ensure a timely processing and avoid REST errors because the payload is to big.
You can configure the output with the following options:
- Output (Format)
- Screen
- Attachment (CSV)
This option will attach the result as a CSV file using a separator between the values if they are different - Attachment (CSV) Columns
This option will attach the result as a CSV file with two columns per field (left and right)
- Output (Filter)
- All differences (only on Left instance or Right instance or are different)
- Left only
- Right only
- Difference only
- All
- Key (Fields)
These fields are used as a combined key to identify if the records is the same on both instances. If empty the SysId is taken as key field. - Output (Fields)
These fields are used in the compare action, all other fields are ignored and not compared for differences. If empty all fields are used that exist on the table of the Left instance.
Use Case: Compare the count of records in tables
Select this option to compare the count of records using either the local instance or a remote instance (left) and compare the records that fulfill the same Filter Condition on a remote instance (right).
You can configure the output with the following options:
- Output (Format)
- Screen
- Attachment (CSV)
- Output (Filter)
- All differences (only on Left instance or Right instance or are different)
- Left only
- Right only
- Difference only
- All
Use Case: Compare the local history of a table
Select this option to create snapshots and compare the history of records using the local instance. This is useful if your operations team needs to ensure that deployments done by the development teams are not braking existing instance specific configurations (by accidentally changing instance specific settings either because of human errors, like including an instance specific system property, connection or credential in the package or by design issues like in REST and SOAP messages where the there is a mix of code and instance specific configuration if your not careful).
The Compare Tool uses two different methods to compare the records:
- Versions
If the table is derived from the Application File (sys_metadata) table and has the "Update Synch" attribute set, the Compare Tool records the currently active version as a baseline in the snapshot. - XML Attachments
For all other tables the selected records are converted to an attachment and stored on the Compare Configuration record for each snapshot. The hash for each XML is then used as a baseline in the snapshot.
You can configure the output with the following options:
- Table (Left)
This is the table for which you want to create the snapshots and compare the history later. It is important to understand that if you change this table, the existing snapshots will be invalid and you have to start again with the process to get meaningful results. There are two tables that have a specific functions:
Tables [sys_db_object]
Compare Configuration [x_snc_compare_tool_config]
This will be explained in more detail below. - Filter Condition
This filter condition selects the records for which you want to create the snapshots and compare the history. It is important to understand that if you change this filter condition, the existing snapshots will be invalid and you have to start again with the process to get meaningful results. - Output (Format)
- Screen
- Attachment (CSV)
- Output (Filter)
- All differences (only on Left instance or Right instance or are different)
- Left only
- Right only
- Difference only
- All
The following actions are required:
- Take Snapshot
Before you can compare you need to take a snapshot. This will use the methods described above to take a snapshot based on the configuration and store it on the Compare Configuration record as a hidden attachment. If successful it will return the current size of the cumulated attachments for your information. You can take as many snapshots as you want, but keep an eye on the size and cleanup snapshots that are no longer required. - Compare
One you have created a snapshot you can do your configuration (manual changes, deployments, upgrades etc.) and then run the "Compare" action to compare the current state (or a selected snapshot) with the latest state (or a selected snapshot) and the result is presented based on your output configuration. - Cleanup Snapshots
If you no longer need older snapshots you can use the Cleanup Snapshot link to delete all the snapshots and related attachments but the last one. - Show Progress Workers
If you need to cancel a progress worker you can use this link to open the running progress workers and set the status to "Cancelled".
Use Case: Compare the local history of the Table [sys_db_object] table
Select the configuration above and set the Table (Left) to Table [sys_db_object] to create snapshots and compare the history of tables in your local instance. This is useful if your operations team needs to know if new tables have been added or tables that contained no records in the past have been populated in order to configure data privacy settings (either for Clone Excludes setup or to configure the Data Classifications.
Additionally to the options available for other local history tables you can select:
-
If set to true will additionally capture the count of records for each table.
Use Case: Run multiple Compare Configurations simultaneously
Select the configuration required and set the Table (Left) to Compare Configuration [x_snc_compare_tool_config] to run multiple actions at once. This is useful if you want to take snapshots and compare the results for multiple tables at once instead of having to go to each Compare Configuration record and do the actions one by one. To have a meaningful result the query is automatically restricted to the same Type of action either when you run it or when you save the record.
The options selected on the parent are passed to the child Compare Configurations and the result of the Compare action is only available on the parent while the snapshots for example are stored on the child records.
You can see the progress of the action for each individual Compare Configuration:
Open topics
Apart from further testing and creating meaningful ATF tests I have identified the following areas for improvement:
- Provide more demo data
- Add direct links / URLs for other compare types (currently only implemented for History)
- Improve the behavior of the progress bar to be more accurate and granular
- Run huge compare jobs simultaneously and use multiple background runners
- Make report generation more generic and allow for "plugins" to be written
Request for help
This was and is a complex project and the possible use-cases and failure modes are many. Please give me feedback either via mail or as comment on this page or on the GitHub repository.
14 Comments
