Add to Update Set Utility Documentation and Custom Configurations

[Michael Ritchie]

@Ben.hollifield , Co-Founder of Yansa Labs, has been maintaining the very popular Add to Update Set Utility in ServiceNow Share. The Add to Update Set Utility is a set of scripts that methodically add related records to an update set "automagically".  This Utility has been used for several years by 1000's of customers, partners, and employees.  Behind the scenes when you click the "Add to Update Set" Related Link on a record or List Drop-down Action, it checks a list of pre-defined tables and will not only add the record you are currently viewing, but add other important related records too automatically.  The full list of pre-defined tables can be found in the description on Share.  

Please note: This is a community created utility and any support related questions for the Add to Update Set Utility should be directed to the Discussion section of the Add to Update Set Utility item within Share. ServiceNow Support cannot provide assistance for this utility.

How to use the Add to Update Set Utility:

Once this utility is loaded into your instance, users with the admin role will be able to selectively add records to update sets.  A global "Add to Update Set" UI Action is included with this solution.

• Form Related Link: This will appear on any record that has "Related Links" towards the bottom of the form.
• 
• List Drop-down Action: Add to Update Set will also appear as a List Action in the drop down on every list allowing you to add multiple records to an update set as well.
• 
• Via Script: in the event you have many records that need to be added to an update set where the List Action isn't convenient, elevate your role to security_admin and navigate to System Definition \ Scripts - Background and execute a script to add records to your update set.  Unfortunately the GlideUpdateManager2() API that is utilized by this utility doesn't support asynchronous background processing so Fix Scripts and Scheduled jobs are not supported; Scripts-Background is the best method.  Below is a simple example script that will add all active Users to an update set:

var user = new GlideRecord("sys_user");
user.addQuery("active", true);
user.query();
while (user.next()) {
	new global.addToUpdateSetUtils().addToUpdateSet(user);
}

Multiple Application Scope Configurations:

For those of you familiar with "forcing" or "adding" items to an update set either with the Add to Update Utility or Force to Update and working with multiple Scopes, you know the importance of ensuring the Customer Updates (sys_udpate_xml) are in the right update set.  If not you won't be able to commit the update set in the target instance since a Preview Problem will appear complaining "Cannot commit Update Set X because: Update scope id "Y" is different than update set scope id "Z".  I have been partnering with Ben to enhance this useful utility so it also works with multi-scoped applications such as Human Resources Service delivery and I posted a blog article in the HRSD community called Migrating Configurations via Add to Update Set Utility with more details.

The Add to Update Set Utility accommodates multiple application scopes and inserts records into an update set matching the record's scope.  As the utility is adding records to the update set, it checks the record's scope against the currently selected update set for a match.  If the record is in a different scope, the current Update Set record (sys_update_set) is cloned but the Application associated is set to match the record's scope.  Utilizing Update Set Batching, the update sets are "batched" together for an easier migration.  Please refer to the Update Set Batching documentation for more details on this feature that was introduced in the Jakarta release.

Note: All instances created since Jakarta have this plugin enabled by default but any instance prior may need to activate this plugin.

• If multiple update sets are leveraged by Add to Update set, the original set will be renamed with a "Batch Child" suffix to note that it is a child update set.
• A parent update set will be created of the same name with a "Batch Parent" suffix.  No customer updates will be added to the Parent Update set, only the child update sets.
• Any additional scoped update sets created will have the same name plus "Batch Child" and also linked as children to the "Batch Parent" update set.

Let me provide an example.  Lets say you are working on ITSM enhancements and you have several catalog items that are in the Global application scope and a Virtual Agent Conversation that is in the ITSM Virtual Agent Conversations application scope.  You start by creating a new update set in the Global application scope, lets call it ITSM Release 11/2019.  Then you navigate to each of your catalog items and click the Add to Update Set Related Link to add them to your update set.  Next you navigate to your Virtual Agent conversation by entering sys_cb_topic.list and open the conversation you created and click the Add to Update Set Related Link.

• All of the Global catalog item components will be added to the 'ITSM Release 11/2019' update set.
• When the Utility attempts to add the Virtual Agent conversation it will detect that record is in the ITSM Virtual Agent Conversations application scope.
• The 'ITSM Release 11/2019' update set will be renamed to 'ITSM Release 11/2019 Batch Child'
• A new update set will be created in the Global scope (same scope as your original update set) called 'ITSM Release 11/2019 Batch Parent' and linked as the parent to the 'ITSM Release 11/2019 Batch Child'
• A new update set will be created in the ITSM Virtual Agent Conversations scope called 'ITSM Release 11/2019 Batch Child' and also linked as a child to the 'ITSM Release 11/2019 Batch Parent' update set.
• Then once you are done adding configurations to your update set, navigate to the 'ITSM Release 11/2019 Batch Parent', mark it complete and then click the Export Update Set Batch to XML Related Link to export a single XML file to import into your target instance.
• Then in your target instance in order to Preview a Batch of Update sets, you must navigate into the "batch base" in order to Preview and Commit it.  Please look for or Add the "Remote Batch Base" column in your Retrieved Update Set list, find 'ITSM Release 11/2019 Batch Parent', and click into it.

Confirmation Messaging:

The confirmation message after adding a configuration item has been enhanced to provide more information.

• Update set(s) utilized - clickable links to all update sets utilized.  This is useful when multiple application scopes are detected while adding configurations to the update set.
• Table list - list of tables from which configurations were pushed to the update set.  This is useful to confirm whether the Add to Update Set Utility includes functionality to include the related components for the configuration you added to an update set.  If a single record was added only one table will be shown.  Please post a comment here or in Share if related components are missing.
• Warnings - the Add to Update Set Utility can really only go so far though certain situations are useful to be warned about:
• • Custom Field Mappings in Catalog Items and HR PDF Templates:
  • • If custom "u_" fields are utilized in catalog item variables or HR PDF Templates a warning will appear letting you know.  The linked dictionary records will not be automatically added since these custom fields may already exist in the target instance.  If the custom field is new, you will also need to add that Dictionary (sys_dictionary) record to the update set.
  • Flow being added to an update set on an instance prior to New York.  Prior to the NY release, Flow update sets were comprised of multiple records.  The NY release flattened this out and now a single Customer Update (sys_update_xml) record is created for all the Flow components.  The Add to Update Set Utility will only work for releases NY and greater and will warn you if you are attempting to add a Flow to an update set on an instance prior to NY.
  • NLU Model's Protection Policy is set.  When adding a Virtual Agent Conversation to an update set all the related components including the NLU model will be added.  If the NLU Model's protection policy is set this means it either came from a Plugin activation or from the ServiceNow Store.  As a result those NLU Models will not be added and you will be warned about that.

Scope Script Execution:

The Kingston release introduced Restricted Caller Access to scoped applications.  This is an additional security feature that allows administrators of a scoped application to control which applications can access data within that application.  At the Table (sys_db_object) level, you can specify what CRUD (create, read, update, and delete) access other application scopes have on that table.  The Human Resources Service Delivery application for example utilizes this feature to protect data within its application tables.  As a result of these security features, the Add to Update Set Utility was enhanced to execute scope scripts so that configuration from restricted tables can be included in an update set.  The Global scoped addToUpdateSetUtils Script Include will query for other addToUpdateSetUtils Script Includes in other application scopes and execute the checkTable function within those scripts.  This gets around the security since queries to the restricted tables are being executed within that same scope.

The Add to Update Set Utility on ServiceNow Share includes pre-built functionality for the Human Resources and Employee Service Center capabilities.  This is a separate update set called "Add to Update Set HR v1.0.xml" that you can find in the Related Files section in Share.

If you have a custom application that has similar application access restrictions, you can create your own addToUpdateSetUtils Script Includes in the proper scope and the Global addToUpdateSetUtils Script Include will leverage that script.  Please use the HR scripts as examples where you include a checkTable() function that will then call sub-functions based on table name.

Custom Script Execution:

The ability to call custom scripts has included in the latest version.  This is useful for the following use cases and many more:

• You need to modify a value in a record prior to it being included in an update set.  Example: You create a knowledge or other record that has a reference to a User and you want to prevent an error in the target instance where that user may not exist in the target.  You can leverage custom script to intercept this and modify the record prior to inserting into the update set.
• You have custom tables and relationships with out of the box tables and you also want to include those records in your update set.
• You have custom configurations with dependencies that you wish to automate pushing into update sets.

In the saveRecord() function of the Global addToUpdateSetUtils you will see code that will look for an execute Script Includes named addToUpdateSetUtilsCustom.  These are optional scripts you can create within your instance to add custom processing to the Add to Update Set Utility.  If you have application access restrictions setup you can create multiple addToUpdateSetUtilsCustom Script Includes in your various application scopes.  Below is a very simple example script that I utilize internally to set the user fields in a knowledge record to out of the box demo users.

var addToUpdateSetUtilsCustom = Class.create();
addToUpdateSetUtilsCustom.prototype = {
	initialize: function() {
	},
	
	checkTable: function(tableRec) {
		/*
		 * If the table extends another table and you prefer to get all extensions, uncomment the default processParentTable = true and the if
		 * statement below it and set the table and function.
		 */
		
		var continueProcessing = true;
		var processParentTable = false;
		
		var tableName = tableRec.getTableName();
		switch (tableName) {
			case "table_name":
				this._addCustomRecord(tableRec, tableName);
				continueProcessing = false;
				break;
			default:
				processParentTable = true;
				break;
		}
		
		if (processParentTable) {
			// Check for table needs at parent table level
			var tableBase = new global.addToUpdateSetUtils()._getTableBase(tableName);
			switch (tableBase) {
				case "kb_knowledge":
					this._cleanKBFields(tableRec, tableName);
					break;
			}
		}
		
		return continueProcessing;
	},
	
	//Sanitize KB fields
    _cleanKBFields: function(tableRec, tableName) {
        var adminSysID = "6816f79cc0a8016401c5a33be04be441"; // User System Administrator's sys_id
		
		if (tableRec.isValidField("modified_by")) {
			tableRec.modified_by = adminSysID;
		}
		
		if (tableRec.isValidField("author")) {
			tableRec.author = adminSysID;
		}
    },
	
	_addCustomRecord: function(tableRec, tableName) {
		/* Add custom code here
		 * To add a record to the update set include the following:
		 * this._addToSet(tableRec);
		*/
	},
	
	_addToSet: function(tableRec) {
		//Must set the third parameter to false to prevent an infinite loop
		new global.addToUpdateSetUtils().saveRecord(tableRec, true, false);
	},
	
	type: 'addToUpdateSetUtilsCustom'
};

• Within the checkTable() function be sure to set the continueProcessing variable to false with each match so that the Global addToUpdateSetUtils will stop processing custom scripts if a match is found.
• In the example above, you have the ability to look for table extensions and have a single function for the inherited tables.  This is useful for situations where you are adding records of extended tables.  You will see that processParentTable is set to true in the default portion of the switch statement if no other matches were found and it will check the table extensions for a match.
• It is important to copy the _addToSet() function from this example.  As noted you must pass "false" as the last parameter of the API call back to the Global addToUpdateSetUtils otherwise you will encounter an infinite loop situation.  This parameter basically tells the addToUpdateSetUtils to not process custom scripts again. 

Executing a Script After Loading an Update Set:

After an update set is committed in a target instance, sometimes you may need to execute a script in that instance.  This is often done with Fix Scripts where the Fix Script is included in the update set and then someone manually executes that script after the update set is loaded.  An example on where this may be required is if you add a new required field to a form and you need to backfill all of the existing records with a value.

Included in the latest version is a stub function called addScheduledJob() that allows you to dynamically add a Scheduled Job (sys_trigger) record to the update set. This was added because NLU Models were included in the latest release.  When migrating NLU Models, even though the model may be published in the source instance, it is loaded in an unpublished state in the target instance.  This is because you must Train and then Publish the NLU Model in each instance.  Since those are API calls that you can execute via script, this process is automated by the Add to Update Set Utility.  The use of this function is leveraged in the _addNLUModel() function where a Scheduled Job is added dynamically to the update set to Publish the NLU Model.

• There is no way to control the order in which Customer Updates (sys_update_xml) records are loaded in the target instance.
• To get around this a Scheduled Job is included which is set to execute immediately when it is committed in the target instance.  This Scheduled Job's purpose is to create another Scheduled Job that will then run 60 seconds later - something we can control.  Idea is the update set shouldn't take longer than 60 seconds to load and thus the NLU Model should be committed by the time the second Scheduled Job executes which will then Train and Publish the NLU Model.

This same construct can certainly apply to other situations so I wanted to call out that this feature exists.  Feel free to look at the example code within the _addNLUModel() function and leverage it yourself in addToUpdateSetUtilsCustom Script Includes.  I will admit I struggled with the fact that a script is executing a script and all the quotation escaping situations so it could probably be refactored in a future release.  If you have any better ideas please post them in comments below.

Enjoy!