Introducing ServiceNow Data Separation application

AttilaVarga — Thu, 31 Aug 2023 13:31:43 GMT

A new feature has arrived with the Vancouver release, related to the security area, called Data Separation. It sounds very interesting, but I need to highlight, that this application is supported only for Project Portfolio Management. I hope that this will be extended in the future. I tried this solution and collected all my experiences, which I would like to detail in this article. I'm going to write about all the important features and limitations, I believe it will help people to decide whether to introduce it to customers.

Data Separation application in general

The following types of entities are supported by the application:

Demand
Project
- Project Task
Resource Plan
Cost Plan
- Cost Plan Breakdown

It also supports records, related items, planning consoles, workbenches, and reports.

How it can be setup

Data Separation is available in the store, which has to be installed first.

Because the functionality is related to the Strategic Portfolio Management product, there are two pre-requisites:

PPM (Project Portfolio Management) Standard
Portfolio Planning

Roles

Two roles are coming with Data Separation:

sn_ds.ds_admin
- This role is used for setup and maintaining the application
sn_ds.ds_privileged_user
- Data separation rules are not evaluated, all records are visible for users who have this role.

Tables

The following two tables are created during the installation of Data Separation:

Supported entity (sn_ds_data_separation_config)
- This table stores the entities where the Data Separation functionality can be enabled and used.
Entity-Group mapping (sn_ds_entity_group_mapping)
- The records of this table connect the Lens entities with user groups, which manage the visibility of records.

Besides these two tables, the third important one is the Lens (sn_align_core_lens). The Lens object is used for Portfolio and Strategic Planning. It helps to build up different levels of entities as a parent-child format. This structure is used by Data Separation to give access to an object to somebody, who is at a higher level of the hierarchy.

Further information can be found in Lenses and portfolio plans and Lenses in Strategics planning.

Domain Separation and Data Separation together

The two features are not mutually exclusive. Domain Separation gives a higher level of data segregation and Data Separation adds another layer to it, which is set on the group membership level.

Limitations

The following list contains the limitations of the application, which I faced with and read about:

Data Separation application can be used only for Strategic Portfolio Management.
Data Separation does not ignore the ACLs and Query Business Rules. (For example, reports with aggregate numbers may not be data separated.)
Based on some special circumstances the application can cause performance issues in the system.
Only one active Lens can be in the system, which means that it is quite hard to implement complex data separation logic. ServiceNow does not allow enabling multiple Lens for data separation.

How it works

⚠️Important information

As mentioned before, Data Separation is based on ACLs and Query Business rules. To increase the performance, ServiceNow uses session variables in the Query Business rules. When the DS configuration changes, these session variables are not invalidated. To re-calculate the values, it is needed to close the browser and open it again!

The logic behind the functionality is simple. There is a key field, called Data separation groups (data_separation_groups). This is a list-type field, which is a reference to the Group (sys_user_group) table. The content of this field manages the visibility of object(s).

There are two specific groups, that are used for the Data Separation:

DS Privileged Users Group
This group is added to the Data separation groups field automatically in all cases when the data separation logic is evaluated on the record.
Group for Non Data Separated Entities
This Group is added to all records, where the data Separation logic is not used or not interpreted.

These two groups have no members and it is highly recommended to NOT delete them if Data Separation is enabled because it can break system functionality.

If the DS is enabled, all records should contain one of these groups, if the functionality is enabled on the actual table.

The Data separation groups field is read-only and managed by business rules/scripts.

To summarise the purpose of these groups, they give access to the ticket (Group for Non Data Separated Entities) or restrict it (DS Privileged Users Group).

The second layer of the restriction is the evaluation logic, which consists of Query Business Rules and ACLs.

⚠️Important information

If the Strategic Portfolio Management product is heavily customised, my recommendation is to create a well-structured plan for the implementation taking into consideration the review of ACLs and Business Rules.

Configure Data Separation

As a first step, the entities have to be selected where Data Separation will be used on.

Then, the Lens structure has to be built up, which is followed by the Lens Entity and Group record relationship definitions. Finally, the configuration has to be synchronised with the Data Separation.

In order to understand how it works the best is to see some examples so the next part of the article is about going through some of them.

Base setup

The following configuration was set in ServiceNow for testing purposes:

Roles

Name	Description
it_project_manager	This role gives full (read/write) access to PPM application. All projects can be opened and edited.
sn_ppm_read	This roles gives read-only access to all PPM related objects

Groups

Name	Role
Project mgr test 1	it_project_manager
Project mgr test 2	it_project_manager
Project mgr test 3	it_project_manager
RO Project users	sn_ppm_read
HR Project managers	it_project_manager

Personas

Name	Department	Group
Abraham Lincoln	IT Operations	Project mgr test 1
Alissa Mountjoy	IT Infrastructure	Project mgr test 2
Amos Linnan	IT Services Management	Project mgr test 3
Allyson Gillispie		RO Project users
Andrew Jackson	HR Services	HR Project managers

Organisational structure

Data separation Lens structure

Based on the organisational structure of Test Company, the following lens structure was created:

To use this data structure in the Data Separation logic, it has to be active, and the Data separation enabled checkbox has to be checked.

The configuration can be modified anytime. Once a modification is done, the value of ‘Is data synched’ becomes false, which means that the corresponding object visibility settings (in our case Projects / Project tasks) have to be synchronised.

The Lens setup is done. Now we have to create the mapping, which defines the relationship between Lens entity records and Groups. Based on our Lens structure, the following mapping definition was created:

As a first and most important action is synchronisation. Any modifications on the entity-group mapping definitions, have to be synchronised.

When the Sync data separation button is clicked, the system schedules an asynchronous job to update the value of the Data separation groups field on all affected records.

Need to wait a couple of seconds until the synchronisation is done. Now the configuration is ready to use. Let’s see how it works in the system.

As a fist action Abraham Lincoln submits a new project:

Because the content of the Department field was changed, the data separation logic was executed and two groups were set in the Data separation groups field, based on the Entity-Group mapping definition:

DS Privileged Users Group
Project mgr test 1

So the only users able to open the record are the members of the Project mgr test 1 group. There are two exceptions:

The user has sn_ds.ds_privileged_user role
The user is a member of a group, which is defined to a higher level of hierarchy in the Entity-Group mappings

Remember, the DS Privileged Users Group has no member, the purpose of this is to be an indicator of restriction.

To check that the separation functionality works well, using Alissa Mountjoy’s account the project record is not visible, by either opening the record directly or listing it:

⚠️Important information

I would like to highlight a dangerous factor. Users can easily close out themselves from the record. By default, the Department field is auto-populated, but editable. If the user changes the record, the system re-calculates the value of the Data separation groups field. If the user is not a member of any groups in the list, the record becomes hidden for the user.

Let’s take another example. Allyson Gillispie is a member of the IT Management group. Members of this group are managers of the IT Business Unit. Allyson has the sn_ppm_read role, which gives her access to all projects, but she should only see the IT BU related and the public projects.

In order to set it up, a new Entity-Group mapping record has to be added to the mapping table.

Now the mapping connects the group with the Business Unit record. It means that the users will be able to see all projects, which are part of departments under IT BU.

Let’s create a new project and add it to the HR Services department. Now we have two projects, one is related to the HR Services department the other to the IT Operations.

Each persona can see different records in the system:

Abraham Lincoln

Andrew Jackson

Allyson Gillispie

If a new project is created for the IT Service Management department, Allyson will also see that one.

Data separation on the Project Task level

In order to use Data Separation on the Task level, the first step is to enable it in the Supported Entities table.

When a Project task is created, the Data separation groups field is filled automatically, and the visibility logic is the same as in case of Project.

Closure

These examples are related to Project Management, but the functionality is the same in case of other types of objects as well.

From the configuration and maintenance point of view, we need to keep in mind, that there are two key fields used in the logic:

Data separation groups, which is responsible for the visibility of records
Lens identifier field (Bottom entity on planning item), which can be one of the following:
- Business application
- Business service
- Department
- Execution system
- External identifier
- Owner
- Portfolio
- Primary goal
- Service offering

One of these fields will be the key, to how the groups will be calculated. Not all fields are available on each type of record, selecting the correct one should be carefully considered at the design stage.

Data Separation can be set up easier on an out-of-the-box or not heavily customised SPM product. The lead time of implementation might be too long in other cases.

My opinion is that the functionality covers several use cases, but not everything. For example, it is not possible to ad-hoc involve multiple departments in the project using the OOTB available components.

The project object has a lot of relationship elements, but just a few are covered by Data Separation.

This is just the first version, I believe several great features will come with the later releases.

Re: Introducing ServiceNow Data Separation application

nikhilzagad — Fri, 08 Dec 2023 15:25:39 GMT

Great information. Fantastic effort. Much appreciated

Re: Introducing ServiceNow Data Separation application

Attila Beregvar — Wed, 03 Jan 2024 11:03:10 GMT

Well detailed and useful information, thanks for the effort!

Re: Introducing ServiceNow Data Separation application

mcastoe — Wed, 07 Feb 2024 16:58:29 GMT

If you are not using an out of the box Lens, for example say Business Unit is your bottom entity, then you will likely need to update the script include, sn_ds.DSLensLeafEntityMapping, to add your bottom entity reference. Also, you should make this configuration change before you do anything other configuration in Data Separation as you will end going in circles trying to sync things.

In the example below, we have our Data Separation Lens such that Business Unit is the bottom entity. We added Business Unit [u_business_unit] to sn_align_core_planning_item and setup appropriate Lens integration as well. We now change to Data Separation scope and open teh script include sn_ds.DSLensLeafEntityMapping. Modify the LEAF_LENS_ENTITY_MAP and then save. See example below in bold blue.

//"lens_leaf_Entity" : "field_name"

DSLensLeafEntityMapping.LEAF_LENS_ENTITY_MAP = {

"pm_project": {

"primary_goal": "primary_goal",

"department": "department",

"product": "product",

"pm_portfolio": "primary_portfolio",

"strategic_program": "strategic_program",

"u_business_unit": "business_unit"

"dmn_demand": {

"primary_goal": "primary_goal",

"department": "department",

"product": "product",

"pm_portfolio": "portfolio",

"strategic_program": "strategic_program",

"u_business_unit": "business_unit"

}

Re: Introducing ServiceNow Data Separation application

AttilaVarga — Thu, 08 Feb 2024 10:14:06 GMT

@mcastoe, thanks for sharing this information, this is very useful for extending the capability of Data Separation!

Re: Introducing ServiceNow Data Separation application

Suzanne Jones2 — Mon, 17 Jun 2024 13:45:40 GMT

Thanks for the information - good article. @mcastoe does ServiceNow have a roadmap for Data Separation enhancements for future releases?

Thanks for your help,

Suzanne

Re: Introducing ServiceNow Data Separation application

mcastoe — Mon, 17 Jun 2024 17:38:26 GMT

@Suzanne Jones2 ,

they are definitely working on it but there are new platform filtering and access capabilities which we are waiting for. it will likely be the X or even Y release before we see added capabilities in Data Separation. If you have data separation use case you would like to share with us, please post here or just in the SPM forum, and i will keep a special eye on this thread and make sure our Product Manager get the use case.

thank you,

mark

Re: Introducing ServiceNow Data Separation application

Brian94 — Wed, 14 Aug 2024 17:20:37 GMT

Thanks for the information. It is very helpful. I do have a question though. I am still a little unclear how to utilize the "Bottom entity on planning item" field in the Lens configuration. If I were to change that on the OOTB Lens "Organization" from "Department" to "Primary Goal", would that now mean my data separation is dependent on "Primary Goal"? Also, would I need to change or add an entity level to the Lens?

post Introducing ServiceNow Data Separation application in Developer articles

Introducing ServiceNow Data Separation application

Data Separation application in general

How it can be setup

Roles

Tables

Domain Separation and Data Separation together

Limitations

How it works

Configure Data Separation

Base setup

Data separation on the Project Task level

Closure

Re: Introducing ServiceNow Data Separation application

Re: Introducing ServiceNow Data Separation application

Re: Introducing ServiceNow Data Separation application

Re: Introducing ServiceNow Data Separation application

Re: Introducing ServiceNow Data Separation application

Re: Introducing ServiceNow Data Separation application

Re: Introducing ServiceNow Data Separation application