All About Data Resources in UI Builder

[Brad Tilton]

Data Resources in UI Builder

A data resource [sys_ux_data_broker] in UI Builder mediates data between a component and the ServiceNow server. The data resource brings data from the server to the component, and it brings updated data back to the server based on interactions with the component.

Why Data Resources?

In the early days of the Next Experience UI framework, all component handled their own data communication with the server (we call this type of component a Connected Component). Connected components were/are easy to set up but they lack the flexibility to be reused for different use cases and a side effect is that it’s difficult for us to optimize the data, caching, or network calls. Each instance of a connected component is a little black box that operates as a mini application. Any flexibility is controlled by the component author’s use of inputs and feature flags. This makes it more difficult for end users to consistently define the layout, composition, or interactions of these components, even if internally they are composed of reusable low-level components.

Because of that, we’ve been moving towards what we call Presentational Components that don’t handle their own data communication with the server. With these components, you bind data to the component’s properties and then the component uses events to communicate with other components or to send data back to the server.

But how do we get data from the server to feed into these presentational components? Well, that’s where Data Resources come into play. Data resources are defined using UXF metadata models and are dynamically generated at runtime which means that there is no hardcoded JavaScript and users are free to define any kind of server interactions they wish. These data resources can be defined by users, instantiated, and chained together to fetch rich and complex data sets that can be exposed to components.

Since we know what data is needed and how it's used, we can make optimizations in dealing with it. First, because the data is separated from the presentational components, multiple components can now hook into a shared state which means we can now de-dupe data that is shared and only fetch it once. Also, if the UI uses data from the tail end of a chained (composite) data resource, we fetch the entire chain as a single query. The endpoint processes the data we need and only returns the data we use in the UI. If data is used at various points through this chain, we return all the necessary data at each step. Our engineering teams are also continually looking at ways to optimize performance in UI Builder and data resources are a large part of that.

Types of Data Resources

Now that we know why we need them, let’s explore the different types of data resources.

GraphQL lets you write and/or use a GraphQL query against a ServiceNow GraphQL API. If you’re building one of these you’ll need to specify the query body, but that’s about it.

REST lets you execute a REST request against a ServiceNow REST endpoint. You’ll need to specify an endpoint, query params, and a request body.

Scriptlet runs sandboxed, vanilla, client-side JavaScript. It is primarily used for transforming the output from one of the data resources above into something usable by your components. The scriptlet is more preferable than a transform because the scriptlet loads and runs faster due to not loading any external APIs.

Transform runs any ServiceNow server-side JavaScript. If you already have a library of script includes for your app you can use this type of data resource to consume that library.

Composite data resources are a combination of other data resources chained together. Using a composite instead of being the outputs of one DR to another allows us to optimize the performance. It’s especially useful for GraphQL+Scriptlet combinations.

Controllers are new in Tokyo and, when combined with preset-enabled components, can be automatically added to a page when the component is added. For now, this is only usable by ServiceNow’s internal engineering teams.

Read vs Mutate

Each type of data resource can either read or mutate (write) data. This is controlled by a checkbox on the data resource record called Mutates server data which changes how UI Builder handles the data resource.

Read [Mutates server data = false]:

Runs when:

• The page loads
• Any client state parameters bound to its properties change
• The refresh event is triggered.
  

To pass data to its inputs you can bind data to its properties as shown below. In the example you can see that the Table property is hardcoded to my recipe table, but the Record property, which is expecting a record sys_id, is bound to the recipeId URL parameter. If that was a client state parameter and it changed it would cause the data resource to refresh.

Mutate [Mutates server data = true]:

Runs when:

• Its Execute event is triggered
  

To map data to its inputs you can pass data when the data resource is executed. You’ll see that, even though this data resource has properties, they’re not available when you add it to the page.

If we look at the Execute event handler you’ll see we can map its 4 properties there.

Because mutating data carries so much more risk than querying data it’s important to be very intentional about creating and using data resources. If your DR does write to the server it is very important that the checkbox is checked so it doesn’t try to run on page load, etc.

Anatomy of a Data Resource

Properties: Each data resource can have its own set of unique properties, which are basically the inputs for the DR. You can find a list of types and the JSON schema here: Component and Data Resource Properties in UI Builder

Script/Request body: Based on the properties and whether you select Script or Query this is where you’ll specify what your data resource does with the properties provided. This can be a request body, GQL query, or script that queries or mutates data.

Events: Every data resource can fire a series of events. Each of them fire an event when they’re initiated, when they succeed, or if they fail.

For a data resource that queries data, these are called Data Fetch and show up on the Events tab in the Data panel in UIB.

For a data resource that mutates data, these are called Operation and show up on the Events tab in the Data panel in UIB

These data resource events can be very helpful when developing in UIB. Many times you’ll want to run a script or show a message based on the results of a data fetch and attaching an event handler to the Succeeded event is a great way to do that. You may also want to introduce some error handling and messaging and using the Failed event is useful for those use cases.

Common Data Resources

If you’ve opened the data panel in UI Builder you may have been overwhelmed with the number of data resources that show up. They are organized by Application they were created in, but most are in the Global tab. There are a few really common data resources that I generally use when developing in UI Builder:

1. Look up record – based on a sys_id, look up a single record.
2. Look up records – based on the properties you specify, look up a set of records.
3. Create record – this mutate DR allows you to create a single record.
4. Update record – this mutate DR allows you to update a single record.

Honorable mention: Look up Properties/User/User Preferences

Additional Information

If you’re looking for even more information on Data Resources in UI Builder, check out the following links:

• Docs: Data resources in UI Builder
• Developer Learning: Data Resources in UI Builder
• Developer Blog: UI Builder - Data Resources

Stay tuned for an upcoming Data Resources Best Practices article.

[JagjeetSingh]

The data source which mutates server data is often triggered by event or client script. The output of such data source is not immediately available to use and is only available after you trigger it. To read the output you can use the event payload sent by "Operation Succeeded" event.

For example you can trigger your client script from this event and in client script you can read data like below.

const output = event.payload.output.data;

[Shakeel Shaik]

Thanks @Brad Tilton for sharing 🙂

[Stephanie35]

Update: I always forget about payloads. I was able to make a decision on the payload: (event.payload.data.output.data.GlidRecord_Mutation.insert_<tablename> == null, means table was not inserted).

I then used api.emit("NOW_UXF_PAGE#ADD_NOTIFICATIONS, ...

However, I wanted the notification to come up on the modal and not on the main page.

Anyway, wanted to share the info just in case someone else came across this question.

I came across this article while researching data resources and the operation failed event. We have a business rule on a table to prevent users from creating duplicate entries. While some users can create entries on the table through the an event on a page, other users can create records using the List page template, therefore opening a form to add. We felt a business rule would manage both well.

I'm not sure how or if we can return a operation failed indicator from the business rule to our create data resource. We have added a gs.adderrormessage for a failure and gs.addinfomessage for success and that does appear in the page, but both show up with the blue informational shading. Looking at the message element on the page, it appears to use the now-event of type info for both messages. I'd really like to use the the events on the data resource to manage the messages though.

This article has me thinking, but I just haven't connected all the dots yet. Any ideas?

[Tae Kyung Lee]

It seems that non-mutate data resource cannot use "Only when invoke" mode despite of it can be selected.

Is this on purpose or product bug? My instance is Tokyo patch 5

[Community Alums]

@Tae Kyung Lee  Did you ever get an answer to this? trying to use link to destination at the end of several events that been linked, but i triggers immediately. 

[Tae Kyung Lee]

@Community Alums No, I didn't. So I am using mutate data resource only.

[Daniel R2]

Hi @Brad Tilton - I am currently trying to configure the contextual side panel using the UI builder.

The aim is to show only the variables that are part of a request within a menu option on the contextual sidebar. So far we have created a page, created a variant and we have added the 'Display only form' component with our custom view to show the variable editor attached to the forms. We are then using a look up data source to the case table.

As you can see in the below screenshot, we can see the variables fields for the specific case (CS0001003) which is great, and this is how we want it to look. However it will not work dynamically. For e.g. when changing to another case on the workspace? How can we fetch the sysid of the open record so that this functionality works dynamically? I believe I am missing a configuration somewhere. if possible to provide an answer/guidance here it would be greatly appreciated. 

We have seen other posts which refers to adding @context.props.sysId to the Sys ID field. However this did not work and actually made what is shown below disappear.

[Brad Tilton]

@Daniel R2 I don't think you actually need a data resource here. It sounds like you just need the sysid from the parent page. There is a method you have to use when passing parameters from a parent page into a viewport subpage (which is what you're doing with the contextual side panel). I outline that method in this response. All you need to pass to the display only form is the sysid which you already should have from the parent page.

If you run into issues with this method please create a new post and tag me.

[Daniel R2]

@Brad Tilton - Thanks Brad for taking the time to reply! it is much appreciated! 

We did go with the method of passing the parameters from the parent page into the viewport subpage, however we did not get any success. We tried the method again from fresh, however still had no luck. We could see the view we wanted however was not working dynamically on case sys Id.

I have tagged you in my colleagues community post, where screenshots are provided of our set up and what it currently looks like. If you have the time to take a look please, again it would be greatly appreciated, thank you

You can also navigate directly to the post below.

UI builder - Unable to show variables in contextual sidebar in configurable workspace

[Bernard Esterhu]

@Brad Tilton  thanks a lot for the guide, very useful.

I am picking up a problem with one of the data resources I have created. I have a data resource that retrieves a list of services that I want to show in a dropdown list. We have a very large number of services (around 35000), so I have modified the max_results parameter from 1000 to 40000. However, this change seems to have made the page very resource heavy. Saving the page in UI Builder takes extremely long, and loading the actual page in the browser also take very very long. Is this the expected behavior?

[Brad Tilton]

Hi @Bernard Esterhu that is a LOT of results to return from a data resource and then populate into a dropdown. You might consider using a typeahead search with your data resource and not executing it until a term is searched or using some pagination on your data resource so it's not returning so much data at once. 

[Jamie Stroud]

@Brad Tilton so, any word on the issue raised by @Tae Kyung Lee about how non-mutated data resources with properties bound to client state params will refresh as soon as those params change values, making the idea of configuring your DR as "Only when invoked" pointless? Having to make data retrieval DRs be "Mutates data = true" just to get around this seems wrong.

[Brad Tilton]

@Jamie Stroud I believe all data resources and components will refresh when their properties change. Only when invoked prevents the data resource from running on page load and in a lot of other scenarios outside of the properties changing. If you don't want the data resource to run in a scenario where its props might change you should change the way you're binding data to the props. A client state parameter may come into play here where instead of directly binding data to the DR prop you bind a CSP to the DR prop and control when you're setting that CSP.

I'll confirm this with the engineering team and update the article to make it clearer.

[Jamie Stroud]

@Brad Tilton thx for the blazing fast response. For us SN devs out here, good to know you guys are supporting IRL as well as on video and blog posts. Good man!

And yea, as your suggestion states, I literally just tested that concept and it does work, but feels clumsy. My use case is a simple one: Page (well, sub-page loaded in a viewport, to be precise) with a single input and button. User enters search term and clicks the button to trigger the search (a custom TRANFSORM DR). I need a state param to just hold the value of the input as the user sets is (Input value set). And I created a separate one to bind to the DR prop. On click of my button, I update the state param for the DR prop with the value of the input state param, and viola.

But I think that design would fall apart if you had, say, 3 inputs all holding values that would be passed as DR props because as soon as you go to api.setState on the first DR state param of the 3, the DR itself would refresh. Seems like unnecessary complexity to do something that should be pretty simple which is to fire a data retrieval exactly when, and ONLY when, you want it to.

Thx again for the support.

[Jamie Stroud]

@Brad Tilton and anyone else, I have a follow-up on this that's bugging me. Seems like a bug related to Data Resources. Here's the scenario:

• I have a "Standard" record page (so we're talking Record Controller).
• I have a subpage loaded into a viewport component on the record page (in the side panel).
• In the subpage, I have a page property defined and bound to @data.record.form.fields so I can have access to the user interacting with the fields on the form.
• I have a TRANSFORM Data Resource with parameters bound to a Client State param, set to evaluate "Only when invoked".
• I handle the "Page Property Changed" event of the subpage with a script so when the user changes the value of a field on the form, my page property bound to data.record.form.fields will have changed. I only care about a subset of the form fields for my subpage so I only update my client state param if one of those field's changes.
• PROBLEM: My data resource's "Data Fetch Initiated" is firing whenever a user changes ANY field on the form, even when I do not update my client state param that is bound to the DR's inputs. Even when I kill the "Page Property Changed" event handler all together! So the whole "Only when invoked" thing seems to not be working as expected.

[Arnoud Kooi]

Still a great resource!

Little troubleshooting tip:

If the succeeded or other events of a Lookup Record(s) data source don't trigger, there may be a field in the list of return fields that has been deleted from the table.

[Vadzim Surin1]

@Brad Tilton thanks a lot for your guide!
I'd like to update multiple records in UI Builder.
I guess I need to use a Update Multiple Records data resource.
I know the scope of sys IDs to to be updated, but I'm not sure of the syntax.
I think I shoud write something like this in the script:

api.data.update_multiple_records_1.execute({
        table: 'TABLE_NAME,
        templateFields: 'FIELD_NAME=' + FIELD_VALUE
});

But I don't know how to provide a condition to specify records to by updated. Which property is used for the condition?

[Arnoud Kooi]

@Vadzim Surin1 
A trick to understand this is add the datasource event handler via the UI and assign some random values and save 

Then open the page definition via the Hamburger Menu > Developer

In the Composition (or Internal Event Mapping) field, search the data source name
You should see the generated payload you can mimic in your script. Looks like you are looking to add a query property to your JSON!

(Don't forget to add the event and datasource)

[Vadzim Surin1]

@Arnoud Kooi  thank you very much. Your suggestion is very helpful.

[aaron_anderson]

Why aren't there ever examples of how we pass a property to the data resource script? It shows we can set up a property and use it in a script such as input.sysId, but how does the script get passed the sysId to begin with? Where is that step demonstrated?

[Brad Tilton]

Hi @aaron_anderson, there is no step to pass a property to the script. When you define the property it is automatically available in the script through the input object.

[Jay Gamblin]

@Brad Tilton I'm wiring up the Calendar component using a Transform with Mutates server data=true so I can pass in parameters to filter the results.  I placed a Typeahead multi component above the Calendar and am using the value(s) of that to feed into my data resource input.  

I used a client event of the Typeahead to update a client state parameter with the values selected.  In the client script I am calling the data resources execute command. Like so:

       api.data.get_training_events_1.execute({
            filter: newQuery
        });

This all seems to work as I am able log the values coming into the data resource's inputs to the script log. I am even logging the gr.getEncodedQuery() to the script log and it looks correct. I wouldn't think anything else is necessary for the Calendar component to refresh with filtered data.  However the data appears the same as page load.  Upon further inspection of the logs, it appears that the data resource is executed twice and the second time reverts to the initial query with no input (filter).  

What am I missing?

[Bartosz Glowac1]

Hello,

thanks for the useful article.

I've been trying to set up simple data resource to create record. I have stucked at the stage of choosing a table, as I want to create record in sys_user_preference table. Other tables as incident, change_request etc. are available in the list, but User Preferences table is not. Any idea where I could modify something (property maybe?) to allow setting sys_user_preference as the table in settings of Create Record Data Resource?

Thanks,

Bartosz

[JanakSinghP]

Thank you  , very useful.