UI Builder Transform Data Source times out

[BurtCrepeault]

Hi everyone,

On Xanadu, I'm working in UI Builder with a custom transform data source, which equates to a Data Broker Server Script (pictured below). The script calls a method in a script include that performs stuff, including some REST calls to a remote system, so it runs a bit long. Everytime I run it with a relatively large amount of data to process, I run into a 30-second timeout:

Cancelling transaction #26946 /api/now/v1/batch (maximum execution time exceeded): Thread Default-thread-51 (burt, 2531BC4B97707A10EDCFB47DE053AFD6), after 30189ms

It appears the Batch API transaction quota is reached (by default, set to 30 seconds). I am, however, not using the batch API explicitly anywhere in my code.

I tried running the same script in a Background script and everything runs smoothly to the end, so this is a problem specific to Data Broker Server Scripts.

I found an attribute in sys_ux_data_broker_transform not normally shown on the form, Batch Exclude, and cried Eureka! but alas, that didn't have any effect, the batch API is still used and still times out.

Google and AI companions are mute on this particular topic, so I thought I'd run it by this community as I don't have access to Now Support resources. Have you encountered anything similar? Is this something that could have been fixed in later SN versions? Is there a way to programmatically bypass the Batch API? Is that Batch Exclude attribute even connected to anything behind the curtain?

[Ranjane_Omkar]

@BurtCrepeault,

We encountered the same issue but have not found a permanent solution. We increased the timeout from 30 to 40 seconds, which resolved the problem in our case. However, the process sometimes still exceeds this limit and gets canceled.

As a workaround, we considered calling an event script from the data broker. This method is not recommended because a high volume of events can cause the event queue to enter a pending state. We did not implement this solution, as our frequent data broker calls (30-40 per minute) were causing the instance to go down.

Regards

----

If this response was helpful, please select "Accept as Solution" and "Helpful." This helps both the community and me.

[BurtCrepeault]

Thanks for the reply, Ranjane,

I was thinking of using an event, only one, not many, so maybe less risk. That is my plan B, to make it asynchronous. The downside is that you can't display or even use results on the UI Buidler page. Or maybe you can, but I'm not sure how.

I'll accept your reply as a solution if nothing else comes up, thanks! 🖐

[kaushal_snow]

@BurtCrepeault ,

The expected behaviour is that it will run under the platform API transaction quota and thus may be cancelled (and emit a log like maximum execution time exceeded: Thread …) before completion; in your case you cannot bypass that quota simply by ticking a Batch Exclude flag, so to address this you should consider off loading the long-running code into an asynchronous process (for example a scheduled job or event based execution) and have the transform data resource return quicker (e.g., a status or partial data), or redesign the logic to retrieve smaller data sets or chunk the calls, because by design transform data resources are expected to execute server side within the standard execution window.......

If you found my response helpful, please mark it as ‘Accept as Solution’ and ‘Helpful’. This helps other community members find the right answer more easily and supports the community.