Data Stream action design considerations

Release version: Yokohama

Updated January 30, 2025

4 minutes to read

Summarize

Summarized using AI

Summary of Data Stream action design considerations

This guidance outlines important considerations for ServiceNow customers when creating and integrating Data Stream actions in flows. Following these best practices helps prevent errors, optimize performance, and ensure smooth operation of Data Stream actions within flow logic.

Show full answer Show less

Considerations when creating a Data Stream action

Pagination Logic: Develop pagination scripts according to the third-party endpoint's format. Use the getNextPage Boolean variable to control pagination flow based on response tokens.
Variable Access and Data Types: Access variables using bracket or dot notation (e.g., variables['getNextPage']). Convert string-type pagination variables to integers for math operations and back to strings as needed.
Preprocessing Scripts: Avoid calling a MID Server or pausing the action in preprocessing scripts. Data Stream actions require a continuous connection and do not support pauses or waits, as these cause errors.
End Condition for Pagination: Always implement a condition that sets getNextPage to false to avoid infinite pagination loops.
Data Size Management: Configure pagination to return pages under 1 GB to prevent size limit issues, especially when using a MID Server. Large pages may require increasing attachment size limits.
Configuration Page Options: Clearing options removes the corresponding step and all associated data from the Data Stream outline.
Retry Policy Constraints: Only fixed interval retry strategies are supported with a maximum total delay of 30 seconds. Data Stream steps do not enter a waiting state during retries.
Nesting: Nesting Data Stream actions is not supported.
Testing: Thoroughly test Data Stream actions before deploying them in production.

Considerations when adding a Data Stream action to a flow

Avoid Actions with Wait Conditions: Do not include actions that pause the flow (such as Ask for Approval or Wait for Condition) in Data Stream For Each logic blocks, as Data Stream requires a constant connection.
Avoid MID Server-Dependent Actions: Actions and steps that always run on a MID Server (e.g., JDBC, PowerShell, SSH steps) cannot be included in Data Stream For Each logic blocks. Verify configurations to ensure actions do not run on a MID Server at runtime.
Performance Optimization: When processing large record sets in For Each logic blocks, disable reporting to reduce resource consumption by setting the com.snc.processflow.reporting.level system property.

Follow these considerations when creating a Data Stream action and when adding a Data Stream action to a flow to prevent errors and performance issues.

Considerations when creating a Data Stream action

Create Data Stream actions with these considerations in mind.

Write pagination logic according to third-party requirements

Evaluate and understand the format required by your third-party endpoint. For example, you may need to write a script that sets the built-in getNextPage variable to true as long as there is a nextPage token in the response. If the response does not contain the token, then set the variable to false. Access variables in script using bracket or dot notation. For example, variables['getNextPage']. This variable only accepts the Boolean data type. The default value is false.

Ensure that preprocessing scripts do not call a MID Server or pause the action.

Data Stream actions require a constant connection to the response stream. They do not support preprocessing scripts that pause the action to call a MID Server or pause the action to wait for a condition or duration. A preprocessing script that pauses the action for any reason returns an error message.

Convert pagination variable data types to perform math operations

Pagination variables only support the string data type. To perform math operations, you must convert the value to an integer, perform any required operations, then convert it back to a string.

variables['offset'] = (parseInt(variables['offset']) + parseInt(variables['limit'])).toString();

Ensure that the pagination script has an end condition

Avoid infinite loops in pagination requests by creating a condition that sets the getNextPage variable to false. Cancel any long-running flows. Always test Data Stream actions before using them in production.

Clear configuration page options carefully

Clearing an option in a configuration page removes the step from the Data Stream outline and deletes all data associated with the step.

Consider the size of each page returned

When configuring pagination options, try to ensure that the size of each page returned is under one GB. This ensures that flow designers who use the action won't encounter size limit issues when retrieving data from a MID Server. If the action returns more than one GB of data per page, the flow designer may need to increase the attachment size limit. For more information, see MID Server support for Data Stream actions.

Note:

Nesting Data Stream actions is not supported.
When using the retry policy, ensure that you are aware of these considerations:
- Only a fixed time interval strategy is honoured for the data stream actions.
- The total delay (count multiplied by the interval) can't exceed 30 seconds. For example, if you want retry for a maximum of 3 times, the maximum delay for each retry is 10 seconds.
- Data stream step doesn’t go into the waiting state during retry.

Considerations when adding a Data Stream action to a flow

Add a Data Stream action to a flow with these considerations in mind.

Do not add actions with wait conditions to a Data Stream For Each logic block

You cannot add an action that pauses the flow to a Data Stream For Each flow logic block. Data Stream actions require a constant connection to the response stream. For example, you cannot use the Ask for Approval or Wait for Condition actions within Data Stream For Each flow logic.

Some actions and steps always pause a flow and cannot be added to Data Stream For Each flow logic blocks. However, some actions and steps may pause the flow depending on how they are configured. Check the action configuration to ensure it does not pause the flow at runtime.


Always pauses the flow (unsupported with Data Stream actions)	Ask for Approval action
	Ask for Approval step
	Wait for Condition action
	Wait for Condition step
May pause the flow depending on configuration	Create Catalog Task action
	Create Task action
	Create Task step

Do not add actions that run on the MID Server to a Data Stream For Each logic block

Some actions and steps always run on a MID Server and cannot be added to Data Stream For Each flow logic blocks. However, some actions and steps may run on a MID Server depending on how they are configured. Check the action configuration to ensure it does not run on the MID Server at runtime.


Always runs on a MID Server (unsupported with Data Stream actions)	JDBC step
	PowerShell step
	SSH step
May run on a MID Server depending on configuration	REST step
	Script step
	SOAP step

Turn off reporting when consuming a large number of records in the For Each logic block

Iterating over a large number of records can be resource intensive, especially when the For Each logic block includes complex actions for each iteration. To avoid performance issues, turn off reporting using the com.snc.process_flow.reporting.level system property. For more information, see Workflow Studio flow system properties.