How to access “response_body” in pagination script for data stream action?

[vikasmaurya187]

Hi all,

I'm currently working on building a Data Stream action for an API that supports pagination. However, I've encountered a couple of issues.

Firstly, the API returns the following response, which includes an array of user objects:

[
{
// object containing user details
},
{
// object containing user details
},
{
// object containing user details
},
// More user objects...
{
"nextToken": 20
},
{
"count": 1184
}
]

The problems are:

1. I'm unable to find a JSONPath expression that can directly access the "count" and "nextToken" fields in the Pagination Variable.

2. Although the ServiceNow documentation mentions that "In the pagination script, the pageResponse object contains the response_headers, the response_body, and a status code", I'm unable to access the "response_body" in the "pageResponse" object. Upon stringify and logging "pageResponse" in the pagination script, I could only see the following fields:{
   "item_count": "",
   "error_message": null,
   "item": "",
   "response_headers": {
   // object containing header information
   },
   "status_code": 200,
   "count": "",
   "ds_count": "",
   "error_code": 0,
   "page_count": ""
   }

Due to these issues, I'm unable to build pagination logic for the Data Stream Action. Any solutions?

ServiceNow documentation link : https://docs.servicenow.com/bundle/vancouver-integrate-applications/page/administer/integrationhub/t...

[Kaylin Lee]

1. Can I ask what JSONPath expressions you've tried? Looking at your data in a JSONPath Evaluator it seems like $..nextToken and $..count could potentially do the trick. If not, you could also try using bracket notation if you haven't already: $..["nextToken"] or $[".."]["nextToken"]

Are you setting the expression in the Expression field on the Pagination variable with "Response Body" set as the "Next Value From" selection?  

2. Just curious - do you need to access the response body before the splitter step? I know that there are use cases where you may need to access the data in the response body in a script step for the pagination or for something else. But if not, you can first use your pagination variables in the pagination variables script to set up the pagination.

For example:

And then in the splitter step is where you can define the path for your response body. I have a similar data set to yours in one of my custom Data Stream actions and have been able to retrieve the data using just the root object JSONPath symbol, $.

In the the Script Parser step I parse and define my outputs. These are the Object keys in the response.

For example, an object in my data set looks like:

[

    {

        "id": 910,

        "lessonType": “test”,

        "title": “test”,

        "description": “test”,

        "courseId": 9499,

        "visibility": "public",

        "creationDate": "2024-05-09T04:07:20Z",

        "requiredReview": false,

        "scoreLimit": 0.75,

        "estimatedDuration": 90,

        "link": “test”

    },

    {

// another object

    }

]

So in the Script Parser step my defined output variables using the keys above would look like this:

And here is the defined output object:

Then when I go to test I'll receive iterations of object outputs based on the response body data and defined outputs. (Note that data stream tests return the first 20 items by default.)

I apologize if you do in fact just need the response body in an earlier step in the data stream for some processing.

Let us know either way. Thanks and good luck!