Not able to create a loop for pagination in integration

[Shankha B]

Hi All,

I have a requirement to use pagination for integration. I am very new to this. Need some help.

Context: I have a flow action where I defined the REST step and calling the same flow action from a scheduled job.

I have written pagination code in the REST Step in the flow. Basically I am using one input variable called ContinuationToken in that action. The API supports the ContinuationToken. So for the first time ContinuationToken will be empty and after that the loop for the REST API will run till the continuationToken gets empty.  ContinuationToken is a token that will be used for recursive call of API. I can get the continuationToken from response body.  My limit is 100. Please refer to the code that I have written in a scripted step before the REST step.

(function execute(inputs, outputs) {
var prop = gs.getProperty('xyz');
var config = JSON.parse(prop);
var endPoint = "endpoint" + config.TenantId + "/oauth2/token";

var continuationToken = inputs && inputs.continuationtoken ? inputs.continuationtoken : null;
var response;

do {
var sm = new sn_ws.RESTMessageV2();

sm.setEndpoint(endPoint);
sm.setHttpMethod("post");
sm.setRequestHeader("Accept", "Application/json");

var requestBody = "grant_type=" + config.GrantType + "&client_id=" + config.ClientId + "&client_secret=" + config.ClientSecret + "&resource=" + config.Resource;
if (continuationToken){
requestBody += "&continuationtoken=" + continuationToken; 
}
sm.setRequestBody(requestBody);

response = sm.execute();
var responseBody = response.getBody();
var resp = new global.JSONParser().parse(responseBody);

continuationToken = resp.continuationtoken || null; // modified to lower case t in token

} while (continuationToken)

var accessToken = resp.access_token;
outputs.accesstoken = accessToken;
})(inputs, outputs);

My limit is defined in the Request Body of the REST step.

From this code I am not able to get all the data from the third party system. If I changing the limit to 5 then I am only getting 5 data. I suppose the loop is not working properly. My scheduled Job is only used to call the action and get the response and set the data in a table. Could you please suggest what can be done or where do I need to add the code.

[Tony Chatfield1]

Hi, normally an integration solution that supports pagination will have some way to identify the number of records to be queries\returned and an offset to identify where to start the current query. As an example the SNC table API uses Query parameters 'sysparm_limit' and 'sysparm_offset'.

Table API | ServiceNow Developers

The simplest way to use these (or similar) would be to define an offset variable before a loop containing your REST query (including mapping your offset reference) and then index the offset variable by the 'limit' value at the end of each loop.

What are you integrating, what options do you have for similar.

[Subhashis Ratna]

Hello @Shankha B  

Could you please try this logic , I have updated it ..

The loop for making API calls is inside the "do-while" loop, and you're using the same "continuationToken" for the subsequent requests.

(function execute(inputs, outputs) {
var prop = gs.getProperty('xyz');
var config = JSON.parse(prop);
var endPoint = "endpoint" + config.TenantId + "/oauth2/token";


var continuationToken = inputs && inputs.continuationtoken ? inputs.continuationtoken : null;
var response;

do {
var sm = new sn_ws.RESTMessageV2();

sm.setEndpoint(endPoint);
sm.setHttpMethod("post");
sm.setRequestHeader("Accept", "Application/json");

var requestBody = "grant_type=" + config.GrantType + "&client_id=" + config.ClientId + "&client_secret=" + config.ClientSecret + "&resource=" + config.Resource;
if (continuationToken){
requestBody += "&continuationtoken=" + continuationToken; 
}
sm.setRequestBody(requestBody);

response = sm.execute();
var responseBody = response.getBody();
var resp = new global.JSONParser().parse(responseBody);

continuationToken = resp.continuationtoken || null; // modified to lower case in token

} while (continuationToken)

var accessToken = resp.access_token;
outputs.accesstoken = accessToken;
})(inputs, outputs);

If you have any more requests or need further clarification, feel free to let me know!

Mark Correct if this solves your issue and also mark Helpful if you find my response worthy based on the impact.

Thank you!
Subhashisr7