Pagination in Rest API for retrieving records from third party

[aamir1]

Hi All,

I need to retrieve data from a third party and I am able to achieve that but we need to apply pagination to get the data in small numbers. I am not familiar with Pagination concept, could anyone please help me how to achieve this. I am able to achieve the 1st record but getting issues in putting it in a loop.

Here are the codes-

Test: function(){
var now = new Date().toUTCString();
var dt=now.split("GMT");
var date=dt[0]+" -0000";
var method=["GET"];
var path=["/admin/v1/users"];
var param={"limit":1000,"offset":0};
var end="api-****.duosecurity.com";
var sig = this.sign(method, path, param,date);
var req = {
method: method,
headers: {
'Date': sig.date,
'Authorization': sig.authorization
}
};
try {
var r = new sn_ws.RESTMessageV2();
r.setEndpoint("https://api-****.duosecurity.com/admin/v1/users");
r.setRequestHeader('Content_type','application/x-www-form-urlencoded');
r.setRequestHeader('Authorization',req.headers.Authorization);
r.setHttpMethod(req.method);
r.setRequestHeader('Date',req.headers.Date);
var response = r.execute();
var responseBody=response.getBody();
responseBody = JSON.parse(responseBody);
var httpStatus = response.getStatusCode();

if(httpStatus == "200"){
return responseBody;
}
}
}

here is the response that i am getting-

{"metadata": {"next_offset": 1000, "total_objects": 4532}, "response": [................], "stat": "OK"}

[Tony Chatfield1]

Hi, you have a parm of 1000 set, but your result suggests you only have 100 records returnef so maybe you need to review this incase it is an issue. Once you have a result you should be able to iterate through it and extract a records data with each loop From memory ( and on my phone) so may not be 100% but try something like var response = r.execute(); var parsedData = new global.JSON().decode(response.getBody)); gs.info(parsedData.records.length); for(I=0; I < parsed data.records.length; I++) { destinationField = parsedData.records[I].fieldName; }

[aamir1]

Hi Tony,

Thanks for the response!!

Yeah, the next_offset is 1000 only, It was my mistake while typing. As per your code, we are retrieving all the records in one go then we are iterating through that data. But in pagination, we get results in batches right?

[Ankita Kolhe]

Hi @aamir1 ,

I'm having the same requirement. Did you able to achieve it?

Please help me on the same.

Thanks,

Ankita

[Tony Chatfield1]

So your query returns X number of records, you process them then offset you query by the previous offset +X and query again. As long as you query and then offset consistently you can basically run a query loop (to manage your offset) and within each itteration a loop (as described above) to process the query results.