Data source pagination with non-sequential response?

[NWTMark]

Hi all,

So I'm working on an integration from an external service, and the pagination is not as you would expect it. Instead of sequential (i.e. return 100 results, offset set to 101, return from 101-201, offset set to 202... etc.), it's the next "ID", returned in the pagination section of the response as the value 'after'. This is a typical response (only one "product" shown for clarity, but the responses have 100 in as you'd expect with a limit set to 100).

Excuse the screenshot instead of code tags, as it kept being flagged as spam.

Any ideas how I can pull this value out and get it picked up to be used as the offset value for the next page? I've been coming up against brick walls here. I can do this using a datastream action, but I want to avoid Enterprise level Integration Hub if I can.

[arielgritti]

Hi @NWTMark 

You can define it under the Response Body in the Pagination Section, as:

$.paging.next.after and scripting it to change the getNextPage value to true in case this value contains an ID. If not, you should change getNextPage to false.

(function paginate(variables, pageResponse) {
    // Use this script to set pagination variables or 
    // manipulate variables extracted by XPath or JSON Path

    moreData = pageResponse.response_headers['x-foundation-continuation'];

    //var jsonResponse = pageResponse.getJSON();
    //var jsonResponse = JSON.parse(variables.pageResponse);
    //variables.nextPage = jsonResponse.x-foundation-continuation;

    if ((variables.moreData) && (variables.times < variables.maxTimes)) {
        variables.getNextPage = true;
        variables.times = variables.times +1;
    }
    else {
        variables.getNexPage = false;
    }

})(variables, pageResponse);

Please mark helpful or correct if I helped you.
This action will help other members with similar issues to find a solution.
Thanks
Ariel

[NWTMark]

Hi Ariel,

This is how I'm doing it at the moment, I assume that's in a data stream action? It's the only place I've seen it. I want to avoid a data stream action unfortunately (as it requires IH Enterprise), and want to use a data source instead (with type set to REST). It has rudimentary pagination of the box, but was hoping for some way to configure this to the above scenario.

[arielgritti · Accepted solution]

Hi @NWTMark 

Yes, I'm testing the Data Stream functionality (I know this is high-cost license stuff, unfortunately).

By "code" you can define a REST message, call it and parse the "token" in the answer to be used into the next request call.

Something like this:

//Getting the "token" in "x-foundation-continuation" response header to know if there is more records to show
 var continuationRecords = response.getHeader("x-foundation-continuation");

//Calling a function to set the header for the next request call with the value of "x-foundation-continuation"
    if (continuationRecords) {
        retrieveExtendedRecords(continuationRecords);
    }

//retrieveExtendedRecords function
function retrieveExtendedRecords(continuationRecords) {

    try {

        //Other stuff I can't share here
r.setRequestHeader("x-foundation-continuation", continuationRecords);
    } catch (ex) {
        var message = ex.message;
        gs.error("Error:" + message);
    }
}

Please mark helpful or correct if I helped you.
This action will help other members with similar issues to find a solution.
Thanks
Ariel