How to get all results from a Paginated API using REST Data Source

[Morgan Castle]

I am currently trying to retrieve all records for a Paginated API using the REST Data source type and then using the ETL plug in to transform the information onto the appropriate CMDB Tables. However, I am not sure how to gain all the records from each page without defining a new REST call for each one. I selected the checkbox for "Support Pagination" on the Data Source Record for the Rest Data Source and set the Offset and Limit fields that appear. I am not sure where else I should be integrating them or if I should reference them somewhere in the Request Action. Please let me know if anyone has advice or guidance on this and where to make adjustments or scripting advice. I have already validated a subset of the API data by specifying the specific page I wanted to gather records from and successfully put them on the software table. Thanks.

[#2]

Checking the support pagination checkbox will continually update the limit and offset input variables to the rest action:

Once the response is empty it will stop incrementing those values and be done sending requests.

It is up to you to write a script or pass these values to the REST API you are attempting to pull data from.  Unfortunately there is no standard for paginated APIs so each implementation will vary.  The key is you can take those inputs to the rest action optionally manipulate them via a script step, and then pass some form of the output to the REST API call to get a page.  The datasource will take care of iterating and populating the import set table.

[VaranAwesomenow]

you can use this approach

https://www.servicenow.com/community/csm-blog/varanawesomenow-load-accounts-using-accountapi/ba-p/23...

[jaheerhattiwale]

@Morgan Castle Once you check the "Support Pagination" checkbox and set the offset and limit. you just need to trigger the API once the offset and limit will be updated and all the data is fetched. So you don't need to bother about the pagination.

[SN Emy]

Hey, Did you find a solution for this?

Thanks,