How can I use pagination in an Import Data Source Rest Step

[Bill77]

I'm trying to build an Import integration that retrieves node data from a REST api. I can make the API call and get data, but I need to paginate the calls so I don't crash the platform I'm retrieving from with a REST call that tried to get everything at once.

My Data Source type is REST and the format is Parse By Script. The data the API returns is JSON, but it comes back as an array and Servicenow doesn't like that, so I have to parse it with my own script. 

When I enable pagination it does the requests I expect it to do, but then I can't tell the step to stop doing additional requests. When the platform I'm integrating with gets a request for a page with no data, it will return an empty array and a 200 status code. This seems to prompt Servicenow to just keep requesting additional pages.

Is there a way in the script step to tell Servicenow that there's no more data and to please stop requesting pages?

[Rogers Cadenhe1]

If the problem is that the platform you're integrating with doesn't return an error status code when a request has no data, you could create a scripted REST service and scripted REST resource on your instance that makes the call to the platform and either returns the full response with a 200 status code (when there is data) or an empty response with a 400 Bad Request error code (when there is none). Then your REST data source calls your REST API instead of the platform.

[Bill77]

I had wanted some time ago to create a scripted rest service for a different reason, but I couldn't find a way to create that service on my instance in a way that was allowed to do the outbound HTTP request. I kept getting errors that the Glide HTTP object wasn't allowed in my context. If there's a way to do it I'll certainly consider doing that.

[Rogers Cadenhe1]

Can you share an example of the full REST URL of an outbound call to the platform requesting paginated records (with the server and private details removed)?

[Bill77]

Does this help?

The Script Step there just reads a couple system tables to get the configuration values where I'm storing the host name to call out to and the authentication token.