Best practice for iterating through a table with the Table API

Tomav · 3 weeks ago

Hello everyone,

I’m working on an integration where I need to iterate through a large table (for example, incident) using the REST Table API. My goals are:

To get responses as quickly as possible (low latency per page).

To avoid over-consuming instance resources (not put unnecessary load on the DB or the node).

Right now, I’m paging through the records with sysparm_limit and either sysparm_offset or keyset style queries (e.g., ORDERBYsys_id + sys_id>{last} watermark).

Before I settle on an approach, I’d like to ask the community:

* What’s considered best practice for iterating through large ServiceNow tables via API?

* Are there recommendations on page size (sysparm_limit) that balance speed with instance impact?

* Is keyset pagination (using sys_id or sys_updated_on) preferred over using sysparm_offset?

* Are there any other tips to ensure I’m getting the data efficiently without overloading the instance?

Thanks in advance for any guidance!

M Iftikhar · 3 weeks ago

Hi @Tomav,

For iterating through large ServiceNow tables via the REST Table API, the definitive best practice is to use keyset pagination instead of sysparm_offset. Keyset pagination involves sorting the results by an indexed, unique field (like sys_id or sys_updated_on) and using the last record's value from one page to create the query for the next page (e.g., sysparm_query=sys_id>last_sys_id_from_previous_page). This is far more efficient as it allows the database to start the next query exactly where the last one ended, whereas sysparm_offset forces the database to re-query and discard all preceding records, leading to progressively slower response times as the offset increases.

For page size (sysparm_limit), a value between 500 and 2,000 is typically a good balance, but it's best to test and monitor instance performance. To further optimize, always use sysparm_fields to request only the necessary columns, apply a specific sysparm_query to filter the dataset as much as possible, and use sysparm_exclude_reference_link=true to reduce the payload size.

Hope this helps!

Thanks & Regards,
Muhammad Iftikhar
If my response helped, please mark it as the accepted solution and helpful so others can benefit as well.

Thanks & Regards,
Muhammad Iftikhar

If my response helped, please mark it as the accepted solution so others can benefit as well.

View solution in original post

Ankur Bawiskar · 3 weeks ago

@Tomav

I won't recommend Table API as it exposes lot of data unnecessarily.

Why not use Scripted REST API which gives you control over what you can share as API response?

My thoughts

1) keep limited fields in sysparm_fields

2) keep pagination and offset as per ServiceNow recommendations

If my response helped please mark it correct and close the thread so that it benefits future readers.

Regards,
Ankur
✨ Certified Technical Architect || ✨ 9x ServiceNow MVP || ✨ ServiceNow Community Leader