ServiceNow Table API returning fewer record while streaming

Sanchit3
Tera Contributor

3 weeks ago

Hi folks,

We have identified a data discrepancy issue where the ServiceNow Table API intermittently returns fewer records than expected when multiple API calls are executed in parallel. The occurrence is entirely random with no noticeable patterns observed either in the offset where the data loss happened or in the number of records.

For example, a request for 500 records may only yield 472 records. Because the API returns no error or exception, the data stream is being silently truncated. Our implementation appends offset and limit parameters to the Table API to fetch the required datasets, and the records are read sequentially in a streaming fashion using Gson's JsonReader object.

We are seeking a deeper understanding of why this occurs, as well as any official ServiceNow documentation describing this issue or its potential resolution. A similar behavior has been documented in the community thread: Table REST API returns duplicates / inconsistent data sets but a definitive root cause has not yet been identified.

Thanks,
Sanchit

0 Helpfuls
  • 308 Views
1 REPLY 1

Kieran Anson
Kilo Patron

a week ago

Hi Sanchit,

I'm hoping the below explains what feels like unexpected behaviour, but is expected functionality. However the randomness does sound odd. The below should be deterministic.

 

When you use sysparm_limit in your query, you're not controlling the limit of returned record. Instead, sysparm_limit is the cap of records pulled from the database before ACL evaluation. If ACL evaluation restricts access, then the returned number is less. E.g

  1. GET API call to /api/now/table?sysparm_limit=30
  2. NOW pulls 30 records from the database
  3. 30 records are passed for ACL evaluation, 5 rows dropped as they're not accessible
  4. Response contains 25 rows, even though there are further records that could have been returned

This behaviour is a good thing, it's a security guarantee. But is annoying. Below is the description from the documentation

 

"This limit is applied before ACL evaluation. If no records return, including records you have access to, rearrange the record order so records you have access to return first."

 

When you further call sysparm_offset, are you passing the current record count? Or basing it on sysparm_limit you previously provided? 

 

Additionally, are you supplying a ORDERBY as part of your query?

0 Helpfuls