Records missing from Table API results

[MikeD314]

I have been developing an application automate data exports from ServiceNow. Mostly everything works great but there is one issue that has been been bothering me for some time.

When querying the Table API often response pages do not contain the number of records expected. I believe this is due to ACL restrictions. The problem is that these hidden records are not reflected in the X-Total-Count HTTP header. Therefor when my application checks that all of the records exist in the local repository it finds a mismatch. Those records can't be accessed so it throws it into a loop of trying to fix the missing records until it gives up (using a simple max_retries count).

Is there any way to determine the number of records that have been hidden from a result page? I know the web interface will (at least sometimes) show the info as "X number of rows were removed by security constraints". But that information does not seem to be present in the raw JSON data or in the response HTTP headers.

My only idea for now is to keep a count of how many records are expected per page and then how many are missing. This will probably work but it seems overly hacky when ServiceNow knows the number of records it has hidden. This problem is also present when accessing the Attachments API. (I assume it will affect other web service calls as well but those are the two I've worked with so far.)

Is there any way to get the number of records hidden (overall or per-page) for a Table API request?

[Chandu Telu]

HI

 Hidden records from the any child table ?

Please refer the below link it will help you

https://community.servicenow.com/community?id=community_question&sys_id=bd7f8725dbdcdbc01dcaf3231f961949

[MikeD314]

I'm unclear as to what exactly you're asking... but from testing we did ages ago it seemed like the missing records were being hidden by ACL rules. It didn't seem related to child tables or any other data relationship.

I'm not sure what exactly the rules are but I'd like a solution where that didn't matter. Getting a correct record count needs to be possible while still abiding by any security rules that are in place.

[MikeD314]

I read the link you've posted but it doesn't seem related. I'm not paging data by the number of records I receive. I use the LINK HTTP header to find the ;rel=next URL. I then order the results by sys_id to avoid getting duplicate records across pages (hopefully).

The issue here is that records are hidden not duplicated. So I request 5000 records but only get 4998 because 2 of them are hidden. Presumably they are hidden by ACL rules but it could be anything.

The problem then is that this is not shown in the X-Total-Count header which leads to record count discrepancy when the export is finished. There doesn't seem to be any way to the actual record count that I will have in the end once ACL rules are taken into account.

[asifnoor]

Hi,

X-total-count basically returns the total number of rows that query returned. The ACL restrictions after the query is performed, hence the x-total-count does not consider them.

You should use sysparm_offset and sysparm_limit to fetch the records and once the result returns 0, then it means the records are over and you skip the loop.

Mark the comment as a correct answer if this answers your question.