Rest API limiting to 500 records

[Susan Davidson]

Prefacing with

Someone else wrote this, I am new to Rest APIs

We have a scheduled script that connects to Airwatch to get information on Apple and Android devices. 

For whatever reason when this runs it is only bringing in 500 records to the import set. I have tried increasing the Page < number and increasing the devices.length but it's not changing anything - still only returns 500 records (there are about 8000 in airwatch)

Can anyone point me in the right direction of what i need to do to get all the 8000 records in to the import set from this script (i tried creating a second script and then got 1000 records.. but created a 3rd and it still stayed at 1000)

 var pageNumber = 0;

    airWatchDevice(pageNumber);

    function airWatchDevice(pageNumber) {

        try {
            var r = new sn_ws.RESTMessageV2('Airwatch PROD', 'Device Information');
            r.setStringParameter('pageNumber', pageNumber);
            var response = r.execute();
            var responseBody = response.getBody();
            var httpStatus = response.getStatusCode();
            if (httpStatus == 200) {

                var assetArray = JSON.parse(responseBody);
                var devices = assetArray.Devices;
                var restGR = new GlideRecord('u_airwatch_integration_import');
                //Query extended import set rows table
                //Loop through the json chucks until 0 remain creating import set rows to be transformed
                for (var i = 0; i < devices.length; i++) {
                    restGR.initialize();
                    restGR.u_serial_number = devices[i].SerialNumber;
                    restGR.u_mac_address = devices[i].MacAddress;
                    //restGR.u_username = devices[i].UserName;
                    // Getting and referencing username
                    var user_id = devices[i].UserName;
                    var user = new GlideRecord('sys_user');
                    user.addQuery('user_name', user_id);
                    user.query();
                    if (user.next())
                        restGR.u_username = user.sys_id;
                    // Getting the location info
                    var location_id = devices[i].LocationGroupId.Id.Value;
                    //gs.info(location_id + ' airwatch location id');
                    var location = new GlideRecord('cmn_location');
                    location.addQuery('u_location_group_id', location_id);
                    location.query();
                    if (location.next())
                        restGR.u_location = location.sys_id;
                    // Getting os version;model description; name; lastseen
                    restGR.u_os_version = devices[i].OperatingSystem;
                    restGR.u_name = devices[i].DeviceFriendlyName;
                    restGR.u_lastseen = devices[i].LastSeen;
                    restGR.u_model = devices[i].Model;
                    // Getting and transforming Operating system
                    var platform = devices[i].Platform;
                    if (platform == 'Android')
                        restGR.u_operating_system = 'Android';
                    if (platform == 'Apple')
                        restGR.u_operating_system = 'ios';
                    if (platform == 'AppleOsX')
                        restGR.u_operating_system = 'Mac OS/X';
					if (platform == 'Apple iOS')
                        restGR.u_operating_system = 'ios';
					if (platform == 'Apple macOS')
                        restGR.u_operating_system = 'Mac OS/X';

                    restGR.insert();
                }
                if (devices.length >= 1000 && pageNumber < 6) {
                    pageNumber++;
                    airWatchDevice(pageNumber);
                }

            } else
                gs.info('airwatch error :::>>>  ' + responseBody);

        } catch (ex) {
            var message = ex.message;
        }
    }

[Allen Andreas]

Hello,

Have you confirmed that the payload has more data...it's just that the import set isn't being built with more than 500 rows?

Have you checked the maximum rows for the import set table/system property?

[Susan Davidson]

Yes, if i run it more than once (with the same script) it has ended up at 8000 rows plus. Have other integrations running and the import sets there are 40,000 rows plus with no issues.
Staging table (custom) is fine too for rows

[Allen Andreas]

Hi,

Were you able to double-check how many records are coming in each payload? Just to verify?

Could you add logging to log the devices.length so you can see what happens through each loop?

It sounds like it must be some sort of pagination issue going on. You're saying that the entire script will finish, you'll only get 1,000 records on the first run, then you run it again, and you're actually getting a different 1,000 the next time? Or the same 1,000 as the first? As your page number and all that would be back to 0 again...so that part is slightly confusing.

In any case, is there anything that may be in rest message that could be causing any issues that you see?

Do the page numbers start at 0 at the endpoint? Or 1?

Do all page numbers have equal to or more than 1,000 devices on it?

Are there only 5 pages or 6, I suppose if you start with 0?

If you don't mind, I'd check the few things I've mentioned here. Thanks!

[Susan Davidson]

500 coming in each payload. If i run it again, immediately, the same 500 come over. If i wait a few hours its some of the same records and some new ones.
Page numbers start at 0. 500 records per page apparently.

I actually found something similar for another application online which said that some rest APIs will only return 500 records each rest call made. So i duplicated the rest call, started the page count at 1, then another and started at 2, another at 3 and so on.. did 20 of them in total. Ran them all concurrently and all 8000 records came over 😐

While it's not pretty it is now at least getting and updating the records and as they are scheduled scripts they can all run at the same time, all load to the same import set and use the same transform map. It seems kinda crazy that i can't get them all with one script but apparently can be an issue not only with this integration so i am going to roll with it unless anyone can offer a prettier solution!

Thanks for looking at it - if you have any other suggestions than what i have done i'll be glad to look at them.