using data stream action in flow designer with a data source and having issues

[wingetc]

using a data stream action with a  data source to pull in data.  It's working except for  2 issues,  

1.) There are around 57,000 records in the view and I'm only pulling in around 42,000 records

here is the pagination code:

var offset = parseInt(variables.offset);

var limit = parseInt(variables.limit);

var totalCount = parseInt(pageResponse.response_headers['x-total-count'] )

var nextOffset = offset + limit;

if (nextOffset < totalCount) {

   variables.getNextPage = true;

   variables.offset = offset + limit;

   variables.offset = variables.offset.toString();

} else {

   variables.getNextPage = false;

}

x_total-count is about 1,200,000 records,  however,  I was told with  the parameter I'm passing should return around 57,000 records

2.)  there is a field that is name/value pairs.

example:

"Metadata":{"APRMID":"7238","Application_Name":"Quality Transformation Web Tool","Application_Location":"OnPremise","Critical_Certificates":"false","Nationwide_Issued":"true","Support_Group":"Run-Support-Apps-Net","followup":"true"}

when I pulled this data with a datasource action,  it came in to the import set table,  using the data stream action, it does not.

I'm not using the datasource action,  because I couldn't figure out how to get the  pagination step in the action.  So, would be open to trying to get that working as well.

Any help would be appreciated.  

Thanks,

CJ