REST Response problem - Too Large to handle

[Abhinab Achary1]

Hi,

I have a REST API call that is returning JSON that is too big(> 35MB). Now I cannot use the response as usual 

response.getBody() as it will only support till 5 MB and max 10 if used a property.

So I have tried the request.saveResponseBodyAsAttachment(file). to store the large json. unitll here its storing and also getting attached to the record.

Now I need to read that file and do necessary action, while doing that I need to store that big content(> 35MB) atleast to a variable. Now SNOW do not allow string object above 32MB. 

Now what technique should I use for such cases where the whole content of the big file needs to be use to do processing (create / update ) record.

Thanks,

Abhinab

[Ankur Bawiskar]

@Abhinab Acharyya 

Thank you for marking my response as helpful.

Let me know if I have answered your question.

If so, please mark appropriate response as correct & helpful so that this thread can be closed and others can be benefited by this.

Regards
Ankur

[Abhinab Achary1]

The attachment max size is fine. But I have an attachment attached to the record that has size > 35MB. Now lets says that attachment has json that will be converted to records in tables. . 

What is the best way to achieve this, as String or var in javascript can accommodate max till 32 MB.

Thanks,

Abhinab

[Ankur Bawiskar]

Hi,

break the file in chunks.

Also as I mentioned you should ensure you get limited data from 3rd party using pagination so that it would become easier to parse it

Regards
Ankur

[Abhinab Achary1]

Hi,

Its a whole json content in the file with a structure that may vary.

How should this breaking up into smaller files should be done ?

Thanks,

Abhinab

[Akshatha Ballal]

Look at the rest api documentation of the third party tool if there's an option to split the response (For example, if you are getting all the details of every device in the response, look if there are any other queries to get only the ID/unique identifier of all devices. You can later make use of these IDs to make rest call iteratively to the system to get individual details)

If not, at the end of URL, try putting &page-size=1000 and create one more query and give  &page-size=1000&skip=1000 and so on. Not sure if this kind of filter is supported in application you're querying. Try to find a similar one.