REST Response problem - Too Large to handle

[Abhinab Achary1]

Hi,

I have a REST API call that is returning JSON that is too big(> 35MB). Now I cannot use the response as usual 

response.getBody() as it will only support till 5 MB and max 10 if used a property.

So I have tried the request.saveResponseBodyAsAttachment(file). to store the large json. unitll here its storing and also getting attached to the record.

Now I need to read that file and do necessary action, while doing that I need to store that big content(> 35MB) atleast to a variable. Now SNOW do not allow string object above 32MB. 

Now what technique should I use for such cases where the whole content of the big file needs to be use to do processing (create / update ) record.

Thanks,

Abhinab

[Ankur Bawiskar]

Hi,

you should ask the 3rd party team on how you can consume their API with the help of pagination and bring in smaller chunk of data

Regards
Ankur

[Abhinab Achary1]

Hi @Ankur Bawiskar 

The third-party tool do-not have option to have them in chunks due to serialization problem. Along with the data, the sequence is also important which is not guaranteed if chunks are made my thirdparty tool(AWS Config)

So this is something that needs to be done in Servicenow.

I have seen it was done in python, How can we achieved something similar in servicenow.

Thanks,

Abhinab

[Ankur Bawiskar]

Hi,

Unless their API supports pagination and is supporting to send the chunk in sequence you cannot do anything

Regards
Ankur

[Abhinab Achary1]

@Ankur Bawiskar 

I have an idea, not sure if it will work.

Transfer the big file to a MID Server, that has some JAVA programs or some other language capable to of handling such things and then return those to Servicenow

Second thought, to Convert JSON into excel and then use transform map.

but How can we convert JSON to Excel in Servicenow

[Ankur Bawiskar]

Hi,

yes that is an option.

I would suggest this

1) consume the 3rd party endpoint from Mid Server only

2) in the mid server using java code send small chunks back to ServiceNow and then write business rule on ecc_queue table and parse the json

Another approach

1) send the big file to Mid Server and then return back the smaller chunks

2) but I doubt on how much large file can be sent over to Mid Server

OR

Another approach

1) Consume the Attachment API from mid server script and get the big file directly by passing the table name, record sys_id

2) then send smaller chunks

Regards
Ankur