REST Response problem - Too Large to handle

[Abhinab Achary1]

Hi,

I have a REST API call that is returning JSON that is too big(> 35MB). Now I cannot use the response as usual 

response.getBody() as it will only support till 5 MB and max 10 if used a property.

So I have tried the request.saveResponseBodyAsAttachment(file). to store the large json. unitll here its storing and also getting attached to the record.

Now I need to read that file and do necessary action, while doing that I need to store that big content(> 35MB) atleast to a variable. Now SNOW do not allow string object above 32MB. 

Now what technique should I use for such cases where the whole content of the big file needs to be use to do processing (create / update ) record.

Thanks,

Abhinab

[Bala Krishna3]

Can you perform paging for received data?

[Ankur Bawiskar]

Hi,

you can get the attachment data as plain text and then parse the json

Script to get attachment file content

Scoped:

var attachment = new GlideSysAttachment();
var incidentSysID = '0e352ec80745ac54540bf2508c1ed0b7';

var agr = attachment.getAttachments('sc_cart_item', incidentSysID);

if (agr.next()) {
  var attachmentContent = attachment.getContent(agr);
  gs.info('Attachment content: ' + attachmentContent);
}

Global:

var sysIdOfRecord = '0e352ec80745ac54540bf2508c1ed0b7';

var grAttachment = new GlideRecord('sys_attachment');

grAttachment.get('table_sys_id', sysIdOfRecord);

var ga = new GlideSysAttachment();

var base64EncodedData = GlideBase64.encode(ga.getBytes(grAttachment));

var data = GlideStringUtil.base64Decode(base64EncodedData);

gs.info("Information Available in File: " + data);

Regards
Ankur

[Abhinab Achary1]

Hi @Ankur Bawiskar 

I am doing this

var attachment = new GlideSysAttachment();

var agr = attachment.getAttachments(tableName, recordID);

if (agr.next()) {

var attachmentContent = attachment.getContent(agr);

gs.info('Attachment content: ' + attachmentContent);

var toObj = JSON.parse(attachmentContent);

this.logUtil.set(JSON.stringify(toObj), "AttachmentFileContent");

return toObj; --->> THIS IS TOO LARGE to store in variable (around 37MB)
}

anything below that size is working fine

[Ankur Bawiskar · Accepted solution]

Hi,

there will be some limitations

I would suggest to get the response in chunks so that it becomes easier to parse in ServiceNow.

the maximum attachment size that the system will return for base64 encoding for is controlled by this system property

com.glide.attachment.max_get_size property.

If the setting does not exist it will default to maximum 5 MB.

You can change this standard value as per your need. However, as mentioned in the following KB, this may cause some issues.

Large attachments may cause Out Of Memory errors and performance degradation

Regards
Ankur