How can we handle large payloads from a REST API?

[cloudyrobert]

We have an integration that requires us to send a single file as an attachment to an email. The contents of the file are sourced from multiple files. To do this, I'm sending the attachment data to a REST API that merges the files and returns the data for the single attachment.

To avoid tying up the user's session, I'm following the guidance in this article: https://hi.service-now.com/kb_view.do?sysparm_article=KB0716391.

This works fine for very small files. For anything even moderately-sized (4 MB? Not sure of the exact cut-off), the payload that is returned from the REST API (which is much larger than the final size of the encoded file, so a 4 MB file might be 10 MB--again, not sure exactly) is larger than the size allowed by the "com.glide.attachment.max_get_size" property, and I can't parse the payload to encode the file. The ECC queue input record shows this error: "Payload attachment exceeds the limit of 5242880 bytes set by system property com.glide.attachment.max_get_size." The script pulls back null for the attachment contents for the payload.txt file attached to the ECC queue record using this logic (that works fine when that error isn't thrown):

var grAttach = new GlideRecord('sys_attachment');
grAttach.addQuery('table_sys_id', 'ECC QUEUE RECORD SYS_ID HERE');
grAttach.query();

if (grAttach.next()) {

	var attachID = grAttach.sys_id;

	var sa = new GlideSysAttachment();
	var binData = sa.getBytes(grAttach);
	strData = Packages.java.lang.String(binData);
		
	gs.log('strData: ' + strData.substr(0,1000)); // this is empty
}

According to this article, I shouldn't increase the allowed size: https://hi.service-now.com/kb_view.do?sysparm_article=KB0610485.

What options do I have?

[Ankur Bawiskar]

Hi Relliott,

Is it possible for you to divide the file in chunks and then combine at once when the last chunk is received

Regards

Ankur

[cloudyrobert]

Possible, yes, but not ideal. I believe we'd then have to rely on the external service to push the file back to us in separate calls from the response.