How can I read from a very large attachment?

[Shane J]

Via REST, we receive a HTTP response from an integration we have setup that is pulling back >32MB of data (and only expected to be larger).  I do not have any ability to paginate.

We can no longer read the HTTP response due to size.

I thought having the response get created as an attachment instead would be a workaround (which I've been able to do), but that's presenting its own issues.  Now that I have the attachment, I can't seem to actually do anything with it.

The 'base' level script is below - it at least returns as an object.

var attachSysID = 'd64cb9c9db016d90e8d780430596194c';
var inputStream = new GlideSysAttachment().getContentStream(attachSysID );
var reader = new GlideTextReader(inputStream);
gs.print(typeof reader);

I haven't been able to iterate through that object.

I tried using what was in this video: https://www.youtube.com/watch?v=3toS32dvMg0&t=23s and ran into a String size limitation.

Am I going about this entirely wrong?  Is there a better way to deal with a massive HTTP response?
Or am I just attacking the reading of the attachment incorrectly?

[Tony Chatfield1]

Hi, you state that you cannot paginate, what tool\platform are you using? how are you scripting your query?
Using the API's would be fair easier than a complicated attachment workaround, which would probably quickly run into size issues of it's own and increases your technical debt.

[Shane J]

It's just a pretty straightforward GET using the MID Server (see below).  This is a homegrown solution, and let's just say the API isn't - robust.

 try { 
 var r = new sn_ws.RESTMessageV2('GroundControl', 'Default GET');

//override authentication profile 
//authentication type ='basic'/ 'oauth2'
//r.setAuthenticationProfile(authentication type, profile name);

//set a MID server name if one wants to run the message on MID
r.setMIDServer('MY_MID_SERVER');

//if the message is configured to communicate through ECC queue, either
//by setting a MID server or calling executeAsync, one needs to set skip_sensor
//to true. Otherwise, one may get an intermittent error that the response body is null
//r.setEccParameter('skip_sensor', true);

 var response = r.execute();
 var responseBody = response.getBody();
 var httpStatus = response.getStatusCode();
}
catch(ex) {
 var message = ex.message;
}

[somdev]

Hello Shane,

I am facing the same problem. I also got the initial workaround by putting it as an attachment. But then I faced the same issue. It seems ServiceNow is not capable of reading the huge data in one go. I see you have tried GlideTextReader but couldn't solve. Let's see what we can come up with. 

Thanks. 

[fil]

To iterate through the stream input you need you use the function readLine(). Watch the first 10 or 11 minutes of this video https://www.youtube.com/watch?v=Wcz0PixUneQ&t=235s

var attachSysID = 'd64cb9c9db016d90e8d780430596194c';
var inputStream = new GlideSysAttachment().getContentStream(attachSysID );
var reader = new GlideTextReader(inputStream);
var ln = ' ';
while((ln = reader.readLine()) != null) {
gs.info(ln);
};