How can I read from a very large attachment?

[Shane J]

Via REST, we receive a HTTP response from an integration we have setup that is pulling back >32MB of data (and only expected to be larger).  I do not have any ability to paginate.

We can no longer read the HTTP response due to size.

I thought having the response get created as an attachment instead would be a workaround (which I've been able to do), but that's presenting its own issues.  Now that I have the attachment, I can't seem to actually do anything with it.

The 'base' level script is below - it at least returns as an object.

var attachSysID = 'd64cb9c9db016d90e8d780430596194c';
var inputStream = new GlideSysAttachment().getContentStream(attachSysID );
var reader = new GlideTextReader(inputStream);
gs.print(typeof reader);

I haven't been able to iterate through that object.

I tried using what was in this video: https://www.youtube.com/watch?v=3toS32dvMg0&t=23s and ran into a String size limitation.

Am I going about this entirely wrong?  Is there a better way to deal with a massive HTTP response?
Or am I just attacking the reading of the attachment incorrectly?

[fil]

Watch the first 10 or 11 minutes of this video https://www.youtube.com/watch?v=Wcz0PixUneQ&t=235s. You need to use the function readLine()

var attachSysID = 'd64cb9c9db016d90e8d780430596194c';
var inputStream = new GlideSysAttachment().getContentStream(attachSysID );
var reader = new GlideTextReader(inputStream);
var ln = ' ';
while((ln = reader.readLine()) != null) {
gs.info(ln);
};

If anyone knows how to use XMLDocument2 with a stream, then please let me know

[surajmesinane1]

Facing the same issue, Did anyone really have a solution for this. We are trying to query data from Splunk and they said they cant paginate the data we are asking.