Create object from XML attachment larger than 5MB

mariochanes · ‎10-06-2015

Hi everyone, I'm looking to pick some brains and see if anyone has attempted to parse a large(atleast 45mb) xml attachment on a record.

Initially I was working with 'GlideSysAttachment().getBytes()' and I was getting exactly what I needed, now that my files are larger than 5mb, getBytes() is being maxed out and wont process files larger than 5mb to protect JVM memory or something memory related.

Here is an example of me creating my parsable object from a xml file <5MB

var sa = new GlideSysAttachment();

var binData = sa.getBytes(gr);

var string = new Packages.java.lang.String(binData);

var xmlhelp = new XMLHelper();

var obj = xmlhelp.toObject(string.toString());

JSUtil.logObject(obj, '');

After looking around the internet and the platform I found a few other ways that I could possibly create an object from larger files.

The example that got me to where I am now is the last comment on this post from Jamie Douglas, https://community.servicenow.com/thread/154141

Here's where I start to get stuck, when I try to run Jamies posted code.

var att = '4a072a906f251100e60f50be5d3ee419'; // sys_id of csv attachment
var attIs = Packages.com.glide.ui.SysAttachmentInputStream(att.toString());
var inDIS = new Packages.java.io.DataInputStream(attIs);
var br = new Packages.java.io.BufferedReader(new Packages.java.io.InputStreamReader(inDIS));
var line = br.readLine();
var lineArray = line.split(",");

As soon as line 3 get processed the script errors out. I'm guessing this is related to Package Calls having been replaced with Script Objects?

Once I ran into this problem I started looking around and found ExtractTermsFromAttachment which pointed me towards a couple script includes named 'Attachment' & 'ExtractTermsFromAttachment'.

ExtractTermsFromAttachment contained a function called getTerms, but it returns "".

Next I saw that the 'Attachment' script include has a getTerms function and uses 'GlideAttachmentIndexDocument' to do the work for the getTerms function. (correct me if I'm wrong)

This is where I'm at so far, if I come up with anything more I'll update you guys.

Update: Here is a list of community posts I've been looking at to help. These links may be helpful to someone in the future.

Doesn't work for me?

Re: Re: How to send attachment to Jira from Servicenow

Hi John,

Size limits on processors?

Reading attachment data and entering into a table

Getting contents of an attachment

Reading attachment data and entering into a table

Is there a way to determine the hash value of an incoming attachment to SN? In general, trying to av...

mariochanes · ‎10-13-2015

I have come across 2 solutions to address this problem, hopefully this is helpful for you guys.

Solution 1:

At the moment I'm working on solving my problem using import sets and transform maps with scripts.

Using an import set allowed me to turn my 40+MB XML file into many smaller records.

Now I have smaller data records to work with, I need to move the code I've already developed into a transform map & create a few business rules.

I have a pretty good feeling I'll have my solution ready by using import sets and TMs by the end of the day.

Solution 2:

This is the code answer. I still need to test it out.

I plan to try both methods so I can find the most efficient solution.

"If you are using the mid-server for this, you can use Packages calls and they will not be deprecated and wrapped with Glide functions.




Use the StringBuilder class for large buffers of data, it works really well, but I do not think there is a glide wrapper for those string builder functions. (You might be able to find it, but I did not see it documented).




Now as for the glide function goes, iterate over it with something like this: (not working code)."




InputStream in = //   this is the object stream with your 45mb file..



bos = new ByteArrayOutputStream();
int bytesRead;
int totalBytes = 0;
byte[] buffer = new byte[1024];
while ((bytesRead = in.read(buffer)) != -1) {
bos.write(buffer, 0, bytesRead);
totalBytes += bytesRead;
}
in.close();
System.out.println("base64 encoded bytes: " + totalBytes);

View solution in original post

Pradeep Sharma · ‎10-06-2015

Hi Mario,

You may find the below thread helpful.

https://community.servicenow.com/thread/204747

mariochanes · ‎10-07-2015

Thank you Pradeep Sharma, I remember having looked at this post on my hunt for the answer.
I'll give look this over and see if I missed anything useful.

mariochanes · ‎10-07-2015

By the way, to get line 2 to work I needed to alter Jamies example to:

'var inStream = new GlideSysAttachmentInputStream(theSysID.toString());'

Here's where I start to get stuck, when I try to run Jamies posted code.

var att = '4a072a906f251100e60f50be5d3ee419'; // sys_id of csv attachment
var attIs = Packages.com.glide.ui.SysAttachmentInputStream(att.toString());
var inDIS = new Packages.java.io.DataInputStream(attIs);
var br = new Packages.java.io.BufferedReader(new Packages.java.io.InputStreamReader(inDIS));
var line = br.readLine();
var lineArray = line.split(",");

As soon as line 3 get processed the script errors out. I'm guessing this is related to Package Calls having been replaced with Script Objects?

mariochanes · ‎10-08-2015

I'm going to be trying to use the following packages.java.io functions to replace what DataInputStream was doing.
Since I don't see DataInputStream in this list below I'm going to focus on getting the data out of the attachment using one of these functions.

*** Script: function >> BufferedOutputStream

*** Script: function >> PrintWriter

*** Script: function >> BufferedReader

*** Script: function >> OutputStream

*** Script: function >> File

*** Script: function >> ByteArrayInputStream

I got these to print out using this for loop.

var class = Packages.java.io;

for (var x in class ) {

if (typeof class[x] !== "function"){

gs.print(typeof class [x] + " >> " + x + " >> " + class[x]);

}else{

gs.print(typeof class[x] + " >> " + x);

}