I was tasked with an integration between our cloud-based ServiceNow environment and our on-prem IBM FileNet content management system. The integration ties into our TDLC process where evidence files are required to be attached to submissions. Rather than keep the attachments in ServiceNow, we move them off to FileNet where appropriate retention policies are applied.
In a nutshell, when someone attaches documents to an rm_doc record, a business rule will kick off a script action to upload the attachments to FileNet. When that process is finished, a work note is written and the attachments are deleted. There's more to it, of course, but that's the gist of the process.
This is what the end result looks like:
And the work notes that get added to the rm_doc record:
There are Java and .Net API's available to work with FileNet automation but I didn't have the luxury of using them directly from ServiceNow. I only had the SOAP WSDL to work with and I couldn't find much documentation about the various functions.
The SOAP message function I used is AddDocumentRequest. Here's what that looks like:
<soap:Envelope xmlns:soap="http://www.w3.org/2003/05/soap-envelope" xmlns:ecg="http://www.nervoussow.com/ECGService-v1.0" xmlns:ns="http://www.nervoussow.com/ECGService/xsd/2012/01">
<soap:Header/>
<soap:Body>
<ecg:AddDocumentRequest>
<ns:RequestHeader>
</ns:RequestHeader>
<ns:Truncated>${is_truncated}</ns:Truncated>
<ns:RestartKey>${restart_key}</ns:RestartKey>
<ns:Document>
<ns:ClassName>${class_name}</ns:ClassName>
<Properties>
<ns:Name>Description</ns:Name>
<ns:Value>${desc}</ns:Value>
</Properties>
<Properties>
<ns:Name>UploadDate</ns:Name>
<ns:Value>${date_time}</ns:Value>
</Properties>
<ContentSegments>
<ns:Content>${file_contents}</ns:Content>
<Properties>
<ns:Name>FileName</ns:Name>
<ns:Value>${file_name}</ns:Value>
</Properties>
</ContentSegments>
</ns:Document>
</ecg:AddDocumentRequest>
</soap:Body>
</soap:Envelope>
Note: I left out a lot of properties to keep it short.
First Attempt
The first version of the integration passed the entire base64 encoded attachment in the SOAP request for the ${file_contents} value. Yeah, that wasn't a great idea. Only attachments around 8 MB or smaller would upload. For any file bigger than that, the FileNet response would return an error essentially saying that it received no content. As it turns out, the problem was that strings are limited to 32 MB and the base64 encoded file contents exceeded that, so ${file_contents} ended up as an empty string.
Second Attempt
The second version included a custom .Net application that used the FileNet .NET SDK to accommodate larger files. This application was installed on our MID server hosts and ran as a Windows service. Instead of using the SOAP function as the first version did, this version exported the attachment, along with a supporting metadata file, to a specific folder on the MID server host. The metadata file was simply JSON that gave FileNet extra information about the file, including values to various properties in the SOAP message function. Every 5 minutes, the Windows service would look in that folder for files to upload. Upon a successful upload, the attachment file was deleted and the metadata file was updated with a Document ID from the FileNet response. The metadata file was then moved to a processed folder.
Meanwhile, back in ServiceNow land, every 10 minutes, a scheduled job would check that processed folder. For each file it found, it crafted a URL to the FileNet document and associated that with the source record (the rm_doc record where the attachment was added). It then deleted the attachment.
For the most part, that process worked and handled large attachments. However, when there were many attachments being added at the same time, that scheduled job didn't do so well at finalizing the processed metadata. Another issue is that it was yet another extra thing to support.
Third Attempt
The third and current version of the integration uses chunking to send the entirety of the base64 content in chunks. It took me a while to find the right documentation that explained it. Now, the files are uploaded directly from ServiceNow and the process supports large files. The .Net service on the MID server hosts is no longer needed nor is the scheduled job that ran every 10 minutes.
How I Did It
To implement chunking with FileNet, we use the same SOAP message function but include the Truncated and RestartKey elements. We must somehow break apart the base64 encoded file content into parts (chunks). All but the last chunk should set the ${is_truncated} value to true. It should be false for the last chunk. With each chunk sent, except for the last one, a RestartKey value will be included in the FileNet response. You pass that value into the next chunk upload. When all chunks have been uploaded successfully, FileNet will return a Document ID.
The Business Rule
The business rule is where it all starts. It's a before-insert business rule that runs on the sys_attachment table. It determines the source record (the rm_doc record where the document was attached), gets some supporting data and build the metadata used for the SOAP message function variable substitution.
(function executeRule(current, previous /*null when async*/ ) {
var source_record = new GlideRecord(current.table_name);
if (source_record.get(current.table_sys_id)) {
var fn_data = {
"desc": "Some Description",
"submitter": current.sys_created_by.toString(),
"class_name": "SomeClass",
"group_type": "SomeGroupType",
};
// fire event with the attachment record's sys_id and FileNet data for the SOAP request
gs.eventQueue('filenet.document.upload', current, current.getValue('sys_id'), JSON.stringify(fn_data));
}
else {
gs.info('FileNet BR: Trigger upload to FileNet (RM) failed to find source record.');
}
})(current, previous);
As you can see, the filenet.document.upload event is triggered and parameters are passed accordingly.
The Script Action
This is where the work is done. Hopefully, my comments are sufficient.
(function() {
// gather the event parameters and start building things out
var attachment_id = event.parm1.toString();
var fn_data = JSON.parse(event.parm2);
var fnutils = new FileNetUtils();
var attachment = new GlideRecord('sys_attachment');
if (attachment.get(attachment_id)) {
var source_record = new GlideRecord(attachment.table_name);
if (source_record.get(attachment.table_sys_id)) {
// get default attachment metadata
var attachment_data = fnutils.getDefaultAttacmentMetaData(attachment_id);
// get the base64 encoded data from the attachment
var strutil = new GlideStringUtil();
var gsis = GlideSysAttachmentInputStream(attachment_id);
var ba = new Packages.java.io.ByteArrayOutputStream();
gsis.writeTo(ba);
ba.close();
// and the entire file is this guy
var base64EncodedData = strutil.base64Encode(ba.toByteArray());
// initialize some variables
var document_key = '';
var response_body = '';
var r = '';
fnutils.addNote('FileNet upload started for file ' + attachment_data.file_name + '.', source_record);
// check file size and don't chunk if 7MB or smaller
var max_bytes = gs.getProperty('integrations.filenet.max_upload_bytes') || '7340032';
if (parseInt(attachment_data.size_bytes) <= parseInt(max_bytes)) {
response_body = fnutils.sendSOAPChunk(base64EncodedData, false, '', attachment_data, fn_data);
// r will contain some key data from the response
r = fnutils.getKeyFromResponse(response_body);
document_key = r.document_key;
}
else {
// the file is larger than 7MB so let's do the truffle-shuffle
// initialize some more variables
var restart_key = '';
var chunk = '';
var truncated = true;
var i = 0;
do {
// get a small chunk of the entire base64 encoded string that is the file
chunk = base64EncodedData.substr(i*fnutils.attachment_chunk_size, fnutils.attachment_chunk_size);
// now send that chunk to FileNet
// the initial request won't have a restart key, as that's returned from the first response
response_body = fnutils.sendSOAPChunk(chunk, truncated, restart_key, attachment_data, fn_data);
// r will contain some key data from the response
r = fnutils.getKeyFromResponse(response_body);
restart_key = r.restart_key;
// r.bytes is the bytes value returned from the response
// using that to determine if there's more data to send
truncated = (parseInt(r.bytes) >= fnutils.response_bytes) ? true : false;
// if there are no more chunks to send, we're done
if (parseInt(r.bytes) == 0 && r.document_key) {
chunk = 'COMPLETE';
document_key = r.document_key;
}
i++;
}
while (chunk != 'COMPLETE');
}
// upload process should return a document key if successful...
if (!document_key) {
fnutils.addNote('Received an invalid response from FileNet:\n\n' + response_body, source_record);
}
// build the URL for the
var query_path = '/navigator/bookmark.jsp?desktop=XYZ&repositoryId=XYZ&version=released&docid=';
var url = fnutils.craftURL(document_key, query_path);
if (fnutils.updateFNUploadTable(url, attachment_data)) {
fnutils.addNote('Successfully uploaded attachment ' + attachment_data.file_name + ' to FileNet.', source_record);
if (fnutils.deleteAttachment(attachment_id)) {
fnutils.addNote('Attachment ' + attachment_data.file_name + ' has been removed by automation.', source_record);
}
else {
fnutils.addNote('Automation failed to remove attachment ' + attachment_data.file_name, source_record);
}
}
else {
fnutils.addNote('Failed creating record for FileNet upload!', source_record);
}
}
}
else {
fnutils.logger('No attachments found for this record!');
}
})();
The Script Include
To keep this article short, the script include can be found here.
In conclusion, it works. 😀 Like I said,, information for doing this was quite sparse. Hopefully this article will help someone who's tasked with a similar challenge. Personally, I'm not a fan of the SOAP protocol and I don't know much about how FileNet works. I only learned what I needed to know. If what I'm doing here is horrible, please let me know. I won't be offended.
