HowTo: Encrypted Catalog variables, at least how we did it and the libraries we used. SN please read...

[DrewW]

So I would like to start with if you are here to understand what we did and how we use it please skip down to the what you need, otherwise read on for my why and a little bit of a rant that I hope ServiceNow reads.   I also hope that people find this of use since I did not find much on the community that addressed Catalog Vars and also the fact they show in the text logs in clear text.   I also find password vars annoying and really did not want to use them for data entry.

Recently we had a request to add a catalog item which requires the user to enter an MRN.   So we need to start encrypting this data.   Now what, what do we do, whats the easiest way to go about this and so on and so forth.   So we found that we had several options but in came down to two, first is we can use and encrypted field, second is Edge Encryption Proxy (EEP).   Both have there good and bad, but neither of them support UI Pages and catalog variables.   I have been told the EEP will support cat vars but no one has mentioned UI Pages.   Ok well that is frustrating, what are we supposed to do with our password reset???   The other issue we had is that why do I have two extremes????   Simple encrypted field or EEP which as far as I am concerned is a giant cog wheel stuffed into the middle of my ServiceNow configuration.   Yes EEP is relatively simple to use once setup but now I have to worry about the people who maintain the firewall making changes and causing down time and making sure that external and internal techs can get at it.   Along with making sure the hardware, OS and other items are maintained.   For what, one encrypted field.   One you say?   Why just one?   Well you cannot search on an encrypted field, whether it was EEP or an encrypted column, this makes sense to me so I do not want to encrypt the Description, work notes or Short description.   This is do to the threats from the Service Desk to lynch me if they could not find records based on searches of those fields.   A solution to this is EEP's tokens but again that's a big cog wheel that we cannot buy servers for and do not want in the middle of our config.

So ultimately we wanted one solution that will take care of our cat vars and also our UI Pages.   The reason we want to include UI Pages is the system writes all of the URL's and there parameters to the text logs that are on disk of the server.   This is the only reason for this solution is we do not want the data put into the logs on the server because they are in clear text.   Yes they roll over every 45 days I believe it is but if someone gets the text log I would prefer to make it at least a little difficult to get the data.   We do not consider EEP to be a good solution because it adds this big cog wheel into our config when one of the whole reasons we want to use ServiceNow is that if we have an outage or aliens steal the building we can just have people work from home and still get at the ticketing system.   EEP does not allow that.   Also if EEP is not going to support UI Pages how am I supposed to import sensitive user data to use in the password reset process.

What I would like from ServiceNow,

1 - Let me set a list of URL's the server will not record in the log.   This will eliminate some of my issues.

2 - Allow me to run a job as a user WITHOUT impersonating that user.   Yes I will supply a user name and password so the process can login.   This should allow me to setup an account that has an encryption context and use it to do some work with out having to setup a web service and make a call to that web service to do that work.

3 - Allow journal fields to have there values changed in a before business rule without causing a double entry.   This would simplify somethings we are thinking of doing around "bad" data.

I know that someone is going to think that this is also a bit of a cog wheel in my ServiceNow config but its also all contained in the instance and right now I definitely prefer that to a bigger one outside of my instance.

On to the what and how.

What you need

If you like you can just download the attached update set and import it.

You will need the AES-JS from GitHub, both version 3 and 2.1.   Version 3 will not run server side but 2.1 will and they are compatible.

GitHub - ricmoo/aes-js: A pure JavaScript implementation of the AES block cipher and all common mode...

If you want to encrypt csv files for import you will want to download and install Node.js on your MID server.   Once you do you can just run "npm install aes-js" to install the AES-JS library for use in the csv file encryption, there is a sample below.

Download | Node.js

How we used it

PLEASE NOTE THAT ALL MY EXAMPLES USE THE CODE IN THE ATTACHED UPDATE SET.   I created a helper class called AESJSUtil to simplify things, its included below.

So what you can do with the aes-js is take v3 and make it a global UI Script.   Then take v2.1 and make it a script include.   This will make the aesjs object available both on the server and in the UI when ever you need to use them.   You will then need to setup a helper class like I did so that you can get the encryption keys over to the client only when they are needed.

For a catalog variable you use an on submit script like this.

function onSubmit() {
   if(g_form.mandatoryCheck()){
   var tv = g_form.getValue("patient_mrn_encrypt");
   var ev = AESJSUtil.encryptValue(tv);
   //Check to make sure that the value returned is not the same
   //as the value passed in.   This should indicate that the value
   //was encrypted.
   if(tv != ev){
   g_form.setValue("patient_mrn_encrypt", ev);
   } else {
   alert("We're sorry but we where unable to encrypt the MRN field so we cannot allow the submitmition of this request, please contact the HFHS Service Desk (534900).");
   }
   }
}

If you have all of your vars that will be encrypted have "_encrypt" at the end you can then use an before insert business rule on the Requested Item table like this to move the data to an Encrypted Column.

(function executeRule(current, previous /*null when async*/) {
   for(var v in current.variables){
   if(v.length > 8 && v.search("_encrypt") != -1){
   var et = current.variables[v].toString();
   current.variables[v] = "Value stored in the encrypted data field.";
   current.u_encrypted_data.setDisplayValue(getVarLabel(v) + ": " + AESJSUtil.decryptValue(et));
   }
   }
   function getVarLabel(name){
   var set = new GlideappVariablePoolQuestionSet();
   set.setRequestID(current.getValue("sys_id"));
   set.load();
   var vs = set.getFlatQuestions();
   var iter = vs.iterator();
   while (iter.hasNext()) {
   var item = iter.next();
   if(item.getName() == name){
   return item.getLabel();
   }
   }
   }
})(current, previous);

For CSV files that you want to encrypt certain columns then you can use a Scripted Web service to get the keys and another one to trigger the job that will import the data.   From there you can use the below javascript to encrypt the needed data using the Node.js environment.   This code is not in the attached update set.

const aesjs = require('aes-js');
const fs = require('fs');
const https = require('https');
// write to file
var txtFileo = "C:/inetpub/ftproot/PSHR/sncps.csv";
var txtFilen = "C:/inetpub/ftproot/PSHR/sncpsen.csv";
var firstLineHeaderSkipped = false;
//make sure the new file does not exist.
fs.access(txtFilen, fs.constants.W_OK, function (err) {
       if (!err) {
               fs.unlink(txtFilen);
       }
});
getKeys(function(data){
       fs.readFileSync(txtFileo).toString().split('\n').forEach(function (line) {
               str = line.toString();
               if (str != "") {
                       if (!firstLineHeaderSkipped) {
                               firstLineHeaderSkipped = true;
                       } else {
                               str = encryptLine(data.result.key, data.result.iv, str);
                       }
                       fs.appendFileSync(txtFilen, str + "\n");
               }
       });
       fs.access(txtFileo, fs.constants.W_OK, function (err) {
               if (!err) {
                       fs.unlink(txtFileo);
               }
       });
       startJob(function (data) {
               if (data.result != "success")
                       console.log(data.error);
       });
});
function encryptLine(key, iv, str) {
       var vals = str.split(",");
       var nstr = "";
       if (vals.length <= 1)
               return str;
       //Encrypt Column 1, 2 and 4
       for (var i = 0; i < vals.length; i++) {
               var s = "";
               if (i != 0 && i != 3) {
                       // Convert text to bytes (text must be a multiple of 16 bytes)
                       vals[i] += ('                               ').substr(vals[i].length % 16);
                       var textBytes = aesjs.utils.utf8.toBytes(vals[i]);
                       var aesCbc = new aesjs.ModeOfOperation.cbc(key, iv);
                       var encryptedBytes = aesCbc.encrypt(textBytes);
                       s = aesjs.utils.hex.fromBytes(encryptedBytes);
               } else {
                       s = vals[i];
               }
               if (nstr)
                       nstr += "," + s;
               else
                       nstr += s;
       }
       return nstr;
}
function getKeys(success) {
       var options = {
               host: 'myinstance.service-now.com',
               path: '/api/hefhs/encryption/keys',
               auth: 'SomeAccount:MyRediculuslyLongPasswordThatIUseOnThursdays.',
               method: 'GET'
       };
       var req = https.request(options, function (res) {
               res.setEncoding('utf8');
               var responseString = '';
               res.on('data', function(data){
                       responseString += data;
               });
               res.on('end', function () {
                       var responseObject = JSON.parse(responseString);
                       success(responseObject);
               });
       }).end();
}
function startJob(success) {
       var options = {
               host: 'myinstance.service-now.com',
               path: '/api/hefhs/encryption/runjob/SYSIDOfJob',
               auth: 'SomeAccount:MyRediculuslyLongPasswordThatIUseOnThursdays.',
               method: 'GET'
       };
       var req = https.request(options, function (res) {
               res.setEncoding('utf8');
               var responseString = '';
               res.on('data', function (data) {
                       responseString += data;
               });
               res.on('end', function () {
                       var responseObject = JSON.parse(responseString);
                       success(responseObject);
               });
       }).end();
}

This is my AESJSUtil script include.   Needs to be client callable.   Some of this I included in the UI Script to make things easier.

var AESJSUtil = Class.create();
AESJSUtil.prototype = Object.extendsObject(AbstractAjaxProcessor, {
       //Returns both key and iv in hex format delimited by a :.
       getKeyIVFromPropHex: function () {
               var ge = new GlideEncrypter();
               var key = gs.getProperty("hfhs.default.encryption.key");
               var iv = gs.getProperty("hfhs.default.encryption.iv");
               return "" + ge.decrypt(key) + ":" + ge.decrypt(iv);
       },
       type: 'AESJSUtil'
});
AESJSUtil.getRandomKeys = function (keyLength) {
       if (keyLength != "16" && keyLength != "24" && keyLength != "32") {
               throw "Key length must be 16, 24 or 32";
               return;
       }
       var l = Number(keyLength);
       var keys = {};
       keys.key = [];
       keys.iv = [];
       for (var i = 0; i < l; i++) {
               // 1 in 5 chance that the next random number is chosen for our key.
               // This helps obscure the key by selecting values at random points in time instead of one immediately following the other.
               while (Math.floor((Math.random() * 256) % 5) != 0) { }
               keys.key[i] = Math.floor(Math.random() * 256);
               keys.iv[i] = Math.floor(Math.random() * 256);
       }
       return keys;
};
AESJSUtil.getEncryptedRandomKeys = function (keyLength) {
       var keys = AESJSUtil.getRandomKeys(keyLength);
       var ge = new GlideEncrypter();
       keys.keyen = ge.encrypt(aesjs.util.convertBytesToString(keys.key, "hex"));
       keys.iven = ge.encrypt(aesjs.util.convertBytesToString(keys.iv, "hex"));
       return keys;
};
AESJSUtil.getKeyFromProp = function () {
       var ge = new GlideEncrypter();
       var key = gs.getProperty("hfhs.default.encryption.key");
       return aesjs.util.convertStringToBytes("" + ge.decrypt(key), "hex");
};
AESJSUtil.getIVFromProp = function () {
       var ge = new GlideEncrypter();
       var iv = gs.getProperty("hfhs.default.encryption.iv");
       return aesjs.util.convertStringToBytes("" + ge.decrypt(iv), "hex");
};
AESJSUtil.encryptValue = function (text, key, iv) {
       if (!key)
               key = AESJSUtil.getKeyFromProp();
       if (!iv)
               iv = AESJSUtil.getIVFromProp();
       // Convert text to bytes (text must be a multiple of 16 bytes)
       text += ('                               ').substr(text.length % 16);
       var textBytes = aesjs.util.convertStringToBytes(text);
       var aesCbc = new aesjs.ModeOfOperation.cbc(key, iv);
       var encryptedBytes = aesCbc.encrypt(textBytes);
       var encryptedHex = aesjs.util.convertBytesToString(encryptedBytes, "hex");
       return encryptedHex;
};
AESJSUtil.decryptValue = function (encryptedText, key, iv) {
       if (!key)
               key = AESJSUtil.getKeyFromProp();
       if (!iv)
               iv = AESJSUtil.getIVFromProp();
       var encryptedBytes = aesjs.util.convertStringToBytes(encryptedText, "hex");
       var aesCbc = new aesjs.ModeOfOperation.cbc(key, iv);
       var decryptedBytes = aesCbc.decrypt(encryptedBytes);
       var decryptedText = aesjs.util.convertBytesToString(decryptedBytes);
       return decryptedText.trim();
};
AESJSUtil.decryptValueWithGE = function (v) {
       var ge = new GlideEncrypter();
       return ge.decrypt(v);
};
AESJSUtil.encryptValueWithGE = function (v) {
       var ge = new GlideEncrypter();
       return ge.encrypt(v);
};
AESJSUtil.scrubText = function (textValue) {
       var tv = "" + textValue;
       var matches = tv.match(/\d{7,12}|\d{3}[^0-9!\n]\d\d[^0-9!\n]\d{3,}/gi);
       if (matches) {
               matches = (new ArrayUtil()).unique(matches);
               for (var i = 0; i < matches.length; i++) {
                       var ev = AESJSUtil.encryptValue(matches[i]);
                       tv = tv.replace(matches[i], "{ENCRYPTED:" + ev + "}");
               }
       }
       return tv;
};
AESJSUtil.descrubText = function (textValue) {
       var tv = "" + textValue;
       var matches = tv.match(/\{ENCRYPTED:.*?\}/gi);
       if (matches) {
               matches = (new ArrayUtil()).unique(matches);
               for (var i = 0; i < matches.length; i++) {
                       var ev = matches[i].toString().replace("{ENCRYPTED:", "").replace("}", "");
                       var dv = AESJSUtil.decryptValue(ev);
                       tv = tv.replace(matches[i], dv);
               }
       }
       return tv;
};

[kevinclark-7EL]

Wow.   Thanks for shatring! So if I'm reading this correctly, the challenge is that Variables aren't supported by column-level encryption?   I'm investigating this for our instance at the moment, and this is the first to have come up with something approaching what we're trying to do.   Seems like a huge solution for a single field to be encrypted.   I haven't got a MID server I can play with in this environment, so this might not be suitable, but I'll see where we get.

[DrewW]

I would not call this a huge solution, but I can see how it may look that way.   As for a MID server that is not needed unless you want to import data that you would like encrypted before it goes to the instance.

There are also several other issues that have come up since I put this out here.   So your only option out of box is a password field, which masks itself from the log which is good but what we found is that it is not secured in any way once it is on the cart.   So if you have users who put something on the cart and do not order it right a way or ever for some reason the data you where trying to protect is clearly visible in the cart table, not good, at least as far as we are concerned.   So we changed the catalog item functions to encrypt on submit and decrypt on load if there is an encrypted value in the fields we are trying protect.   This way the cart is protected also and the user will be able to see what they entered if they edit the item on there cart.

I also after reviewing the EEP docs again just to make sure I was not going out of my way for nothing found that they say it does not support journal fields either which means no work note encryption so that puts the last nail in that for the time being.   So we adapted the above to work with the work notes and email because we have teams that add work notes that have data that we need to encrypt.

[Reyes]

for service catalog variables, the mask variable will be encrypted. In the mask variable's type specification, you will need to enable encryption

https://docs.servicenow.com/bundle/newyork-it-service-management/page/product/service-catalog-manage...

[DrewW]

Yes it will but they you cannot see what you are typing and if you are typing a fair bit of text that does not do you any good.  Also you would have to move it to another place so the tech can see what was entered which defeats the whole idea.  You could use client script to manipulate the field to make it not masked then masked when it is submitted and so forth.

This also was not just about cat vars because there was need in other places also.

But with all of that said since this was done and posted I would suggest that anyone needs something like this they just buy the Database Encryption the ServiceNow started to offer about 1 year after I did this.  It takes care of the issue and you will not have to mess with any of this.