Converting XML to JSON failing with gs.xmlToJSON when XML contains #&13;

[codycotulla]

Hi,

I'm trying to convert XML strings to JSON objects using gs.xmlToJSON()

I've found that if the XML contains a character entity such as 
 then the conversion fails.

For example

var xTest = "<a>Hello &#13; World</a>";

var obj = gs.xmlToJSON(xTest);

// obj is null;



var xmlDoc = new XMLDocument2();

xmlDoc.parseXML(xTest);

var tOrF = xmlDoc.isValid();

// tOrF is true 

I know that the XML is valid because the XMLDocument2.isValid() method is OK.

I also know that if I take out the & in 
 then the conversion works.

var xTest = "<a>Hello &#13; World</a>";
xTest = "<a>Hello #13;  World</a>";

var obj = gs.xmlToJSON(xTest);

// obj is an Object.

Has anyone seen this problem? Is there a work around?

Any help is appreciated.

Thanks,

Cody

[Simon Christens]

You cannot parse & into XML as it needs to be converted.

This can be done by:

var xmlDoc = new XMLDocument2();
xmlDoc.createElementWithTextValue("a", "Hello 
 World");

This is a sample to generate XML from an Incident:

	var gr = new GlideRecord('incident');
	gr.get('1775976bdb20e300b182793ebf9619fb');
	
	var fields = new GlideRecordUtil().getFields(gr); //First we get all fields
	fields.sort();
	
	var xmlDoc = new XMLDocument2();
	var xmlPayload = xmlDoc.createElement(gr.getTableName());
	var xmlNode;
	
	//var fields = ['short_description', 'description', 'state', 'sys_id'];
	
	for(var i = 0; i < fields.length; i++){
		
		if(gr.getValue(fields[i]) != 'null' && gr.getValue(fields[i]) != null && gr.getValue(fields[i]) != ''){
			
			if(gr.getElement(fields[i]).getED().getInternalType() == 'boolean'){
				
				xmlNode = xmlDoc.createElementWithTextValue(fields[i], gr.getDisplayValue(fields[i]));
				
			}else{
				
				xmlNode = xmlDoc.createElementWithTextValue(fields[i], gr.getValue(fields[i]));
			}
			
			if(gr.getElement(fields[i]).getED().getInternalType() == 'reference'){
				
				xmlNode.setAttribute('display_value', gr.getDisplayValue(fields[i]));
			}
		}else{

			xmlNode = xmlDoc.createElement(fields[i]);	
		}
		xmlPayload.appendChild(xmlNode);
	}
	
	gs.print(xmlPayload.toString());

If you want to manually convert the chars then see this as an example:
https://stackoverflow.com/questions/7918868/how-to-escape-xml-entities-in-javascript

[Simon Christens]

I have tested alittle more and it seems interresting:

First:

var xTest2 = "<a>Hello &#13; World</a>";

This cannot be parsed into a XMLDocument because it needs a root node. If it havent got a root node then its taken as a string.

var test = '<root>' + xTest2 + '</root>';
var xml = new XMLDocument2();
xml.parseXML(test);

Now we got a XMLDocument

var obj = gs.xmlToJSON(xml.toString());

This still fails though

var helper = new XMLHelper(xml.toString());
var obj = helper.toObject();

But with XMLHelper API we are able to convert the XML to an Object and it will also understand the conversion of "
" and convert it to "\r"

In general im not a fan of gs. methods (time, conversions etc) and i prefer to use the built in API's to do the job

[codycotulla · Accepted solution]

Hi Simon,

Thanks for the suggestions! However, I'm not able to use the XMLHelper because I'm in a scoped application.

It seems that the gs.xmlToJSON() command should work because 
 is a valid character entity. It appears all over the payload of ecc_queue records. 

I decided to go a "kludgy" route and strip out the white space characters that are common. To that end I created a script include that converts Xml to Json and that converts Json To XML. The Json to XML part, I copied from the XMLHelper Script include from ServiceNow.

Below is the content of the script include. It's called OkXMLHelper because ... it's OK for what I need!

When it comes to converting Xml to JSON. There are differences between how this works and how XMLHelper works. I've documented the differences in the description of the Script Include which I've attached to this comment. Along, with a scheduled job for testing.

var OkXMLHelper = Class.create();
OkXMLHelper.prototype = {
	
	initialize : function() {
	},
	
	toObject : function(s){
		this.toObject = function(s){
			s = this.fixXml(s);
			var obj = gs.xmlToJSON(s);
			return obj;
		};
		this.fixXml = function(s){
			//The gs.xmlToJSON() method doesn't handle character entities.
			//So I'm replacing the common ones for whitespace characters that appear
			//in the ecc_queue payload strings that I'm primarily working with.
			if(s.match(/&#/ig) != null){
				s = s.replace(/
/ig, "\r");
				s = s.replace(/
/ig, "\n");
				if(s.match(/&#/ig) == null){
					return s;
				}
				s = s.replace(/	/ig, "\t");
				s = s.replace(/	/ig, "\t");
			}
			
			return s;
		};
		return this.toObject(s);
		
	},
	
	
    toXMLStr : function(o, leaveBlanks) {
		/*
			This was copied from the XMLHelper script include provided by ServiceNow.
		*/
        var toXml = function(v, name, ind) {
		   if (typeof(v) == "function")
			  return "";
		   
            var xml = "";
            if (v instanceof Array) {
                for ( var i = 0, n = v.length; i < n; i++)
                    xml += ind + toXml(v[i], name, ind + "\t") + "\n";
            } else if (typeof (v) == "object") {
                var hasChild = false;
                xml += ind + "<" + name;
                for ( var m in v) {
                    if (m.charAt(0) == "@")
                        xml += " " + m.substr(1) + "=\"" + v[m].toString() + "\"";
                    else
                        hasChild = true;
                }
                xml += hasChild ? ">" : "/>";
                if (hasChild) {
                    for ( var m in v) {
                        if (m == "#text")
                            xml += v[m];
                        else if (m == "#cdata")
                            xml += "<![CDATA[" + v[m] + "]]>";
                        else if (m.charAt(0) != "@")
                            xml += toXml(v[m], m, ind + "\t");
                    }
                    xml += (xml.charAt(xml.length - 1) == "\n" ? ind : "") + "</" + name + ">";
                }
            } else {
                xml += ind + "<" + name + ">" + v.toString() + "</" + name + ">";
            }
            return xml;
        };

        xml = "";
        for ( var m in o)
            xml += toXml(o[m], m, "");
		
		if (leaveBlanks)
			return xml;
		
        return xml.replace(/\t|\n/g, "");
    },


	type : "OkXMLHelper"
};

Also if you are interested in running a scheduled job in the debugger, which is what I used for testing this, see Test Scheduled Jobs(and Fix Scripts) in the Script Debugger

Thanks,

Cody

[FilipVacula]

Hi Cody,

I've been dealing with the task of converting a XML to an JS Object within a scoped application 2 times recently, and found your response helpful. There are 2 improvements I have put in place:

1. Trim the XML string. If there were empty lines at the end of it, xmlToJSON was throwing an exception.

2. Instead of replacing just a selected few entities, I've made the replacement generic, which gets rid even of unexpected XML entities.

The resulting code:

function xmlToObject(xmlString) {
	var repairedXmlString = xmlString.replace(/&#(\d{1,6});/g, function(match, asciiCode) {
		return String.fromCharCode(asciiCode);
	});
	return gs.xmlToJSON(repairedXmlString.trim());
}

Just for reference, there are 2 HI Known Errors related to this topic:

KB0688278

KB0784264

Hope this helps!

Filip