Trying to understand GlideRecord.next()

[Javier Arroyo]

This article presents an alternative explanation to the visible behavior of GlideRecord.next() when looping through a Record Set and setting values in an Object or Array. By now we’ve all likely seen that the Object or Array will only hold the value of the last object in the loop. Values of the last record retrieved from GlideRecord.next() are duplicated for the total count of records returned by GlideRecord.query() in the Object or Array.

The GlideRecord below demonstrates the behavior. The query returned 56 records, looping through each, then extracting the record number during each iteration shows the final array only displays the very last item in the record set. That result is the behavior which I’ll look to explain in this article.

var incident = new GlideRecord('incident');
incident.addActiveQuery();

incident.query();

var incidentNames = [];

while( incident.next() ) {
   incidentNames.push( incident.number);

}

gs.debug( incidentNames );

// INC0007002,INC0007002,INC0007002,INC0007002,INC0007002,INC0007002,INC0007002,INC0007002,INC0007002,INC0007002,INC0007002,INC0007002,INC0007002,INC0007002,INC0007002,INC0007002,INC0007002,INC0007002,INC0007002,INC0007002,INC0007002,INC0007002,INC0007002,INC0007002,INC0007002,INC0007002,INC0007002,INC0007002,INC0007002,INC0007002,INC0007002,INC0007002,INC0007002,INC0007002,INC0007002,INC0007002,INC0007002,INC0007002,INC0007002,INC0007002,INC0007002,INC0007002,INC0007002,INC0007002,INC0007002,INC0007002,INC0007002,INC0007002,INC0007002,INC0007002,INC0007002,INC0007002,INC0007002,INC0007002,INC0007002,INC0007002 (sys.scripts extended logging)

Before presenting my thoughts, lets look into the prevailing interpretation of the what’s happening. The interpretation states that because of some quirky behavior of JavaScript’s Pass-By-Reference, the value must be retrieved by accessing the properties field’s method getValue(), or some variation of type conversion to extracts the precise object value. It is further explained that because GlideRecord.fieldName such as incident.number returns an Object, the only possible value will be that of the very last record, as each push(incident.number) call isn’t really extracting a value but pointer to a reference.

I find the existing explanation to misrepresent the effect, and not grounded on concrete JavaScript knowledge. Perhaps looking at this behavior through an ECMAScript lens will yield a closer representation. Because I’m no expert on the behavior between JavaScript and Java, I’ll purely look at this from a JavaScript perspective. In my humble opinion, GlideRecord.next() should have been implemented differently, therefore, I consider the behavior in question a buggy implementation of GlideRecord.next()

We’ll start by addressing the existing theory of a Quirky Pass-By-Reference JavaScript behavior.

If GlideRecord is a Java Class, then how does Pass-By-Reference (PBR) come into play? Does it even come into play, as it rests outside the Rhino Engine? By the result, I say PBR doesn’t have a thing to do with anything; and more importantly, because JavaScript uses an Evaluation Strategy called Call-By-Sharing, then it’s impossible that some quirky behavior of PBR is to blame.

Furthermore, when declaring var gr = new GlideRecord(‘incident’) a memory location is allocated to gr. This fact means that gr will always point to the same memory location, further distancing the blame from PBR. But is it reasonable to presume that GlideRecord.next() is creating a brand new gr during each .next call setting a brand new memory location, and why PBR is the right answer? Were that the case, each GlideRecord.field would indeed point a different object in a different memory location, thereby returning the correct value for each object. The logical outlook is that it is highly unlikely for the platform to be replacing the entire gr record with a brand new memory location. So, what is really happening inside the method GlideRecord.next? Is the call being handled by Rhino or by Java? If it’s a Java class, then Rhino is not involved as Rhino is a JavaScript engine. So, I don’t concretely know what’s happening inside GlideRecord.next.

But, lets still try to figure out how PBR can be responsible. For it to be, an evaluation strategy must be be identified AND come into play. But, there doesn’t seem to be an evaluation strategy involved at all. GR is not being passed into a function call anywhere, it is its own class, doing the work itself. GlideRecord.next is an instance method that doesn’t accept an argument. If it did it would be GlideRecord.next(GlideRecord). Because .next doesn’t accept an argument, Pass-By-Reference is never involved. Lastly, if an evaluation strategy did come into effect then, it would be Call-By-Sharing and the values retrieved by each consecutive .next() call would undoubtedly point to the correct record, bypassing a need to call getValue to get the correct value. JavaScript’s evaluation strategy is known as Call-By-Sharing, it differs in that all values in JavaScript are memory locations, there by dictating that JS is a Call-By-Value language, and that the values are memory locations. When passed as arguments into functions, the memory location can not be overridden.

My interpretation is that GlideRecord is mutating values instead of pointing each .next call to a different record. This is either because of a misguided implementation, or a technical limitation. Either way, the effect is definitely that values are mutating.

Mutation means changes of properties, Objects when passed into functions can not change memory reference; they can only mutate as stated above. Such as:

var p = {
  name: 'me'
};

function transfromp(obj) {
   obj = {
     name: 'another'
  };
}

transfromp(p);
gs.debug( global.JSON.stringify(p)); // {name: ‘me’}

Notice that although p was passed into function transformp, and the object rewritten, p is still pointing to the initial memory location even after transformp has rewritten the object. This points to one of ECMAScript mandates: Object memory locations can not be overwritten when passed into a function.

On the same toke, if the code above bypassed Call-By-Sharing all together and changed the value in the same context. Then the memory location would take on the newly set value, such as:

var p = { name: 1};
p = { name: 2};

gs.debug( global.JSON.stringify(p)); // {name: 2}

The knowledge of the above seems to indicate that each .next() is indeed mutating.

This is what is known thus far.

1. PBR is not a thing in JavaScript, so it can’t be a quirky implementation to blame.
2. Call-By-Sharing isn’t involved because .next() doesn’t accept parameters .next() isn’t being rewritten in a new memory location
3. an object in the same context can be re-written.
4. A new object rewritten would point to a brand new memory location yielding the value of the last set statement
5. Still sort of obscure is why getValue, toString, or + "" do return the correct answer.

The answer lies in JavaScript’s behavior. Primitives are immutable so that when the operations above are called, JavaScript returns a copy of the value, not the value itself. This means that a brand new value pointing to a brand new memory location is returned and stored into the array or object.

When directly accessing a property of the field, such as the case with Array.push( GlideRecord.getValue(fieldName) ), getValue doesn’t work because it’s directly grabbing the object value, but because the value being retrieved is turned into a primitive, thereby becoming a whole new value in anew memory location. That tidbitlikely indicates to the only technically correct and logical answer: bugs and new values in memory locations.

If the presumed explanation of Call-By-Reference was to be accepted then the only outcome based on the current implementation of GlideRecord.next() for getValue is to return the value of the very last item through the loop, just as GlideRecord.fieldName does, rather than the correct value. The reasoning is that getValue belongs to the very last Object (which can’t be another object at all but the very same memory location pointing to an object with mutated values). That a GlideRecord.field is an Object, as opposed to a primitive makes no difference when an object points to the very same memory location. This can only mean that the problem is elsewhere; that place is, in my limited insight into the platform, a buggy implementation of .next(), even GlideRecord as a whole.

The function call should instead work like it is expected, be it in a Call-By-Reference or Call-By-Sharing. An entirely new object returned so that it work such as

var objs = [{name: 1},{name: 2},{name: 3},{name: 4},{name: 5},{name: 6},{name: 7}];
var names = [];
objs.forEach( function getValues(item) {
 names.push( item.name );
 items.push(item);
});

items //[{"name":1},{"name":2},{"name":3},{"name":4},{"name":5},{"name":6},{"name":7}]
names // [1, 2, 3, 4, 5, 6, 7]

The above outcome is what is expected in a Call-By-Reference or Call-By-Sharing evaluation strategy. Without a code base to analyze, one can only interpret the behavior and surmise that it all boils down to GlideRecord mutation, a bug, or some sort of technical impediment between Java and JavaScript.

Well, even in the absence of a code base to understand, and this article is incorrect, the explanation looks to be logical and grounded in JavaScript. At least hopeful to further clear the current interpretation.

[John Caruso]

The thing most people don't grok is that GlideRecord properties (i.e., field getter/setters) return GlideElements, not primitive values.

Below is a mock GlideRecord implementation that (intentionally) exhibits the same behavior seen in ServiceNow. I imagine this is pretty close to what the Java code is doing, i.e., creating a GlideElement per column and simply mutating its value as the cursor over the query result set advances.

This also demonstrates why '==' is widely used instead of the JS best practice (outside of ServiceNow) '===' when comparing GlideElement values.

var db = {
  incident: {
    schema: { number: { type: "string" } },
    data: [{ number: "INC0000001" }, { number: "INC0000002" }]
  }
};

function GlideElement(schema, value) {
  this.internalType = schema.type;
  this.value = value;
}

GlideElement.prototype.toString = function() {
  return this.value.toString();
};

function GlideRecord(tableName) {
  this._schema = db[tableName].schema;
  this._data = db[tableName].data;
  this._set = [];
  this._index = 0;
  for (var field in this._schema) {
    this[field] = new GlideElement(this._schema[field], null);
  }
}

GlideRecord.prototype.query = function() {
  this._set = this._data.filter(function(record) {
    // TODO: eval this.encodedQuery
    return true;
  });
};

GlideRecord.prototype.next = function() {
  if (this._index >= this._set.length) return false;
  var record = this._set[this._index++];
  for (var field in record) this[field].value = record[field];
  return true;
};

var result = [];
var gr = new GlideRecord("incident");
gr.query();

while (gr.next()) {
  var el = gr.number;

  console.log(el.toString());
  // eslint-disable-next-line eqeqeq
  console.log("el == 'INC0000001'", el == "INC0000001");
  console.log("el === 'INC0000001'", el === "INC0000001");
  console.log(el);

  result.push(el);
}

console.log(JSON.stringify(result));

[Javier Arroyo]

John,

Loved the post. And, the sandbox, simply wonderful. Got no idea how you did it. Love it. 

Agreed. As long as the Object mutates (with objects) and context remains the same, the memory location during the final iteration can only be that very last one. It's default JS behavior. One can only surmise that a closure might just yield the correct result each time around despite the mutation... given that closures wraps the actual value. Sort of the same context gotcha seen with setTimeout inside a for loop. The output, unless wrapped inside a context, will always give the unexpected result.

var arr = ['a', 'b', 'c', 'd'];
for (var i = 0; i < arr.length; i++) {
  setTimeout(function() {
    console.log('Index: ' + i + ', element: ' + arr[i]);
  }, 1000);
}

The output will be the line below as many times it loops. 
Index: 4, element: undefined

But, if it is placed inside a closure, the correct values will be presented. 

[John Caruso]

As long as you're pushing gr.number into an array, whether inside a closure or not, you're pushing the same object reference over and over, namely the reference to the single GlideElement object created for the number column.

The only way to push the value itself is to call gr.getValue('number') or gr.number.toString() on the GlideElement returning a new (string) object.

[Javier Arroyo]

It sure is as you state.

Clarifications for future readers:

A reference isn't being pushed into the array, rather a value. That value is the in memory address of the Object. The behavior is known as Call-by-Sharing, the evaluation rule for JS. This is why a value/primitive is the only way to access the value, they method calls turn the object into primitives creating new memory locations/new primitives (primitives aren't Objects).

getValue and toString both return the string primitive, rather than String object. The difference being that a primitive is always parsed, where as String object isn't.

this can be seen by: 

gs.debug( typeof gr.getValue('number') ); //=> string

and

var stringObject = new String("some string");

gs.debug( typeof stringObject ); //=> object

to turn stringObject into a string use the String.valueOf() method.