Diagnosing Pseudo-paused Mid Servers

[tim_broberg]

I've had a few issues with mids getting stuck where they respond only to SystemCommands and heartbeats.

They're in some kind of pause state, but it's not obvious what that state is.

This is my tool to figure out what's going on without having to squint at heap dumps all day.

This mid server script include captures the state of the relevant variables.

var MidState = Class.create();

(function() {
MidState.prototype = Object.extendsObject(AProbe, {
    initialize : function(probe) {
        this.probe = probe;
        this.output = "";
    },

	run : function() {
		var mid = Packages.com.service_now.mid.MIDServer.get();
		var validityChecker = Packages.com.service_now.mid.security.MIDValidityChecker.get();
		this.output = 'Paused: ' + mid.isPaused() + ', Operational State: ' + mid.getOperationalState() + ', Valid: ' + validityChecker.getIsValid();
		this.createOutputResult(this.output);
	},

	type: "MidState"
});

})();

I have a Probe named MidState with topic = JavascriptProbe and ECC queue name = MidState.

I run Test probe on the probe with the mid server I need to investigate and it reports back what the internal state variables of the mid are.

Known behaviors I'm currently watching for in our environment:

1. Normal: Paused: false, Operational State: UP, Valid: true
2. PRB1250701: Paused: false, Operational State: UP, Valid: false
3. PRB1247167: Paused: true, Operational State: UPGRADE_CHECK, Valid: true

PRB1250701 will clear up if you pause and unpause the mid server. I think the other one might also.

PRB750509 is the other big one, where network glitches during credential reload cause it to stick forever. This is fixed in K / JP3 / IP9. A credential reload will clear it up, which you can do by touching any credential.

Hope it helps,

    - Tim.