Hi everyone, I'm facing an issue with a MID Server that has been in the Validating state

[AvinashU]

Hi everyone,

I'm facing an issue with a MID Server that has been in the Validating state for more than 12 hours.

Environment

• MID Server: mid_demo
• Used for AWS Cloud Discovery

Current observations

• MID Server process is running on the EC2 instance.
• ECC Queue Processed count continues to increase (e.g., 770 → 781 → 802).
• The MID Server is actively processing other ECC messages.

However, I also see the following ECC Queue records:

TopicCommandQueueState
SystemCommand	restartService	input	Ready
SystemCommand	restartService	output	Processed

I observed similar behavior for the Invalidate (delete_mid_keypair) SystemCommand as well:

• Input record remains Ready.
• Output record is Processed.

Questions

1. Is it normal for a MID Server to remain in Validating for more than 12 hours?
2. Is it expected that the SystemCommand input record (restartService or delete_mid_keypair) remains in Ready while the corresponding output record is Processed?
3. Should the input record eventually change to Processed, or is this expected behavior for internal SystemCommands?
4. Has anyone encountered this issue before, and what was the root cause or resolution?

Any guidance would be greatly appreciated.

[Vikram Reddy · Accepted solution]

Hi @AvinashU,

No on both counts. Twelve-plus hours in Validating is not normal, and an input record parked in Ready while its matching output sits in Processed is not expected behavior either. That combination almost always means the MID Server did what it was told, but the instance never finished consuming the reply, it's not a sign that restartService or delete_mid_keypair actually failed on the box.

Think about what Validate is doing under the hood: delete_mid_keypair drops the old keys, the MID reissues a new keypair, restartService bounces the Wrapper service, and the instance re-validates on the next check-in. On a healthy MID that whole handshake is a few minutes, not half a day. The fact that mid_demo is still chewing through other ECC Queue traffic normally (your Processed count climbing) tells you the process itself is fine, the problem is isolated to how the instance is handling the response for those two specific SystemCommands.

A few things I'd check before assuming the worst:

• Compare the exact case of "mid_demo" in the ecc_queue.agent field on the stuck input record against the name field on the mid_server record. A known ServiceNow issue causes input records to hang in Ready forever when there's even a single case mismatch between the two.
• Pull agent.log (not just wrapper.log) on the MID host around the timestamps of the restartService and delete_mid_keypair commands, look for exceptions right after the service comes back up.
• Confirm the MID host's OS timezone matches the instance timezone. Clock/timezone drift between the two is one of the more common causes of validation hanging indefinitely.
• Check whether agent_keystore.jks is out of sync with config.xml. If it is, the rekey can appear to run without ever completing cleanly.
• As an immediate unstick, try Pause MID then Resume MID from Related Links on the mid_server record, that's been reported to flip Validated back to Yes without a reinstall.
• If that doesn't move it, stop the Windows/Linux service, rename or delete the keystore folder, restart the service to regenerate it fresh, then hit Validate again.

If a rekey plus keystore reset still doesn't clear it, post the agent.log lines around that restartService timestamp, that'll usually tell us if it's a cert/network issue or purely the instance not consuming the response.

Thank you,
Vikram Karety
Octigo Solutions INC