MID server continually doing credentials_reload but not processing. All are in ready state, every few seconds.

Aoife Lucas
Giga Expert

‎06-06-2019 06:14 AM

MID server appears to be stuck in a loop.  It keeps issuing a credentials reload command but it never gets processed so it does it again and again, every few seconds.  The ECC Queue has them all in the ready state, never processing.

Any ideas?

 

Marshall+

Labels:
0 Helpfuls
  • 3,057 Views
6 REPLIES 6

tim_broberg
ServiceNow Employee
ServiceNow Employee
In response to elemonnier

‎03-24-2020 04:12 PM

I did have a case like this.

 

Do you see a line like this in your agent log?

01/02/20 17:54:35 (746) StartupSequencer WARNING *** WARNING *** Unable to update public key, will try again next MID Server restart

If so, try rekeying the mid server from the mid server record on the instance.

They recently (Orlando?) started rekeying the public key every startup, which can get out of synch.
    - Tim.

elemonnier
Tera Expert
In response to elemonnier

‎03-25-2020 08:00 AM

SNOW support got back to me last night. It's related to a bug PRB1342894: https://hi.service-now.com/kb_view.do?sysparm_article=KB0748823

If you have Discovery installed and you create an OAuth 2.0 Application Registry, the MID servers will be flooded with credentials_reload.

It's quite a pain to clean up if it has been broken for a while, as it builds up an enormous amount of credentials_reload records in the ECC queue. I had over 5 million records I had to set to "error" while the MID servers were all stopped.

0 Helpfuls