MID Servers getting authentication issues

[MarkM1172412930]

We are migrating our MID Servers from vmWare to Azure. In the process I have new MID Servers that I am trying to configure. They got connected to the instance and stayed up for at most 10 min. Now all of them have the following in the logs:

User [username] cannot be authenticated! Please ensure the MID server user exists on Instance and has the proper roles.

I know that the account exists and the password is good as the existing MID Servers are using them. I have even tried to shut down one of the existing ones that uses that account to see if that is the issue, but the new one still will not connect.

Wonder if anyone else has run into this issue. I have configured many MID Server before and never had this one happen. I do have a HI Ticket open with Service Now Support as well, but they are not able to track it down just yet.

[Tanushree Maiti]

Hi @MarkM1172412930 

Hope you have gone through this KB and validated each steps:

KB0597574 Troubleshooting MID Server user authentication issues 

Also check that your MID server's ip is whitelisted under IP Access Control .

[Shahjay]

Hi @MarkM1172412930 ,

How did you build the Azure boxes? If they were cloned from the vmWare VMs, or you copied the agent directory across, that's more likely your problem than the account. A copied install carries three things that don't survive the move: the keystore in agent/keystore, the name and mid_sys_id parameters in config.xml, and the already-encrypted mid.instance.password. The MID comes up looking fine on that copied identity and then gets refused a few minutes later, which is roughly what you're describing. The auth wording in the log is a bit of a red herring in that case.

If that's what happened, on one of the new boxes:

Stop the service.
Rename agent/keystore so a fresh one gets generated.
In config.xml, give it a new name, clear mid_sys_id, and put the password back in as plain text.
Start it, then validate the record from MID Servers > Servers.
The MID encrypts that password with its own key on first start, so a blob copied from another host won't decrypt. Worth checking for a leftover record with the same name too — a name or sys_id collision gets a MID kicked after it has already connected (KB0743043).

Before rebuilding anything, though, look at what the instance actually saw. ecc_agent_issue is where the instance logs MID user login and connectivity problems, so filter it on the new MID's name. If there's nothing there and no failed login against the user, the request never landed and you're chasing a network problem, not credentials.

On Tanushree's IP Access Control point, one thing to watch on Azure: outbound traffic doesn't necessarily leave on a single address. A NAT gateway or a load balancer frontend pool can hand you several, so allowing the one IP you tested from works right up until a connection goes out on a different one. Add the whole egress range. KB0864024 covers MID Servers and IP Address Access Control specifically.

Something else that would fit your symptom neatly is the newer basic auth restriction. Existing MIDs keep running on sessions they already hold, so they look healthy while a brand new login on the same account gets a 401 — which is exactly the contradiction you're hitting. Check glide.authenticate.basic_auth.restriction.active and .enforce, and look for that user in the Basic Auth User Exceptions table. The mid_server role is supposed to inherit the access, but if a revoke decision or the default decision caught the account you'd see this.

One thing I'd steer away from: don't tick Web service access only on the MID user as a fix. There's a known error against that (KB0598957) where MID validation fails with it set. If you need that account explicitly allowed for basic auth, the snc_basic_auth_api_access role is the cleaner lever.

The rest of the account checks are quick — active, not locked out, mid_server role, local ServiceNow user rather than an LDAP or SSO one. If the password has special characters in it, swap it for letters and numbers; the config.xml encryption doesn't handle all of them cleanly. And while support are still looking, give the new Azure MID its own dedicated user. If it stays up you've proved it was the shared account or a copied credential, and if it still drops at ten minutes you've narrowed it to the host.

[Ambuj Tripathi]

Please check if this user has 

snc_basic_auth_api_access

role assigned directly or indirectly through mid_server role or its its marked as machine user. At least one of these should enable it do make API calls.