Authentication failure with the local MID server service credential

[RRolling]

Yesterday a workflow randomly stopped working and now errors out with "Authentication failure with the local MID server service credential"

Credentials and MID server are fine. They are still working on other workflows but one for SCCM software distribution.

I'm concerned about the output being Null and don't understand why it stopped working.

Any tips?

Robert

[bernyalvarado]

That's odd, do you have multiple MID Servers supporting this workflow? Have you tried restarting the MID Server that had the authentication failure? Have other workflows succeed with that MID Server that failed?

Thanks,

Berny

[Daniel Evans]

Hey Berny,

Thanks for another set of eyes. In our dev environment, we only have a single MID server, but we have fallback MID servers in our production environment. We encounter the same error in both environments.

We have tried restarting the MID server and this didn't have any effect.

Other workflows are working utilizing the MID service account.

-

Daniel

[Daniel Evans]

Just in case somebody finds this in the future, it turns out that this was an issue specifically related to OAuth 2.0 credentials in MADRID. This is touted to be fixed in Orlando/New York. I spent a lot of time troubleshooting other things, and had even tried disabling all other 'windows' type credentials, but I never thought to check OAuth 2.0 credentials.

I knew that our MID server was validated successfully, but the error message below was still something we were seeing in the logs. I Googled this error from our MID server agent:

com.snc.automation_common.integration.exceptions.AutomationIOException: Unable to retrieve data from instance. This MID may not be validated.

This brought me to https://hi.service-now.com/kb_view.do?sysparm_article=KB0748823, which ultimately resolved our issues.

Description

If there is an OAuth 2.0 type Credential [oauth_2_0_credentials] added to the list of Credentials in the instance, the instance will no longer return any credentials to the MID server when the MID Server tries to re-load the credential list. The MID Server will no longer be able to run any probes that requires credentials. This includes all Discovery, Orchestration, Event Management Connector probes, and others.

The MID Server agent logs will show a "SEVERE *** ERROR *** An error occurred while decrypting credentials from instance" when running each affected probe.

Steps to Reproduce

1. Create a new OAuth 2.0 credential using an OOB OAuth Entity Profile (e.g. Google default_profile) on an instance with at least one MID running.
2. Verify that an ECC queue output record has been processed by the MID(s) with the topic credentials_reload. The agent logs will also confirm this ran.
3. Verify that in the agent log file for the MID(s) there is an exception that looks like the following. This is the exception thrown when no key element is found in the response (i.e., the response is empty)

SEVERE *** ERROR *** An error occurred while decrypting credentials from instance
com.snc.automation_common.integration.exceptions.AutomationIOException: Unable to retrieve data from instance. This MID may not be validated.   <<<=== Red Herring - The MID Server is Validated.
	at com.glide.util.MIDServerInfoPayloadDecrypter.decryptPayload(MIDServerInfoPayloadDecrypter.java:25)
	at com.service_now.mid.creds.provider.standard.StandardCredentialsProvider.loadCredentials(StandardCredentialsProvider.java:289)
	at com.service_now.mid.creds.provider.standard.StandardCredentialsProvider.load(StandardCredentialsProvider.java:256)
	at com.service_now.mid.creds.provider.standard.StandardCredentialsProvider.init(StandardCredentialsProvider.java:275)
	at com.service_now.mid.creds.provider.MIDCredentialsConfigProvider.getCredentialsProvider(MIDCredentialsConfigProvider.java:58)
	at com.snc.commons.credentials.CredentialsProviderFactory.getCredentialsProvider(CredentialsProviderFactory.java:30)
	...

Lines below this in the stack trace will be specific to the probe running.

Workaround

This problem has been fixed. There is no workaround available. If you are able to upgrade, review the Fixed In section to determine the latest version with a permanent fix your instance can be upgraded to.

The workaround is to ensure the affected MID Servers do not have any OAuth Credentials set up for them:

• Deactivating all OAuth 2.0 Credentials in the Credentials table will workaround this issue, however they will then not be available to the features that require them.
• If OAuth Credentials are still needed to be Active for integrations or other features, set the OAuth 2.0 Credential record field "Applies To:" to "Specific MID Servers" and leave the field "MID servers" as empty. This will make the credential inaccessible from any mid server and only available for instance. Note that, even for OAuth 2.0 credentials to be used on instance, the flow should be executed as system user and not session user.
• Please make sure to restart the mid service manually from the mid host if you see the same issue even after de-activating the OAuth 2.0 credentials.

NOTE: The same symptom/error could also be due to PRB1305469 Excluding table-per-class (TPC) extended tables from a clone can cause orphaned Discovery...

If you also have thousands of source=credentials_reload jobs backed-up in the ECC Queue, then you probably also experiencing this, which requires additional steps to resolve:
PRB1411442 / KB0829702 OAuth 2.0 credential that is configured with extremely short TTL (<1 minute) ...

Thanks,

Daniel Evans