Post-Clone Auth Failure: getDERCert null and StorageEncrypter no Key-Provider

Selva Arun · 20m ago

After DEV Clone — All API Authentication Fails (OAuth + Basic Auth): getDERCert bytes null / StorageEncrypter no Key-Provider

Environment:

Trigger: PROD → DEV instance clone
Affected: OAuth client_credentials, Basic Auth, service account API access

Symptoms

After cloning PROD down to DEV and re-committing Update Sets, all external API authentication fails:

OAuth client_credentials token request to oauth_token.do returns:

{"error_description": "access_denied", "error": "server_error"}

Basic Auth to /api/now/table/... returns:

{"error": {"message": "User is not authenticated", "detail": "Required to provide Auth information"}, "status": "failure"}

Response header shows X-Is-Logged-In: false
No OAuth errors appear in syslog — the failure happens before application logging

The confusing part: everything in the instance looks correct. The database view exists, ACLs are in place, the service account has the right roles, and a live impersonation test inside Scripts - Background returns data successfully. The configuration is not the problem.

Root Cause

The system log reveals the real issue — look for these lines on every failed auth attempt:

getDERCert: bytes is null!
Unable to load certificate code-signing-key_<hash>
Unable to load certificate code-signing-key_<hash>
StorageEncrypter: no registered Key-Provider for hashKey: a8o

ServiceNow's Key Management Framework (KMF) uses code-signing certificates and encryption key providers to:

Sign and verify password hashes (Basic Auth)
Encrypt and decrypt OAuth client secrets (OAuth)

When a clone is performed, the key material is intentionally not transferred (it is instance-specific for security reasons). The cloned DEV instance retains references to key IDs from PROD, but the actual cryptographic material does not exist in DEV. As a result, every credential validation attempt fails silently — ServiceNow cannot sign or verify anything.

This is why:

Regenerating the OAuth secret doesn't help (the new secret is encrypted with a broken key provider)
Setting a new password on the service account doesn't help (the hash can't be signed)
A brand new user with minimal roles fails identically (it's not user-specific)
SN calling itself via sn_ws.RESTMessageV2 Basic Auth also returns 401 (rules out network/WAF)

How to Confirm

Run this in Scripts - Background after a failed auth attempt and look for the certificate errors in the output:

var gr = new GlideRecord('sys_log');
gr.addQuery('message', 'CONTAINS', 'getDERCert');
gr.orderByDesc('sys_created_on');
gr.setLimit(5);
gr.query();
while (gr.next()) {
    gs.print(gr.getValue('sys_created_on') + ' | ' + gr.getValue('message'));
}

If you see getDERCert: bytes is null entries, the KMF infrastructure is broken on your instance.

Resolution

This cannot be fixed at the application or admin level. It requires the ServiceNow platform team or a Now Support (HI) ticket:

Navigate to Key Management Administration in the SN navigator
Requires KMF Admin role — standard admin cannot access this
Platform team needs to reinstall the KMF plugin on the DEV instance
If any nodes are in a retired state, restore them first
After the fix: regenerate the OAuth client secret and reset service account passwords (old encrypted values are tied to the broken keys and must be refreshed)

Reference: KMF cryptographic module key missing — ServiceNow Community