KMF Errors for SSO and API calls: :Key id kmf_file_keystore_imk NOT found in repo for registry KMFMasterKeyRegistry Failure getting key from the KMF."

BabyYoda · ‎06-24-2022

A new instance was setup not long ago. Let's call it instance B. Then another instance--call it instance A--was cloned down to instance B. One more note to add is ServiceNow did push down the multiSSOv2 update. It's likely not a factor but it's worth mentioning. From here on out, instance B and instance will refer to the same box.

I can't say for sure when these KMF errors started. Too much has time has passed such that I can't scan logs going back to the creation of the instance. It probably started after the clone operation but I can't be certain on that.

Here's the KMF error stacktrace (I removed the Key Id, in case it is sensitive data):

---

Found keyId : [redacted] in store : com.glide.kmf.KMFDBInstanceKeyStore@1e47857
Found wrapped key in repo. Attempting to unwrap.
Key id kmf_file_keystore_imk NOT found in repo for registry KMFMasterKeyRegistry
Failure getting key from the KMF. Key ID: [redacted], Error: com.glide.kmf.commons.CryptoOperationException: Error decrypting key bytes, key not present.: com.glide.kmf.commons.CryptoOperationException: Error decrypting key bytes, key not present.: com.glide.kmf.KMFFormattedKeyUnwrappingOperation.decryptKeyBytes(KMFFormattedKeyUnwrappingOperation.java:56)
com.glide.kmf.LocalRegistryWrapper.unwrapKey(LocalRegistryWrapper.java:41)
com.glide.kmf.AKMFKeyRegistry.getKeyById(AKMFKeyRegistry.java:100)
com.glide.kmf.KMFInstanceKeyRegistry.getKeyByType(KMFInstanceKeyRegistry.java:254)
com.glide.kmf.KMFInstanceKeyRegistry.getKeyByType(KMFInstanceKeyRegistry.java:232)
com.glide.kmf.LocalRegistryWrapper.computeHMAC(LocalRegistryWrapper.java:47)
com.glide.kmf.AKMFKeyRegistry.validateHmac(AKMFKeyRegistry.java:333)
com.glide.kmf.KMFModuleKeyRegistry.getKeyById(KMFModuleKeyRegistry.java:158)
com.glide.kmf.KMFModuleKeyRegistry.getKeyByType(KMFModuleKeyRegistry.java:85)
com.glide.kmf.KMFOutputBaseBuilder.getKMFOutputObject(KMFOutputBaseBuilder.java:103)
com.glide.kmf.KMFOutputStringBuilder.apply(KMFOutputStringBuilder.java:22)
com.glide.kmf.KMFGlideEncrypter.decrypt(KMFGlideEncrypter.java:179)
com.glide.util.Encrypter.decrypt(Encrypter.java:105)
com.glide.util.GlideEncrypter.decrypt(GlideEncrypter.java:68)

...

HMAC validation failed for: [redacted] : com.glide.kmf.AKMFKeyRegistry$KeyRegistryException: Error while computing HMAC at com.glide.kmf.LocalRegistryWrapper@920d8d. Could not retrieve hmac key from com.glide.kmf.KMFInstanceKeyRegistry: com.glide.kmf.AKMFKeyRegistry$KeyRegistryException: Error while computing HMAC at com.glide.kmf.LocalRegistryWrapper@920d8d. Could not retrieve hmac key from com.glide.kmf.KMFInstanceKeyRegistry: com.glide.kmf.AKMFKeyRegistry.validateHmac(AKMFKeyRegistry.java:335)
com.glide.kmf.KMFModuleKeyRegistry.getKeyById(KMFModuleKeyRegistry.java:158)
com.glide.kmf.KMFModuleKeyRegistry.getKeyByType(KMFModuleKeyRegistry.java:85)
com.glide.kmf.KMFOutputBaseBuilder.getKMFOutputObject(KMFOutputBaseBuilder.java:103)
com.glide.kmf.KMFOutputStringBuilder.apply(KMFOutputStringBuilder.java:22)
com.glide.kmf.KMFGlideEncrypter.decrypt(KMFGlideEncrypter.java:179)
com.glide.util.Encrypter.decrypt(Encrypter.java:105)
com.glide.util.GlideEncrypter.decrypt(GlideEncrypter.java:68)

...

string may not be encrypted : Could not fetch key information for encryption

---

The KMF errors occur are related to the GlideEncrypter API. So far, SSO and REST API calls trigger the errors. Direct login works OK. Not using SAML at the moment for SSO. Using Digest Token SSO. Basically, I think anything that uses KMF authentication is potentially affected. GlideEncrypter API for sure. I haven't tried anything else besides SSO and REST API, both of which are not working due to these KMF errors.

For SSO, I was able to narrow it down to the getSecretKey method in the MultiSSO_Abstract_Core Class:

getSecretKey : function (data){
var cred = new String(data);
var e = new GlideEncrypter();

this.debug("MultiSSO_Abstract_Core > getSecretKey > data: " + data);

var jsCred = cred + '';
var decryptedData = e.decrypt(jsCred);
return decryptedData ;
},

I grabbed the code and executed the script in a sandbox:

// SSO -- GlideEncrypter API Decrypt method
// Comes from MultiSSO_Abstract_Core.getSecretKey method
var data = "{Secret Key data}";
var cred = new String(data);
var e = new GlideEncrypter();

var jsCred = cred + '';
var decryptedData = e.decrypt(jsCred);

The above script produces the same KMF error I pasted earlier in this post. Similarly, a REST API script produces the same KMF error; no need to paste that in.

I opened a case with ServiceNow Support. They said that KMF plugin was installed multiple times. At my request, they reinstalled the KMF plugin. However, it didn't fix the errors. I didn't double check their work but I did verify that the KMF plugin was reinstalled.

There is a KB Article on duplicate Keys, as a result of cloning without preserving KMF related data, and I believe the resolution is ServiceNow Support reinstalls the KMF plugin. I was expecting that to solve this issue but it didn't.

Has anyone experienced anything similar and what was your resolution?

is this so-called KMFMasterKeyRegistry registry accessible on the instance or is it closed-source code? Is it on the instance or is it onn the Data Center instance?

If there was a way to tinker with the registry and repair the repo, that could resolve this problem. There must be something that can be done. I don't know what ServiceNow Support has done in past cases where KMF reinstall didn't fix the problem. Maybe it's a logic hardware with their DB or the DB itself. I feel like creating a new node and performing a clone down again, ensuring KMF data is excluded this time, might be the step I have to take but I'm hoping someone here or even at ServiceNow has other ideas on what can be tried next before having to scorch Earth and start all over again.

Thank you for your help in advance.

Brian Bakker · ‎06-27-2022

Hello BabyYoda,

Please review KB1115784: Unable to decrypt Key Store password after upgrading to San Diego. Otherwise, let me know your case ID, so that I can review it.

Best regards,

Brian

BabyYoda · ‎06-28-2022

Hey Brian,

Thank you for responding to this thread! I appreciate it! And I appreciate the KB Article link (more on that in a sec).

The instance release in question is Rome and not San Diego. I'm not getting an Access Denied error to the crypto module in the stacktrace, however. ServiceNow Support came back with the KMF plugin was not properly activated and, thus, not properly installed, even though it's saying it's installed. There were/are multiple KMF plugin installs, as well. They tried reinstalling the KMF plugin and that didn't fix it. That's when they seemingly uncovered that the KMF plugin was not properly activated when the instance came online So, they're going to try activating the plugin first and then installing it once more. They had me run the following one-line of code in a sandbox to showcase that the KMF plugin was partially installed, as the result came back undefined with errors for me:

gs.info((new sn_kmf_ns.KMFPluginHealth()).getHealth());

I'm not 100% sure they're right on this. I have a personal Rome instance where SSO and API calls are working fine. Tested in a sandbox script and no issue with KMF. I paste it in the same one-line of code above in a personal instance sandbox and I get the same error. There's nothing in the sn_kmf_ns namespace. It comes back as an object. But everything else inside of it returns 'undefined'. Yet, SSO and API auth works fine. Either there's a separate issue going on with Rome instances or that's not the right test. Maybe that line of code only works in San Diego? I don't have a personal San Diego instance to test that theory out, though.

Going back to your KB Article: As I said, I'm on Rome. However, comparing the instance in question and a personal Rome instance finds that the instance in question contains 5 rejection policies and the personal instance has 0 rejection policies (all GlideEncrypter API related) How are these rejection policies created? Are they created after I try SSO/API auth or are they created some other way such as install of the KMF plugin? I haven't tried deleting the rejection policies but I'm wondering if that's the issue. And if it is, why and how are they created?

Let's see what Support has to say. If reactivation and reinstall doesn't fix it, I can privately message you the Case Number, if that's even available to do so.

Thanks for your help!

Kind regards,

BabyYoda