MFA broken after clone down

[Tim Grindlay]

Our Development and UAT environments require local login with MFA. We have a clone preserver setup for the user_multifactor_auth table and during our last upgrade to San Diego we cloned our DEV and UAT environments.

Post clone, we could log into the instances with our pre-clone DEV and UAT MFA codes from our authenticator apps fine. For an unrelated reason we needed to clone our UAT environment again (several update sets committed in error) and after this second clone (a few days after the initial clone) no-one was able to login using their MFA codes and we were effectively locked out of the instance.  

Luckily we had a clean up script that re-enables our email, and sends them to a shared email account so we were able to get a temporary MFA code (For accounts that had an email, because you don't get that option if your account doesn't have one!)

While troubleshooting we discovered that clicking the 'Receive a code via email' link, brings the old codes back to life. You don't have to use the code that was sent to the email. I also have a suspicion that an account that is locked out and then unlocked also brings them back, but I haven't thoroughly tested this as once the codes are working I have to find another user that had pre-clone MFA codes setup to test with.

I raised a case with the now Support but got nowhere. They suggested re-cloning, resetting the 'Enable multi-factor authentication' flag on the user profile or using the 'Receive a code via email' link, but these are all workarounds. Posting to see if anyone else has come across this.

[Joe Wong]

Don't have an answer for you, but just wanted to add my experience here.  We had the same issue just last week when we cloned our Prod over to our pre-prod instance.  Everyone with MFA setup login failed.  Lucky, I forgot to turn my MFA back on after initiating the clone, so I was able to have admin access to our pre-prod instance and reset everyone's MFA status.

What I wanted to test but only thought of after the fact was, would I have been able to log into the pre-prod instance with my prod MFA credential?  If this is truly a cloning issue and for whatever reason, SN cloned the prod MFA key over the our pre-prod instance.  If anyone is in the same situation, maybe they are try using the MFA from the instance that was being clone and see if it works.

[Tim Grindlay]

Thanks Joe, always appreciate some insight! 

We've added preserve and exclude entries to the clone profile for the user_multifactor_auth table in order to retain our codes.

That being said I've just checked our configuration and there are two entries for these tables in the clone preserver and table excludes - I wonder if that has something to do with it. Maybe ServiceNow added these after we added ours? Do you have entries for this table in the clone profile you used?

As a side note, I'm guessing you turned off your MFA when initiating the clone because your authentication was failing when you have to supply admin credentials for the target instance? If you add your MFA code to the end of the password, you don't need to turn off your MFA.

[Charlotte Pakes]

Hi Tim,

We're facing similar issues with cloning recently and also have a case open with ServiceNow support. Clones and maintaining MFA for our local admins had been working successfully for a long time - since we first implemented it a few years ago and many clones completed since - but in the last few months we keep hitting this issue. Just this morning it has happened again. We experimented by having 1 admin delete their MFA context in sub-Prod pre-clone and we left mine in place. He could get in by resetting up MFA, but I couldn't, my authenticator codes were rejected.

Your comment about an additional preserver having been added - I've looked and we have this too though. I do wonder if this is causing an issue. 

I'll update if we get to the bottom of it.

Thanks,

Charlotte

[Tim Grindlay]

Thanks for the post Charlotte! Do let us know how you get on 🙂