The New Way for MID Servers to Authenticate: Private-Key JWTs

[ChristianW29113]

The New Way for MID Servers to Authenticate: Private Key JWTs

As ServiceNow, and the industry at large, moves toward a future without Basic Authentication, we're changing how MID Servers connect to your ServiceNow instance. This is rolling out now in Zurich Patch 11 and Australia Patch 4 for Commercial Cloud instances. Testing for Regulated Markets is in progress.

The Problem with Basic Authentication

In practice, maintaining best practices with Basic Auth is difficult. ServiceNow recommends using one ServiceNow user per MID and regular password rotation, but this is a painful process to maintain especially for those of you with 100s of MID Servers. And since this password is just plain text, it is easy for the secret to be left copied in a notepad or email somewhere. This leaves the environment in a situation where one (easily) leaked credential has a large blast radius.

What's Changing: ServiceNow Certificates

We're introducing authentication using JSON Web Tokens (JWT), backed by private keys managed by the Instance PKI (IPKI). Here's generally how it works:

• On first startup, your MID Server authenticates with a one-time registration key.
• The MID Server generates a private key and certificate signing request (CSR), then requests a certificate.
• Your Instance PKI issues a certificate bound to that MID Server.
• From that point on, the MID authenticates using a short lived JWT, not a shared password.
• Certificates rotate every 45 days so there's no manual upkeep once a MID is set up.

Modified Onboarding Experience

This rolls out alongside a simpler command line installer. With these changes, validation happens automatically and there is no need to create a new ServiceNow user for every MID Server or configure config.xml manually. The only thing you need to bring is the service account/user details to run the MID Server process itself. One thing to keep in mind for now: you'll still need to assign applications and capabilities to the MID Server yourself after setup. A patch to streamline this is coming soon.

This new flow is available through the ITOM Infra Services Workspace. If you need the previous onboarding page, it's still there for now at mid_server_download_ui.do.

Impact of Change

• New MIDs using JWTs do not need a unique user or manual validation.
• Every MID Server gets its own verifiable machine identity.
• Credentials expire and rotate automatically.
• Private Key is much less likely to leak in plain text compared to user password.

What You Need to Do

When you're ready, you can migrate existing MID Servers from Basic Auth to JWT using the Auth-Type migration tab in the ITOM Infra Services Workspace. It's a one-click migration through the instance itself. I would recommend testing it in sub-prod on a single MID Server first before rolling it out further. 

Longer term, Basic Auth will be disabled by default on ServiceNow instances. That timeline hasn't been determined yet, but we will share it well in advance. I would recommend bringing new MID Servers up with JWT instead of Basic Auth (or migrating them soon after they come up, if that is easier with your MID Server creation processes). 

If you have questions, please reach out to me directly at j.christian.wu <at> servicenow.com.

[Thomas Bohan]

How do non-ITOM customers available of this please

I am an ITSM Pro customer with MID Servers.
I don't even know what ITOM Infra Services Workspace is.

[ChristianW29113]

Hey Thomas, non-ITOM customers will be able to take advantage of this. ITOM Infra Services Workspace is a unlicensed store app is installed out of box for all customers. After upgrading to Zurich Patch 11 or Australia Patch 4, log in with the `agent_admin` role and it will be visible under workspaces tab.

[JaineelPetiwale]

@ChristianW29113 @ Can we use this with AD Service Accounts ? Whose password is authenticated using realtime LDAP Authentication

[ChristianW29113]

Hi Jaineel, this JWT is used for the MID Server -> Instance authentication. This replaces the login for your ServiceNow user account (on sys_user), separate from your Windows service account or AD/LDAP authenticated accounts. I might have misunderstood what you are asking, can clarify further if needed.

[WaiYin Leong]

Hi @ChristianW29113 

I would like to understand how this Auth-Type Migration work. Is there any documentation inside ServiceNow Doc?

Reason being we have a MID Server that is reside in a very restrictive environment and security application is blocking the migration, and now the mid server is stuck at Certificate Issued.

Thank in advance.

WaiYin

[Jan Sierens]

Thanks for the article, Christian.

A few questions on how this relates to mTLS, which is already supported for MID-to-instance authentication but isn’t mentioned here:

1. Will mTLS remain a fully supported alternative for MID Server authentication once Basic Auth is disabled by default, or is JWT intended to replace it over time?
2. Does JWT registration and certificate rotation work through a proxy that performs TLS inspection? If not, which endpoints need to be excluded from inspection?
3. Can a MID Server switch from JWT back to mTLS (or the other way around) without reinstalling it?

[DanT44586748758]

Hi @ChristianW29113, 

Do you have a guide for upgrading a Mid to use JWT Auth? I am testing this out in one of my lower instances but getting an error "Exception during certificate signing. UI". Trying to find guide of things I need to check for this migration to work but everything is leading me back to this page.

[OMT]

Hi,

In my organisation, we have had a problem for a while that, after a security patch, we have to re-validate our Mid servers, otherwise they do not work and we cannot see this in our Mid server overview because everything is "green".

We opened a case and the engineer working on it provided us with this link to a community article.

He said that if we change our authentication method from Basic Auth to JWT, this problem will no longer occur.

I have a couple of questions that I couldn't find the answers to in the article.

Firstly, is there any official documentation, or is there none?

Secondly, if there is no official documentation, how can we ensure that we are doing the right thing?

Thirdly, what should we do if we encounter error messages? Can we revert to Basic Auth, and if not, what are our options? Would reinstalling the Mid Servers with Basic Auth help?

Is this really one click action oder do we have to do something on our side on our mid server? 

I would really appreciate it if you could help me here.

Thanks in advance,

Tahir

[lmundere]

Hi, does it mean that we can disable/delete the MID server accounts after migration

[ChristianW29113]

@WaiYin Leong : Docs on how this works under the hood are being worked on right now, planned to come out in November. If your security team has specific questions, please email me and I can answer them.

@Jan Sierens : mTLS will remain fully supported, JWT is not intended as a replacement. JWT cert registration and rotation will work through a proxy that inspects. I haven't tested switching from JWT to mTLS without reinstalling it.

@DanT44586748758 : Please create a case for this, we will compile common failure scenarios into official documentation. That error message is probably cut off (maybe by the UI): there should be a UID as well as an "Exception:" later in the error. That exception will help support troubleshoot and our future docs.

@OMT : There is currently limited official documentation, but more will be out in November. We are working on that right now. Please test migration on subproduction before trying on production. If you encounter error messages, there is likely a problem on your instance - we will compile a list of common failure scenarios and give troubleshooting tips but based on migration data so far these errors are fairly rare. This migration will only switch you to JWT if everything is successful, if something goes wrong the MID will continue on basic auth. The MID Servers can always be reverted to Basic Auth by modifying config.xml. (setting a password again and disabling IPKI). This is one-click (I guess two technically), there is nothing you need to do on your MID Server except in unexpected circumstances (for example if its pinned to an out of date version).

@lmundere : After migration, your MIDs will continue to use the same mid server accounts they used to use, but instead of using a password they will use a JWT. You cannot disable or delete these accounts after migration. If you want to delete these accounts, it will be a more hands on process - you will have to switch each MID manually to use a shared account. 

For more elaboration, feel free to reach out to me on email as well.

[Savita Govinda]

@ChristianW29113  We observed that a user account "mid_server_default" was created following the Zurich Patch 11 upgrade as part of the ITOM Infra Services Workspace application.

We are unable to disable this user or remove the associated role because there are two Business Rules protecting the record. Any attempt to modify the user results in the following error:

"The MID IPKI user record cannot be modified."

Additionally, we have enabled EMSelfMonitoring for our MID Servers, which has generated an incident with the following error:

"User mid_server_default with mid_server role not associated with a MID Server. No login attempts within reporting period."

We would like to understand the purpose of creating this mid_server_default account. Typically, organizations create and manage their own MID Server service accounts, and all MID Servers are configured to use those accounts.

Even after migrating to JWT authentication, the service account configured on our MID Servers remains unchanged. Therefore, we are trying to understand the intended use of this newly created account , can we disbale the BR and the newly created mid service account so we dont gets incidents created from EMSelfMonitoring?

[Jason D3]

Just curious, we use scripts to build multiple servers at once. How would this work with the new standard?

[Shubham47]

Can someone help me with document/video links to set it this up for the first time?

[RGreene]

How does the JWT change how we recover a Mid-Server after an Upgrade goes horribly wrong and roll back fails?
With over 200 Mid-Servers between the environments we experience 1-5% failure rate on every Upgrade, Patch or Hotfix.