Setting Up Mutual TLS (mTLS) Authentication Between a MID Server and Your Instance

Vishnu-K
Kilo Sage

Article

Why I wrote this

I recently configured mutual TLS (mTLS) authentication end-to-end between a ServiceNow MID Server and its instance  replacing username/password login with certificate-based authentication. This post is the complete, verified walkthrough of that setup: every command, every setting, and the two mistakes that cost real troubleshooting time before I tracked them down.

If you're planning a new mTLS install, or migrating an existing MID Server off basic auth, this should save you the debugging cycle I went through.

 

What mTLS actually buys you

In normal HTTPS, only the server proves who it is. The MID Server still authenticates the old-fashioned way — a username and password sitting in config, transmitted on every request.

Mutual TLS flips this around. During the TLS handshake itself, the instance asks the MID Server for a certificate. The MID Server presents one, the instance verifies it was signed by a CA it trusts, maps the certificate to a user, and the connection is authenticated — before a single HTTP request is even sent. No password ever crosses the wire, and there's nothing sitting in a config file for someone to steal.

The trade-off is operational complexity: you're now managing a certificate lifecycle instead of a password reset. This article is about making that complexity predictable.

 

Environment

Two Windows Server 2022 hosts, same AD domain:

  • Server 1 — Domain Controller + Certificate Authority
  • Server 2 — MID Server host

Standard instance on ADCv2 infrastructure with the com.glide.auth.mutual plugin active.

 

Step 1 - Confirm your instance can actually support this

Before touching any certificates, verify the instance side is ready:

curl -I https://<your-instance>.service-now.com/

Look for Server: snow_adc in the response headers — this confirms ADCv2. Then check:

curl -I https://<your-instance>.service-now.com/adcv2/supports_tls

A 200 OK confirms TLS support at the load balancer. Also confirm the plugin is active:

VishnuK_0-1786713308266.png

 

If either check fails, stop here and engage ServiceNow Support, mutual auth for MID connections is enabled at the infrastructure level and isn't something you can toggle from the instance UI.

 

Step 2 - Build the internal CA

For an internal deployment, an Enterprise Root CA via AD Certificate Services is the simplest path. On the Domain Controller:

Install-AdcsCertificationAuthority `
    -CAType EnterpriseRootCA `
    -CryptoProviderName "RSA#Microsoft Software Key Storage Provider" `
    -KeyLength 2048 `
    -HashAlgorithmName SHA256 `
    -CACommonName "Kaptius-CA" `
    -Force

Note: the GUI post-install wizard can hit an unrelated IIS redirection.config error on the Credentials step in some Server 2022 builds. The PowerShell cmdlet above sidesteps it entirely.

VishnuK_1-1786713396938.png

 

Step 3 - The certificate template (this is where most setups go wrong)

Duplicate the built-in Workstation Authentication template — never edit the built-in ones directly. Name it something like MIDServerClientAuth.

 

Two settings actually matter:

Extensions tab confirm Client Authentication is present under Application Policies.

VishnuK_3-1786713862478.png

Subject Name tab , change this from "Build from this Active Directory information" to "Supply in the request."

 

VishnuK_4-1786713893370.png

This second one is the mistake that catches almost everyone. If you leave it on "Build from AD," the CA silently overwrites whatever Subject you specify in your CSR with the requesting computer's AD identity. You'll end up with a certificate whose Subject is your server's hostname instead of something like CN=MID Server mTLS Client — and since ServiceNow maps the certificate to a user based on Subject, your user mapping breaks in a way that's very non-obvious to debug from the symptoms alone.

Publish the template (New → Certificate Template to Issue) and confirm it's listed:

 

VishnuK_5-1786713933759.png

 

Step 4 - Generate the certificate on the MID Server host

From C:\mtls\, create an INF file describing the request:

[Version]
Signature="$Windows NT$"

[NewRequest]
Subject = "CN=MID Server mTLS Client, O=YourOrg, OU=ITOM"
KeySpec = 1
KeyLength = 2048
Exportable = TRUE
MachineKeySet = TRUE
ProviderName = "Microsoft RSA SChannel Cryptographic Provider"
ProviderType = 12
RequestType = PKCS10
KeyUsage = 0xa0

[EnhancedKeyUsageExtension]
OID = 1.3.6.1.5.5.7.3.2  ; Client Authentication

Exportable = TRUE matters — without it you can't pull the private key back out later to build the PEM bundle.

Generate, submit, and accept:

certreq -new midserver-mtls.inf midserver-mtls.csr
certreq -submit -attrib "CertificateTemplate:MIDServerClientAuth" -config "<CA-host>\Kaptius-CA" midserver-mtls.csr midserver-mtls.crt
certreq -accept midserver-mtls.crt

VishnuK_6-1786714980635.png

 

Export as PFX:

certutil -exportpfx -p "Temp123!" my "<thumbprint>" midserver-mtls.pfx

VishnuK_7-1786715067656.png

 

Step 5 - Convert to PEM and assemble the bundle

The MID Server's certificate management tool only accepts PEM, not PFX. Install OpenSSL for Windows, then extract:

openssl pkcs12 -in midserver-mtls.pfx -nocerts -nodes -out midserver-mtls.key
openssl pkcs12 -in midserver-mtls.pfx -clcerts -nokeys -out midserver-mtls-leaf.crt


VishnuK_8-1786715189827.png

VishnuK_9-1786715227663.png

 

Export your root CA cert (Base-64 X.509) from the CA console, copy it to the MID host, then assemble everything into a single bundle — leaf, then root, then key, in that order. PowerShell handles this more cleanly than copy /a, which tends to leave OpenSSL's commentary lines in the file and breaks strict PEM parsers:

 

$leaf = Get-Content midserver-mtls-leaf.crt | Where-Object { $_ -notmatch "Bag Attributes|localKeyID|friendlyName|Microsoft|subject=|issuer=|Key Attributes|X509v3" }
$root = Get-Content kaptius-root.pem
$key  = Get-Content midserver-mtls.key | Where-Object { $_ -notmatch "Bag Attributes|localKeyID|friendlyName|Microsoft|Key Attributes|X509v3" }
$leaf + $root + $key | Set-Content -Encoding ascii midserver-bundle.pem

VishnuK_10-1786715258749.png

VishnuK_11-1786715268680.png

VishnuK_12-1786715278448.png

 

Step 6 - Instance-side configuration

Two records need creating:

CA certificate at sys_ca_certificate.list — Type = CA Cert, Format = PEM, attach your root cert, confirm Publish Status shows Active.

VishnuK_0-1786715458243.png

 

Client certificate at sys_user_certificate.list — attach the leaf cert, link it to your MID user. If the upload rejects a .crt extension, rename it to .pem — same content, different filter behavior.

VishnuK_1-1786715505489.png

 

Confirm the linked user has the mid_server role:

VishnuK_2-1786715563901.png

 

Step 7 - Import into the MID keystore (the second big pitfall)

manage-certificates.bat -a defaultsecuritykeypairhandle "C:\mtls\midserver-bundle.pem"

The alias has to be exactly defaultsecuritykeypairhandle. Not the certificate's CN, not a descriptive name this exact string, because it's what the MID Server's TLS code specifically looks for at handshake time.

Here's why this is worse than a normal typo: if you use any other alias, manage-certificates.bat reports success. The keystore genuinely does get updated. Everything looks fine right up until the MID Server tries to actually use it at which point the JVM can't find anything under the expected alias, silently falls back to a DummyX509KeyManager, and sends an empty certificate list during the handshake. From the instance side this looks exactly like a plain authentication failure — a normal 401, no TLS-layer error at all — which sends you straight down the wrong debugging path (checking passwords, roles, CA trust) instead of the actual cause.

 

Verify:

manage-certificates.bat -l
manage-certificates.bat -g defaultsecuritykeypairhandle

VishnuK_0-1786942824002.png

 

VishnuK_1-1786942925594.png

 

Step 8 - Confirm it's actually working

Start the MID Server service and check wrapper.log. This is the log line that tells you everything:

Before the fix (wrong alias):

X509KeyManager class: sun.security.ssl.DummyX509KeyManager
No X.509 cert selected for [EC, EdDSA, RSASSA-PSS, RSA, DSA]
"Certificates": <empty list>

VishnuK_2-1786942953616.png

 

After the fix (correct alias):

X509KeyManager class: sun.security.ssl.SunX509KeyManagerImpl
Produced client Certificate handshake message (
  "subject" : "CN=MID Server mTLS Client, OU=ITOM, O=YourOrg"
Produced CertificateVerify handshake message

VishnuK_3-1786942977825.png

 

That CertificateVerify line is the proof  it's a signature produced with the private key, confirming the MID Server actually possesses it, not just a copy of the public cert.

Finally, check the instance:

VishnuK_4-1786944147480.png

 

One thing people miss after it's working

If your MID Server came up under mTLS from the very first connection (rather than migrating from an existing basic-auth MID), it will not get its capabilities auto-assigned. The Autobind job that normally sets Discovery/Orchestration/etc. runs on a code path tied to the first basic-auth handshake, which never happens here. You have to add capabilities to the MID Server record manually.

 

Quick troubleshooting reference

Symptom Likely cause

Clean 401 User Not Authenticated, no SSL exception anywhere in the logWrong keystore alias - see Step 7
SSLHandshakeException: bad_certificate / unknown_caCA record on instance isn't Active, or wrong CA entirely
InvalidKeyException: Missing key encoding on importTried to import PFX directly, or key is PKCS#1 instead of PKCS#8
Certificate Subject shows your hostname, not what you specifiedSubject Name tab left on "Build from AD" - see Step 3
MID shows Up but Discovery/other jobs failCapabilities weren't auto-assigned - add manually

 

Reference

Full command list and file paths used throughout this setup are summarized in the appendix below for quick copy-paste reference.

Happy to answer questions in the comments especially if you hit a variant of the alias issue that manifested differently than what I've described here.

 

Appendix Command Quick Reference

# Generate CSR
certreq -new midserver-mtls.inf midserver-mtls.csr

# Submit CSR to internal CA
certreq -submit -attrib "CertificateTemplate:MIDServerClientAuth" -config "<CA-host>\<CA-name>" midserver-mtls.csr midserver-mtls.crt

# Accept issued certificate
certreq -accept midserver-mtls.crt

# Export as PFX
certutil -exportpfx -p "<password>" my "<thumbprint>" midserver-mtls.pfx

# Extract private key and leaf cert (OpenSSL)
openssl pkcs12 -in midserver-mtls.pfx -nocerts -nodes -out midserver-mtls.key
openssl pkcs12 -in midserver-mtls.pfx -clcerts -nokeys -out midserver-mtls-leaf.crt

# Import into MID keystore — alias name is mandatory and exact
manage-certificates.bat -a defaultsecuritykeypairhandle "C:\mtls\midserver-bundle.pem"

# Verify
manage-certificates.bat -l
manage-certificates.bat -g defaultsecuritykeypairhandle
 

 

1 REPLY 1

Pratiksha
Mega Sage

Helpful