In the following chapters want to share the experience we made during a Cloud Encryption setup project that we had together with a sensitive banking customer in Switzerland. We encountered several issues during that process that will be pointed out in the following chapters and how you might be able to prevent for in future projects.
Context:
Early this year (2022), a ServiceNow customer working in the banking industry submitted an inquiry to the Customer Security & Trust (CST) team. The customer had the requirement for the encryption of all customer production and sub-production instances with a proper data-at-rest encryption mechanism. This was necessary due to the bank would like to extend the usage of the Now Platform to other business areas within the company as well as due to changes in bank regulatory requirements and updated internal policies affecting the processing of sensitive data.
In an initial session, we had listened to the customer use cases and criticality of data they wanted to process on the platform in the future. Related to that information we gave the customer an overview of the encryption functionality ServiceNow provides. In the end, the customer decided to move to the Platform Encryption bundle, providing two data-at-rest approaches on a volume level (Cloud Encryption) as well as on field level (Column Level Encryption Enterprise). Bring-Your-Own-Key was one of their major requirements, which both solutions offer. To set the foundation they wanted to start with implementing Cloud Encryption of all of their instances.
How does Cloud Encryption work in ServiceNow’s backup procedure
One major topic in the conversation we had was about the possible impact of Cloud Encryption to their backups.
Instances with Cloud Encryption follow ServiceNow’s standard backup process. Backups are encrypted with AES-256 using randomly generated encryption keys for every backup. Data is encrypted and not the disc encryption for the backups. A symmetric key is generated and then wrapped with a public key that all servers have access to. The wrapped symmetric key is archived with the backup. The private key is stored on the keysecure appliances.
The preparation phase and provisioning process:
In the kick-off meeting, ServiceNow and the customer defined the migration to Cloud Encryption for their 19 instances (without any existing data-at-rest encryption in place at that point). Starting with their sub-prod environments (DEV, Test...) and finally prod.
ServiceNow has created an automated workflow to support the instance activation Cloud Encryption (CE) process. Users with the customer admin and partner admin role can request this Service Catalog item to create a Change, which is completed using end-to-end automation.
For new customer with new instances, customers with a license for Platform Encryption or ServiceNow Vault may have their new instances provisioned with Cloud Encryption.
Before existing customers with existing instances can start with the implementation project, the following steps have to be done ServiceNow internally:
- First, the Customer must purchase the Platform Encryption SKU or the ServiceNow Vault SKU to obtain the entitlement to Cloud Encryption.
- Capacity Management of ServiceNow is informed by the Platform Security PM of the customer’s interest in moving to Cloud Encryption and ensures specialized hardware for the customer is provisioned. The worst-case scenario is 60-days for obtaining machines for use with Cloud Encryption, which will be for moving a large number and/or large sized instances to Cloud Encryption.
- As soon as the specialized hardware has been provisioned, the AE/Sales teamwill be notified of the hardware availability by the Platform Security PM.
- The AE/sales informed the customer that approval is there to start with the request to move instances to Cloud Encryption.
NOTE: Customers should not start the instance move request before the provisioning process has been completed.
After confirmation from ServiceNow, the customer starts the instance move request for Cloud Encryption for their first sub-production instance, using ServiceNow’s preconfigured automation we provide via a Service Catalogue item (see screenshot below).
Figure 1 Enable Cloud Encryption item in the service catalogue
After the Service Catalog item has been selected, customers choose their instance and a maintenance window (see screenshot below)
Figure 2 Select instance and maintenance windows for the activation of CE
Upload process of the customer-managed-key to the Now platform
Since the beginning, the customer told us they wanted to use a Customer Managed Key (BYOK) ito align with their internal compliance department. For Cloud Encryption (CE), as well as Column Level Encryption Enterprise (CLEE), customers can use the standard platform “Key Management Framework”. It is important to highlight here that customers are responsible for storing and protecting this symmetric key in their environment .
The customer was irritated because after the first sub-prod instance was created a ServiceNow key was generated for the encryption. Clarification with product management confirmedthis is an intended behavior. After the initial encryption with a ServiceNow key, customers can generate and use their own key in the platform, using the “Switch to Customer Managed Key” action.
For compatibility, you’re a Customer Managed Key must have the following attributes:
- Key type: Advanced Encryption Standard (AES) algorithm-based symmetric key.
- Key size: 256 bits (32 bytes)
Key wrapping requirements:
- RSA encryption algorithm
- Optimal Asymmetric Encryption Padding (OAEP)
- SHA-256 hash function (RSAES_OAEP_SHA_256)
- Encoded using Base64 algorithm
These details are included in our product documentation at
You should ensure this information is known by the customer’s security team in a timely manner.
To protect the symmetric key during transport from the customer environment to ServiceNow, key exchange procedure is available. Customers can download the wrapping certificate file from their instance and extract the public key to use when wrapping their symmetric key. . After the wrapping has been done, customers can upload the wrapped key to the platform. The symmetric key is unwrapped by the private key on the platform and can then be used.
The customer faced an issue with key wrapping process. It wasn’t possible for the customer to wrap the symmetric key with the public key according to the procedure in our product documentation. After several meetings (also with ServiceNow’s product management team), the syntax the customer used in OpenSSL wasn’t correct and there is a different approach on Mac OS devices and Windows machines. On Mac OS devices you have to use full paths in OpenSSL commands. After consulting our product management, we got an updated documentation we could share with the customer showing the correct syntax to wrap the customer key in Mac environments. Please see the following excerpt from that document:
_ _ _
If you’re on a Mac, ensure you have OpenSSL installed (in addition to the Mac default install of LibreSSL). One possible way of quickly installing OpenSSL on a Mac is by using the command ‘brew install openssl’ but use whichever way your organization mandates or allows. The reason for this is because future steps require you to define a hash function of ‘sha256’ which LibreSSL does not support, but OpenSSL does. LibreSSL is needed for Mac system functions, so we eventually need to tell the computer to use OpenSSL from a different directory than the default directory (which is pointing to LibreSSL).
If you’re on a Mac, make sure to use the full path again for OpenSSL when constructing this command. For example, the complex command looks like this in the product documentation:
- ‘openssl pkeyutl – encrypt -pubin -inkey publickey_<keyname>.PEM -in <keyname.bin> -out wrapped_key_material -pkeyopt rsa_padding_mode:oaep -pkeyopt sa_oaep_md:sha<128 or 256>’
- Using specifics, and the full path for openssl, the command would look like this:
- ‘/usr/local/opt/openssl/bin/openssl pkeyutl -encrypt -pubin -inkey publickey_572d2ef3874c6150af4f0e16dabb35e2.PEM -in mykey.bin -out wrapped_key_material -pkeyopt rsa_padding_mode:oaep -pkeyopt rsa_oaep_md:sha256’
The reason we are using the full path for openssl on a Mac is because the default “OpenSSL” on Macs is actually LibreSSL, which is used for other system functions. LibreSSL does not support the sha256 command we used in Windows, so we need to ensure we are using OpenSSL. By using the full path, we are specifying that the computer uses the actual, different OpenSSL version that was manually installed without affecting the LibreSSL version that is used for other system purposes.
_ _ _
After that issue has been clarified, the customer was able to wrap the symmetric key with the public key and upload the wrapped key again to the platform. The replacement of the ServiceNow key with the Customer Managed Key worked without any failures.
Rollout Cloud Encryption on 19 instances
The customer asked us for a more convenient approach to activate CE on the remaining instances. Because the effort of the initial, manual process, was too much to perform on all 18 instances remaining. We decided to make use of instance cloning and validated the behavior using clones in conjunction with Cloud Encryption together with our product management team. The behavior is as follows:
CE-encrypted Instance à Non-CE Instance: The target will be encrypted with a ServiceNow Managed key after the clone is completed. Customer can replace this key with their own CMK.
Non-CE Instance à CE-encrypted Instance: The target will no longer be encrypted after the clone is completed.
CE-encrypted Instance -> CE-encrypted Instance: No changes regarding encryption of data. However, using a “preserver” functionality on the target instances ensures the key for the target instance (no matter whether SNMK or CMK) stays the same and is not replaced by the one used on the source instance.
With the approach of cloning, the customer was able only within a few weeks to setup Cloud Encryption on all 18 remaining instances.
Cloud Encryption Withdraw & Resupply feature (Optional add-on SKU)
When we started customer conversations, they told us the need for having the Withdraw & Resupply (W&R) feature for Cloud Encryption. This functionality ensures that after a Customer Managed Key is withdrawn from the instance by the customer, no access to data is possible anymore, neither for ServiceNow nor for the customer. Therefore, this is a functionality where we have increased the awareness to use it only in an emergency (like data theft from the platform) use cases.
Because the customer was still on the San Diego release at that point and W&R was released for Utah, we agreed on a workaround process. The intention of this process was to ensure the customer can withdraw the key using the ServiceNow Support until the customer has updated all their instances to Utah.
The customer wanted to validate this functionality on a sub-prod instance before activating it in production. They wanted to evaluate the “resupply” part of that process in particular. To do so, ServiceNow provided a corresponding catalogue item that allows customer to submit the wrapped symmetric key (see screenshot). However, during the upload process of the symmetric key, which is necessary for restarting the instance by our Now Support team, a failure occurred.
Figure 3 Resupply catalogue item accessible on service catalogue
At first it was unclear of the reason for the failure. After our Support team invested more time in investigation, we uncovered the customer used a wrong symmetric key. It is important to understand that key being resupplied is the same key that was used and then withdrawn. As noted in our product documentation at https://docs.servicenow.com/bundle/vancouver-platform-security/page/administer/encryption-dare/task/...:
Danger: You can only perform a restore operation with the same key that was withdrawn. If you want to rotate to another key, you must do so after restoring the key that was withdrawn.
After the customer uploaded the right key, the instance could be restarted as expected without further problems.
In summary, the whole migration process took more than 6 months with a lot of internal and external meetings.
There are some take aways that might be helpful in future customer calls, to prevent issues for the migration project and make the whole process smoother and more convenient, for both parties.
Key take aways:
- Ensure the customer understands that there is no possibility to test Cloud Encryption in a dedicated environment beforehand since we use specialized hardware hosted in our production environment in a separate zone, so we won’t expose test machines to customers for testing purposes.
- However, we have PowerPoint slides based on a Reprise demo showing Cloud Encryption Key Management and the withdraw feature (see references)
- As soon as a customer has paid for the SKU, the Capacity Management team is getting a notice, with a worst-case scenario of 60 days for obtaining machines for use with Cloud Encryption. In general, the Capacity Management team won’t need the full-time span to prepare, however customers should plan with that worst-case scenario in mind.
- Customers that want to use their own key must ensure they generate a symmetric key with correct properties. Also be aware that generating keys with OpenSSL might be differ on Mac and Windows. The security team of the customer should be involved in an early stage.
- Cloud Encryption can only be purchased together with Column Level Encryption Enterprise in the Platform Encryption bundle. Platform Encryption is part of ServiceNow Vault as well.
- The Withdraw & Resupply feature (optional add-on SKU for Cloud Encryption) has a huge impact on the operation of the instance. Customers should understand that this functionality should only be used as a last resort.
- The whole ServiceNow organization has learned to manage Cloud Encryption and how to solve problems quickly. You can expect to get comprehensive help from CS&T/Product Management/Support if you face problems.
References:
- Process for requesting Cloud Encryption (https://support.servicenow.com/kb?id=kb_article_view&sysparm_article=KB1117369)
- FAQ for Cloud Encryption (https://support.servicenow.com/kb?id=kb_article_view&sysparm_article=KB1181374#mcetoc_1eivfhctq2k)
- Reprise demo for Cloud Encryption (https://servicenow.sharepoint.com/:p:/s/PlatformFoundationOutbound/EWvZ5IvCwmtKp4PkP-NJiTUBgcBrmY5eh...)
- ServiceNow Product Documentation (https://docs.servicenow.com/bundle/vancouver-platform-security/page/administer/encryption-dare/conce... )
1 Comment
