Oh Crap Moments and How to Protect PROD

[Chris Pearson]

Disclaimer: The ideas, thoughts, and opinions in this post reflect my own views and do not necessarily represent the views of my employer, Accenture. 

Most ServiceNow developers and admins who've been in the game for a few years probably could tell you their 'oh crap' story. Mine was just due to not paying attention...going too fast...not reading the warning pop-up well enough. And what happened? I accidentally deleted all Business Rules on the Incident table from the production instance.  

Oh. Crap. 

Now, thankfully this was in a situation where we had NOT gone live yet, and I had another instance in the stack that I could export those Business Rules to XML and import them into the instance that I wiped them from.  

Whew! 

Unfortunately, not all developers are as lucky as I was.  

If there's one single thing that all 'oh crap' moments have in common it is this: The person who says 'oh crap' was in the production instance using a login which has full admin rights. So, how can we reduce these 'oh crap' moments? Establish a system where your developers and administrators are not doing their 'day job' with their admin accounts. 

There are a few ways to accomplish this and I'll talk about two versions...a low tech version and a high tech version. 

Low Tech Version: 

This version of protecting production from accidental admin activity relies heavily on process and trusting your administrators and development team. 

1. Create a local account in ServiceNow for all administrators/developers which has full admin/security_admin permissions 
2. Strip the 'day to day' accounts in ServiceNow of your administrators/developers of admin and security_admin permissions, leaving only the necessary roles to do their daily work (itil, etc.) These accounts usually authenticate via SSO or LDAP Authentication (not a local account). 
3. Establish a process for your administrators/developers where they agree to ONLY use their local admin account in production for scenarios where they NEED to be admins (code promotions, etc). For all other scenarios, they are to use their 'day to day' account. 

High Tech Version: 

The high tech version uses only one user account for each of your development team members, but when needed, the developer can request production admin access and the system will auto-provision it to them via a catalog item. 

1. Create a catalog item which is only available to your administrators and developers which allows them to request 'admin access' to production. The item should allow the requester to select the start time of when they need their admin access to begin, how long they believe they'll need admin permissions for, and why they need that level of access. The item should also force the requester to select a Task which is related to why they need this type of access (is it an Incident that they're researching?, a Change Request because they're pushing code?, etc.) 
   1. The item should fire an approval to the manager of the developer team 
   2. Once approved, the workflow should auto-provision the role to that user's account and set a timer in order to revoke admin access after the appropriate duration selected in the catalog item. 
   3. Emails are sent to the requester to inform them when admin access has been provisioned and also when it has been revoked. 
2. Create a Cleanup Script which executes in the lower environments after each clone in order to identify all of your administrators/developers and provision the admin and security_admin roles to each of them. 

The result, after implementing either of these versions will be a production environment which is less vulnerable to those 'oh crap' moments of accidental admin activity.