Automatically restart service in event management - auto remediation

[Henrik Jutterst]

Is there anyone here who has set up a Flow to automatically restart a service in Event Management?

1. What is the best practice for this?
2. What if there is a Windows Server or a Linux Server - there are two separate SubFlows for this as I see?
3. Where do I kick of these SubFlows and how does the SubFlow know the name of the Service to restart?
4. Are there any permissions needed for this for the system users that try to remediate this?

Kind regards

[Patrick DeCarl1]

Hey Henrik, 

We do this for a lab internal here at ServiceNow. We use ServiceNow instance to auto deploy VMs with mid servers deployed so internal team members can learn / practice their ITOM skills. We use ITOM health to track the health of the service of the mid server service. OOTB there are a few alert rules "Windows server actions" that has actions with subflows that will start, stop and restart a service. We took these OOTB services and just created a new to  subflow restart the mid server service, we pass in via the subflow the service name from the event, and the subflow will make the call to the device via powershell (windows) or ssh(linux) and restart the service, if it doesn't work, we then turn the alert into an incident. 

2. It depends on how you want to maintain alert rule and flows. For us we used one alert rule and subflow. Alert rule looks for the window/linux services and then triggers the subflow under the action section. The subflow has a if check to see what OS the device is running, if windows or linux and based on that we will make a powershell or ssh call to restart the service. 
3. Alert rule is where the magic happens. You define the rule, the filter you are looking for, Event source, what service you looking for, and then the action section you define the subflow to call. 
4. Just the same permission as the user who would remote into the box and run the cmds. 

[Henrik Jutterst]

Hi @Patrick DeCarl1 and thank you so much for the helping hand here! 🎊
Although I haven't solved my automation yet, it feels like I'm getting closer thanks to your feedback.

Background

We are trying to start up a service on a host if we receive events to ServiceNow that a service is down. We have both Linux and Windows environment so ideally we are looking for restart automation whether or less it's Linux or Windows hosts.

Findings

I found an Alert Management Rule called "Windows server actions" / "Linux server actions" and here I can see multiple SubFlows, but these are all triggered manually. But again this is only for Windows...

Figure 01 - OOB Alert Management Rule for Windows Services

Question 1.

Is there any OOB SubFlow or Alert Management Rule for Event Management to detect if it's a Windows or Linux host and then perform the restart actions like Figure 01, or how is this handled? Or do we need to build that our self?

Question 2.

Is it possible or even recommended to have an automation of starting a service if it's down? Is setting the Execution to "Automatic" all that's needed here?

Here are the OOB rules that I have in our instance:

Figure 02 - OOB Alert Management Rules

[Patrick DeCarl1]

Q1 -- If I understand you question, an OOTB rule to defect if a service is down? No, you will need to create a rule for a service or services you want to manage and then take those OOTB subflows and use or copy and add too. 

Q2.  I already answer in my first reply.  Yes its possible, and it depends on your environment and if this is something you want to do. 

[Henrik Jutterst]

Q1 -- No. ServiceNow will receive the event if a service is down (from Splunk in this case) with service name and host name in the description of the Alert. But we monitor both Linux and Windows hosts. The node (host) data is matched to the correct CI in Alert with successful host lookup.

But from here, do we need to create an Alert Management Rule with a SubFlow to detect Windows or Linux, and then use OOB SubFlows to brick back the Service from either Windows or Linux - or is there a SubFlow for this already.

Alert:

Q2 -- Thanks!