MID Server fails after patch update – systemctl shows failed, but wrapper and java are running

[NithianandanBV]

Hello Team,

After applying a recent patch automatically to multiple Linux-based MID Servers, we have noticed a recurring issue:

• systemctl status snow.service shows FAILED

• But processes like wrapper-linux-x86-64 and java are still running in the background

• manual kill -9 <PID> followed by systemctl restart snow resolves the issue

This is happening when ever the SNOW mid-servers are Upgraded automatically !!

Note - We have deployed a shell script, which use to check the status of the service for every 15 minutes. if its failed state it will restart it. When it tried to restart through shell script we got the below error.

Job for snow.service failed because the control process exited with error code.
See "systemctl status snow.service" and "journalctl -xeu snow.service" for details.

Manual kill -9 <PID> followed by systemctl restart snow resolves the issue

Questions:

1. Is this a known issue with the MID Server patching process?

2. Are there official recommendations from ServiceNow to handle this reported issue ?

3. Has anyone implemented a better/cleaner workaround?

Looking for inputs from those who faced similar issues and please share the workaround adapted for the same. Thanks in advance!

[steven_yap]

I encountered exactly the same problem when the instance pushed down Xanadu Patch9b to the midserver. The midserver was running Xanadu Patch9a but the upgrade seemed to have gone through. However, the systemctl status <service.name> showed it FAILED. The wrapper/java process was running and the instance had no problem communicating. But reboot or trying to restart the service was unsuccessful. 

After looking through the various directories in /opt/servicenow/mid/agent and looking through the scripts, I finally found the source of the problem.

The script mid.sh seems to keep starting the process and creating the file /opt/servicenow/mid/agent/work/mid.pid. It should have created a PID file with the midserver's service name as in <midserver service name>.pid.

The problem, in my case, was the file /opt/servicenow/mid/agent/bin/mid.shconf_override was not properly maintained after the auto patch upgrade. It was missing quite a number of information. I put a sample below for your reference.

-------------------------------------------------------------------------------------------------------------------------------------

#APP_NAME="this_mid_app_name
#APP_LONG_NAME="this_mid_long_app_name"

# Put a user name here if you want to run the service by a non-root user.
# The specified user is going to be the owner of the entire agent directory.
# If leaving empty, the owner is unchanged.
RUN_AS_USER=

# Put a group name here who is going to be the owner group of the entire agent directory.
# If leaving empty, the group is unchanged.
GROUP_NAME=

-------------------------------------------------------------------------------------------------------------------------

After I populated these fields with the appropriate information, the service was started normally. Everything else worked even after reboots.

Hope this helps those affected.