This is just posted for shared community knowledge, I've already solved it.
I came into work this morning to reports of a MID Server not running for our dev instance but the test and prod instances were fine. I knew we recently had a patch applied to our dev instance and we were planning to check it and then roll it out across our test then prod instances.
After trying to restart the service, I noticed it was starting, then stopping just a couple of minutes later. So after enabling logging in the dev mid server...
[ Open %mid_server_install_directory%\agent\config.xml and add <parameter name="debug.logging" value="true" /> ]
I restarted the service, waited for it to stop/stall and then examined the logs...
[ %mid_server_install_directory%\agent\logs\agent0.log.0 ]
I read through the logs and found that the MID server was trying to auto-upgrade itself to match the new patch version of the instance.
The instance was on:
Build name: HEAD (Fuji)
Build date: 09-01-2015_1202
Build tag: glide-fuji-12-23-2014__patch7-hotfix5-09-01-2015
I found a number of errors in the log. I will list them here to assist with searching...
- 09/28/15 09:40:04 (575) StartupSequencer Enabling monitor: AutoUpgrade
- 09/28/15 09:40:04 (575) StartupSequencer Enabling monitor: IdleConnectionMonitor
- 09/28/15 09:40:04 (637) ECCQueueMonitor.15 Monitor query: state=processing^queue=output^agent=mid.server.MIDSERVERNAMEREMOVED^sys_created_on>=2015-08-29 09:39:58^ORDERBYsys_created_on
- 09/28/15 09:40:05 (027) LogStatusMonitor.60 stats threads: 27, memory max: 495.0mb, allocated: 129.0mb, used: 30.0mb, queued: 0 probes, processing: 0 probes
- 09/28/15 09:40:05 (105) StatusMonitor.600 Enqueuing: P:\Program Files\ServiceNow\MIDServers\MIDSERVERNAMEREMOVED\agent\work\monitors\ECCSender\output\ecc_queue.af0f0011e832020057eeea3499477418.xml
- 09/28/15 09:40:05 (651) ECCSender.1 Sending ecc_queue.af0f0011e832020057eeea3499477418.xml
- 09/28/15 09:40:05 (760) AutoUpgrade.3600 Deleted path from upgrade marker, `C:\Users\SERVICEACCOUNTNAMEREMOVED\AppData\Local\Temp\1443428388993-0`.
- 09/28/15 09:40:05 (776) AutoUpgrade.3600 SEVERE *** ERROR *** ExecuteException: Process exited with an error: 1060 (Exit value: 1060)
- 09/28/15 09:40:05 (776) AutoUpgrade.3600 Checking to see if MID server needs to upgrade.
- 09/28/15 09:40:06 (021) AutoUpgrade.3600 Packages refreshed.
- 09/28/15 09:40:06 (021) AutoUpgrade.3600 Current packages:
- 09/28/15 09:40:06 (021) AutoUpgrade.3600 Installed: [mid-core.2015-07-13-1834.universal.universal.zip, mid-jre.2015-07-13-1834.windows.x86-64.zip]
- 09/28/15 09:40:06 (021) AutoUpgrade.3600 Assigned: [mid-core.2015-09-01-1202.universal.universal.zip, mid-upgrade.2015-09-01-1202.universal.universal.zip]
- 09/28/15 09:40:06 (021) AutoUpgrade.3600 Missing: [mid-core.2015-09-01-1202.universal.universal.zip, mid-upgrade.2015-09-01-1202.universal.universal.zip]
- 09/28/15 09:40:06 (021) AutoUpgrade.3600 Downloaded: [mid-core.2015-09-01-1202.universal.universal.zip, mid-upgrade.2015-09-01-1202.universal.universal.zip]
- 09/28/15 09:40:06 (258) AutoUpgrade.3600 Packages refreshed.
and then...
- 09/28/15 09:40:08 (676) AutoUpgrade.3600 Extracting package P:\Program Files\ServiceNow\MIDServers\MIDSERVERNAMEREMOVED\agent\package\incoming\mid-upgrade.2015-09-01-1202.universal.universal.zip to C:\Users\SERVICEACCOUNTNAMEREMOVED\AppData\Local\Temp\1443429606289-0
- 09/28/15 09:40:08 (863) AutoUpgrade.3600 Stopping MID server. Bootstrapping upgrade.
- 09/28/15 09:40:14 (776) Gobbling stderr: cmd.exe /C bin\glide-dist-upgrade.bat start Gobbled: The syntax of the command is incorrect.
- 09/28/15 09:40:14 (822) Gobbling stderr: cmd.exe /C bin\glide-dist-upgrade.bat start Gobbled: ERROR | wrapper | 2015/09/28 09:40:14 | The ServiceNow Platform Distribution Upgrade (SERVERNAME-INSTANCENAME1) service is not installed - The specified service does not exist as an installed service. (0x424)
The bit where it is looking for SERVERNAME-INSTANCENAME1 seems odd to me. We've given our services a custom name via the wrapper.conf file as we have multiple MID servers on one virtual server. It seems that it's looking for a service based on a name pattern rather than interrogating the configuration first.
Whilst this may look odd and the error message is not seemingly related; after doing some research with trial and error, it appears that the issue is that the service account which the service is configured to run under, does not have suitable permissions to administratively perform the installation of the upgrade. I'm sure there is a specific privilege to grant somewhere but without spending more time on it, we implemented a solution.
The solution we implemented was to add the service account to the local administrators group on the server. Restarting the service then allowed the auto-upgrade to take place without issue and it has stayed alive.
Anyway, I hope this helps someone out there. If anyone knows the exact permission it actually requires, I'd be interested to hear about it!
Thanks,
Daniel
Daniel Critchley. Acora Ltd.
