MID Server will not start after instance upgrade

[Daniel A-C]

This is just posted for shared community knowledge, I've already solved it.

I came into work this morning to reports of a MID Server not running for our dev instance but the test and prod instances were fine. I knew we recently had a patch applied to our dev instance and we were planning to check it and then roll it out across our test then prod instances.

After trying to restart the service, I noticed it was starting, then stopping just a couple of minutes later. So after enabling logging in the dev mid server...

[ Open %mid_server_install_directory%\agent\config.xml and add   <parameter name="debug.logging" value="true" />   ]

I restarted the service, waited for it to stop/stall and then examined the logs...

[ %mid_server_install_directory%\agent\logs\agent0.log.0   ]

I read through the logs and found that the MID server was trying to auto-upgrade itself to match the new patch version of the instance.

The instance was on:

Build name: HEAD (Fuji)

Build date: 09-01-2015_1202

Build tag: glide-fuji-12-23-2014__patch7-hotfix5-09-01-2015

I found a number of errors in the log. I will list them here to assist with searching...

• 09/28/15 09:40:04 (575) StartupSequencer Enabling monitor: AutoUpgrade
• 09/28/15 09:40:04 (575) StartupSequencer Enabling monitor: IdleConnectionMonitor
• 09/28/15 09:40:04 (637) ECCQueueMonitor.15 Monitor query: state=processing^queue=output^agent=mid.server.MIDSERVERNAMEREMOVED^sys_created_on>=2015-08-29 09:39:58^ORDERBYsys_created_on
• 09/28/15 09:40:05 (027) LogStatusMonitor.60 stats threads: 27, memory max: 495.0mb, allocated: 129.0mb, used: 30.0mb, queued: 0 probes, processing: 0 probes
• 09/28/15 09:40:05 (105) StatusMonitor.600 Enqueuing: P:\Program Files\ServiceNow\MIDServers\MIDSERVERNAMEREMOVED\agent\work\monitors\ECCSender\output\ecc_queue.af0f0011e832020057eeea3499477418.xml
• 09/28/15 09:40:05 (651) ECCSender.1 Sending ecc_queue.af0f0011e832020057eeea3499477418.xml
• 09/28/15 09:40:05 (760) AutoUpgrade.3600 Deleted path from upgrade marker, `C:\Users\SERVICEACCOUNTNAMEREMOVED\AppData\Local\Temp\1443428388993-0`.
• 09/28/15 09:40:05 (776) AutoUpgrade.3600 SEVERE *** ERROR *** ExecuteException: Process exited with an error: 1060 (Exit value: 1060)
• 09/28/15 09:40:05 (776) AutoUpgrade.3600 Checking to see if MID server needs to upgrade.
• 09/28/15 09:40:06 (021) AutoUpgrade.3600 Packages refreshed.
• 09/28/15 09:40:06 (021) AutoUpgrade.3600 Current packages:
• 09/28/15 09:40:06 (021) AutoUpgrade.3600     Installed: [mid-core.2015-07-13-1834.universal.universal.zip, mid-jre.2015-07-13-1834.windows.x86-64.zip]
• 09/28/15 09:40:06 (021) AutoUpgrade.3600     Assigned: [mid-core.2015-09-01-1202.universal.universal.zip, mid-upgrade.2015-09-01-1202.universal.universal.zip]
• 09/28/15 09:40:06 (021) AutoUpgrade.3600     Missing: [mid-core.2015-09-01-1202.universal.universal.zip, mid-upgrade.2015-09-01-1202.universal.universal.zip]
• 09/28/15 09:40:06 (021) AutoUpgrade.3600     Downloaded: [mid-core.2015-09-01-1202.universal.universal.zip, mid-upgrade.2015-09-01-1202.universal.universal.zip]
• 09/28/15 09:40:06 (258) AutoUpgrade.3600 Packages refreshed.

and then...

• 09/28/15 09:40:08 (676) AutoUpgrade.3600 Extracting package P:\Program Files\ServiceNow\MIDServers\MIDSERVERNAMEREMOVED\agent\package\incoming\mid-upgrade.2015-09-01-1202.universal.universal.zip to C:\Users\SERVICEACCOUNTNAMEREMOVED\AppData\Local\Temp\1443429606289-0
• 09/28/15 09:40:08 (863) AutoUpgrade.3600 Stopping MID server. Bootstrapping upgrade.
• 09/28/15 09:40:14 (776) Gobbling stderr: cmd.exe /C bin\glide-dist-upgrade.bat start Gobbled: The syntax of the command is incorrect.
• 09/28/15 09:40:14 (822) Gobbling stderr: cmd.exe /C bin\glide-dist-upgrade.bat start Gobbled: ERROR   | wrapper   | 2015/09/28 09:40:14 | The ServiceNow Platform Distribution Upgrade (SERVERNAME-INSTANCENAME1) service is not installed - The specified service does not exist as an installed service. (0x424)

The bit where it is looking for SERVERNAME-INSTANCENAME1 seems odd to me. We've given our services a custom name via the wrapper.conf file as we have multiple MID servers on one virtual server. It seems that it's looking for a service based on a name pattern rather than interrogating the configuration first.

Whilst this may look odd and the error message is not seemingly related; after doing some research with trial and error, it appears that the issue is that the service account which the service is configured to run under, does not have suitable permissions to administratively perform the installation of the upgrade. I'm sure there is a specific privilege to grant somewhere but without spending more time on it, we implemented a solution.

The solution we implemented was to add the service account to the local administrators group on the server. Restarting the service then allowed the auto-upgrade to take place without issue and it has stayed alive.

Anyway, I hope this helps someone out there. If anyone knows the exact permission it actually requires, I'd be interested to hear about it!

Thanks,

Daniel

Daniel Critchley. Acora Ltd.

[JBark]

MID Server Installation - ServiceNow Wiki

Section 3

Network account: Install the MID Server with the proper account, either local or domain administrator.

We did the same, just makes it   easier to do everything.

[Daniel A-C]

Hi Jeffrey,

Thanks for pointing that out, I must admit, I hadn't spotted that in the wiki! Our MID servers have been running for a while now with just a domain user account which was granted 'log on as a service' rights.

The reason it was done this way is that it matches our default security policy of 'grant the minimum rights necessary for any service account at all times'. We then extend the rights on an as-needed basis.

In this case, I was surprised because a) the logs didn't suggest it was a permissions issue and b) the MID server has been running successfully and meeting all other requirements (up until now) without issue.

But it makes sense that if it is going to self-install an upgrade, then it would need administrative rights over at least the machine that it is running on.

Thanks again,

Daniel

[JBark]

I swung for the fence and asked for Domain Admin rights for my Discovery account. Of course I didn't get that, but asking for local admin afterwards became a lot easier

I understand and agree with the policy of "minimum rights necessary" Luckily the documentation was explicit.

[johnnyjava]

We ran into similar problems during a recent upgrade from Eureka to Geneva. Our windows MID Servers were running as a domain account which was a member of Local Administrators group on their boxes, however someone in Security had removed some of the domain accounts privileges. Once they were restored, the upgrade was rather smooth for the MID Servers - with the exception of all the Discovery Schedules re-starting when the Scheduler began running again. Once I re-canceled all the Discovery Schedules the MID Instances began to auto upgrade.