Why I got this error: Error; job finished with status ERROR: SSH channel 116 timeout in state TERMINAL_REQUESTED. (timeout.channel_ready = 120000)

[HugoFirst]

We're evaluating the Fuji Patch 10 version of Discovery and I had problems with a probe, The Linux Storage Probe.

It kept returning this error:

Error; job finished with status ERROR: SSH channel 116 timeout in state TERMINAL_REQUESTED. (timeout.channel_ready = 120000)

I'll skip the long story and give you the quick rundown on the reason and the fix.

The reason revolves around the account used by ServiceNow's midserver to login to Linux systems to run probes.   This account needs entries in the sudo permission file to run certain commands and our admins utilize the "full path" to the executable for our entries.

When our Linux admins setup the sudo permissions, they followed the guidance provided by ServiceNow in this document:

link: Credentials - ServiceNow Wiki  

In this document, certain commands are listed as being in /usr/bin.   In our case, some commands actually are found in /sbin, NOT /usr/bin.

So when the Linux Storage probe attempts to run the command, it prompts for a password for a command which will NOT be allowed to run under sudo.

This hangs the script and results in the timeout., _and_ the error mentioned above.

The fix is to have the Linux admin change the sudo entry for any command not actually in the expected   location to use the actual location.   In our case, use /sbin/multipath instead of /usrt/bin/multipath.

I wanted to post this in the event that someone else runs into this problem.   It might save them some time debugging.

[tim_broberg]

FYI, that error message indicates lack of traffic on an individual probe. A hung sudo prompt is one possible cause of many which may include a stalled connection, timeout set too low for the commands you're running, exhaustion of the max sessions per host with slow commands, host configuration errors, etc.

A good first step for debugging is to try to reproduce the behavior with the openssh client: "ssh tbroberg@timspc ./mycommand.sh". If it fails with openssh also, then you have an easier to debug problem in the command itself or the server config.

To get a client debug log, set mid.ssh.debug = (IP address of interest), reproduce the problem, delete mid.ssh.debug, capture the logs on the mid server page, download the agent log, and squint at it. You may require expert help if you get as far as looking at this log, be it technical or psychological. The log is very detailed, but if you really need to get to the bottom of the problem, the information is all there.

Server log for openssh can be collected by setting debug_level in /etc/ssh/sshd_config to debug (or debug3), and restarting the sshd service, then recreating the failure, setting loglevel back to info and restarting, and collecting the log which is generally in /var/log/secure.

    - Tim.

[james wells]

Just ran across this error because Fdisk -l (Linux - Storage probe) started then hung on the server. Tried increasing the Mid Server SSH channel timeout but no luck. Hoping I can fix this on the server...

[tim_broberg]

Yes, if it's hanging on the server, then this timeout is the best case outcome for the probe.

Does it run to completion? I.e., if you run "ssh -t jwells@jimsmachine sudo fdisk -l" (for appropriate values of "jwells" and "jimsmachine"), does it finish?

[Sean Sparshott]

link: Credentials - ServiceNow Wiki

Link fixed.