Migration from J2SSH to SNCSSH

[tim_broberg]

Eureka adds a new SSH library, SNCSSH or "ServiceNow SSH,"   which is gradually replacing the older J2SSH library or "Legacy SSH."

J2SSH limitations:

1. no longer being supported by the open source community
2. the set of negotiable algorithms is steadily aging, for example there is no support for diffie-hellman-group14-sha1 and aes256-ctr
3. large number of threads per connection strains server resources
4. limited number of simultaneous probes per IP, by default three
5. maintainability of the J2SSH code is less than optimal
6. poor resource locking causes stalls during connection
7. scripts are scp'ed to the target system in a tmp directory, which can fail when no temp space is available

New features of SNCSSH:

1. nio-based architecture greatly reduces the number of threads
2. improved error messages
3. "deferred" debug logging, collects full debug information for each connection, displaying it only when that connection encounters an error
4. number of simultaneous probes limited only by server's MaxSessions, by default seven
5. where possible, scripts are handled as bourne-shell functions, reducing the number of operations required and removing the requirement to execute scripts from the temp directory

Terminal vs channels

Under the hood, sncssh uses ssh connections in a different way from j2ssh, which may result in some subtle differences in behavior.

A "terminal" is like the ssh connection you're probably used, a persistent connection where you type one character at a time, and watch for a command prompt to know it's your turn to type.

A "channel" is an internal feature of SSH sessions that allows multiple operations to be conducted over the same session, including execution of commands. This often goes by the term, "multiplexing" in ssh implementations.

j2ssh

• creates a pool of terminals, with one terminal used per probe
• each terminal has its own session to the server in question, so it's a heavy object
  
• commands are executed by sending a sequence of characters through the input stream of the terminal, and results are gleaned from the output stream.
• state from one probe may remain in the terminal when a subsequent probe is run. (This is why you cannot end a probe with "exit." It closes the terminal.)

sncssh

• creates a new channel for each operation run
• all the channels for the same host share a session, so it they are light objects
• commands to be executed are handled as discrete requests to the channel, one command per channel
• the lifetime of the channel is a single command, with no state carried over from previous commands

Selection of sncssh vs j2ssh

SNCSSH is enabled by default for new customers in Eureka and Fuji. Existing customers retain the same j2ssh.

You may select which library you prefer for the whole instance, for individual mid servers, and probe by probe.

Boolean attributes enabling sncssh, in order of descending precedence:

Probe - use_snc_ssh

Mid Server - (Discovery | Orchestration) / Mid Servers / (Your individual mid server) / Configuration Parameters / mid.ssh.use_snc

Instance - Mid Server / Properties / mid.property.ssh.use_snc

Detection of sncssh

When sncssh is in use, the probe parameter use_snc_ssh will be set to true in the ecc_queue record.

[StephenHey]

Are there plans to add support to the SNCSSH for rbash or other restricted shells?   Also, in cases where PATH cannot be set, it appears that SNCSSH tries to set PATH now on every call (unlike J2SSH) and fails for servers that restrict the setting of PATH.

[tim_broberg]

Terrific questions, Stephen.

The thinking behind the shell check was:

1. It's better to explicitly fail when we don't support a shell than it is to subtly fail with obscure symptoms that take hours in support to debug.
2. Administrators have control over what the shell is, so they can easily switch to one of the supported ones.

The existing sncssh shell support:

1. direct support for sh, bash, ksh, csh, and tcsh
2. in fuji, the allow_unsupported_shells probe parameter to allow running individual probes (without path or script support)

Having said that, I think rbash is a special case. Sure, you can switch to another shell, but you can't get the security you wanted with rbash.

To rbash it work, there are two issues:

1. You can't set PATH, or it errors out. This can be handled by clearing mid.ssh.set_path, but that's a little awkward.
2. It's not on the list of supported bourne-compatible shells.

I will put in a feature-request for the ability to edit the list of bourne-compatible shells in general, and rbash support in specific.

What other restricted shells are people using?

Thanks for the feedback!

    - Tim.

[tim_broberg]

FYI, the list of bourne-compatible shells is now editable with the mid.ssh.shells_supported parameter which defaults to ksh,sh,bash.

If you wanted to add dash, for example, you would set it to ksh,sh,bash,dash.

    - Tim.

[Ankush13]

I changed from J2SSH to SNCSSH and found that it was considerably faster. Great article indeed!