Migration from J2SSH to SNCSSH

[tim_broberg]

Eureka adds a new SSH library, SNCSSH or "ServiceNow SSH,"   which is gradually replacing the older J2SSH library or "Legacy SSH."

J2SSH limitations:

1. no longer being supported by the open source community
2. the set of negotiable algorithms is steadily aging, for example there is no support for diffie-hellman-group14-sha1 and aes256-ctr
3. large number of threads per connection strains server resources
4. limited number of simultaneous probes per IP, by default three
5. maintainability of the J2SSH code is less than optimal
6. poor resource locking causes stalls during connection
7. scripts are scp'ed to the target system in a tmp directory, which can fail when no temp space is available

New features of SNCSSH:

1. nio-based architecture greatly reduces the number of threads
2. improved error messages
3. "deferred" debug logging, collects full debug information for each connection, displaying it only when that connection encounters an error
4. number of simultaneous probes limited only by server's MaxSessions, by default seven
5. where possible, scripts are handled as bourne-shell functions, reducing the number of operations required and removing the requirement to execute scripts from the temp directory

Terminal vs channels

Under the hood, sncssh uses ssh connections in a different way from j2ssh, which may result in some subtle differences in behavior.

A "terminal" is like the ssh connection you're probably used, a persistent connection where you type one character at a time, and watch for a command prompt to know it's your turn to type.

A "channel" is an internal feature of SSH sessions that allows multiple operations to be conducted over the same session, including execution of commands. This often goes by the term, "multiplexing" in ssh implementations.

j2ssh

• creates a pool of terminals, with one terminal used per probe
• each terminal has its own session to the server in question, so it's a heavy object
  
• commands are executed by sending a sequence of characters through the input stream of the terminal, and results are gleaned from the output stream.
• state from one probe may remain in the terminal when a subsequent probe is run. (This is why you cannot end a probe with "exit." It closes the terminal.)

sncssh

• creates a new channel for each operation run
• all the channels for the same host share a session, so it they are light objects
• commands to be executed are handled as discrete requests to the channel, one command per channel
• the lifetime of the channel is a single command, with no state carried over from previous commands

Selection of sncssh vs j2ssh

SNCSSH is enabled by default for new customers in Eureka and Fuji. Existing customers retain the same j2ssh.

You may select which library you prefer for the whole instance, for individual mid servers, and probe by probe.

Boolean attributes enabling sncssh, in order of descending precedence:

Probe - use_snc_ssh

Mid Server - (Discovery | Orchestration) / Mid Servers / (Your individual mid server) / Configuration Parameters / mid.ssh.use_snc

Instance - Mid Server / Properties / mid.property.ssh.use_snc

Detection of sncssh

When sncssh is in use, the probe parameter use_snc_ssh will be set to true in the ecc_queue record.

[sandeshsrikanta]

I found the error logging in SNCSSH to be better compared to J2SSH but had an issue with sudo.

The lsof command in Unix - Application Dependency Mapping probe runs perfectly fine in   J2SSH but comes back with a message 'The sudo command is not present, or user has no right to use it'.   Should I be doing something more?

[Ankush13]

It's not the ssh version that is causing this, so you have configured it correctly 🙂 Next steps will be to check the server that is throwing this error. Looks like lsof needs entry in /etc/sudoers..

[tim_broberg]

Hmm... the sncssh library tries to be smart enough to parse your sudoers config to see if it has permissions to run the command. If it doesn't it just runs the command without sudo and hopes for the best.

Can you login to that machine with your discovery user, run "sudo -l", and report the output? This will display what sudo permissions are configured.

    - Tim.

[sandeshsrikanta]

I ran a 50 server test with both J2SSH and SNCSSH. I get this error on almost all servers while running with SNCSSH whereas in J2SSH run the command runs normally.

Tim, I will try and and get the output for the sudo permissions thing if that brings more light.

[Ankush13]

If it runs with J2SSH, then I believe the permissions are there. Can you take a test server and try putting NOPASSWD in /etc/sudoers for lsof.