Cannot connect, status is TCP_CONNECTION_DROPPED. Connection unexpectedly closed by SSH server:

[pavanyuvi]

Hi All,

When we are trying to run discovery , we are getting the below issue on classify.

Active couldn't classify.

Cannot connect, status is TCP_CONNECTION_DROPPED. Connection unexpectedly closed by SSH server:

Regards,

Pavan

[Larry Youngqui2]

Pavan,

Assuming that the target is a *nix device, and you have access to the target machine through any network firewalls, do you also have the MID Servers listed in the etc/host.allow to grant local firewall access?

https://unixsherpa.com/solution/setting-up-hosts-allow/

[psb]

Just found out you are speaking from experience. Thanks for this

[tim_broberg]

Larry has a very likely explanation. There are some other possibilities.

In general, try to replicate what the mid server is doing from a standard ssh client and see how that goes.

Do ssh -vv username@hostname and see what happens. If it just hangs up on you, you have some firewall issues to sort out.

Otherwise, you may be having issues with some security application losing patience with Servicenow. If you go to the credential and run test credential, does it work?

You can also bump up the loglevel on the server and see what's in the server logs (usually /var/log/secure).

Finally, you can enable mid.ssh.debug on the mid, recreate the failure, turn debug back off, and look in agent/logs/agent0.log.0 for the gory details on when the disconnection happened from the client side.

1. It it's almost immediate, it's probably a firewall.
2. If it's after kexinit exchange, there may be a problem finding acceptable algorithms.
3. If it's after a userauth request, we're probably trying so many credentials some security tool thinks we're hacking your system.

Once we have a handle on probable causes, we can look at how to fix the problem.
    - Tim.

[geoffhotchkiss]

We hit this problem recently and found it's due to how ServiceNow SSH discovery works. From our support case we opened:

This issue here is "TCP_CONNECTION_DROPPED Connection unexpectedly closed by SSH server" after only a single unsuccessful credential attempt. The target should not be closing the connection. Since the target is initiating the closing of the connection, no more credentials are tried by ServiceNow.

This problem was happening on CIs that have not been discovered in a very long time and are trying to be discovered using a different credential than was previously used. In other words, previously discoveryuser1 was used but now we want discoveryuser2 to be used (for these specific CIs only).

For us, the connection was getting closed due to our access.conf config. We allow a very specific set of users access to certain servers and we deny everyone else. We expect discoveryuser1 to be used on linuxservertype1, discoveryuser2 to be used on linuxservertype2, and so on.

Because of credential ordering details, ServiceNow tried to use discoveryuser1 on linuxservertype2. But we don't allow discoveryuser1 in access.conf for linuxservertype2. So PAM denies the login and ServerNow sees this as the connection being dropped, refuses to try discoveryuser2 (or any other creds), and moves on with trying to discover other servers.

I was able to work around this by temporally changing the order of the credentials so that discoveryuser2 is tried before discoveryuser1. That allowed the credential affinities to get created for the servers that are expecting discoveryuser2. But the problem still exists. We're not yet sure how to resolve this long term and we're thinking about how we should solve this. This will be a problem for all of our future CIs.