mjhjmtu

Recently, we have introduced several new hosts on our server. Remote connectivity was working fine, till the OS on the server was re-installed.

After checking the logs from both the server and client side, we have found the below error message appearing on the client side causing the connection to be closed by the client side.

Error: Unexpected SSH layer 3 GLOBAL_REQUEST message received

After raising the debug level on the server side to DEBUG3, here below are the logs, it seems that the server is sending packet type 51 just before the disconnection. Also, you will find below the logs extracted from another node where the SSH is working fine.

 debug3: fd 5 is not O_NONBLOCK


debug1: Forked child 2652.
debug3: send_rexec_state: entering fd = 8 config len 1085
debug3: ssh_msg_send: type 0
debug3: send_rexec_state: done
debug3: oom_adjust_restore
debug1: Set /proc/self/oom_score_adj to 0
debug1: rexec start in 5 out 5 newsock 5 pipe 7 sock 8
debug1: inetd sockets after dupping: 3, 3
Connection from 10.115.33.65 port 51585 on 10.58.126.35 port 22
debug1: Client protocol version 2.0; client software version Nakina_SSH_1.0
debug1: no match: Nakina_SSH_1.0
debug1: Local version string SSH-2.0-OpenSSH_7.4
debug1: Enabling compatibility mode for protocol 2.0
debug2: fd 3 setting O_NONBLOCK
debug3: ssh_sandbox_init: preparing seccomp filter sandbox
debug2: Network child is on pid 2653
debug3: preauth child monitor started
debug1: SELinux support enabled [preauth]
debug3: ssh_selinux_change_context: setting context from 'system_u:system_r:sshd_t:s0-s0:c0.c1023' to 'system_u:system_r:sshd_net_t:s0-s0:c0.c1023' [preauth]
debug3: privsep user:group 74:74 [preauth]
debug1: permanently_set_uid: 74/74 [preauth]
debug3: ssh_sandbox_child: setting PR_SET_NO_NEW_PRIVS [preauth]
debug3: ssh_sandbox_child: attaching seccomp filter program [preauth]
debug1: list_hostkey_types: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
debug3: send packet: type 20 [preauth]
debug1: SSH2_MSG_KEXINIT sent [preauth]
debug3: receive packet: type 20 [preauth]
debug1: SSH2_MSG_KEXINIT received [preauth]
debug2: local server KEXINIT proposal [preauth]
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1 [preauth]
debug2: host key algorithms: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
debug2: ciphers ctos: aes128-ctr,aes192-ctr,aes256-ctr [preauth]
debug2: ciphers stoc: aes128-ctr,aes192-ctr,aes256-ctr [preauth]
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1 [preauth]
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1 [preauth]
debug2: compression ctos: none,zlib@openssh.com [preauth]
debug2: compression stoc: none,zlib@openssh.com [preauth]
debug2: languages ctos: [preauth]
debug2: languages stoc: [preauth]
debug2: first_kex_follows 0 [preauth]
debug2: reserved 0 [preauth]
debug2: peer client KEXINIT proposal [preauth]
debug2: KEX algorithms: diffie-hellman-group14-sha1,diffie-hellman-group1-sha1 [preauth]
debug2: host key algorithms: ssh-rsa,ssh-dss [preauth]
debug2: ciphers ctos: aes256-cbc,aes256-ctr,aes192-cbc,aes192-ctr,aes128-cbc,aes128-ctr,3des-cbc,3des-ctr,blowfish-cbc,blowfish-ctr [preauth]
debug2: ciphers stoc: aes256-cbc,aes256-ctr,aes192-cbc,aes192-ctr,aes128-cbc,aes128-ctr,3des-cbc,3des-ctr,blowfish-cbc,blowfish-ctr [preauth]
debug2: MACs ctos: hmac-sha1,hmac-md5,hmac-sha1-96,hmac-md5-96 [preauth]
debug2: MACs stoc: hmac-sha1,hmac-md5,hmac-sha1-96,hmac-md5-96 [preauth]
debug2: compression ctos: none [preauth]
debug2: compression stoc: none [preauth]
debug2: languages ctos: [preauth]
debug2: languages stoc: [preauth]
debug2: first_kex_follows 0 [preauth]
debug2: reserved 0 [preauth]
debug1: kex: algorithm: diffie-hellman-group14-sha1 [preauth]
debug1: kex: host key algorithm: ssh-rsa [preauth]
debug1: kex: client->server cipher: aes256-ctr MAC: hmac-sha1 compression: none [preauth]
debug1: kex: server->client cipher: aes256-ctr MAC: hmac-sha1 compression: none [preauth]
debug1: kex: diffie-hellman-group14-sha1 need=32 dh_need=32 [preauth]
debug3: mm_request_send entering: type 120 [preauth]
debug3: mm_request_receive_expect entering: type 121 [preauth]
debug3: mm_request_receive entering [preauth]
debug3: mm_request_receive entering
debug3: monitor_read: checking request 120
debug3: mm_request_send entering: type 121
debug1: kex: diffie-hellman-group14-sha1 need=32 dh_need=32 [preauth]
debug3: mm_request_send entering: type 120 [preauth]
debug3: mm_request_receive_expect entering: type 121 [preauth]
debug3: mm_request_receive entering [preauth]
debug3: mm_request_receive entering
debug3: monitor_read: checking request 120
debug3: mm_request_send entering: type 121
debug2: bits set: 1037/2048 [preauth]
debug1: expecting SSH2_MSG_KEXDH_INIT [preauth]
debug3: receive packet: type 30 [preauth]
debug2: bits set: 1021/2048 [preauth]
debug3: mm_key_sign entering [preauth]
debug3: mm_request_send entering: type 6 [preauth]
debug3: mm_key_sign: waiting for MONITOR_ANS_SIGN [preauth]
debug3: mm_request_receive_expect entering: type 7 [preauth]
debug3: mm_request_receive entering [preauth]
debug3: mm_request_receive entering
debug3: monitor_read: checking request 6
debug3: mm_answer_sign
debug3: mm_answer_sign: hostkey proof signature 0x55aa18906680(271)
debug3: mm_request_send entering: type 7
debug2: monitor_read: 6 used once, disabling now
debug3: send packet: type 31 [preauth]
debug3: send packet: type 21 [preauth]
debug2: set_newkeys: mode 1 [preauth]
debug1: rekey after 4294967296 blocks [preauth]
debug1: SSH2_MSG_NEWKEYS sent [preauth]
debug1: expecting SSH2_MSG_NEWKEYS [preauth]
debug3: receive packet: type 21 [preauth]
debug1: SSH2_MSG_NEWKEYS received [preauth]
debug2: set_newkeys: mode 0 [preauth]
debug1: rekey after 4294967296 blocks [preauth]
debug1: KEX done [preauth]
debug3: receive packet: type 5 [preauth]
debug3: send packet: type 6 [preauth]
debug3: receive packet: type 50 [preauth]
debug1: userauth-request for user root service ssh-connection method none [preauth]
debug1: attempt 0 failures 0 [preauth]
debug3: mm_getpwnamallow entering [preauth]
debug3: mm_request_send entering: type 8 [preauth]
debug3: mm_getpwnamallow: waiting for MONITOR_ANS_PWNAM [preauth]
debug3: mm_request_receive_expect entering: type 9 [preauth]
debug3: mm_request_receive entering [preauth]
debug3: mm_request_receive entering
debug3: monitor_read: checking request 8
debug3: mm_answer_pwnamallow
debug2: parse_server_config: config reprocess config len 1085
debug3: mm_answer_pwnamallow: sending MONITOR_ANS_PWNAM: 1
debug3: mm_request_send entering: type 9
debug2: monitor_read: 8 used once, disabling now
debug2: input_userauth_request: setting up authctxt for root [preauth]
debug3: mm_start_pam entering [preauth]
debug3: mm_request_send entering: type 100 [preauth]
debug3: mm_inform_authserv entering [preauth]
debug3: mm_request_send entering: type 4 [preauth]
debug3: mm_inform_authrole entering [preauth]
debug3: mm_request_send entering: type 80 [preauth]
debug3: mm_auth2_read_banner entering [preauth]
debug3: mm_request_send entering: type 10 [preauth]
debug3: mm_request_receive_expect entering: type 11 [preauth]
debug3: mm_request_receive entering [preauth]
debug3: mm_request_receive entering
debug3: monitor_read: checking request 100
debug1: PAM: initializing for "root"
debug1: PAM: setting PAM_RHOST to "10.115.33.65"
debug1: PAM: setting PAM_TTY to "ssh"
debug2: monitor_read: 100 used once, disabling now
debug3: mm_request_receive entering
debug3: monitor_read: checking request 4
debug3: mm_answer_authserv: service=ssh-connection, style=
debug2: monitor_read: 4 used once, disabling now
debug3: mm_request_receive entering
debug3: monitor_read: checking request 80
debug3: mm_answer_authrole: role=
debug2: monitor_read: 80 used once, disabling now
debug3: mm_request_receive entering
debug3: monitor_read: checking request 10
debug3: mm_request_send entering: type 11
debug2: monitor_read: 10 used once, disabling now
debug3: send packet: type 53 [preauth]
debug1: userauth_send_banner: sent [preauth]
debug2: input_userauth_request: try method none [preauth]
debug3: userauth_finish: failure partial=0 next methods="publickey,password" [preauth]
debug3: send packet: type 51 [preauth]
Connection closed by 10.115.33.65 port 51585 [preauth]
debug1: do_cleanup [preauth]
debug3: PAM: sshpam_thread_cleanup entering [preauth]
debug3: mm_request_send entering: type 124 [preauth]
debug3: mm_request_send entering: type 122 [preauth]
debug3: mm_request_receive_expect entering: type 123 [preauth]
debug3: mm_request_receive entering [preauth]
debug3: mm_request_receive entering
debug3: monitor_read: checking request 124
debug3: mm_request_receive entering
debug3: monitor_read: checking request 122
debug3: mm_request_send entering: type 123
debug1: monitor_read_log: child log fd closed
debug3: mm_request_receive entering
debug1: do_cleanup
debug1: PAM: cleanup
debug3: PAM: sshpam_thread_cleanup entering
debug1: Killing privsep child 2653

Here are the logs for the working node

 debug1: Forked child 30048.
 Set /proc/self/oom_score_adj to 0
 debug1: rexec start in 5 out 5 newsock 5 pipe 7 sock 8
 debug1: inetd sockets after dupping: 3, 3
 Connection from 10.115.33.64 port 53628 on 10.58.105.46 port 22
 debug1: Client protocol version 2.0; client software version Nakina_SSH_1.0
 debug1: no match: Nakina_SSH_1.0

 debug1: Enabling compatibility mode for protocol 2.0
 debug1: Local version string SSH-2.0-OpenSSH_6.6.1

 debug1: SELinux support enabled [preauth]
 debug1: permanently_set_uid: 74/74 [preauth]
 debug1: list_hostkey_types: ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
 debug1: SSH2_MSG_KEXINIT sent [preauth]
 debug1: SSH2_MSG_KEXINIT received [preauth]


 debug1: kex: client->server aes256-cbc hmac-sha1 none [preauth]
 debug1: kex: server->client aes256-cbc hmac-sha1 none [preauth]
 debug1: kex: diffie-hellman-group14-sha1 need=32 dh_need=32 [preauth]
 debug1: kex: diffie-hellman-group14-sha1 need=32 dh_need=32 [preauth]
 debug1: expecting SSH2_MSG_KEXDH_INIT [preauth]

 debug1: SSH2_MSG_NEWKEYS sent [preauth]
 debug1: expecting SSH2_MSG_NEWKEYS [preauth]
 debug1: SSH2_MSG_NEWKEYS received [preauth]

 debug1: KEX done [preauth]
 debug1: userauth-request for user root service ssh-connection method none [preauth]
 debug1: attempt 0 failures 0 [preauth]
 debug1: PAM: initializing for "root"
 debug1: PAM: setting PAM_RHOST to "10.115.33.64"
 debug1: PAM: setting PAM_TTY to "ssh"
 debug1: userauth_send_banner: sent [preauth]
 debug1: userauth-request for user root service ssh-connection method password [preauth]
 debug1: attempt 1 failures 0 [preauth]
 debug1: PAM: password authentication accepted for root
 debug1: do_pam_account: called
 Accepted password for root from 10.115.33.64 port 53628 ssh2
 debug1: monitor_child_preauth: root has been authenticated by privileged process
 debug1: monitor_read_log: child log fd closed
 debug1: SELinux support enabled
 debug1: PAM: establishing credentials
 pam_unix(sshd:session): session opened for user root by (uid=0)
 debug1: Entering interactive session for SSH2.
 debug1: server_init_dispatch_20
 debug1: server_input_channel_open: ctype session rchan 0 win 131072 max 32768
 debug1: input_session_request
 debug1: channel 0: new [server-session]
 debug1: session_new: session 0
 debug1: session_open: channel 0
 debug1: session_open: session 0: link with channel 0
 debug1: server_input_channel_open: confirm session
 debug1: server_input_channel_req: channel 0 request pty-req reply 1
 debug1: session_by_channel: session 0 channel 0
 debug1: session_input_channel_req: session 0 req pty-req
 debug1: Allocating pty.
 debug1: session_pty_req: session 0 alloc /dev/pts/4
 debug1: SELinux support enabled
 debug1: server_input_channel_req: channel 0 request shell reply 1
 debug1: session_by_channel: session 0 channel 0
 debug1: session_input_channel_req: session 0 req shell
 Starting session: shell on pts/4 for root from 10.115.33.64 port 53628
 debug1: Setting controlling tty using TIOCSCTTY.
 debug1: PAM: reinitializing credentials
 debug1: permanently_set_uid: 0/0