mjhjmtu

I am using SSH tunnels launched via autossh at boot on a device installed at remote locations. Up to now everything was working fine but yesterday the connection partly went down and autossh wouldn't do its thing. And I would love to figure out what happened to prevent it ever happening again. I have to deal with SLAs and all.

Here are the details:

THE SETUP

Remote site:

-Raspbian on a raspberry pi (debian wheezy)
-Autossh running at boot connects to our server and establishes 3 ssh tunnels
Tunnel 1: regular tunnel forwarding a local port on the raspberry pi to a server in our office
Tunnel 2: reverse tunnel opens a port on our server pointing to a device on the remote network
Tunnel 3: reverse tunnel opens a port from our server's loopback (127.0.0.1) adapter pointing to the ssh port of the raspberry pi on its loopback adapter.

So basically with this setup I have to ssh in our server from which I can ssh into each remote devices.

SSH Server:

-Debian Wheezy
-No firewall/iptables on the server, we use another box for this

THE PROBLEM

Both reverse tunnels (tunnels 2 and 3) stopped working. The output of

sudo netstat -atp | grep ssh

Showed that the only port still opened was for tunnel # 1

When I killed the ssh process on the server for tunnel # 1 autossh did its thing and all tunnels were reconnected but I would prefer if I didn't have to do this.

Is it normal behavior for autossh? Does autossh have issues monitoring reverse tunnels?

SYSLOG from the remote device

Sep 7 06:25:06 remote_hostname rsyslogd: [origin software="rsyslogd" swVersion="5.8.11" x-pid="1661" x-info="http://www.rsyslog.com"] rsyslogd was HUPed
Sep 8 06:25:05 remote_hostname rsyslogd: [origin software="rsyslogd" swVersion="5.8.11" x-pid="1661" x-info="http://www.rsyslog.com"] rsyslogd was HUPed
Sep 8 10:24:57 remote_hostname autossh[1428]: ssh exited with error status 255; restarting ssh
Sep 8 10:24:57 remote_hostname autossh[1428]: starting ssh (count 2)
Sep 8 10:24:57 remote_hostname autossh[1428]: ssh child pid is 24161
Sep 8 16:53:53 remote_hostname autossh[1428]: ssh exited with error status 255; restarting ssh
Sep 8 16:53:53 remote_hostname autossh[1428]: starting ssh (count 3)
Sep 8 16:53:53 remote_hostname autossh[1428]: ssh child pid is 30541
Sep 9 06:25:05 remote_hostname rsyslogd: [origin software="rsyslogd" swVersion="5.8.11" x-pid="1661" x-info="http://www.rsyslog.com"] rsyslogd was HUPed

You are using various ways to create log output,

but do not mention the most important ones:

The command ssh itself has a "verbose" option -v.

And more interesting, it also has a "verry verbose" option -vv.

Oh, and "verry verry verbose", -vvv, which may actually be too verbose!

Does it give more insight to run ssh with options -v, -vv, or -vvv?

Showing how much verbosity the options enable:

$ ssh -v localhost true |& wc -l 
56
$ ssh -vv localhost true |& wc -l
122
$ ssh -vvv localhost true |& wc -l
282