mjhjmtu

I have an embedded Linux system with different programs running from start-up. Some of these programs write to stdout and stderr with printf().

When I first log into the system via SSH, I can see all of these messages appear on the terminal.

However , when I open another SSH session after this, of course the processes are still running but nothing is getting wrote to that terminal. This makes me confused about how stdout and stderr actually work. I thought they would write to all terminals open?

If you're using the first SSH login to connect to "a serial debug interface", as you mentioned in the comments, then your embedded device is simply using the serial port as a console device. That's a common solution on Linux systems with no video display device at all, and actually the way classic Unix computers worked by default.

On start-up, the boot messages are output on /dev/console, and some of the started programs don't fully detach from the terminal device they've started on, and keep sending messages to it.

Technically this would be "a failure to fully daemonize" on the application's part if the application messages are unneeded, or "a half-a**ed way to implement application logging" if the messages are useful and the sysadmin is expected to write start-up scripts to direct the messages to a log file or some other useful destination. Sadly, this is relatively common. Cluttering up the console device with application messages can make it difficult for the sysadmin to use the console to troubleshoot any problems that cause the network connection to fail.

(The part below this was written before the fact that a serial debug interface was involved in that first SSH session was mentioned...)

The standard input, output and error streams are normally connected to one terminal device only. The only obvious special case would be /dev/console, which should cause the output on all physical console devices... but that does not include pseudo-TTYs, and a SSH connection will always be using a pseudo-TTY (or no TTY at all), never a "real" TTY device.

If your "programs running from start-up" are really started up at boot time, and not at your first login, then this is an interesting puzzle.

If your embedded system uses legacy (BSD-style) pseudo-TTY devices, then I think I know how that might be possible. (In other words, does your system have pairs of device nodes named like /dev/pty?? and /tty?? respectively, or does it have the modern Unix98 style PTY devices like /dev/ptmx and /dev/pts/<number>?)

At start-up, whatever script or system is managing your boot process might assign the standard output & error of all your services to the first pseudo-TTY, where some form of boot log process would capture them for logging and then pass them on to the actual console TTY device. When the boot process is complete, the boot log process ends and releases the pseudo-TTY master side. But some of your start-up programs might not be releasing their side of the pseudo-TTY, and keep sending output to it.

When your first SSH session is established, the sshd daemon gets the first unallocated pseudo-TTY master device to be used for the connection... but with the BSD-style pseudo-TTY devices, it will be the same device node that was used in the boot logging. As a result, the first SSH session will get a pseudo-TTY device that will already have the start-up programs attached to the slave side of it, and so the output of those programs will appear in the SSH session. Other SSH sessions will get a "clean" pseudo-TTY device, and so the problem will not occur with them.

With modern Unix98-style pseudo-TTY devices, each new user of PTY devices will get a guaranteed-unique PTY device pair, so this kind of thing could only happen with BSD-style PTY devices.