I am want to stream a linux terminal to my own program, and as far as I understand this is done by opening /dev/ptmx to start a new pts, I have tested this and this does indeed work (it creates a new file in /dev/pts). But I am not sure how I am supposed to actually read and write to this terminal. Writing directly to /dev/pts/(pts number) I just get an input/output error. Also, am I supposed to open /dev/ptmx and /dev/pts/(pts number) at the same time with the same program. Am I supposed to somehow open a shell first?

I find this stuff kinda confusing and I have not been able to find much information except for this man page http://man7.org/linux/man-pages/man4/pts.4.html

There are two distinct parts to this sort of thing.

1. As you have determined, you open the master side of the pseudo-terminal and this creates a slave-side device file that can be opened. The ptsname() library function allows something with an open file descriptor for the master side to determine this device name.


2. The same or another program, in a different process, opens the slave side, treating it exactly as real and virtual terminals are treated by the login subsystem: setting that process as a session leader; setting the slave side as the controlling terminal for the session; and setting standard input, output, and error as open file descriptors to the slave side. It then chain loads whatever interactive program is appropriate, which can indeed be a shell amongst other things.

The second part cannot proceed, on several operating systems, until the first part has called the grantpt() and unlockpt() library functions. Interlocks in the kernel prevent the slave side from being openable until these have happened on the master side.

Interestingly, these functions (which date from AT&T Unix System 5 Release 4) have proven to be unnecessary. They result from implementations where the slave side device is created owned by the wrong user account and with the wrong permissions, or even older implementations where slave side devices are persistent character device nodes (whose permissions and ownership persist from what they were last set to) and are not created on the fly, resulting in windows of opportunity in various circumstances for an attacker program run by other users to gain access to the terminal. But nowadays some operating system kernels simply give the slave side devices the appropriate ownership and permissions right from the get-go, allowing these functions to essentially be no-operations as a consequence. FreeBSD and OpenBSD both work this way nowadays. Unfortunately, despite several rumblings to this effect by kernel developers over the years, Linux is not one such kernel.

The second part is intentionally vague about exactly what different process this is. A common architecture is for the master-side process to fork(), call ptsname(), open the slave device, and close the master side file descriptor. This is what pty-run in the nosh toolset, out of which one can build tools such as ptybandage and ptyrun, does. It's how script, GNU Screen, tmux, and GUI terminal emulators such as XTerm work.

But this is not a necessary thing. As long as it knows the filename to open, the slave side process does not need to be a fork()ed child of the master side process. Indeed, in not being it does away with any need for knowledge about the master side or that the terminal is a pseudo-terminal in particular.

In my user-space virtual terminal subsystem, for example, the process running console-terminal-emulator creates a symbolic link with a known fixed name pointing to the slave device filename. Entirely separate service processes use the known fixed name to open the slave side device without need to know the exact /dev/pts/N name that the kernel happened to use each time. These service processes operate identically to similar service processes attached to kernel virtual terminals, and largely the same as service processes attached to real terminals. The slave side of a pseudo-terminal is, after all, designed to work just like the other two sorts of terminals.

Further reading

• Jonathan de Boyne Pollard (2014). pty-get-tty. nosh Guide. Softwares.


• Jonathan de Boyne Pollard (2014). pty-run. nosh Guide. Softwares.


• Jonathan de Boyne Pollard (2014). console-terminal-emulator. nosh Guide. Softwares.


• Jonathan de Boyne Pollard (2018). The head0 user-space virtual terminal. nosh Guide. Softwares.


• 
  grantpt(), unlockpt(), and ptsname(). §3. FreeBSD Library Functions Manual. 2008-08-20.


• What are the responsibilities of each Pseudo-Terminal (PTY) component (software, master side, slave side)?


• How do I come by this pty and what can I do with it?


• Mocking a pseudo tty (pts)


• Eric W. Biederman (2015-12-11).
  devpts: Sensible /dev/ptmx & force newinstance. Linux kernel mailing list.


• Eric W. Biederman (2016-04-08).
  devpts: Teach /dev/ptmx to find the associated devpts via path lookup. Linux kernel mailing list.


• Jonathan de Boyne Pollard (2016). "Daniel J. Bernstein's ptyget toolset". djbwares. Softwares.


• Jonathan de Boyne Pollard (2010). Daniel J. Bernstein on TTYs in UNIX. Frequently Given Answers.

Not so sure about what do you mean by "streaming a Linux terminal" . You didn't include any code.
You should open /dev/ptmx to get a pty master fd, and get a slave device fd (by ioctl or open, depending on the kernel version you have) then pass it to your shell, so your shell will read/write it as stdxxx.
Directly writing to a slave device will not return error, check your code again, maybe you closed your master fd.