mjhjmtu

Reading up on an answer to find out which process registered inotify watchers, I executed the following commands

echo 1 | sudo tee /sys/kernel/debug/tracing/events/syscalls/sys_exit_inotify_add_watch/enable
sudo cat /sys/kernel/debug/tracing/trace 

The resulting output from the trace file had a header that indicated that the first column should have the name of the process (task?), along with the PID:

# _-----=> irqs-off
# / _----=> need-resched
# | / _---=> hardirq/softirq
# || / _--=> preempt-depth
# ||| / delay
# TASK-PID CPU# |||| TIMESTAMP FUNCTION
# | | | |||| | |
 gmain-1715 [004] .... 23200.386116: sys_inotify_add_watch -> 0xfffffffffffffffe

Surprisingly, none of the PIDs listed in the many lines of output exist when I grep for them using ps -ef | grep $THE_PID (where $THE_PID would be 1715). Also, the task name is unknown to me and does not exist in the ps output either.

Further, the entire list is all tuples like gmain-<some number>. Not the expected postgres or node. So what is this gmain process? And what are these PIDs that do not exist in /proc/? Are they historic by the time I try accessing them?

I see gmain is a thing in these kernel docs by Theodore Ts'o on tracing, so it's something.

These are not processes. These are tasks. Linux works in terms of tasks. You do not see their IDs in a process list because these tasks happen to be threads within a process. They are the worker threads of some multi-threaded GIO process somewhere. You'll find them in the task/ subdirectory of the /proc/<process> subdirectory (example /proc/860/task/926).

As suspected, this gmain process had something to do with GTK or Gnome, but the thing to note is that it wasn't a process at all, but a thread (the gtk main loop)! That's also why it didn't show when grepping ps.

I understood this when using the -q option to ps, which lets you list the pids you are interested in. The pid that came up wasn't the one I passed as an option at all, but still showed when I did pstree -p, which made me think this was perhaps some kind of thread.

Using my newfound knowledge, I found that I could list all the threads, that also have PIDs (is this the right name?) of their own by supplying ps with the -L option.

Example: sudo ps -efL -q 906.

This (and pstree) led me to find out that the thread belonged to NetworkManager).