How are PIDs assigned in RHEL7?

What is the probability that a PID used before boot will be assigned to a process after the reboot?

In other words, how statistically conceivable is it - on RHEL7 - that the same PID would be assigned to a process that starts after a machine's reboot, which is identical to a PID that was used before the reboot.

Edit:

Example: before a reboot, there is daemon A with PID 544, the machine is rebooted, and after the reboot, daemon A starts, and gets assigned PID 544.

If the init system is deterministic then it is highly likely that daemons started by that init system will start with the same pid across reboots, as the same code will be run each boot (though with randomization from the occasional fsck or selinux relabels or other such not-on-every-boot code).

However, systemd on RHEL7 is not deterministic: PIDs are allocated in sequence but systemd runs tasks in parallel possibly across multiple cores possibly waiting for various hardware or network services. Let's see what happens before and after to the process list:

$ ps axo pid,command | sort -n > before
$ sudo reboot
...
$ ps axo pid,command | sort -n > after
$ comm -12 before after | grep -v \[
comm: file 1 is not in sorted order
comm: file 2 is not in sorted order
 PID COMMAND
 1 /usr/lib/systemd/systemd --switched-root --system --deserialize 21
 745 /usr/lib/systemd/systemd-logind
 864 /usr/sbin/NetworkManager --no-daemon
$ 

So that's two processes (excepting the obvious not-randomized "PID eins", and kernel threads marked with [) with the same PID. Across 30 reboots recording much the same information, it appears systemd is pretty good at randomizing the pids; during those reboots /usr/lib/systemd/systemd-logind came up at:

PID PERCENTAGE
733 5%
734 5%
737 15%
739 5%
743 5%
746 5%
748 5%
749 5%
752 10%
753 10%
755 5%
758 5%
760 5%
764 5%
771 5%
773 5%

The data was captured with a startup service:

[Unit]
Description=recordpidorder
After=getty.target

[Service]
Type=oneshot
ExecStart=/root/pidandboots

[Install]
WantedBy=multi-user.target

that ran:

#!/bin/bash
NF=/root/sequence
[[ ! -e "$NF" ]] && echo 0 > "$NF"
CUR=$(( 1 + $(< "$NF") ))
ps haxo pid,command > "/root/pidorder$CUR"
[[ $CUR -gt 30 ]] && mv /root/pidandboots /root/pidandboots.done
echo "$CUR" > "$NF"
reboot

Once the system is up and running process creation order will randomize itself as not-at-@reboot cron jobs fire, users login and run various different commands, etc. This will depend on the system, how many PIDs are created on it, etc.

So yes it is statistically conceivable that a daemon will come up at the same PID on a RedHat system that uses systemd. However, the odds will vary depending on the hardware and exact services the system offers.

Short answer

100%

(I am not suggesting that it will be the same image, though init is always PID=1. Just that after a process dies, at some point its PID will be reused. )

Longer answer

They can be reused before reboot. They count up until all possible PIDs are used then start again, avoiding the ones in use.

At reboot they restart at 1 (the process init is always 1).

This is a general answer that should be true for ALL Unixs.

There is a 100% certainty that a given PID will be reused after a reboot, but there is no guarantee that a given program will get the same PID after a reboot that it had before the reboot (and in most cases, it won't).

Linux, like other UNIX systems, uses a dead-simple method for figuring out what PID a newly created process has, it just grabs the next one in numerical order after the PID used for the last process started that isn't in use. When it hits the cap specified by /proc/sys/kernel/pid_max, it starts at 1 again. Note that this sysctl is user tunable, but from a practical perspective it just limits the total number of processes on the system.

Now, there are a couple of specific exceptions to this rule. On boot, whatever your system uses as init (systemd by default on RHEL, but you can specify an arbitrary binary on the kernel command line) is always started as PID 1. On Linux specifically, PID 2 is always kthreadd, which is the core kernel process responsible for starting all other kernel threads, and a lot of kernel threads that get started very early will often consistently have the same PID across reboots too (for example, PID 4 will almost always be the softirq handler thread for CPU 0 on x86 systems), but this is dependent on the hardware configuration of the system.