mjhjmtu

What happens if a Linux, let’s say Arch Linux or Debian, is installed with no swap partition or swap file. Then, when running the OS while almost out of RAM, the user opens a new application. Considering that this new application needs more RAM memory than what’s needed, what will happen?

What part of the operating system handles RAM management operations, and can I configure it to behave differently?

The Linux kernel has a component called the OOM killer (out of memory). As Patrick pointed out in the comments the OOM killer can be disabled but the default setting is to allow overcommit (and thus enable the OOM killer).

Applications ask the kernel for more memory and the kernel can refuse to give it to them (because there is not enough memory or because ulimit has been used to deny more memory to the process). If overcommit is enabled then an application has asked for some memory and was granted the amount but if the application writes to a new memory page (for the first time) and the kernel actually has to allocate memory for this but cannot do that then the kernel has to decide which process to kill in order to free memory.

The kernel will rather kill new processes than old ones, especially those who (together with their children) consume much memory. So in your case the new process might start but would probably be the one which gets killed.

You can use the files

/proc/self/oom_adj
/proc/self/oom_score
/proc/self/oom_score_adj

to check the current settings and to tell the kernel in which order it shall kill processes if necessary.

Keep in mind that the kernel can toss out text pages to make more room in memory. Text pages are memory that contains executable code. This can cause a kind of thrashing, as pages are tossed out and subsequently loaded from executables on disk.

Hauke Laging’s answer focuses on the “out of memory” processes killer,
and barely mentions the fact that things can simply fail. 
You mention a scenario in which “the user opens a new application”. 
The standard mechanism for “a new application” being opened
is for some process (typically a shell or a window manager) to “fork”,
to create a new process that is a copy of the existing one,
and for the new process to call “exec”,
to replace itself with the new program. 
Either can fail for lack of memory.

From fork(2):

ERRORS

^{(i.e., list of possible failure conditions)}

   ︙



ENOMEM



fork() failed to allocate the necessary kernel structures
because memory is tight.

   ︙

And similarly execve(2):

‍

   ︙



ENOMEM



Insufficient kernel memory was available.

   ︙

This Google search shows other system functions
that can return the “ENOMEM” error.

So it’s possible that you (the user) will simply get an error message
when you try to start a new program running,
or otherwise initiate a new activity, when memory is low.