mjhjmtu

we are using many hadoop clusters

for now we are using swap size - 16G

 free -g
 total used free shared buff/cache available
 Mem: 125 1 123 0 0 123

 Swap: 15 0 15

but as all know swap is very slow memory that actually use the disk

in some hadoop component as zookeeper server , swap should be disables!!!

but I have the feeling that in hadoop machine swap should be the min

I think I need to decrease it to 4G ,

but I want to get more opinions

You can minimize usage of swap by playing with swappiness. You can get current value with command:

cat /proc/sys/vm/swappiness

If you want to change it edit /etc/sysctl.conf and add line like

vm.swappiness = 1

(by Cloudera recommendations) and execute

sysctl -p 

(as root)

The general recommendation is: If RAM is more than 1 GB, swap size should be at least equal to the square root of the RAM size and at most double the size of RAM

One can also find a similar guideline from Redhat.

What if you simply have a system which has 512GB or more of DDR4 RAM and you install linux on one smaller disk that is 500gb, or 300gb, or 80gb?

the definition of swap (someone please correct if wrong) is it is a partition on disk that served the purpose back in the day when the system ran out of RAM (when there was only 32 MB or 1 GB of it) then the system could make use of swap to stay alive when RAM was full; back then it made sense because disk storage was often always larger, more economical, and more available than RAM and it could be easy to run out of 32 MB of RAM.

Fast forward to the 21st century, RAM is now inexpensive and home pc's can easily have 16GB or more of RAM; and servers can easily have 256GB or more of RAM; simple Dell servers (and many others) offer options up to 3TB of RAM nowadays for example. RAM can easily exceed the amount of disk space available.

My opinion, based on experience and some facts:

• any guideline for swap space is now obsolete with 21st century computing hardware.


• does your android phone (linux) have swap space? consider the computing power and performance of smart phones with 16,32,64 GB of (flash?) memory compared to laptops/pc from ? years ago [with swap space partitioned].


• you can install many linux distro's on many different kinds of storage medium


• you are not required to have a SWAP partition for linux to run


• what matters is how much RAM there is, and understanding the limits of the system based on the amount of available RAM.


• Swappiness = a kernel parameter that defines how much and how often your Linux kernel will copy RAM contents to swap. You want to let the operating system do whatever it wants all within the confines of RAM; if RAM never reaches 100% then this parameter is almost pointless since nothing will ever be written to whatever is defined as swap.


• if you have 256GB of RAM and you run out, any amount of swap is pointless at thit point... 2gb of swap on HDD or SSD does not make sense having 256GB of RAM... having N gb of swap on disk is then just a waste of space, if you have N gb of swap the system will still choke and basically become unresponsive should RAM get to 100% (it has pretty much always been this way)


• If you think you need swap, then you need more RAM. If your system does not allow for enough RAM, then you need a newer/better system that will provide enough RAM capacity.


• your question should simply be what is the right size for swap?. whether it is hadoop, linux, or windows, should be irrelevant if we agree the definition of swap as in the swap partition is some storage space that only gets used by the operating system when RAM is full.


• also look up disk caching and understand it... linux ate my ram. Always use RAM (so have as much as possible) for the best performance. Use the free command in linux and observe the value for cached in addition to used and free.


• I hate swap.

For changing the swap size you would need to first disable the swap:

swapoff (swap partition)

sudo dd if=/dev/zero of=(swap partition) bs=1M count=(newsize in MB) oflag=append conv=notrunc

mkswap (swap partition)

swapon (swap partition)

The point is that you have to always try to avoid the swap usage; usage that does not matter so much in the ammount of disk space; is true that if you have a bigger swap space, you can handle more processes, but the aim is to try to avoid the use of swap, as it will worse the general system performance.

Anyway, that does not mean that you have to disable it, it is always recommendable to have some disk space reserved for swaping for emergencies. The general recomendation is:

If RAM is more than 1 GB, swap size should be at least equal to the square root of the RAM size and at most double the size of RAM

So if it is possible try apply that rule, as long as you hard disk has enough space.