mjhjmtu

Is it normal for a kvm virtual machine to get more memory allocated? I'm afraid that when there is not much memory left the 'over-committed' memory can be an 'uncontrolled factor' that could cause a terrible out-of-memory killing.

I made a kvm vm using libvirt and saw weird symptoms, it gets much more virtual memory than I set it to. the 'SZ' part of 'ps' command starts with a small amount, less than 1GiB. but it can grow much bigger later with time past. and even when the 'SZ' says a small number, the output of 'numastat ' always says a big number. - I gave 1 GiB of RAM to the VM, and numastat says 1.5+ -

root@peta-0001-02:~# ps -elfww | sed -ne '1p; /qemu-system/p'
F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD
6 S oneadmin 26811 1 1 80 0 - 812801 poll_s 15:42 ? 00:00:12 /usr/bin/qemu-system-x86_64 -name one-35 -S -machine pc-i440fx-xenial,accel=kvm,usb=off,mem-merge=off -m 1024 -realtime mlock=on -smp 2,sockets=2,cores=1,threads=1 -object memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/dev/hugepages/libvirt/qemu,share=no,size=1073741824,host-nodes=0,policy=bind -numa node,nodeid=0,cpus=0-1,memdev=ram-node0 -uuid df4830f6-5a0d-42d0-b04a-192115d2512e -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-one-35/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device usb-ehci,id=usb1,bus=pci.0,addr=0x5 -device nec-usb-xhci,id=usb2,bus=pci.0,addr=0x6 -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-scsi-pci,id=scsi0,num_queues=2,bus=pci.0,addr=0x4 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x7 -drive file=/var/lib/one//datastores/104/35/disk.0,format=raw,if=none,id=drive-scsi0-0-0-0,cache=none,discard=unmap -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1 -drive file=/var/lib/one//datastores/104/35/disk.1,format=raw,if=none,id=drive-ide0-0-0,readonly=on -device ide-cd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -netdev tap,fd=28,id=hostnet0,vhost=on,vhostfd=30 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=02:00:c0:a8:04:0c,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/domain-one-35/org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -vnc 0.0.0.0:35,password -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -msg timestamp=on
0 S root 31260 27986 0 80 0 - 3755 pipe_w 15:53 pts/0 00:00:00 sed -ne 1p; /qemu-system/p
root@peta-0001-02:~# numastat 26811

Per-node process memory usage (in MBs) for PID 26811 (qemu-system-x86)
 Node 0 Total
 --------------- ---------------
Huge 1024.00 1024.00
Heap 21.88 21.88
Stack 0.13 0.13
Private 628.00 628.00
---------------- --------------- ---------------
Total 1674.01 1674.01

...
things were the same even when I had the memballoon feature disabled. -Yes, I use libvirt-

<devices> <memballoon model='none'/> </devices>

...
not using hugepages doesn't really make a big difference.

root@peta-0001-02:~# ps -elfww | sed -ne '1p; /qemu-system/p'
F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD
6 S oneadmin 3800 1 1 80 0 - 1021714 poll_s 14:44 ? 00:00:13 /usr/bin/qemu-system-x86_64 -name one-35 -S -machine pc-i440fx-xenial,accel=kvm,usb=off,mem-merge=off -m 1024 -realtime mlock=on -smp 2,sockets=2,cores=1,threads=1 -uuid 530a2e87-6426-497a-b7bb-4e0d49ed1e26 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-one-35/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device usb-ehci,id=usb1,bus=pci.0,addr=0x5 -device nec-usb-xhci,id=usb2,bus=pci.0,addr=0x6 -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-scsi-pci,id=scsi0,num_queues=2,bus=pci.0,addr=0x4 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x7 -drive file=/var/lib/one//datastores/104/35/disk.0,format=raw,if=none,id=drive-scsi0-0-0-0,cache=none,discard=unmap -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1 -drive file=/var/lib/one//datastores/104/35/disk.1,format=raw,if=none,id=drive-ide0-0-0,readonly=on -device ide-cd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -netdev tap,fd=28,id=hostnet0,vhost=on,vhostfd=30 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=02:00:c0:a8:04:0c,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/domain-one-35/org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -vnc 0.0.0.0:35,password -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 -msg timestamp=on
0 S root 11005 27986 0 80 0 - 3755 pipe_w 15:02 pts/0 00:00:00 sed -ne 1p; /qemu-system/p


root@peta-0001-02:~# numastat 3800

Per-node process memory usage (in MBs) for PID 3800 (qemu-system-x86)
 Node 0 Total
 --------------- ---------------
Huge 0.00 0.00
Heap 21.76 21.76
Stack 0.13 0.13
Private 1701.06 1701.06
---------------- --------------- ---------------
Total 1722.95 1722.95

...
The test environment was an ubuntu box.

root@peta-0001-02:~# uname -a
Linux peta-0001-02 4.10.0-42-generic #46~16.04.1-Ubuntu SMP Mon Dec 4 15:57:59 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

root@peta-0001-02:~# cat /proc/cmdline 
BOOT_IMAGE=/boot/vmlinuz-4.10.0-42-generic root=UUID=371a1314-677a-4b88-b804-b1278ff4ac6e ro isolcpus=1,2,3,8,9,10,11 hugepagesz=1G hugepages=0 hugepagesz=2M hugepages=2048

...
Here's another interesting case in a system with two NUMA nodes in which the memory over-commitment occurs at every node, thus severe over-commitment:

root@peta-0001-06:~# ps -elfww | sed -ne '1p; /qemu/p'
F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD
6 S oneadmin 24960 1 27 80 0 - 1648605 poll_s 13:51 ? 00:03:25 /usr/bin/qemu-system-x86_64 -name one-38 -S -machine pc-i440fx-xenial,accel=kvm,usb=off,mem-merge=off -m 1024 -realtime mlock=on -smp 2,sockets=2,cores=1,threads=1 -object memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/dev/hugepages/libvirt/qemu,share=no,size=1073741824,host-nodes=1,policy=bind -numa node,nodeid=0,cpus=0-1,memdev=ram-node0 -uuid 87a23172-daf3-4605-92f2-50055f9370b9 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-one-38/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device usb-ehci,id=usb1,bus=pci.0,addr=0x5 -device nec-usb-xhci,id=usb2,bus=pci.0,addr=0x6 -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-scsi-pci,id=scsi0,num_queues=2,bus=pci.0,addr=0x4 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x7 -drive file=/var/lib/one//datastores/0/38/disk.0,format=raw,if=none,id=drive-scsi0-0-0-0,cache=none,discard=unmap -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1 -drive file=/var/lib/one//datastores/0/38/disk.1,format=raw,if=none,id=drive-ide0-0-0,readonly=on -device ide-cd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -netdev tap,fd=28,id=hostnet0,vhost=on,vhostfd=30 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=02:00:c0:a8:04:0b,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/domain-one-38/org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -vnc 0.0.0.0:38,password -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -msg timestamp=on
0 R root 32103 26834 0 80 0 - 4075 - 14:04 pts/2 00:00:00 sed -ne 1p; /qemu/p

root@peta-0001-06:~# numastat 24960

Per-node process memory usage (in MBs) for PID 24960 (qemu-system-x86)
 Node 0 Node 1 Total
 --------------- --------------- ---------------
Huge 0.00 1024.00 1024.00
Heap 15.97 5.91 21.88
Stack 0.13 0.00 0.13
Private 676.95 595.42 1272.37
---------------- --------------- --------------- ---------------
Total 693.05 1625.34 2318.38

root@peta-0001-06:~# free -h
 total used free shared buff/cache available
Mem: 5.8G 5.0G 483M 104K 310M 407M