Debian Server: Out of Memory: kill process [closed]
Clash Royale CLAN TAG#URR8PPP
up vote
1
down vote
favorite
I run a debian Webserver with nextcloud. Every night a backup job (storeBackup) is scheduled and it only succeeds in very few cases.
But every night the journal is full of OOM Errors (see below).
The machine has 16G RAM, therefore I am wondering, what is so much RAM consuming.
Output of journalctl -p err -e for one night:
Jun 21 03:28:01 balkonien kernel: Out of memory: Kill process 4723 (clamd) score 19 or sacrifice child
Jun 21 03:28:01 balkonien kernel: Killed process 4723 (clamd) total-vm:760960kB, anon-rss:49044kB, file-rss:6504kB, shmem-rss:0kB
Jun 21 03:35:00 balkonien kernel: Out of memory: Kill process 22111 (forkit) score 12 or sacrifice child
Jun 21 03:35:00 balkonien kernel: Killed process 22139 (loolkit) total-vm:1488072kB, anon-rss:53768kB, file-rss:0kB, shmem-rss:0kB
Jun 21 03:38:11 balkonien kernel: Out of memory: Kill process 8246 (loolkit) score 13 or sacrifice child
Jun 21 03:38:11 balkonien kernel: Killed process 8246 (loolkit) total-vm:1488072kB, anon-rss:58388kB, file-rss:25816kB, shmem-rss:0kB
Jun 21 03:40:49 balkonien kernel: Out of memory: Kill process 10165 (loolkit) score 13 or sacrifice child
Jun 21 03:40:49 balkonien kernel: Killed process 10165 (loolkit) total-vm:1488072kB, anon-rss:58336kB, file-rss:26224kB, shmem-rss:0kB
Jun 21 03:47:38 balkonien kernel: Out of memory: Kill process 12575 (loolkit) score 13 or sacrifice child
Jun 21 03:47:38 balkonien kernel: Killed process 12575 (loolkit) total-vm:1488072kB, anon-rss:58212kB, file-rss:26224kB, shmem-rss:0kB
Jun 21 03:47:39 balkonien kernel: Out of memory: Kill process 22111 (forkit) score 12 or sacrifice child
Jun 21 03:47:39 balkonien kernel: Killed process 22111 (forkit) total-vm:1487792kB, anon-rss:53880kB, file-rss:2080kB, shmem-rss:0kB
Jun 21 03:48:45 balkonien kernel: Out of memory: Kill process 16652 (forkit) score 17 or sacrifice child
Jun 21 03:48:45 balkonien kernel: Killed process 16783 (loolkit) total-vm:1488068kB, anon-rss:350976kB, file-rss:27500kB, shmem-rss:0kB
Jun 21 04:07:00 balkonien kernel: Out of memory: Kill process 16652 (forkit) score 12 or sacrifice child
Jun 21 04:07:00 balkonien kernel: Killed process 16801 (loolkit) total-vm:1488068kB, anon-rss:349752kB, file-rss:7536kB, shmem-rss:0kB
Jun 21 04:07:00 balkonien kernel: Out of memory: Kill process 16652 (forkit) score 12 or sacrifice child
Jun 21 04:07:00 balkonien kernel: Killed process 16652 (forkit) total-vm:1487720kB, anon-rss:349536kB, file-rss:18508kB, shmem-rss:0kB
Jun 21 04:49:00 balkonien kernel: Out of memory: Kill process 27040 (forkit) score 12 or sacrifice child
Jun 21 04:49:00 balkonien kernel: Killed process 27082 (loolkit) total-vm:1488068kB, anon-rss:102080kB, file-rss:428kB, shmem-rss:0kB
Jun 21 04:49:00 balkonien kernel: Out of memory: Kill process 27040 (forkit) score 12 or sacrifice child
Jun 21 04:49:00 balkonien kernel: Killed process 27088 (loolkit) total-vm:1488068kB, anon-rss:101984kB, file-rss:416kB, shmem-rss:0kB
Jun 21 04:56:00 balkonien kernel: Out of memory: Kill process 12165 (loolkit) score 12 or sacrifice child
Jun 21 04:56:00 balkonien kernel: Killed process 12165 (loolkit) total-vm:1488068kB, anon-rss:100852kB, file-rss:7140kB, shmem-rss:0kB
Jun 21 04:56:00 balkonien kernel: Out of memory: Kill process 27040 (forkit) score 12 or sacrifice child
Jun 21 04:56:00 balkonien kernel: Killed process 27040 (forkit) total-vm:1487720kB, anon-rss:99072kB, file-rss:60kB, shmem-rss:0kB
Jun 21 05:07:01 balkonien kernel: Out of memory: Kill process 13137 (forkit) score 12 or sacrifice child
Jun 21 05:07:01 balkonien kernel: Killed process 13170 (loolkit) total-vm:1488068kB, anon-rss:348584kB, file-rss:840kB, shmem-rss:0kB
Jun 21 05:09:00 balkonien kernel: Out of memory: Kill process 13137 (forkit) score 12 or sacrifice child
Jun 21 05:09:00 balkonien kernel: Killed process 13174 (loolkit) total-vm:1488068kB, anon-rss:340688kB, file-rss:0kB, shmem-rss:0kB
Jun 21 05:09:43 balkonien kernel: Out of memory: Kill process 14902 (loolkit) score 13 or sacrifice child
Jun 21 05:09:43 balkonien kernel: Killed process 14902 (loolkit) total-vm:1488068kB, anon-rss:340476kB, file-rss:26308kB, shmem-rss:0kB
Jun 21 05:54:52 balkonien kernel: Out of memory: Kill process 15401 (loolkit) score 12 or sacrifice child
Jun 21 05:54:52 balkonien kernel: Killed process 15401 (loolkit) total-vm:1488068kB, anon-rss:295604kB, file-rss:1080kB, shmem-rss:0kB
Jun 21 05:56:00 balkonien kernel: Out of memory: Kill process 15519 (loolkit) score 13 or sacrifice child
Jun 21 05:56:00 balkonien kernel: Killed process 15519 (loolkit) total-vm:1488068kB, anon-rss:292004kB, file-rss:25992kB, shmem-rss:0kB
Jun 21 05:56:00 balkonien kernel: Out of memory: Kill process 13137 (forkit) score 12 or sacrifice child
Jun 21 05:56:00 balkonien kernel: Killed process 13137 (forkit) total-vm:1487720kB, anon-rss:291656kB, file-rss:1272kB, shmem-rss:0kB
Jun 21 05:56:07 balkonien kernel: Out of memory: Kill process 27684 (perl) score 10 or sacrifice child
Jun 21 05:56:07 balkonien kernel: Killed process 27688 (perl) total-vm:34744kB, anon-rss:4820kB, file-rss:3848kB, shmem-rss:0kB
Jun 21 05:56:07 balkonien kernel: Out of memory: Kill process 27684 (perl) score 10 or sacrifice child
Jun 21 05:56:07 balkonien kernel: Killed process 27703 (perl) total-vm:34728kB, anon-rss:5364kB, file-rss:3812kB, shmem-rss:0kB
Jun 21 05:56:15 balkonien kernel: Out of memory: Kill process 16287 (forkit) score 13 or sacrifice child
Jun 21 05:56:15 balkonien kernel: Killed process 16287 (forkit) total-vm:1381924kB, anon-rss:248144kB, file-rss:129808kB, shmem-rss:0kB
Jun 21 05:58:18 balkonien kernel: Out of memory: Kill process 16503 (forkit) score 16 or sacrifice child
Jun 21 05:58:18 balkonien kernel: Killed process 16655 (loolkit) total-vm:1488068kB, anon-rss:348072kB, file-rss:25648kB, shmem-rss:0kB
Jun 21 05:59:18 balkonien kernel: Out of memory: Kill process 16503 (forkit) score 16 or sacrifice child
Jun 21 05:59:18 balkonien kernel: Killed process 16675 (loolkit) total-vm:1488068kB, anon-rss:345868kB, file-rss:24916kB, shmem-rss:0kB
Jun 21 06:07:00 balkonien kernel: Out of memory: Kill process 16503 (forkit) score 12 or sacrifice child
Jun 21 06:07:00 balkonien kernel: Killed process 17546 (loolkit) total-vm:1488068kB, anon-rss:342780kB, file-rss:3176kB, shmem-rss:0kB
Jun 21 06:07:00 balkonien kernel: Out of memory: Kill process 16503 (forkit) score 12 or sacrifice child
Jun 21 06:07:00 balkonien kernel: Killed process 16503 (forkit) total-vm:1487720kB, anon-rss:342536kB, file-rss:3128kB, shmem-rss:0kB
Jun 21 06:07:05 balkonien kernel: Out of memory: Kill process 27684 (perl) score 10 or sacrifice child
Jun 21 06:07:05 balkonien kernel: Killed process 27702 (bzip2) total-vm:13624kB, anon-rss:6392kB, file-rss:952kB, shmem-rss:0kB
Jun 21 06:08:45 balkonien systemd[1]: Failed to start systemd-cron daily script service.
Output of ps faux (regarding the backup task):
samuel 9458 0.0 0.0 26016 2212 ? Ss Jun16 0:00 SCREEN -R
samuel 9459 0.0 0.0 25516 5812 pts/2 Ss Jun16 0:00 _ /bin/bash
root 14092 0.0 0.0 45336 3804 pts/2 S+ 12:08 0:00 _ sudo ./storeBackup
root 14093 0.0 0.0 4312 672 pts/2 S+ 12:08 0:00 _ /bin/sh ./storeBackup
root 14099 38.4 1.9 369208 306664 pts/2 SN+ 12:08 33:13 _ perl /usr/local/bin/storeBackup.pl -f /etc/storeBackup/storebackup.conf
root 14103 0.0 0.0 34752 7432 pts/2 SN+ 12:08 0:00 _ perl /opt/storeBackup/lib/stbuLog.pl --readFile /tmp/prLog-54a4c1c781f71013 --out /var/log/storeBackup/storeBackup.log --maxFilelen 1000000 --noOfOldFiles 5
root 14116 1.6 0.0 13624 7500 pts/2 SN+ 12:08 1:22 _ bzip2
root 14117 0.0 0.0 34732 7328 pts/2 SN+ 12:08 0:00 _ perl /opt/storeBackup/lib/stbuLog.pl --readFile /tmp/prLog-bd0ccc7b5ac7d895 --out /media/Backup/storeBackup/default/2018.06.21_12.08.27/.storeBackup.log --maxFilelen 0 --noOfOldFiles 1
Output of iotop (first lines):
Total DISK READ : 7.06 K/s | Total DISK WRITE : 0.00 B/s
Actual DISK READ: 84.75 K/s | Actual DISK WRITE: 0.00 B/s
TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
14099 be/7 root 7.06 K/s 0.00 B/s 0.00 % 0.06 % perl /usr/local/bin/storeBackup.pl -f /etc/storeBackup/storebackup.conf
1 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % init
2 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kthreadd]
3 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [ksoftirqd/0]
5 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kworker/0:0H]
Output of top (first lines):
top - 13:38:35 up 5 days, 6:38, 1 user, load average: 1,00, 1,15, 1,22
Tasks: 592 total, 2 running, 589 sleeping, 0 stopped, 1 zombie
%Cpu(s): 0,5 us, 25,7 sy, 1,5 ni, 71,8 id, 0,2 wa, 0,0 hi, 0,2 si, 0,0 st
KiB Mem: 15908456 total, 15733624 used, 174832 free, 2555436 buffers
KiB Swap: 12695892 total, 263052 used, 12432840 free. 10882640 cached Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
14099 root 38 18 369208 306664 4920 R 44,5 1,9 34:54.76 perl
47 root 20 0 0 0 0 S 3,3 0,0 67:54.66 kswapd0
18400 root 20 0 26224 3432 2568 R 2,0 0,0 0:00.21 top
7 root 20 0 0 0 0 S 1,6 0,0 29:58.23 rcu_sched
Output of free -h:
total used free shared buffers cached
Mem: 15G 14G 214M 18M 2,4G 10G
-/+ buffers/cache: 2,2G 12G
Swap: 12G 256M 11G
linux debian ram out-of-memory
asked Jun 21 at 7:18
Sammy
136
closed as too broad by Ipor Sircer, schily, Kusalananda, Thomas Nyman, slm♦ Jun 22 at 0:37
Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.
 |Â
show 2 more comments
up vote
1
down vote
favorite
I run a debian Webserver with nextcloud. Every night a backup job (storeBackup) is scheduled and it only succeeds in very few cases.
But every night the journal is full of OOM Errors (see below).
The machine has 16G RAM, therefore I am wondering, what is so much RAM consuming.
Output of journalctl -p err -e for one night:
Jun 21 03:28:01 balkonien kernel: Out of memory: Kill process 4723 (clamd) score 19 or sacrifice child
Jun 21 03:28:01 balkonien kernel: Killed process 4723 (clamd) total-vm:760960kB, anon-rss:49044kB, file-rss:6504kB, shmem-rss:0kB
Jun 21 03:35:00 balkonien kernel: Out of memory: Kill process 22111 (forkit) score 12 or sacrifice child
Jun 21 03:35:00 balkonien kernel: Killed process 22139 (loolkit) total-vm:1488072kB, anon-rss:53768kB, file-rss:0kB, shmem-rss:0kB
Jun 21 03:38:11 balkonien kernel: Out of memory: Kill process 8246 (loolkit) score 13 or sacrifice child
Jun 21 03:38:11 balkonien kernel: Killed process 8246 (loolkit) total-vm:1488072kB, anon-rss:58388kB, file-rss:25816kB, shmem-rss:0kB
Jun 21 03:40:49 balkonien kernel: Out of memory: Kill process 10165 (loolkit) score 13 or sacrifice child
Jun 21 03:40:49 balkonien kernel: Killed process 10165 (loolkit) total-vm:1488072kB, anon-rss:58336kB, file-rss:26224kB, shmem-rss:0kB
Jun 21 03:47:38 balkonien kernel: Out of memory: Kill process 12575 (loolkit) score 13 or sacrifice child
Jun 21 03:47:38 balkonien kernel: Killed process 12575 (loolkit) total-vm:1488072kB, anon-rss:58212kB, file-rss:26224kB, shmem-rss:0kB
Jun 21 03:47:39 balkonien kernel: Out of memory: Kill process 22111 (forkit) score 12 or sacrifice child
Jun 21 03:47:39 balkonien kernel: Killed process 22111 (forkit) total-vm:1487792kB, anon-rss:53880kB, file-rss:2080kB, shmem-rss:0kB
Jun 21 03:48:45 balkonien kernel: Out of memory: Kill process 16652 (forkit) score 17 or sacrifice child
Jun 21 03:48:45 balkonien kernel: Killed process 16783 (loolkit) total-vm:1488068kB, anon-rss:350976kB, file-rss:27500kB, shmem-rss:0kB
Jun 21 04:07:00 balkonien kernel: Out of memory: Kill process 16652 (forkit) score 12 or sacrifice child
Jun 21 04:07:00 balkonien kernel: Killed process 16801 (loolkit) total-vm:1488068kB, anon-rss:349752kB, file-rss:7536kB, shmem-rss:0kB
Jun 21 04:07:00 balkonien kernel: Out of memory: Kill process 16652 (forkit) score 12 or sacrifice child
Jun 21 04:07:00 balkonien kernel: Killed process 16652 (forkit) total-vm:1487720kB, anon-rss:349536kB, file-rss:18508kB, shmem-rss:0kB
Jun 21 04:49:00 balkonien kernel: Out of memory: Kill process 27040 (forkit) score 12 or sacrifice child
Jun 21 04:49:00 balkonien kernel: Killed process 27082 (loolkit) total-vm:1488068kB, anon-rss:102080kB, file-rss:428kB, shmem-rss:0kB
Jun 21 04:49:00 balkonien kernel: Out of memory: Kill process 27040 (forkit) score 12 or sacrifice child
Jun 21 04:49:00 balkonien kernel: Killed process 27088 (loolkit) total-vm:1488068kB, anon-rss:101984kB, file-rss:416kB, shmem-rss:0kB
Jun 21 04:56:00 balkonien kernel: Out of memory: Kill process 12165 (loolkit) score 12 or sacrifice child
Jun 21 04:56:00 balkonien kernel: Killed process 12165 (loolkit) total-vm:1488068kB, anon-rss:100852kB, file-rss:7140kB, shmem-rss:0kB
Jun 21 04:56:00 balkonien kernel: Out of memory: Kill process 27040 (forkit) score 12 or sacrifice child
Jun 21 04:56:00 balkonien kernel: Killed process 27040 (forkit) total-vm:1487720kB, anon-rss:99072kB, file-rss:60kB, shmem-rss:0kB
Jun 21 05:07:01 balkonien kernel: Out of memory: Kill process 13137 (forkit) score 12 or sacrifice child
Jun 21 05:07:01 balkonien kernel: Killed process 13170 (loolkit) total-vm:1488068kB, anon-rss:348584kB, file-rss:840kB, shmem-rss:0kB
Jun 21 05:09:00 balkonien kernel: Out of memory: Kill process 13137 (forkit) score 12 or sacrifice child
Jun 21 05:09:00 balkonien kernel: Killed process 13174 (loolkit) total-vm:1488068kB, anon-rss:340688kB, file-rss:0kB, shmem-rss:0kB
Jun 21 05:09:43 balkonien kernel: Out of memory: Kill process 14902 (loolkit) score 13 or sacrifice child
Jun 21 05:09:43 balkonien kernel: Killed process 14902 (loolkit) total-vm:1488068kB, anon-rss:340476kB, file-rss:26308kB, shmem-rss:0kB
Jun 21 05:54:52 balkonien kernel: Out of memory: Kill process 15401 (loolkit) score 12 or sacrifice child
Jun 21 05:54:52 balkonien kernel: Killed process 15401 (loolkit) total-vm:1488068kB, anon-rss:295604kB, file-rss:1080kB, shmem-rss:0kB
Jun 21 05:56:00 balkonien kernel: Out of memory: Kill process 15519 (loolkit) score 13 or sacrifice child
Jun 21 05:56:00 balkonien kernel: Killed process 15519 (loolkit) total-vm:1488068kB, anon-rss:292004kB, file-rss:25992kB, shmem-rss:0kB
Jun 21 05:56:00 balkonien kernel: Out of memory: Kill process 13137 (forkit) score 12 or sacrifice child
Jun 21 05:56:00 balkonien kernel: Killed process 13137 (forkit) total-vm:1487720kB, anon-rss:291656kB, file-rss:1272kB, shmem-rss:0kB
Jun 21 05:56:07 balkonien kernel: Out of memory: Kill process 27684 (perl) score 10 or sacrifice child
Jun 21 05:56:07 balkonien kernel: Killed process 27688 (perl) total-vm:34744kB, anon-rss:4820kB, file-rss:3848kB, shmem-rss:0kB
Jun 21 05:56:07 balkonien kernel: Out of memory: Kill process 27684 (perl) score 10 or sacrifice child
Jun 21 05:56:07 balkonien kernel: Killed process 27703 (perl) total-vm:34728kB, anon-rss:5364kB, file-rss:3812kB, shmem-rss:0kB
Jun 21 05:56:15 balkonien kernel: Out of memory: Kill process 16287 (forkit) score 13 or sacrifice child
Jun 21 05:56:15 balkonien kernel: Killed process 16287 (forkit) total-vm:1381924kB, anon-rss:248144kB, file-rss:129808kB, shmem-rss:0kB
Jun 21 05:58:18 balkonien kernel: Out of memory: Kill process 16503 (forkit) score 16 or sacrifice child
Jun 21 05:58:18 balkonien kernel: Killed process 16655 (loolkit) total-vm:1488068kB, anon-rss:348072kB, file-rss:25648kB, shmem-rss:0kB
Jun 21 05:59:18 balkonien kernel: Out of memory: Kill process 16503 (forkit) score 16 or sacrifice child
Jun 21 05:59:18 balkonien kernel: Killed process 16675 (loolkit) total-vm:1488068kB, anon-rss:345868kB, file-rss:24916kB, shmem-rss:0kB
Jun 21 06:07:00 balkonien kernel: Out of memory: Kill process 16503 (forkit) score 12 or sacrifice child
Jun 21 06:07:00 balkonien kernel: Killed process 17546 (loolkit) total-vm:1488068kB, anon-rss:342780kB, file-rss:3176kB, shmem-rss:0kB
Jun 21 06:07:00 balkonien kernel: Out of memory: Kill process 16503 (forkit) score 12 or sacrifice child
Jun 21 06:07:00 balkonien kernel: Killed process 16503 (forkit) total-vm:1487720kB, anon-rss:342536kB, file-rss:3128kB, shmem-rss:0kB
Jun 21 06:07:05 balkonien kernel: Out of memory: Kill process 27684 (perl) score 10 or sacrifice child
Jun 21 06:07:05 balkonien kernel: Killed process 27702 (bzip2) total-vm:13624kB, anon-rss:6392kB, file-rss:952kB, shmem-rss:0kB
Jun 21 06:08:45 balkonien systemd[1]: Failed to start systemd-cron daily script service.
Output of ps faux (regarding the backup task):
samuel 9458 0.0 0.0 26016 2212 ? Ss Jun16 0:00 SCREEN -R
samuel 9459 0.0 0.0 25516 5812 pts/2 Ss Jun16 0:00 _ /bin/bash
root 14092 0.0 0.0 45336 3804 pts/2 S+ 12:08 0:00 _ sudo ./storeBackup
root 14093 0.0 0.0 4312 672 pts/2 S+ 12:08 0:00 _ /bin/sh ./storeBackup
root 14099 38.4 1.9 369208 306664 pts/2 SN+ 12:08 33:13 _ perl /usr/local/bin/storeBackup.pl -f /etc/storeBackup/storebackup.conf
root 14103 0.0 0.0 34752 7432 pts/2 SN+ 12:08 0:00 _ perl /opt/storeBackup/lib/stbuLog.pl --readFile /tmp/prLog-54a4c1c781f71013 --out /var/log/storeBackup/storeBackup.log --maxFilelen 1000000 --noOfOldFiles 5
root 14116 1.6 0.0 13624 7500 pts/2 SN+ 12:08 1:22 _ bzip2
root 14117 0.0 0.0 34732 7328 pts/2 SN+ 12:08 0:00 _ perl /opt/storeBackup/lib/stbuLog.pl --readFile /tmp/prLog-bd0ccc7b5ac7d895 --out /media/Backup/storeBackup/default/2018.06.21_12.08.27/.storeBackup.log --maxFilelen 0 --noOfOldFiles 1
Output of iotop (first lines):
Total DISK READ : 7.06 K/s | Total DISK WRITE : 0.00 B/s
Actual DISK READ: 84.75 K/s | Actual DISK WRITE: 0.00 B/s
TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
14099 be/7 root 7.06 K/s 0.00 B/s 0.00 % 0.06 % perl /usr/local/bin/storeBackup.pl -f /etc/storeBackup/storebackup.conf
1 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % init
2 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kthreadd]
3 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [ksoftirqd/0]
5 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kworker/0:0H]
Output of top (first lines):
top - 13:38:35 up 5 days, 6:38, 1 user, load average: 1,00, 1,15, 1,22
Tasks: 592 total, 2 running, 589 sleeping, 0 stopped, 1 zombie
%Cpu(s): 0,5 us, 25,7 sy, 1,5 ni, 71,8 id, 0,2 wa, 0,0 hi, 0,2 si, 0,0 st
KiB Mem: 15908456 total, 15733624 used, 174832 free, 2555436 buffers
KiB Swap: 12695892 total, 263052 used, 12432840 free. 10882640 cached Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
14099 root 38 18 369208 306664 4920 R 44,5 1,9 34:54.76 perl
47 root 20 0 0 0 0 S 3,3 0,0 67:54.66 kswapd0
18400 root 20 0 26224 3432 2568 R 2,0 0,0 0:00.21 top
7 root 20 0 0 0 0 S 1,6 0,0 29:58.23 rcu_sched
Output of free -h:
total used free shared buffers cached
Mem: 15G 14G 214M 18M 2,4G 10G
-/+ buffers/cache: 2,2G 12G
Swap: 12G 256M 11G
linux debian ram out-of-memory
asked Jun 21 at 7:18
Sammy
136
closed as too broad by Ipor Sircer, schily, Kusalananda, Thomas Nyman, slm♦ Jun 22 at 0:37
Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.
instead of long useless log, you could had the result of the commandps faux,iotoptopfree -hthat would help way more than your log. Also obviously your backup process eat all the capacity of the system.
– Kiwy
Jun 21 at 7:53
@Kiwy it's not useless as it tells exactly what causes the problem. OOM killer is not a random magic - every process in linux has its calculated score and OOM killer pick the best one to kill. Simplifying is most likely it'll kill the one which consumes biggest amount of memory. More at this topic
– pawel7318
Jun 21 at 8:03
@pawel7318 it still doesn't inform you precissely of what is the cause for your problem, another program than the one causing the issue can be terminated. I already had the case several time. To me it's just an indication that something went wrong at that moment. make a conclusion on those log, is IMO not relevant, i'll read the doc you post though.
– Kiwy
Jun 21 at 8:09
@Kiwy I added the outputs accordingly. Right now the backup script is only running for ten minutes. I can update the outputs in an hour, if it's neccessary. Or is it clear that the backup script is the cause and I should contact them?
– Sammy
Jun 21 at 10:19
is the problem is currently happening ? because you have 11GB of cached memory that could be freed by the system before shooting processes.
– Kiwy
Jun 21 at 10:32
 |Â
show 2 more comments
up vote
1
down vote
favorite
up vote
1
down vote
favorite
I run a debian Webserver with nextcloud. Every night a backup job (storeBackup) is scheduled and it only succeeds in very few cases.
But every night the journal is full of OOM Errors (see below).
The machine has 16G RAM, therefore I am wondering, what is so much RAM consuming.
Output of journalctl -p err -e for one night:
Jun 21 03:28:01 balkonien kernel: Out of memory: Kill process 4723 (clamd) score 19 or sacrifice child
Jun 21 03:28:01 balkonien kernel: Killed process 4723 (clamd) total-vm:760960kB, anon-rss:49044kB, file-rss:6504kB, shmem-rss:0kB
Jun 21 03:35:00 balkonien kernel: Out of memory: Kill process 22111 (forkit) score 12 or sacrifice child
Jun 21 03:35:00 balkonien kernel: Killed process 22139 (loolkit) total-vm:1488072kB, anon-rss:53768kB, file-rss:0kB, shmem-rss:0kB
Jun 21 03:38:11 balkonien kernel: Out of memory: Kill process 8246 (loolkit) score 13 or sacrifice child
Jun 21 03:38:11 balkonien kernel: Killed process 8246 (loolkit) total-vm:1488072kB, anon-rss:58388kB, file-rss:25816kB, shmem-rss:0kB
Jun 21 03:40:49 balkonien kernel: Out of memory: Kill process 10165 (loolkit) score 13 or sacrifice child
Jun 21 03:40:49 balkonien kernel: Killed process 10165 (loolkit) total-vm:1488072kB, anon-rss:58336kB, file-rss:26224kB, shmem-rss:0kB
Jun 21 03:47:38 balkonien kernel: Out of memory: Kill process 12575 (loolkit) score 13 or sacrifice child
Jun 21 03:47:38 balkonien kernel: Killed process 12575 (loolkit) total-vm:1488072kB, anon-rss:58212kB, file-rss:26224kB, shmem-rss:0kB
Jun 21 03:47:39 balkonien kernel: Out of memory: Kill process 22111 (forkit) score 12 or sacrifice child
Jun 21 03:47:39 balkonien kernel: Killed process 22111 (forkit) total-vm:1487792kB, anon-rss:53880kB, file-rss:2080kB, shmem-rss:0kB
Jun 21 03:48:45 balkonien kernel: Out of memory: Kill process 16652 (forkit) score 17 or sacrifice child
Jun 21 03:48:45 balkonien kernel: Killed process 16783 (loolkit) total-vm:1488068kB, anon-rss:350976kB, file-rss:27500kB, shmem-rss:0kB
Jun 21 04:07:00 balkonien kernel: Out of memory: Kill process 16652 (forkit) score 12 or sacrifice child
Jun 21 04:07:00 balkonien kernel: Killed process 16801 (loolkit) total-vm:1488068kB, anon-rss:349752kB, file-rss:7536kB, shmem-rss:0kB
Jun 21 04:07:00 balkonien kernel: Out of memory: Kill process 16652 (forkit) score 12 or sacrifice child
Jun 21 04:07:00 balkonien kernel: Killed process 16652 (forkit) total-vm:1487720kB, anon-rss:349536kB, file-rss:18508kB, shmem-rss:0kB
Jun 21 04:49:00 balkonien kernel: Out of memory: Kill process 27040 (forkit) score 12 or sacrifice child
Jun 21 04:49:00 balkonien kernel: Killed process 27082 (loolkit) total-vm:1488068kB, anon-rss:102080kB, file-rss:428kB, shmem-rss:0kB
Jun 21 04:49:00 balkonien kernel: Out of memory: Kill process 27040 (forkit) score 12 or sacrifice child
Jun 21 04:49:00 balkonien kernel: Killed process 27088 (loolkit) total-vm:1488068kB, anon-rss:101984kB, file-rss:416kB, shmem-rss:0kB
Jun 21 04:56:00 balkonien kernel: Out of memory: Kill process 12165 (loolkit) score 12 or sacrifice child
Jun 21 04:56:00 balkonien kernel: Killed process 12165 (loolkit) total-vm:1488068kB, anon-rss:100852kB, file-rss:7140kB, shmem-rss:0kB
Jun 21 04:56:00 balkonien kernel: Out of memory: Kill process 27040 (forkit) score 12 or sacrifice child
Jun 21 04:56:00 balkonien kernel: Killed process 27040 (forkit) total-vm:1487720kB, anon-rss:99072kB, file-rss:60kB, shmem-rss:0kB
Jun 21 05:07:01 balkonien kernel: Out of memory: Kill process 13137 (forkit) score 12 or sacrifice child
Jun 21 05:07:01 balkonien kernel: Killed process 13170 (loolkit) total-vm:1488068kB, anon-rss:348584kB, file-rss:840kB, shmem-rss:0kB
Jun 21 05:09:00 balkonien kernel: Out of memory: Kill process 13137 (forkit) score 12 or sacrifice child
Jun 21 05:09:00 balkonien kernel: Killed process 13174 (loolkit) total-vm:1488068kB, anon-rss:340688kB, file-rss:0kB, shmem-rss:0kB
Jun 21 05:09:43 balkonien kernel: Out of memory: Kill process 14902 (loolkit) score 13 or sacrifice child
Jun 21 05:09:43 balkonien kernel: Killed process 14902 (loolkit) total-vm:1488068kB, anon-rss:340476kB, file-rss:26308kB, shmem-rss:0kB
Jun 21 05:54:52 balkonien kernel: Out of memory: Kill process 15401 (loolkit) score 12 or sacrifice child
Jun 21 05:54:52 balkonien kernel: Killed process 15401 (loolkit) total-vm:1488068kB, anon-rss:295604kB, file-rss:1080kB, shmem-rss:0kB
Jun 21 05:56:00 balkonien kernel: Out of memory: Kill process 15519 (loolkit) score 13 or sacrifice child
Jun 21 05:56:00 balkonien kernel: Killed process 15519 (loolkit) total-vm:1488068kB, anon-rss:292004kB, file-rss:25992kB, shmem-rss:0kB
Jun 21 05:56:00 balkonien kernel: Out of memory: Kill process 13137 (forkit) score 12 or sacrifice child
Jun 21 05:56:00 balkonien kernel: Killed process 13137 (forkit) total-vm:1487720kB, anon-rss:291656kB, file-rss:1272kB, shmem-rss:0kB
Jun 21 05:56:07 balkonien kernel: Out of memory: Kill process 27684 (perl) score 10 or sacrifice child
Jun 21 05:56:07 balkonien kernel: Killed process 27688 (perl) total-vm:34744kB, anon-rss:4820kB, file-rss:3848kB, shmem-rss:0kB
Jun 21 05:56:07 balkonien kernel: Out of memory: Kill process 27684 (perl) score 10 or sacrifice child
Jun 21 05:56:07 balkonien kernel: Killed process 27703 (perl) total-vm:34728kB, anon-rss:5364kB, file-rss:3812kB, shmem-rss:0kB
Jun 21 05:56:15 balkonien kernel: Out of memory: Kill process 16287 (forkit) score 13 or sacrifice child
Jun 21 05:56:15 balkonien kernel: Killed process 16287 (forkit) total-vm:1381924kB, anon-rss:248144kB, file-rss:129808kB, shmem-rss:0kB
Jun 21 05:58:18 balkonien kernel: Out of memory: Kill process 16503 (forkit) score 16 or sacrifice child
Jun 21 05:58:18 balkonien kernel: Killed process 16655 (loolkit) total-vm:1488068kB, anon-rss:348072kB, file-rss:25648kB, shmem-rss:0kB
Jun 21 05:59:18 balkonien kernel: Out of memory: Kill process 16503 (forkit) score 16 or sacrifice child
Jun 21 05:59:18 balkonien kernel: Killed process 16675 (loolkit) total-vm:1488068kB, anon-rss:345868kB, file-rss:24916kB, shmem-rss:0kB
Jun 21 06:07:00 balkonien kernel: Out of memory: Kill process 16503 (forkit) score 12 or sacrifice child
Jun 21 06:07:00 balkonien kernel: Killed process 17546 (loolkit) total-vm:1488068kB, anon-rss:342780kB, file-rss:3176kB, shmem-rss:0kB
Jun 21 06:07:00 balkonien kernel: Out of memory: Kill process 16503 (forkit) score 12 or sacrifice child
Jun 21 06:07:00 balkonien kernel: Killed process 16503 (forkit) total-vm:1487720kB, anon-rss:342536kB, file-rss:3128kB, shmem-rss:0kB
Jun 21 06:07:05 balkonien kernel: Out of memory: Kill process 27684 (perl) score 10 or sacrifice child
Jun 21 06:07:05 balkonien kernel: Killed process 27702 (bzip2) total-vm:13624kB, anon-rss:6392kB, file-rss:952kB, shmem-rss:0kB
Jun 21 06:08:45 balkonien systemd[1]: Failed to start systemd-cron daily script service.
Output of ps faux (regarding the backup task):
samuel 9458 0.0 0.0 26016 2212 ? Ss Jun16 0:00 SCREEN -R
samuel 9459 0.0 0.0 25516 5812 pts/2 Ss Jun16 0:00 _ /bin/bash
root 14092 0.0 0.0 45336 3804 pts/2 S+ 12:08 0:00 _ sudo ./storeBackup
root 14093 0.0 0.0 4312 672 pts/2 S+ 12:08 0:00 _ /bin/sh ./storeBackup
root 14099 38.4 1.9 369208 306664 pts/2 SN+ 12:08 33:13 _ perl /usr/local/bin/storeBackup.pl -f /etc/storeBackup/storebackup.conf
root 14103 0.0 0.0 34752 7432 pts/2 SN+ 12:08 0:00 _ perl /opt/storeBackup/lib/stbuLog.pl --readFile /tmp/prLog-54a4c1c781f71013 --out /var/log/storeBackup/storeBackup.log --maxFilelen 1000000 --noOfOldFiles 5
root 14116 1.6 0.0 13624 7500 pts/2 SN+ 12:08 1:22 _ bzip2
root 14117 0.0 0.0 34732 7328 pts/2 SN+ 12:08 0:00 _ perl /opt/storeBackup/lib/stbuLog.pl --readFile /tmp/prLog-bd0ccc7b5ac7d895 --out /media/Backup/storeBackup/default/2018.06.21_12.08.27/.storeBackup.log --maxFilelen 0 --noOfOldFiles 1
Output of iotop (first lines):
Total DISK READ : 7.06 K/s | Total DISK WRITE : 0.00 B/s
Actual DISK READ: 84.75 K/s | Actual DISK WRITE: 0.00 B/s
TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
14099 be/7 root 7.06 K/s 0.00 B/s 0.00 % 0.06 % perl /usr/local/bin/storeBackup.pl -f /etc/storeBackup/storebackup.conf
1 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % init
2 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kthreadd]
3 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [ksoftirqd/0]
5 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kworker/0:0H]
Output of top (first lines):
top - 13:38:35 up 5 days, 6:38, 1 user, load average: 1,00, 1,15, 1,22
Tasks: 592 total, 2 running, 589 sleeping, 0 stopped, 1 zombie
%Cpu(s): 0,5 us, 25,7 sy, 1,5 ni, 71,8 id, 0,2 wa, 0,0 hi, 0,2 si, 0,0 st
KiB Mem: 15908456 total, 15733624 used, 174832 free, 2555436 buffers
KiB Swap: 12695892 total, 263052 used, 12432840 free. 10882640 cached Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
14099 root 38 18 369208 306664 4920 R 44,5 1,9 34:54.76 perl
47 root 20 0 0 0 0 S 3,3 0,0 67:54.66 kswapd0
18400 root 20 0 26224 3432 2568 R 2,0 0,0 0:00.21 top
7 root 20 0 0 0 0 S 1,6 0,0 29:58.23 rcu_sched
Output of free -h:
total used free shared buffers cached
Mem: 15G 14G 214M 18M 2,4G 10G
-/+ buffers/cache: 2,2G 12G
Swap: 12G 256M 11G
linux debian ram out-of-memory
asked Jun 21 at 7:18
Sammy
136
I run a debian Webserver with nextcloud. Every night a backup job (storeBackup) is scheduled and it only succeeds in very few cases.
But every night the journal is full of OOM Errors (see below).
The machine has 16G RAM, therefore I am wondering, what is so much RAM consuming.
Output of journalctl -p err -e for one night:
Jun 21 03:28:01 balkonien kernel: Out of memory: Kill process 4723 (clamd) score 19 or sacrifice child
Jun 21 03:28:01 balkonien kernel: Killed process 4723 (clamd) total-vm:760960kB, anon-rss:49044kB, file-rss:6504kB, shmem-rss:0kB
Jun 21 03:35:00 balkonien kernel: Out of memory: Kill process 22111 (forkit) score 12 or sacrifice child
Jun 21 03:35:00 balkonien kernel: Killed process 22139 (loolkit) total-vm:1488072kB, anon-rss:53768kB, file-rss:0kB, shmem-rss:0kB
Jun 21 03:38:11 balkonien kernel: Out of memory: Kill process 8246 (loolkit) score 13 or sacrifice child
Jun 21 03:38:11 balkonien kernel: Killed process 8246 (loolkit) total-vm:1488072kB, anon-rss:58388kB, file-rss:25816kB, shmem-rss:0kB
Jun 21 03:40:49 balkonien kernel: Out of memory: Kill process 10165 (loolkit) score 13 or sacrifice child
Jun 21 03:40:49 balkonien kernel: Killed process 10165 (loolkit) total-vm:1488072kB, anon-rss:58336kB, file-rss:26224kB, shmem-rss:0kB
Jun 21 03:47:38 balkonien kernel: Out of memory: Kill process 12575 (loolkit) score 13 or sacrifice child
Jun 21 03:47:38 balkonien kernel: Killed process 12575 (loolkit) total-vm:1488072kB, anon-rss:58212kB, file-rss:26224kB, shmem-rss:0kB
Jun 21 03:47:39 balkonien kernel: Out of memory: Kill process 22111 (forkit) score 12 or sacrifice child
Jun 21 03:47:39 balkonien kernel: Killed process 22111 (forkit) total-vm:1487792kB, anon-rss:53880kB, file-rss:2080kB, shmem-rss:0kB
Jun 21 03:48:45 balkonien kernel: Out of memory: Kill process 16652 (forkit) score 17 or sacrifice child
Jun 21 03:48:45 balkonien kernel: Killed process 16783 (loolkit) total-vm:1488068kB, anon-rss:350976kB, file-rss:27500kB, shmem-rss:0kB
Jun 21 04:07:00 balkonien kernel: Out of memory: Kill process 16652 (forkit) score 12 or sacrifice child
Jun 21 04:07:00 balkonien kernel: Killed process 16801 (loolkit) total-vm:1488068kB, anon-rss:349752kB, file-rss:7536kB, shmem-rss:0kB
Jun 21 04:07:00 balkonien kernel: Out of memory: Kill process 16652 (forkit) score 12 or sacrifice child
Jun 21 04:07:00 balkonien kernel: Killed process 16652 (forkit) total-vm:1487720kB, anon-rss:349536kB, file-rss:18508kB, shmem-rss:0kB
Jun 21 04:49:00 balkonien kernel: Out of memory: Kill process 27040 (forkit) score 12 or sacrifice child
Jun 21 04:49:00 balkonien kernel: Killed process 27082 (loolkit) total-vm:1488068kB, anon-rss:102080kB, file-rss:428kB, shmem-rss:0kB
Jun 21 04:49:00 balkonien kernel: Out of memory: Kill process 27040 (forkit) score 12 or sacrifice child
Jun 21 04:49:00 balkonien kernel: Killed process 27088 (loolkit) total-vm:1488068kB, anon-rss:101984kB, file-rss:416kB, shmem-rss:0kB
Jun 21 04:56:00 balkonien kernel: Out of memory: Kill process 12165 (loolkit) score 12 or sacrifice child
Jun 21 04:56:00 balkonien kernel: Killed process 12165 (loolkit) total-vm:1488068kB, anon-rss:100852kB, file-rss:7140kB, shmem-rss:0kB
Jun 21 04:56:00 balkonien kernel: Out of memory: Kill process 27040 (forkit) score 12 or sacrifice child
Jun 21 04:56:00 balkonien kernel: Killed process 27040 (forkit) total-vm:1487720kB, anon-rss:99072kB, file-rss:60kB, shmem-rss:0kB
Jun 21 05:07:01 balkonien kernel: Out of memory: Kill process 13137 (forkit) score 12 or sacrifice child
Jun 21 05:07:01 balkonien kernel: Killed process 13170 (loolkit) total-vm:1488068kB, anon-rss:348584kB, file-rss:840kB, shmem-rss:0kB
Jun 21 05:09:00 balkonien kernel: Out of memory: Kill process 13137 (forkit) score 12 or sacrifice child
Jun 21 05:09:00 balkonien kernel: Killed process 13174 (loolkit) total-vm:1488068kB, anon-rss:340688kB, file-rss:0kB, shmem-rss:0kB
Jun 21 05:09:43 balkonien kernel: Out of memory: Kill process 14902 (loolkit) score 13 or sacrifice child
Jun 21 05:09:43 balkonien kernel: Killed process 14902 (loolkit) total-vm:1488068kB, anon-rss:340476kB, file-rss:26308kB, shmem-rss:0kB
Jun 21 05:54:52 balkonien kernel: Out of memory: Kill process 15401 (loolkit) score 12 or sacrifice child
Jun 21 05:54:52 balkonien kernel: Killed process 15401 (loolkit) total-vm:1488068kB, anon-rss:295604kB, file-rss:1080kB, shmem-rss:0kB
Jun 21 05:56:00 balkonien kernel: Out of memory: Kill process 15519 (loolkit) score 13 or sacrifice child
Jun 21 05:56:00 balkonien kernel: Killed process 15519 (loolkit) total-vm:1488068kB, anon-rss:292004kB, file-rss:25992kB, shmem-rss:0kB
Jun 21 05:56:00 balkonien kernel: Out of memory: Kill process 13137 (forkit) score 12 or sacrifice child
Jun 21 05:56:00 balkonien kernel: Killed process 13137 (forkit) total-vm:1487720kB, anon-rss:291656kB, file-rss:1272kB, shmem-rss:0kB
Jun 21 05:56:07 balkonien kernel: Out of memory: Kill process 27684 (perl) score 10 or sacrifice child
Jun 21 05:56:07 balkonien kernel: Killed process 27688 (perl) total-vm:34744kB, anon-rss:4820kB, file-rss:3848kB, shmem-rss:0kB
Jun 21 05:56:07 balkonien kernel: Out of memory: Kill process 27684 (perl) score 10 or sacrifice child
Jun 21 05:56:07 balkonien kernel: Killed process 27703 (perl) total-vm:34728kB, anon-rss:5364kB, file-rss:3812kB, shmem-rss:0kB
Jun 21 05:56:15 balkonien kernel: Out of memory: Kill process 16287 (forkit) score 13 or sacrifice child
Jun 21 05:56:15 balkonien kernel: Killed process 16287 (forkit) total-vm:1381924kB, anon-rss:248144kB, file-rss:129808kB, shmem-rss:0kB
Jun 21 05:58:18 balkonien kernel: Out of memory: Kill process 16503 (forkit) score 16 or sacrifice child
Jun 21 05:58:18 balkonien kernel: Killed process 16655 (loolkit) total-vm:1488068kB, anon-rss:348072kB, file-rss:25648kB, shmem-rss:0kB
Jun 21 05:59:18 balkonien kernel: Out of memory: Kill process 16503 (forkit) score 16 or sacrifice child
Jun 21 05:59:18 balkonien kernel: Killed process 16675 (loolkit) total-vm:1488068kB, anon-rss:345868kB, file-rss:24916kB, shmem-rss:0kB
Jun 21 06:07:00 balkonien kernel: Out of memory: Kill process 16503 (forkit) score 12 or sacrifice child
Jun 21 06:07:00 balkonien kernel: Killed process 17546 (loolkit) total-vm:1488068kB, anon-rss:342780kB, file-rss:3176kB, shmem-rss:0kB
Jun 21 06:07:00 balkonien kernel: Out of memory: Kill process 16503 (forkit) score 12 or sacrifice child
Jun 21 06:07:00 balkonien kernel: Killed process 16503 (forkit) total-vm:1487720kB, anon-rss:342536kB, file-rss:3128kB, shmem-rss:0kB
Jun 21 06:07:05 balkonien kernel: Out of memory: Kill process 27684 (perl) score 10 or sacrifice child
Jun 21 06:07:05 balkonien kernel: Killed process 27702 (bzip2) total-vm:13624kB, anon-rss:6392kB, file-rss:952kB, shmem-rss:0kB
Jun 21 06:08:45 balkonien systemd[1]: Failed to start systemd-cron daily script service.
Output of ps faux (regarding the backup task):
samuel 9458 0.0 0.0 26016 2212 ? Ss Jun16 0:00 SCREEN -R
samuel 9459 0.0 0.0 25516 5812 pts/2 Ss Jun16 0:00 _ /bin/bash
root 14092 0.0 0.0 45336 3804 pts/2 S+ 12:08 0:00 _ sudo ./storeBackup
root 14093 0.0 0.0 4312 672 pts/2 S+ 12:08 0:00 _ /bin/sh ./storeBackup
root 14099 38.4 1.9 369208 306664 pts/2 SN+ 12:08 33:13 _ perl /usr/local/bin/storeBackup.pl -f /etc/storeBackup/storebackup.conf
root 14103 0.0 0.0 34752 7432 pts/2 SN+ 12:08 0:00 _ perl /opt/storeBackup/lib/stbuLog.pl --readFile /tmp/prLog-54a4c1c781f71013 --out /var/log/storeBackup/storeBackup.log --maxFilelen 1000000 --noOfOldFiles 5
root 14116 1.6 0.0 13624 7500 pts/2 SN+ 12:08 1:22 _ bzip2
root 14117 0.0 0.0 34732 7328 pts/2 SN+ 12:08 0:00 _ perl /opt/storeBackup/lib/stbuLog.pl --readFile /tmp/prLog-bd0ccc7b5ac7d895 --out /media/Backup/storeBackup/default/2018.06.21_12.08.27/.storeBackup.log --maxFilelen 0 --noOfOldFiles 1
Output of iotop (first lines):
Total DISK READ : 7.06 K/s | Total DISK WRITE : 0.00 B/s
Actual DISK READ: 84.75 K/s | Actual DISK WRITE: 0.00 B/s
TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
14099 be/7 root 7.06 K/s 0.00 B/s 0.00 % 0.06 % perl /usr/local/bin/storeBackup.pl -f /etc/storeBackup/storebackup.conf
1 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % init
2 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kthreadd]
3 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [ksoftirqd/0]
5 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kworker/0:0H]
Output of top (first lines):
top - 13:38:35 up 5 days, 6:38, 1 user, load average: 1,00, 1,15, 1,22
Tasks: 592 total, 2 running, 589 sleeping, 0 stopped, 1 zombie
%Cpu(s): 0,5 us, 25,7 sy, 1,5 ni, 71,8 id, 0,2 wa, 0,0 hi, 0,2 si, 0,0 st
KiB Mem: 15908456 total, 15733624 used, 174832 free, 2555436 buffers
KiB Swap: 12695892 total, 263052 used, 12432840 free. 10882640 cached Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
14099 root 38 18 369208 306664 4920 R 44,5 1,9 34:54.76 perl
47 root 20 0 0 0 0 S 3,3 0,0 67:54.66 kswapd0
18400 root 20 0 26224 3432 2568 R 2,0 0,0 0:00.21 top
7 root 20 0 0 0 0 S 1,6 0,0 29:58.23 rcu_sched
Output of free -h:
total used free shared buffers cached
Mem: 15G 14G 214M 18M 2,4G 10G
-/+ buffers/cache: 2,2G 12G
Swap: 12G 256M 11G
linux debian ram out-of-memory
asked Jun 21 at 7:18
Sammy
136
edited Jun 21 at 11:39
asked Jun 21 at 7:18
Sammy
136
asked Jun 21 at 7:18
Sammy
136
asked Jun 21 at 7:18
Sammy
136
136
closed as too broad by Ipor Sircer, schily, Kusalananda, Thomas Nyman, slm♦ Jun 22 at 0:37
Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.
closed as too broad by Ipor Sircer, schily, Kusalananda, Thomas Nyman, slm♦ Jun 22 at 0:37
Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.
instead of long useless log, you could had the result of the commandps faux,iotoptopfree -hthat would help way more than your log. Also obviously your backup process eat all the capacity of the system.
– Kiwy
Jun 21 at 7:53
@Kiwy it's not useless as it tells exactly what causes the problem. OOM killer is not a random magic - every process in linux has its calculated score and OOM killer pick the best one to kill. Simplifying is most likely it'll kill the one which consumes biggest amount of memory. More at this topic
– pawel7318
Jun 21 at 8:03
@pawel7318 it still doesn't inform you precissely of what is the cause for your problem, another program than the one causing the issue can be terminated. I already had the case several time. To me it's just an indication that something went wrong at that moment. make a conclusion on those log, is IMO not relevant, i'll read the doc you post though.
– Kiwy
Jun 21 at 8:09
@Kiwy I added the outputs accordingly. Right now the backup script is only running for ten minutes. I can update the outputs in an hour, if it's neccessary. Or is it clear that the backup script is the cause and I should contact them?
– Sammy
Jun 21 at 10:19
is the problem is currently happening ? because you have 11GB of cached memory that could be freed by the system before shooting processes.
– Kiwy
Jun 21 at 10:32
 |Â
show 2 more comments
instead of long useless log, you could had the result of the commandps faux,iotoptopfree -hthat would help way more than your log. Also obviously your backup process eat all the capacity of the system.
– Kiwy
Jun 21 at 7:53
@Kiwy it's not useless as it tells exactly what causes the problem. OOM killer is not a random magic - every process in linux has its calculated score and OOM killer pick the best one to kill. Simplifying is most likely it'll kill the one which consumes biggest amount of memory. More at this topic
– pawel7318
Jun 21 at 8:03
@pawel7318 it still doesn't inform you precissely of what is the cause for your problem, another program than the one causing the issue can be terminated. I already had the case several time. To me it's just an indication that something went wrong at that moment. make a conclusion on those log, is IMO not relevant, i'll read the doc you post though.
– Kiwy
Jun 21 at 8:09
@Kiwy I added the outputs accordingly. Right now the backup script is only running for ten minutes. I can update the outputs in an hour, if it's neccessary. Or is it clear that the backup script is the cause and I should contact them?
– Sammy
Jun 21 at 10:19
is the problem is currently happening ? because you have 11GB of cached memory that could be freed by the system before shooting processes.
– Kiwy
Jun 21 at 10:32
instead of long useless log, you could had the result of the command
ps faux, iotop top free -h that would help way more than your log. Also obviously your backup process eat all the capacity of the system.– Kiwy
Jun 21 at 7:53
instead of long useless log, you could had the result of the command
ps faux, iotop top free -h that would help way more than your log. Also obviously your backup process eat all the capacity of the system.– Kiwy
Jun 21 at 7:53
@Kiwy it's not useless as it tells exactly what causes the problem. OOM killer is not a random magic - every process in linux has its calculated score and OOM killer pick the best one to kill. Simplifying is most likely it'll kill the one which consumes biggest amount of memory. More at this topic
– pawel7318
Jun 21 at 8:03
@Kiwy it's not useless as it tells exactly what causes the problem. OOM killer is not a random magic - every process in linux has its calculated score and OOM killer pick the best one to kill. Simplifying is most likely it'll kill the one which consumes biggest amount of memory. More at this topic
– pawel7318
Jun 21 at 8:03
@pawel7318 it still doesn't inform you precissely of what is the cause for your problem, another program than the one causing the issue can be terminated. I already had the case several time. To me it's just an indication that something went wrong at that moment. make a conclusion on those log, is IMO not relevant, i'll read the doc you post though.
– Kiwy
Jun 21 at 8:09
@pawel7318 it still doesn't inform you precissely of what is the cause for your problem, another program than the one causing the issue can be terminated. I already had the case several time. To me it's just an indication that something went wrong at that moment. make a conclusion on those log, is IMO not relevant, i'll read the doc you post though.
– Kiwy
Jun 21 at 8:09
@Kiwy I added the outputs accordingly. Right now the backup script is only running for ten minutes. I can update the outputs in an hour, if it's neccessary. Or is it clear that the backup script is the cause and I should contact them?
– Sammy
Jun 21 at 10:19
@Kiwy I added the outputs accordingly. Right now the backup script is only running for ten minutes. I can update the outputs in an hour, if it's neccessary. Or is it clear that the backup script is the cause and I should contact them?
– Sammy
Jun 21 at 10:19
is the problem is currently happening ? because you have 11GB of cached memory that could be freed by the system before shooting processes.
– Kiwy
Jun 21 at 10:32
is the problem is currently happening ? because you have 11GB of cached memory that could be freed by the system before shooting processes.
– Kiwy
Jun 21 at 10:32
 |Â
show 2 more comments
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
instead of long useless log, you could had the result of the command
ps faux,iotoptopfree -hthat would help way more than your log. Also obviously your backup process eat all the capacity of the system.– Kiwy
Jun 21 at 7:53
@Kiwy it's not useless as it tells exactly what causes the problem. OOM killer is not a random magic - every process in linux has its calculated score and OOM killer pick the best one to kill. Simplifying is most likely it'll kill the one which consumes biggest amount of memory. More at this topic
– pawel7318
Jun 21 at 8:03
@pawel7318 it still doesn't inform you precissely of what is the cause for your problem, another program than the one causing the issue can be terminated. I already had the case several time. To me it's just an indication that something went wrong at that moment. make a conclusion on those log, is IMO not relevant, i'll read the doc you post though.
– Kiwy
Jun 21 at 8:09
@Kiwy I added the outputs accordingly. Right now the backup script is only running for ten minutes. I can update the outputs in an hour, if it's neccessary. Or is it clear that the backup script is the cause and I should contact them?
– Sammy
Jun 21 at 10:19
is the problem is currently happening ? because you have 11GB of cached memory that could be freed by the system before shooting processes.
– Kiwy
Jun 21 at 10:32