mjhjmtu

I'm running a yocto build with 4.4.91 kernel on an embedded device with 256mb RAM, no swap.

I'm trying to download a large file onto mmc flash via tftp command, during which according to /proc/meminfo my cache grows and free memory goes down to zero, triggering memory allocation issues and watchdog which is set to reset the machine once free space goes down to 4mb.

I've tried adjusting cache pressure, verified kernel memory tunables are set to default and match with the same device running an older build.

I have not seen this problem running an older yocto build with 3.12.10 kernel.

What else can I try to force the kernel to give up cached memory, and what other tunables can I experiment with?

Here's a sample message. I suspect it has something to do with lowmem_reserve and number of pages in highmem.

[ 2471.077819] CPU: 0 PID: 1279 Comm: mmcqd/0 Tainted: G O 4.4.91-g2657e61db2 #3
[ 2471.086520] Hardware name: Generic AM33XX (Flattened Device Tree)
[ 2471.092936] Backtrace: 
[ 2471.095550] [<c0019608>] (dump_backtrace) from [<c0019804>] (show_stack+0x18/0x1c)
[ 2471.103520] r7:00000002 r6:00000000 r5:00000002 r4:02284020
[ 2471.109519] [<c00197ec>] (show_stack) from [<c02892c0>] (dump_stack+0x24/0x28)
[ 2471.117135] [<c028929c>] (dump_stack) from [<c00adc0c>] (warn_alloc_failed+0xe4/0x124)
[ 2471.125483] [<c00adb2c>] (warn_alloc_failed) from [<c00b0d4c>] (__alloc_pages_nodemask+0x814/0x9a4)
[ 2471.135005] r3:00000000 r2:00000000
[ 2471.138775] r6:cd0012c0 r5:dcfdcfdd r4:00000000
[ 2471.143662] [<c00b0538>] (__alloc_pages_nodemask) from [<c00da04c>] (cache_alloc_refill+0x2c0/0x514)
[ 2471.153279] r10:00000000 r9:02088020 r8:00000000 r7:cd0012c0 r6:cd0012c0 r5:cd0004c0
[ 2471.161547] r4:00000000
[ 2471.164222] [<c00d9d8c>] (cache_alloc_refill) from [<c00da538>] (__kmalloc+0xcc/0xf4)
[ 2471.172466] r10:cd60d3a0 r9:cd170218 r8:00000001 r7:400e0013 r6:cd0004c0 r5:cd60c000
[ 2471.180737] r4:02088020
[ 2471.183421] [<c00da46c>] (__kmalloc) from [<c0301728>] (edma_prep_slave_sg+0xa0/0x328)
[ 2471.191757] r7:ffffffff r6:cd5ec148 r5:cd60c000 r4:00000000
[ 2471.197750] [<c0301688>] (edma_prep_slave_sg) from [<c04664b0>] (omap_hsmmc_request+0x454/0x618)
[ 2471.207000] r10:cd60d3a0 r9:cd5ec208 r8:00000000 r7:ffffffff r6:cd5ec148 r5:c0301688
[ 2471.215270] r4:cd5adc00
[ 2471.217954] [<c046605c>] (omap_hsmmc_request) from [<c044ddfc>] (__mmc_start_request+0x48/0xdc)
[ 2471.227114] r10:cd607eac r9:00000000 r8:cd5ec148 r7:cd5ec148 r6:cd5ec148 r5:cd5adc00
[ 2471.235384] r4:cd5ec148
[ 2471.238061] [<c044ddb4>] (__mmc_start_request) from [<c044df7c>] (mmc_start_request+0xec/0x114)
[ 2471.247219] r9:00000000 r8:cd5ec250 r7:cd5ec148 r6:cd5ec148 r5:cd5adc00 r4:cd5ec148
[ 2471.255411] [<c044de90>] (mmc_start_request) from [<c044f2bc>] (mmc_start_req+0x29c/0x3c4)
[ 2471.264113] r5:cd5c3c00 r4:cd5adc00
[ 2471.267904] [<c044f020>] (mmc_start_req) from [<c045e84c>] (mmc_blk_issue_rw_rq+0xc4/0xa88)
[ 2471.276697] r10:cb4eb9d8 r9:245d0001 r8:cd5ec144 r7:cd5ec148 r6:cb4eb9d8 r5:cd5c3c00
[ 2471.284968] r4:cd5ec004
[ 2471.287647] [<c045e788>] (mmc_blk_issue_rw_rq) from [<c045f45c>] (mmc_blk_issue_rq+0x24c/0x4e0)
[ 2471.296806] r10:cb4eb9d8 r9:245d0001 r8:cd5ec000 r7:cd5ec000 r6:cd4f9088 r5:cd5c3c00
[ 2471.305077] r4:cd5ec004
[ 2471.307755] [<c045f210>] (mmc_blk_issue_rq) from [<c045ff84>] (mmc_queue_thread+0xa0/0x164)
[ 2471.316547] r10:00000000 r9:245d0001 r8:ffffe000 r7:cd5ec00c r6:cd4f9088 r5:cd606000
[ 2471.324818] r4:cd5ec004
[ 2471.327508] [<c045fee4>] (mmc_queue_thread) from [<c0055020>] (kthread+0xf8/0x110)
[ 2471.335477] r9:00000000 r8:c045fee4 r7:cd5ec004 r6:cd606000 r5:cd5f6e40 r4:00000000
[ 2471.343680] [<c0054f28>] (kthread) from [<c0015518>] (ret_from_fork+0x14/0x3c)
[ 2471.351284] r8:00000000 r7:00000000 r6:00000000 r5:c0054f28 r4:cd5f6e40
[ 2471.358364] Mem-Info:
[ 2471.360774] active_anon:19476 inactive_anon:2093 isolated_anon:0
[ 2471.360774] active_file:17499 inactive_file:13549 isolated_file:0
[ 2471.360774] unevictable:338 dirty:2923 writeback:1337 unstable:0
[ 2471.360774] slab_reclaimable:2493 slab_unreclaimable:3156
[ 2471.360774] mapped:3711 shmem:2429 pagetables:277 bounce:0
[ 2471.360774] free:2228 free_pcp:80 free_cma:676
[ 2471.396419] Normal free:8912kB min:4096kB low:5120kB high:6144kB active_anon:77904kB inactive_anon:8372kB active_file:69996kB inactive_file:54196kB unevictable:1352ko
[ 2471.445276] lowmem_reserve: 0 0 0
[ 2471.448963] Normal: 1584*4kB (UMC) 28*8kB (MC) 19*16kB (C) 8*32kB (C) 4*64kB (C) 4*128kB (C) 0*256kB 2*512kB (C) 0*1024kB 0*2048kB 0*4096kB 0*8192kB = 8912kB
[ 2471.463895] 33736 total pagecache pages
[ 2471.467930] 65024 pages RAM
[ 2471.470866] 0 pages HighMem/MovableOnly
[ 2471.474901] 2813 pages reserved
[ 2471.478204] 6144 pages cma reserved
[ 2471.482040] edma 49000000.edma: edma_prep_slave_sg: Failed to allocate a descriptor
[ 2471.490114] omap_hsmmc 481d8000.mmc: prep_slave_sg() failed
[ 2471.496008] omap_hsmmc 481d8000.mmc: MMC start dma failure
[ 2471.516691] mmcblk0: unknown error -1 sending read/write command, card status 0x900
[ 2471.525087] blk_update_request: I/O error, dev mmcblk0, sector 2178224
[ 2471.532013] EXT4-fs warning (device mmcblk0p2): ext4_end_bio:330: I/O error -5 writing to inode 13757 (offset 0 size 0 starting block 272279)
[ 2471.545412] Buffer I/O error on device mmcblk0p2, logical block 264454
[ 2471.552629] blk_update_request: I/O error, dev mmcblk0, sector 2178232
[ 2471.559518] EXT4-fs warning (device mmcblk0p2): ext4_end_bio:330: I/O error -5 writing to inode 13757 (offset 0 size 0 starting block 272280)
[ 2471.572931] Buffer I/O error on device mmcblk0p2, logical block 264455
[ 2471.579923] blk_update_request: I/O error, dev mmcblk0, sector 2178240
[ 2471.586837] EXT4-fs warning (device mmcblk0p2): ext4_end_bio:330: I/O error -5 writing to inode 13757 (offset 0 size 0 starting block 272281)
[ 2471.600232] Buffer I/O error on device mmcblk0p2, logical block 264456
[ 2471.607209] blk_update_request: I/O error, dev mmcblk0, sector 2178248
[ 2471.614111] EXT4-fs warning (device mmcblk0p2): ext4_end_bio:330: I/O error -5 writing to inode 13757 (offset 0 size 0 starting block 272282)
[ 2471.627508] Buffer I/O error on device mmcblk0p2, logical block 264457
[ 2471.634468] blk_update_request: I/O error, dev mmcblk0, sector 2178256
[ 2471.641351] EXT4-fs warning (device mmcblk0p2): ext4_end_bio:330: I/O error -5 writing to inode 13757 (offset 0 size 0 starting block 272283)
[ 2471.654744] Buffer I/O error on device mmcblk0p2, logical block 264458
[ 2471.661708] blk_update_request: I/O error, dev mmcblk0, sector 2178264
[ 2471.668590] EXT4-fs warning (device mmcblk0p2): ext4_end_bio:330: I/O error -5 writing to inode 13757 (offset 0 size 0 starting block 272284)
[ 2471.681985] Buffer I/O error on device mmcblk0p2, logical block 264459
[ 2471.688929] blk_update_request: I/O error, dev mmcblk0, sector 2178272
[ 2471.695829] EXT4-fs warning (device mmcblk0p2): ext4_end_bio:330: I/O error -5 writing to inode 13757 (offset 0 size 0 starting block 272285)
[ 2471.709218] Buffer I/O error on device mmcblk0p2, logical block 264460
[ 2471.716174] blk_update_request: I/O error, dev mmcblk0, sector 2178280
[ 2471.723069] EXT4-fs warning (device mmcblk0p2): ext4_end_bio:330: I/O error -5 writing to inode 13757 (offset 0 size 0 starting block 272286)
[ 2471.736467] Buffer I/O error on device mmcblk0p2, logical block 264461
[ 2471.743425] blk_update_request: I/O error, dev mmcblk0, sector 2178288
[ 2471.750308] EXT4-fs warning (device mmcblk0p2): ext4_end_bio:330: I/O error -5 writing to inode 13757 (offset 0 size 0 starting block 272287)
[ 2471.763701] Buffer I/O error on device mmcblk0p2, logical block 264462
[ 2471.770643] blk_update_request: I/O error, dev mmcblk0, sector 2178296
[ 2471.777542] EXT4-fs warning (device mmcblk0p2): ext4_end_bio:330: I/O error -5 writing to inode 13757 (offset 0 size 0 starting block 272288)
[ 2471.790930] Buffer I/O error on device mmcblk0p2, logical block 264463

The issue was related to systemd consuming a lot of memory for journaling. After reducing the amount of memory allocated to tmpfs and systemd journal, the problem went away.