mjhjmtu

I have a test zpool set up with a single, USB-attached 4TB spinning disk which reads at about 40MB/sec.

I have four ~300GB fast SATA2 internal SSDs set up as cache for this disk. I can read about 900MB/sec from this L2ARC.

I am reading a mostly static 400GB dataset from this pool.

The system in question has over 200GB of free memory, which I assume operates at >1500MB/sec for our purposes.

Every 1.0s: zpool iostat -y -v 1 1 ; zpool status -v las2: Tue Oct 16 20:20:07 2018

 capacity operations bandwidth
pool alloc free read write read write
-------------------------------------------------- ----- ----- ----- ----- ----- -----
storage 417G 3.22T 239 2 29.7M 132K
 usb-Seagate_M3_Portable_NM12QHQF-0:0-part1 417G 3.22T 239 2 29.7M 132K
cache - - - - - -
 ata-INTEL_SSDSC2BA400G3_BTTV41450D92400HGN-part3 52.8G 291G 11 0 927K 128K
 ata-INTEL_SSDSC2BA400G3_BTTV423000XA400HGN-part3 52.9G 291G 13 0 1.48M 0
 ata-INTEL_SSDSC2BA400G3_BTTV423406FG400HGN-part3 52.7G 291G 8 0 324K 0
 ata-INTEL_SSDSC2BA400G3_BTTV423406G6400HGN-part3 52.9G 291G 5 0 336K 0
-------------------------------------------------- ----- ----- ----- ----- ----- -----
 pool: storage
 state: ONLINE
 scan: scrub repaired 0B in 1h56m with 0 errors on Sun Oct 14 02:20:56 2018
config:

 NAME STATE READ WRITE CKSUM
 storage ONLINE 0 0 0
 usb-Seagate_M3_Portable_NM12QHQF-0:0-part1 ONLINE 0 0 0
 cache
 ata-INTEL_SSDSC2BA400G3_BTTV41450D92400HGN-part3 ONLINE 0 0 0
 ata-INTEL_SSDSC2BA400G3_BTTV423000XA400HGN-part3 ONLINE 0 0 0
 ata-INTEL_SSDSC2BA400G3_BTTV423406FG400HGN-part3 ONLINE 0 0 0
 ata-INTEL_SSDSC2BA400G3_BTTV423406G6400HGN-part3 ONLINE 0 0 0

errors: No known data errors

AIUI, reads get cached into the ARC in system memory. When that hits some level of "full", these get evicted onto the L2ARC, 1.1TB of fast SSD.

When I run a tar -c . | pv > /dev/null from one of the mostly-unchanging directories on this pool, I see around 40MB/sec - the speed of reads from the underlying, slow HDD.

The problem is, I see it the second time, too! Sometimes it spikes to 80-100MB/sec, but also sometimes it falls to 10MB/sec. I would expect 99% of the reads to come from memory or the fast L2ARC.

What gives? I would expect to see from-RAM speeds as this box has 256GB of system memory and over 200GB of that is free. Failing that, there's still hundreds of gigabytes of L2ARC unused.

What am I doing wrong? Why is this data not being read from the ARC at extremely high speed (>1GB/sec)? Failing that, why is it not being read from the L2ARC (>500MB/sec)? Why am I seeing 40-80MB/sec on a mostly-unloaded system?

Ubuntu bionic 18.04.1 x86_64 system.

Linux las2 4.15.0-36-generic #39-Ubuntu SMP Mon Sep 24 16:19:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

zfs-auto-snapshot/now 1.2.4-1 all [installed,local]
zfs-zed/now 0.7.5-1ubuntu16.4 amd64 [installed,local]
zfsutils-linux/now 0.7.5-1ubuntu16.4 amd64 [installed,local]