mjhjmtu

I want to benchmark a ssd (possibly with encrypted filesystems) and compare it to benchmarks done by crystaldiskmark on windows.

So how can I measure approximately the same things as crystaldiskmark does?

For the first row (Seq) I think I could do something like

LC_ALL=C dd if=/dev/zero of=tempfile bs=1M count=1024 conv=fdatasync,notrunc

sudo su -c "echo 3 > /proc/sys/vm/drop_caches"
LC_ALL=C dd if=tempfile of=/dev/null bs=1M count=1024

But I am not sure about the dd parameters.

For the random 512KB, 4KB, 4KB (Queue Depth=32) reads/writes speed-tests I don't have any idea how to reproduce the measurements in linux? So how can I do this?

For testing reading speeds something like sudo hdparm -Tt /dev/sda doesn't seem to make sense for me since I want for example benchmark something like encfs mounts.

Edit

@Alko, @iain

Perhaps I should write something about the motivation about this question: I am trying to benchmark my ssd and compare some encryption solutions. But that's another question (Best way to benchmark different encryption solutions on my system). While surfing in the web about ssd's and benchmarking I have often seen users posting their CrystelDiskMark results in forums. So this is the only motivation for the question. I just want to do the same on linux. For my particular benchmarking see my other question.

I'd say fio would have no trouble producing those workloads. Note that despite its name CrystalDiskMark is actually a benchmark of a filesysystem on a particular disk - it can't do I/O raw to the disk alone. As such it will always have filesystem overhead in it (not necessarily a bad thing but something to be aware of e.g. because the filesystems being compared might not be the same).

An example based on replicating the output in the screenshot above supplemented by information from the CrystalDiskMark manual (this isn't complete but should give the general idea):

fio --loops=5 --size=1000m --filename=/mnt/fs/fiotest.tmp --stonewall --ioengine=libaio --direct=1 
 --name=Seqread --bs=1m --rw=read 
 --name=Seqwrite --bs=1m --rw=write 
 --name=512Kread --bs=512k --rw=randread 
 --name=512Kwrite --bs=512k --rw=randwrite 
 --name=4kQD32read --bs=4k --iodepth=32 --rw=randread 
 --name=4kQD32write --bs=4k --iodepth=32 --rw=randwrite
rm -f /mnt/fs/fiotest.tmp 

BE CAREFUL - this example permanently destroys the data in /mnt/fs/fiotest.tmp!

A list of fio parameters can be seen on http://fio.readthedocs.io/en/latest/fio_doc.html .

You can use iozone and bonnie. They can do what crystal disk mark can do and more.

I personally used iozone a lot while benchmarking and stress testing devices from personal computers to enterprise storage systems. It has an auto mode which does everything but you can tailor it to your needs.

I'm not sure the various deeper tests make any real sense when considering what you're doing in detail.

The settings like block size, and queue depth, are parameters for controlling the low level input/output parameters of the ATA interface your SSD is sitting on.

Thats all well and good when you're just running some basic test against a drive fairly directly, like to a large file in a simple partitioned filesystem.

Once you start talking about benchmarking an encfs, these parameters dont particularly apply to your filesystem any more, the filesystem is just an interface into something else that eventually backs onto a filesystem that backs onto a drive.

I think it would be helpful to understand what exactly you're trying to measure, because there are two factors in play here - the raw disk IO speed, which you can test by timing various DD commands (can give examples if this is what you want) /without/ encfs, or the process will be CPU bounded by the encryption and you're trying to test the relative throughput of the encryption algorithm. In which case the parameters for queue depth etc aren't particularly relevant.

In both regards, a timed DD command will give you the basic throughput statistics you seek, but you should consider what you're intending to measure and the relevant parameters for that.

This link seems to provide a good guide to disk speed testing using timed DD commands including the necessary coverage about 'defeating buffers/cache' and so forth. Likely this will provide the information you need. Decide which you're more interested in tho, disk performance or encryption performance, one of the two will be the bottleneck, and tuning the non-bottleneck isn't going to benefit anything.

I created a script that tries to replicate the behavior of crystaldiskmark 5 with fio.

#!/bin/bash

LOOPS=5 #How many times to run each test
SIZE=1024 #Size of each test, multiples of 32 recommended for Q32 tests to give the most accurate results.

QSIZE=$(($SIZE / 32)) #Size of Q32Seq tests
SIZE+=m
QSIZE+=m

if [ -z $1 ]; then
 TARGET=$HOME
else
 TARGET="$1"
fi

DRIVE=$(df $TARGET | grep /dev | cut -d" " -f1)

echo "Running Benchmark on: $DRIVE
"

fio --loops=$LOOPS --size=$SIZE --filename=$TARGET/.fiomark.tmp --lockfile=exclusive --stonewall --ioengine=libaio --direct=1 
 --name=Bufread --loops=1 --bs=$SIZE --rw=read 
 --name=Seqread --bs=$SIZE --rw=read 
 --name=Seqwrite --bs=$SIZE --rw=write 
 --name=SeqQ32T1read --bs=$QSIZE --iodepth=32 --rw=read 
 --name=SeqQ32T1write --bs=$QSIZE --iodepth=32 --rw=write 
 --name=4kread --bs=4k --rw=randread 
 --name=4kwrite --bs=4k --rw=randwrite 
 --name=4kQ32T1read --bs=4k --iodepth=32 --rw=randread 
 --name=4kQ32T1write --bs=4k --iodepth=32 --rw=randwrite |&tee $TARGET/.fiomark.txt

SEQR=$(cat $TARGET/.fiomark.txt | grep BW | sed '2!d' | cut -d"(" -f2 | cut -d")" -f1)
SEQW=$(cat $TARGET/.fiomark.txt | grep BW | sed '3!d' | cut -d"(" -f2 | cut -d")" -f1 | cut -d";" -f1)
F12KR=$(cat $TARGET/.fiomark.txt | grep BW | sed '4!d' | cut -d"(" -f2 | cut -d")" -f1)
F12KW=$(cat $TARGET/.fiomark.txt | grep BW | sed '5!d' | cut -d"(" -f2 | cut -d")" -f1 | cut -d";" -f1)
FKR=$(cat $TARGET/.fiomark.txt | grep BW | sed '6!d' | cut -d"(" -f2 | cut -d")" -f1)
FKW=$(cat $TARGET/.fiomark.txt | grep BW | sed '7!d' | cut -d"(" -f2 | cut -d")" -f1 | cut -d";" -f1)
FK32R=$(cat $TARGET/.fiomark.txt | grep BW | sed '8!d' | cut -d"(" -f2 | cut -d")" -f1)
FK32W=$(cat $TARGET/.fiomark.txt | grep BW | sed '9!d' | cut -d"(" -f2 | cut -d")" -f1 | cut -d";" -f1)

echo "
Results from $DRIVE: 

Sequential Read:" $SEQR$(cat $TARGET/.fiomark.txt | grep BW | sed '2!d' | cut -d: -f2 | cut -d, -f1) $(cat $TARGET/.fiomark.txt | grep BW | sed '2!d' | cut -d")" -f2)")"
echo Sequential Write: $SEQW$(cat $TARGET/.fiomark.txt | grep BW | sed '3!d' | cut -d: -f2 | cut -d, -f1) $(cat $TARGET/.fiomark.txt | grep BW | sed '3!d' | cut -d")" -f2)")"
echo "
Sequential Q32T1 Read:" $F12KR$(cat $TARGET/.fiomark.txt | grep BW | sed '4!d' | cut -d: -f2 | cut -d, -f1) $(cat $TARGET/.fiomark.txt | grep BW | sed '4!d' | cut -d")" -f2)")"
echo Sequential Q32T1 Write: $F12KW$(cat $TARGET/.fiomark.txt | grep BW | sed '5!d' | cut -d: -f2 | cut -d, -f1) $(cat $TARGET/.fiomark.txt | grep BW | sed '5!d' | cut -d")" -f2)")"
echo "
4KB Read:" $FKR$(cat $TARGET/.fiomark.txt | grep BW | sed '6!d' | cut -d: -f2 | cut -d, -f1) $(cat $TARGET/.fiomark.txt | grep BW | sed '6!d' | cut -d")" -f2)")"
echo 4KB Write: $FKW$(cat $TARGET/.fiomark.txt | grep BW | sed '7!d' | cut -d: -f2 | cut -d, -f1) $(cat $TARGET/.fiomark.txt | grep BW | sed '7!d' | cut -d")" -f2)")"
echo "
4KB Q32T1 Read:" $FK32R$(cat $TARGET/.fiomark.txt | grep BW | sed '8!d' | cut -d: -f2 | cut -d, -f1) $(cat $TARGET/.fiomark.txt | grep BW | sed '8!d' | cut -d")" -f2)")"
echo 4KB Q32T1 Write: $FK32W$(cat $TARGET/.fiomark.txt | grep BW | sed '9!d' | cut -d: -f2 | cut -d, -f1) $(cat $TARGET/.fiomark.txt | grep BW | sed '9!d' | cut -d")" -f2)")"


rm $TARGET/.fiomark.txt $TARGET/.fiomark.tmp

Which will output results like this:

Results from /dev/sdb5: 

Sequential Read: 555MB/s IOPS=0 (5120MiB/9682msec)
Sequential Write: 133MB/s IOPS=0 (5120MiB/40426msec)

Sequential Q32T1 Read: 555MB/s IOPS=16 (5120MiB/9680msec)
Sequential Q32T1 Write: 136MB/s IOPS=4 (5120MiB/39619msec)

4KB Read: 38.9MB/s IOPS=9486 (5120MiB/138166msec)
4KB Write: 122MB/s IOPS=29.8k (5120MiB/43953msec)

4KB Q32T1 Read: 101MB/s IOPS=24.7k (5120MiB/53131msec)
4KB Q32T1 Write: 80.1MB/s IOPS=19.6k (5120MiB/67001msec

The script when run without arguments will test the speed of your home drive/partition. You can also enter a path to a directory on another hard drive if you would like to test that instead.

I ran into an issue however where fio's sequential writes are being incorrectly handled. They are much slower than on dd, on dd the write speeds are comparable to the reads, but in fio the reads are the same (more or less) as in dd but writes are only at around 1/4 of the available speeds.

Here is an alternative version that tries to replicate crystaldiskmark 6 behavior. (Does not calculate the IOPS for 4KQ8T8, too lazy to create the for loop to tally it up.)

#!/bin/bash

LOOPS=5 #How many times to run each test
SIZE=1024 #Size of each test, multiples of 32 recommended for Q32 tests to give the most accurate results.

QSIZE=$(($SIZE / 32)) #Size of Q32Seq tests
SIZE+=m
QSIZE+=m

if [ -z $1 ]; then
 TARGET=$HOME
else
 TARGET="$1"
fi

DRIVE=$(df $TARGET | grep /dev | cut -d" " -f1)

echo "Running Benchmark on: $DRIVE
"

fio --loops=$LOOPS --size=$SIZE --filename=$TARGET/.fiomark.tmp --stonewall --ioengine=libaio --direct=1 
 --name=Bufread --loops=1 --bs=$SIZE --rw=read 
 --name=SeqQ32T1read --bs=$QSIZE --iodepth=32 --numjobs=1 --rw=read 
 --name=SeqQ32T1write --bs=$QSIZE --iodepth=32 --numjobs=1 --rw=write 
 --name=4kQ8T8read --bs=4k --iodepth=8 --numjobs=8 --rw=randread 
 --name=4kQ8T8write --bs=4k --iodepth=8 --numjobs=8 --rw=randwrite 
 --name=4kQ32T1read --bs=4k --iodepth=32 --numjobs=1 --rw=randread 
 --name=4kQ32T1write --bs=4k --iodepth=32 --numjobs=1 --rw=randwrite 
 --name=4kread --bs=4k --numjobs=1 --rw=randread 
 --name=4kwrite --bs=4k --numjobs=1 --rw=randwrite |&tee $TARGET/.fiomark.txt

SEQR=$(cat $TARGET/.fiomark.txt | grep BW | sed '2!d' | cut -d"(" -f2 | cut -d")" -f1)
SEQW=$(cat $TARGET/.fiomark.txt | grep BW | sed '3!d' | cut -d"(" -f2 | cut -d")" -f1 | cut -d";" -f1)
F12KR=$(cat $TARGET/.fiomark.txt | grep READ | sed '3!d' | cut -d"(" -f2 | cut -d")" -f1)
F12KW=$(cat $TARGET/.fiomark.txt | grep WRITE | sed '2!d' | cut -d"(" -f2 | cut -d")" -f1 | cut -d";" -f1)
FKR=$(cat $TARGET/.fiomark.txt | grep BW | sed '20!d' | cut -d"(" -f2 | cut -d")" -f1)
FKW=$(cat $TARGET/.fiomark.txt | grep BW | sed '21!d' | cut -d"(" -f2 | cut -d")" -f1 | cut -d";" -f1)
FK32R=$(cat $TARGET/.fiomark.txt | grep BW | sed '22!d' | cut -d"(" -f2 | cut -d")" -f1)
FK32W=$(cat $TARGET/.fiomark.txt | grep BW | sed '23!d' | cut -d"(" -f2 | cut -d")" -f1 | cut -d";" -f1)

echo "
Results from $DRIVE: 

Sequential Q32T1 Read:" $SEQR$(cat $TARGET/.fiomark.txt | grep BW | sed '2!d' | cut -d: -f2 | cut -d, -f1) $(cat $TARGET/.fiomark.txt | grep BW | sed '2!d' | cut -d")" -f2)")"
echo Sequential Q32T1 Write: $SEQW$(cat $TARGET/.fiomark.txt | grep BW | sed '3!d' | cut -d: -f2 | cut -d, -f1) $(cat $TARGET/.fiomark.txt | grep BW | sed '3!d' | cut -d")" -f2)")"
echo "
4KB Q8T8 Read:" $F12KR $(cat $TARGET/.fiomark.txt | grep BW | sed '5!d' | cut -d")" -f2)")"
echo 4KB Q8T8 Write: $F12KW $(cat $TARGET/.fiomark.txt | grep BW | sed '15!d' | cut -d")" -f2)")"
echo "
4KB Q32T1 Read:" $FKR$(cat $TARGET/.fiomark.txt | grep BW | sed '20!d' | cut -d: -f2 | cut -d, -f1) $(cat $TARGET/.fiomark.txt | grep BW | sed '20!d' | cut -d")" -f2)")"
echo 4KB Q32T1 Write: $FKW$(cat $TARGET/.fiomark.txt | grep BW | sed '21!d' | cut -d: -f2 | cut -d, -f1) $(cat $TARGET/.fiomark.txt | grep BW | sed '21!d' | cut -d")" -f2)")"
echo "
4KB Q1T1 Read:" $FK32R$(cat $TARGET/.fiomark.txt | grep BW | sed '22!d' | cut -d: -f2 | cut -d, -f1) $(cat $TARGET/.fiomark.txt | grep BW | sed '22!d' | cut -d")" -f2)")"
echo 4KB Q1T1 Write: $FK32W$(cat $TARGET/.fiomark.txt | grep BW | sed '23!d' | cut -d: -f2 | cut -d, -f1) $(cat $TARGET/.fiomark.txt | grep BW | sed '23!d' | cut -d")" -f2)")"


rm $TARGET/.fiomark.txt $TARGET/.fiomark.tmp

I might create a zenity version or something like that if the issue with the write speeds is resolved.