mjhjmtu

I often work on full disk images, I often need to read and write full disk images on not so capable laptops using a USB-3 disk as my temporary medium.

Duplicating a raw image is probably the fastest way, but I have to deal with limited storage space available.

What I need is high throughput and low CPU usage - compression ratio is not that important for me.

I'd like to have a tool that can very lightly compress the images created - mostly encoding empty space on the disk so that I can only store useful data, while making it as fast as possible - hopefully almost as fast as a raw dd duplication.

I tied to use pxz -1, as it can use multiple CPU cores to speed up the compression, hoping for the fastest compression to still be useful. That doesn't however seem to be as fat as I'd like.

Compressing a 500 GB disk image was estimated to take 21 hours:

mint Backup # dd if=/dev/sda bs=100M | pv -ptera -s500G | pxz -1 > Lenovo-Win8-sda.dd.xz
0:09:21 [9.26MiB/s] [6.54MiB/s] [> ] 0% ETA 21:34:46

No compression at all predicted 3 hours to complete the imaging:

mint Backup # dd if=/dev/sda bs=100M | pv -ptera -s500G > Lenovo-Win8-sda.dd
0:00:20 [80.2MiB/s] [49.8MiB/s] [> ] 0% ETA 2:50:20

Using lbzip2 -1 for compresson seems to be slightly faster than pxz -1 with ETA at 15 hours, which is still very slow.

mint Backup # dd if=/dev/sda bs=100M | pv -ptera -s500G | lbzip2 -1 > Lenovo-Win8-sda.dd.bz2
0:00:22 [9.07MiB/s] [9.76MiB/s] [> ] 0% ETA 14:33:38

Are there any faster compression tools for Linux?

LZ4

apt install liblz4-tool

Then you can compress with lz4 command and decompress with lz4 -d.

It defaults to the fastest compression mode.

500 GB from internal hard drive to an external drive over USB 3.0 is estimated to take between 2 and 3 hours:

mint Backup # dd if=/dev/sda bs=100M | pv -ptera -s500G | lz4 > Lenovo-Win8-sda.dd.lz4
0:02:47 [97.5MiB/s] [58.1MiB/s] [> ] 1% ETA 2:24:11

So, this won't be the absolute fastest (It's slower than LZ4, LZOP, and Snappy), but might be fast enough for your usage and will get way better ratios than LZ4 (and transferring less data will save you time too).

ZStandard

Official website: https://facebook.github.io/zstd/

The Debian package is called zstd

Unlike LZ4, it's multi-threaded (for both compression and decompression), and with the lowest compression settings it can easily saturate a USB 3.0 link (which your test with LZ4 is probably already doing), while getting compression ratios on par with the default settings in Gzip.

This sounds like an XY Problem. 
A generic compression tool
isn’t particularly likely to do a good job on a disk image,
because unused space is not guaranteed to be empty. 
(Admittedly, there are ways to zero out unused space in a filesystem.)

You might be better off using a backup tool — such as rsync, Clonezilla,
Time Machine, Duplicity, dump, or even tar, to name a few —
that understands the filesystem format
and backs up only the portions that are used. 
See this question for a discussion of some of them. 
Such an approach would have the advantage that it becomes much easier
to restore the backup to a disk of a different size.

I use Clonezilla and it can create cloned copies or compressed images of a whole drive including the head of the drive with boot sector for BIOS mode, partition table and everything.

A Clonezilla image is a directory with a number of files, where the big files are compressed, and Clonezilla can restore from such an image to a drive of exactly the same size or bigger (but not to a smaller drive). Such a drive behaves like the original one (is bootable etc). If the drive is bigger, and the partition table is GPT, you may have to restore the backup partition table at the tail end of the drive, manually with gdisk or with the script gpt-fix.

Clonezilla is smart enough to identify used blocks and skip free blocks in file systems (I have used it with several linux file systems and Microsoft file systems, where it uses partclone), so if there is a fair pencentage of free blocks, it will be much faster that 'raw' cloning and compression (with or without zeroed out free space).

When I want high compression and very flexible usage without the end user needing Clonezilla, I zero out free space and pipe dd via xz to get an img.xz file. But this process is very slow, and not suitable for your case.