mjhjmtu

This question is about compressing rather large data directories (around 22-23 GiB), specifically the integrity checks of such archives. I do these big backups about once a month, so the time it takes is not an issue here.

I am on Linux Mint 19.1 which is based on Ubuntu 18.04 and I want to use tar in combination with xz with the best checksum or verification available.

Software versions for future reference:

$ tar --version

tar (GNU tar) 1.29

and

$ xz --version

xz (XZ Utils) 5.2.2

Just to note:

• I burn it on M-Disc media.




• I don't mind if it takes long; I stress this.




• If some settings depend on memory, then I have 16 GiB DDR4 in this laptop.

It appears to me, according to the xz man page, I see is no way of setting a solid compression that would maybe help with compression ratio.

Until this day, I just use the simplest way of doing my BluRay backup in one go:

$ tar -cJf ...

It was relatively fast, but almost no compression occurred when compressing some .iso files, which were already compressed inside, and there is still my question about increasing, ... well, a better word would be ensuring the integrity of my archives.

With the help of:

$ xz -l ...

I was able to somewhat analyze the output result + I found the CRC-64 is in place as the checksum method.

Now when writing this, I found one interesting way to specify the compression level, which is of course not set to maximum (9), nor extreme by default:

$ XZ_OPT=-9e tar ...

I also found that I could switch the xz checksum from the default CRC-64 to SHA-256:

$ XZ_OPT='-9 --extreme --check=sha256' tar ...

What I am wondering about now, is the --verify option of tar, from the man page:

attempt to verify the archive after writing it

Because if doing it in one go as seen above, adding this option results in:

tar: Cannot verify compressed archives

Instead, I would have to do something like:

$ tar --create --file="$backup_file" "$dir_to_backup" --verify

And archive it separately with xz afterwards; I am unsure how to make use of it directly?

^^^ Which is my Question ^^^

Example archive with little compression possible, as all of its contents have already been compressed and enclosed in .iso files for instance. Still, I made 1.8 GiB by compressing it, which is nice.

$ xz --list backup--2019-03-09.tar.xz

backup--2019-03-09.tar.xz (1/1)
 Streams: 1
 Blocks: 127
 Compressed size: 22.0 GiB (23,670,187,308 B)
 Uncompressed size: 23.8 GiB (25,546,864,640 B)
 Ratio: 0.927
 Check: SHA-256
 Stream padding: 0 B
 Streams:
 Stream Blocks CompOffset UncompOffset CompSize UncompSize Ratio Check Padding
 1 127 0 0 23,670,187,308 25,546,864,640 0.927 SHA-256 0
 Blocks:
 Stream Block CompOffset UncompOffset TotalSize UncompSize Ratio Check
 1 1 12 0 53,243,880 201,326,592 0.264 SHA-256
 1 2 53,243,892 201,326,592 196,992,880 201,326,592 0.978 SHA-256
 1 3 250,236,772 402,653,184 201,315,216 201,326,592 1.000 SHA-256
 1 4 451,551,988 603,979,776 201,335,864 201,326,592 1.000 SHA-256
 1 5 652,887,852 805,306,368 201,335,864 201,326,592 1.000 SHA-256
 1 6 854,223,716 1,006,632,960 201,334,900 201,326,592 1.000 SHA-256
 1 7 1,055,558,616 1,207,959,552 200,622,316 201,326,592 0.997 SHA-256
 1 8 1,256,180,932 1,409,286,144 201,335,832 201,326,592 1.000 SHA-256
 1 9 1,457,516,764 1,610,612,736 201,335,864 201,326,592 1.000 SHA-256
 1 10 1,658,852,628 1,811,939,328 201,330,732 201,326,592 1.000 SHA-256
 1 11 1,860,183,360 2,013,265,920 199,713,948 201,326,592 0.992 SHA-256
 1 12 2,059,897,308 2,214,592,512 175,761,000 201,326,592 0.873 SHA-256
 1 13 2,235,658,308 2,415,919,104 201,165,620 201,326,592 0.999 SHA-256
 1 14 2,436,823,928 2,617,245,696 201,321,784 201,326,592 1.000 SHA-256
 1 15 2,638,145,712 2,818,572,288 201,326,148 201,326,592 1.000 SHA-256
 1 16 2,839,471,860 3,019,898,880 200,038,544 201,326,592 0.994 SHA-256
 1 17 3,039,510,404 3,221,225,472 200,750,292 201,326,592 0.997 SHA-256
 1 18 3,240,260,696 3,422,552,064 201,310,636 201,326,592 1.000 SHA-256
 1 19 3,441,571,332 3,623,878,656 201,335,864 201,326,592 1.000 SHA-256
 1 20 3,642,907,196 3,825,205,248 201,335,864 201,326,592 1.000 SHA-256
 1 21 3,844,243,060 4,026,531,840 201,335,864 201,326,592 1.000 SHA-256
 1 22 4,045,578,924 4,227,858,432 201,335,864 201,326,592 1.000 SHA-256
 1 23 4,246,914,788 4,429,185,024 201,330,168 201,326,592 1.000 SHA-256
 1 24 4,448,244,956 4,630,511,616 199,487,940 201,326,592 0.991 SHA-256
 1 25 4,647,732,896 4,831,838,208 201,335,172 201,326,592 1.000 SHA-256
 1 26 4,849,068,068 5,033,164,800 201,334,468 201,326,592 1.000 SHA-256
 1 27 5,050,402,536 5,234,491,392 201,170,860 201,326,592 0.999 SHA-256
 1 28 5,251,573,396 5,435,817,984 201,200,520 201,326,592 0.999 SHA-256
 1 29 5,452,773,916 5,637,144,576 191,043,588 201,326,592 0.949 SHA-256
 1 30 5,643,817,504 5,838,471,168 196,939,368 201,326,592 0.978 SHA-256
 1 31 5,840,756,872 6,039,797,760 185,189,612 201,326,592 0.920 SHA-256
 1 32 6,025,946,484 6,241,124,352 201,298,652 201,326,592 1.000 SHA-256
 1 33 6,227,245,136 6,442,450,944 201,137,020 201,326,592 0.999 SHA-256
 1 34 6,428,382,156 6,643,777,536 201,166,576 201,326,592 0.999 SHA-256
 1 35 6,629,548,732 6,845,104,128 201,230,404 201,326,592 1.000 SHA-256
 1 36 6,830,779,136 7,046,430,720 201,265,840 201,326,592 1.000 SHA-256
 1 37 7,032,044,976 7,247,757,312 201,277,512 201,326,592 1.000 SHA-256
 1 38 7,233,322,488 7,449,083,904 201,252,812 201,326,592 1.000 SHA-256
 1 39 7,434,575,300 7,650,410,496 201,268,876 201,326,592 1.000 SHA-256
 1 40 7,635,844,176 7,851,737,088 201,220,980 201,326,592 0.999 SHA-256
 1 41 7,837,065,156 8,053,063,680 201,286,716 201,326,592 1.000 SHA-256
 1 42 8,038,351,872 8,254,390,272 201,162,416 201,326,592 0.999 SHA-256
 1 43 8,239,514,288 8,455,716,864 201,167,132 201,326,592 0.999 SHA-256
 1 44 8,440,681,420 8,657,043,456 201,191,296 201,326,592 0.999 SHA-256
 1 45 8,641,872,716 8,858,370,048 201,234,228 201,326,592 1.000 SHA-256
 1 46 8,843,106,944 9,059,696,640 201,300,992 201,326,592 1.000 SHA-256
 1 47 9,044,407,936 9,261,023,232 198,262,448 201,326,592 0.985 SHA-256
 1 48 9,242,670,384 9,462,349,824 91,113,148 201,326,592 0.453 SHA-256
 1 49 9,333,783,532 9,663,676,416 143,398,764 201,326,592 0.712 SHA-256
 1 50 9,477,182,296 9,865,003,008 178,570,780 201,326,592 0.887 SHA-256
 1 51 9,655,753,076 10,066,329,600 188,475,820 201,326,592 0.936 SHA-256
 1 52 9,844,228,896 10,267,656,192 193,534,136 201,326,592 0.961 SHA-256
 1 53 10,037,763,032 10,468,982,784 157,357,336 201,326,592 0.782 SHA-256
 1 54 10,195,120,368 10,670,309,376 111,495,992 201,326,592 0.554 SHA-256
 1 55 10,306,616,360 10,871,635,968 99,397,052 201,326,592 0.494 SHA-256
 1 56 10,406,013,412 11,072,962,560 192,862,268 201,326,592 0.958 SHA-256
 1 57 10,598,875,680 11,274,289,152 180,224,836 201,326,592 0.895 SHA-256
 1 58 10,779,100,516 11,475,615,744 165,539,672 201,326,592 0.822 SHA-256
 1 59 10,944,640,188 11,676,942,336 170,200,512 201,326,592 0.845 SHA-256
 1 60 11,114,840,700 11,878,268,928 130,293,232 201,326,592 0.647 SHA-256
 1 61 11,245,133,932 12,079,595,520 194,065,376 201,326,592 0.964 SHA-256
 1 62 11,439,199,308 12,280,922,112 187,087,700 201,326,592 0.929 SHA-256
 1 63 11,626,287,008 12,482,248,704 187,057,960 201,326,592 0.929 SHA-256
 1 64 11,813,344,968 12,683,575,296 187,377,228 201,326,592 0.931 SHA-256
 1 65 12,000,722,196 12,884,901,888 199,580,888 201,326,592 0.991 SHA-256
 1 66 12,200,303,084 13,086,228,480 152,912,664 201,326,592 0.760 SHA-256
 1 67 12,353,215,748 13,287,555,072 169,255,644 201,326,592 0.841 SHA-256
 1 68 12,522,471,392 13,488,881,664 187,663,644 201,326,592 0.932 SHA-256
 1 69 12,710,135,036 13,690,208,256 194,150,692 201,326,592 0.964 SHA-256
 1 70 12,904,285,728 13,891,534,848 193,988,696 201,326,592 0.964 SHA-256
 1 71 13,098,274,424 14,092,861,440 195,142,924 201,326,592 0.969 SHA-256
 1 72 13,293,417,348 14,294,188,032 200,672,944 201,326,592 0.997 SHA-256
 1 73 13,494,090,292 14,495,514,624 199,290,184 201,326,592 0.990 SHA-256
 1 74 13,693,380,476 14,696,841,216 199,263,024 201,326,592 0.990 SHA-256
 1 75 13,892,643,500 14,898,167,808 183,145,436 201,326,592 0.910 SHA-256
 1 76 14,075,788,936 15,099,494,400 167,990,704 201,326,592 0.834 SHA-256
 1 77 14,243,779,640 15,300,820,992 196,509,992 201,326,592 0.976 SHA-256
 1 78 14,440,289,632 15,502,147,584 147,559,752 201,326,592 0.733 SHA-256
 1 79 14,587,849,384 15,703,474,176 183,516,020 201,326,592 0.912 SHA-256
 1 80 14,771,365,404 15,904,800,768 135,885,500 201,326,592 0.675 SHA-256
 1 81 14,907,250,904 16,106,127,360 182,531,984 201,326,592 0.907 SHA-256
 1 82 15,089,782,888 16,307,453,952 165,100,712 201,326,592 0.820 SHA-256
 1 83 15,254,883,600 16,508,780,544 161,361,236 201,326,592 0.801 SHA-256
 1 84 15,416,244,836 16,710,107,136 201,335,864 201,326,592 1.000 SHA-256
 1 85 15,617,580,700 16,911,433,728 199,923,388 201,326,592 0.993 SHA-256
 1 86 15,817,504,088 17,112,760,320 193,446,284 201,326,592 0.961 SHA-256
 1 87 16,010,950,372 17,314,086,912 198,499,544 201,326,592 0.986 SHA-256
 1 88 16,209,449,916 17,515,413,504 201,335,864 201,326,592 1.000 SHA-256
 1 89 16,410,785,780 17,716,740,096 201,335,864 201,326,592 1.000 SHA-256
 1 90 16,612,121,644 17,918,066,688 201,335,864 201,326,592 1.000 SHA-256
 1 91 16,813,457,508 18,119,393,280 201,335,864 201,326,592 1.000 SHA-256
 1 92 17,014,793,372 18,320,719,872 201,335,864 201,326,592 1.000 SHA-256
 1 93 17,216,129,236 18,522,046,464 201,335,864 201,326,592 1.000 SHA-256
 1 94 17,417,465,100 18,723,373,056 201,330,604 201,326,592 1.000 SHA-256
 1 95 17,618,795,704 18,924,699,648 201,335,864 201,326,592 1.000 SHA-256
 1 96 17,820,131,568 19,126,026,240 183,578,156 201,326,592 0.912 SHA-256
 1 97 18,003,709,724 19,327,352,832 201,335,864 201,326,592 1.000 SHA-256
 1 98 18,205,045,588 19,528,679,424 201,162,480 201,326,592 0.999 SHA-256
 1 99 18,406,208,068 19,730,006,016 201,333,120 201,326,592 1.000 SHA-256
 1 100 18,607,541,188 19,931,332,608 199,120,720 201,326,592 0.989 SHA-256
 1 101 18,806,661,908 20,132,659,200 201,335,864 201,326,592 1.000 SHA-256
 1 102 19,007,997,772 20,333,985,792 201,335,864 201,326,592 1.000 SHA-256
 1 103 19,209,333,636 20,535,312,384 201,335,772 201,326,592 1.000 SHA-256
 1 104 19,410,669,408 20,736,638,976 201,335,864 201,326,592 1.000 SHA-256
 1 105 19,612,005,272 20,937,965,568 179,075,024 201,326,592 0.889 SHA-256
 1 106 19,791,080,296 21,139,292,160 128,422,864 201,326,592 0.638 SHA-256
 1 107 19,919,503,160 21,340,618,752 186,242,664 201,326,592 0.925 SHA-256
 1 108 20,105,745,824 21,541,945,344 199,270,940 201,326,592 0.990 SHA-256
 1 109 20,305,016,764 21,743,271,936 198,996,836 201,326,592 0.988 SHA-256
 1 110 20,504,013,600 21,944,598,528 175,004,932 201,326,592 0.869 SHA-256
 1 111 20,679,018,532 22,145,925,120 200,046,576 201,326,592 0.994 SHA-256
 1 112 20,879,065,108 22,347,251,712 178,979,024 201,326,592 0.889 SHA-256
 1 113 21,058,044,132 22,548,578,304 199,886,808 201,326,592 0.993 SHA-256
 1 114 21,257,930,940 22,749,904,896 177,371,480 201,326,592 0.881 SHA-256
 1 115 21,435,302,420 22,951,231,488 190,387,928 201,326,592 0.946 SHA-256
 1 116 21,625,690,348 23,152,558,080 194,894,972 201,326,592 0.968 SHA-256
 1 117 21,820,585,320 23,353,884,672 199,717,584 201,326,592 0.992 SHA-256
 1 118 22,020,302,904 23,555,211,264 152,814,704 201,326,592 0.759 SHA-256
 1 119 22,173,117,608 23,756,537,856 199,462,188 201,326,592 0.991 SHA-256
 1 120 22,372,579,796 23,957,864,448 175,908,224 201,326,592 0.874 SHA-256
 1 121 22,548,488,020 24,159,191,040 181,534,208 201,326,592 0.902 SHA-256
 1 122 22,730,022,228 24,360,517,632 195,606,504 201,326,592 0.972 SHA-256
 1 123 22,925,628,732 24,561,844,224 192,220,808 201,326,592 0.955 SHA-256
 1 124 23,117,849,540 24,763,170,816 188,762,236 201,326,592 0.938 SHA-256
 1 125 23,306,611,776 24,964,497,408 173,512,276 201,326,592 0.862 SHA-256
 1 126 23,480,124,052 25,165,824,000 106,397,932 201,326,592 0.528 SHA-256
 1 127 23,586,521,984 25,367,150,592 83,664,288 179,714,048 0.466 SHA-256