How to debug this? This issue has suddenly appeared within the last couple of days. All backups of a website are corrupted.

If the backup is just left as tar, there are no problems, but as soon the tar is compressed as gz or xz I can't uncompress them.

There is a lot of free disk

Local disk space 2.68 TB total / 2.26 TB free / 432.46 GB used

error

tar: Skipping to next header[===============================> ] 39% ETA 0:01:14
tar: A lone zero block at 2291466===============================> ] 44% ETA 0:01:13
tar: Exiting with failure status due to previous errors
 878MiB 0:00:58 [15.1MiB/s] [===================================> ] 44%

And why does it say Skipping to next header? It has never done that before. Something is terribly wrong the some of the files.

There are about 15k pdf, jpg or png files in the directories.

command

pv $backup_file | tar -izxf - -C $import_dir

There must be some data that corrupts the compression.

I have also tried to check the HDD health by doing this:

# getting the drives
lsblk -dpno name

smartctl -H /dev/sda
smartctl -H /dev/sdb

On both drives I get this:

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

How can I find out which files that are corrupting the tar.gz? I just want to delete them.

update

Have now copied all files to another server and I have the exact same issue. I can tar everything and extract it without problems, but as soon I want to compress the files, I can't uncompress them (gz/xz).

Your file is either truncated or corrupted, so xz can't get to the end of the data. tar complains because the archive stops in the middle, which is logical since xz didn't manage to read the whole data.

Run the following commands to check where the problem is:

cat /var/www/bak/db/2017-05-20-1200_mysql.tar.xz >/dev/null
xzcat /var/www/bak/db/2017-05-20-1200_mysql.tar.xz >/dev/null

If cat complains then the file is corrupted on the disk and the operating system detected the corruption. Check the kernel logs for more information; usually the disk needs to be replaced at this point. If only xz complains then the OS didn't detect any corruption but the file is nevertheless not valid (either corrupted or truncated). Either way, you aren't going to be able to recover this file. You'll need to get it back from your offline backups.

I don't see any mentioning of how the broken tar files are created?

You say it's backups from a web site, but the issues you're showing are all when restoring/unpacking, so there (the source) is where you need to put the trouble shooting effort.

If the files can't be uncompressed after moving the backup to another machine/location, they must be either created faulty, or broken in transport.

To locate the source of the error:

• manually create a backup on the web server (without pv and without -i)


• manually test the backup on the web server (without pv and without -i)

If no problems found so far:

• copy the backup from the web server


• test the copied backup on the target machine (without pv and without -i)

If no problems found so far, the backup script doesn't create the archive the same way you did when doing it by hand (and should probably be modified to do what you did manually).

Also, make sure to use the absolute paths of all involved commands. If you have a bad $PATH and/or $LD_LIBRARY_PATH variable and an intruder in the system, you might be using trojaned binaries, which could cause unintentional side-effects.

It could of course also be incompatible tar versions involved, unless both systems are debian. You could try forcing POSIX-mode on both sides.

You're using the flag -i that in its long form is --ignore-zeros.
This is why tar does not complain about the files that are corrupted.
So, if you want to debug your tar file, just remove the -i option and you'll get the list of corrupted files.

There are also 2 other ways to find corrupted files on unix (in general). I quote an answer given in another question.

rsync can be used to copy directories, and is capable of restarting the copy from the point at which it terminated if any error causes the rsync to die.

Using rsync's --dry-run option you can see what would be copied without actually copying anything. The --stats and --progress options would also be useful. and --human-readable or -h is easier to read.

e.g.

rsync --dry-run -avh --stats --progress /path/to/src/ /path/to/destination/

I'm not sure if rsync is installed by default on Mac OS X, but I have used it on Macs so I know it's definitely available.

For a quick-and-dirty check on whether files in a subdirectory can be read or not, you could use grep -r XXX /path/to/directory/ > /dev/null. The search regexp doesn't matter, because output is being discarded anyway.

STDOUT is being redirected to /dev/null, so you'll only see errors.

The only reason I chose grep here was because of its -R recursion option. There are many other commands that could be used instead of
grep here, and even more if used with find.

As reference: Finding corrupted files

The line of reasoning in answer by @MattBianco is what I would methodically follow to solve this particular issue.

Zeroed blocks indicate EOF, but that's dependent on the blocking factor (the default is a compiled constant, typically 20). Tar's --compare|--diff appear to execute with --ignore-zeros (-i) implicitly.

Given the extra complication of pv, I suspect tar -i is causing issues for xz, looking at tar man on blocking factor I'd suggest first removing -i

Then if that doesn't help, replacing with:

--read-full-records --blocking-factor=300

If you're just reading this having googled "tar: A lone zero block at N", and aren't piping anything then try --ignore-zeros.