mjhjmtu

I have a backup server which takes backup of other servers and in few days back the backup server started experiencing error and the not able to take back. I think the error log bellow could describe the problem better than me:

2012-09-10 20:05:40 Aborting backup up after signal PIPE
2012-09-10 20:05:44 Got fatal error during xfer (aborted by signal=PIPE)
2012-09-10 21:00:15 full backup started for directory rootBackup (baseline backup #277)

2012-09-11 10:21:07 Aborting backup up after signal PIPE
2012-09-11 10:21:11 Got fatal error during xfer (aborted by signal=PIPE)
2012-09-11 11:00:24 full backup started for directory rootBackup (baseline backup #277)
2012-09-11 13:22:41 Aborting backup up after signal PIPE
2012-09-11 13:22:52 Got fatal error during xfer (aborted by signal=PIPE)
2012-09-11 14:00:30 full backup started for directory rootBackup (baseline backup #277)for directory rootBackup
2012-09-14 06:38:02 Got fatal error during xfer (Child exited prematurely)
2012-09-14 06:38:08 Backup aborted (Child exited prematurely)
2012-09-14 07:00:27 incr backup started back to 2012-09-11 14:00:30 (backup #278) for directory rootBackup
2012-09-20 14:22:04 Got fatal error during xfer (Child exited prematurely)
2012-09-20 14:22:10 Backup aborted (Child exited prematurely)
2012-09-20 15:00:45 incr backup started back to 2012-09-19 05:00:02 (backup #282) for directory rootBackup
2012-09-20 15:26:29 Got fatal error during xfer (Child exited prematurely)
2012-09-20 15:26:35 Backup aborted (Child exited prematurely)
2012-09-20 16:00:12 incr backup started back to 2012-09-19 05:00:02 (backup #282) for directory rootBackup
2012-09-20 16:27:37 Got fatal error during xfer (Child exited prematurely)
2012-09-20 16:27:43 Backup aborted (Child exited prematurely)
2012-09-20 17:00:09 incr backup started back to 2012-09-19 05:00:02 (backup #282) for directory rootBackup
2012-09-20 17:27:37 Got fatal error during xfer (Child exited prematurely)
2012-09-20 17:27:43 Backup aborted (Child exited prematurely)
2012-09-20 18:00:20 incr backup started back to 2012-09-19 05:00:02 (backup #282) for directory rootBackup
2012-09-20 18:27:49 Got fatal error during xfer (Child exited prematurely)
2012-09-20 18:27:55 Backup aborted (Child exited prematurely)
2012-09-20 19:00:26 incr backup started back to 2012-09-19 05:00:02 (backup #282) for directory rootBackup
2012-09-20 19:28:29 Got fatal error during xfer (Child exited prematurely)
2012-09-20 19:28:36 Backup aborted (Child exited prematurely)
2012-09-20 20:00:32 incr backup started back to 2012-09-19 05:00:02 (backup #282) for directory rootBackup
2012-09-20 20:23:06 Got fatal error during xfer (Child exited prematurely)
2012-09-20 20:23:11 Backup aborted (Child exited prematurely)
2012-09-20 21:00:16 incr backup started back to 2012-09-19 05:00:02 (backup #282) for directory rootBackup
2012-09-20 21:20:30 Got fatal error during xfer (Child exited prematurely)
2012-09-20 21:20:37 Backup aborted (Child exited prematurely)
2012-09-20 22:00:15 incr backup started back to 2012-09-19 05:00:02 (backup #282) for directory rootBackup
2012-09-20 22:15:21 Got fatal error during xfer (Child exited prematurely)
2012-09-20 22:15:26 Backup aborted (Child exited prematurely)
2012-09-20 23:00:21 incr backup started back to 2012-09-19 05:00:02 (backup #282) for directory rootBackup
2012-09-20 23:15:06 Got fatal error during xfer (Child exited prematurely)
2012-09-20 23:15:11 Backup aborted (Child exited prematurely)
2012-09-21 01:00:09 incr backup started back to 2012-09-19 05:00:02 (backup #282) for directory rootBackup
2012-09-21 01:27:47 Got fatal error during xfer (Child exited prematurely)
2012-09-21 01:27:53 Backup aborted (Child exited prematurely)
2012-09-21 02:00:20 incr backup started back to 2012-09-19 05:00:02 (backup #282) for directory rootBackup
2012-09-21 02:34:06 Got fatal error during xfer (Child exited prematurely)
2012-09-21 02:34:13 Backup aborted (Child exited prematurely)
2012-09-21 03:00:21 incr backup started back to 2012-09-19 05:00:02 (backup #282) for directory rootBackup
2012-09-21 03:26:13 Got fatal error during xfer (Child exited prematurely)
2012-09-21 03:26:19 Backup aborted (Child exited prematurely)
2012-09-21 04:00:14 incr backup started back to 2012-09-19 05:00:02 (backup #282) for directory rootBackup
2012-09-21 04:17:05 Got fatal error during xfer (Child exited prematurely)
2012-09-21 04:17:11 Backup aborted (Child exited prematurely)
2012-09-21 05:00:09 incr backup started back to 2012-09-19 05:00:02 (backup #282) for directory rootBackup
2012-09-21 05:27:00 Aborting backup up after signal PIPE
2012-09-21 05:27:02 Got fatal error during xfer (aborted by signal=PIPE)
2012-09-21 06:00:04 incr backup started back to 2012-09-19 05:00:02 (backup #282) for directory rootBackup
2012-09-21 06:19:01 Aborting backup up after signal PIPE
2012-09-21 06:19:02 Got fatal error during xfer (aborted by signal=PIPE)
2012-09-21 07:00:01 incr backup started back to 2012-09-19 05:00:02 (backup #282) for directory rootBackup
2012-09-21 07:11:57 incr backup 283 complete, 138 files, 282332868 bytes, 0 xferErrs (0 bad files, 0 bad shares, 0 other)
2012-09-22 07:00:09 incr backup started back to 2012-09-19 05:00:02 (backup #282) for directory rootBackup
2012-09-22 07:11:31 Got fatal error during xfer (Child exited prematurely)
2012-09-22 07:11:36 Backup aborted (Child exited prematurely)
2012-09-22 08:00:07 incr backup started back to 2012-09-19 05:00:02 (backup #282) for directory rootBackup
2012-09-22 08:16:04 Got fatal error during xfer (Child exited prematurely)
2012-09-22 08:16:10 Backup aborted (Child exited prematurely)
2012-09-22 09:00:05 incr backup started back to 2012-09-19 05:00:02 (backup #282) for directory rootBackup
2012-09-22 09:25:18 Got fatal error during xfer (Child exited prematurely)
2012-09-22 09:25:26 Backup aborted (Child exited prematurely)
2012-09-22 10:00:29 incr backup started back to 2012-09-19 05:00:02 (backup #282) for directory rootBackup
2012-09-22 10:26:23 Got fatal error during xfer (Child exited prematurely)
2012-09-22 10:26:28 Backup aborted (Child exited prematurely)
2012-09-22 11:00:13 incr backup started back to 2012-09-19 05:00:02 (backup #282) for directory rootBackup
2012-09-22 11:23:26 Got fatal error during xfer (Child exited prematurely)
2012-09-22 11:23:32 Backup aborted (Child exited prematurely)
2012-09-22 12:00:13 incr backup started back to 2012-09-19 05:00:02 (backup #282) for directory rootBackup
2012-09-22 12:16:27 Got fatal error during xfer (Child exited prematurely)
2012-09-22 12:16:33 Backup aborted (Child exited prematurely)
2012-09-22 13:00:11 incr backup started back to 2012-09-19 05:00:02 (backup #282) for directory rootBackup
2012-09-22 13:25:22 Got fatal error during xfer (Child exited prematurely)
2012-09-22 13:25:27 Backup aborted (Child exited prematurely)
2012-09-22 14:00:33 incr backup started back to 2012-09-19 05:00:02 (backup #282) for directory rootBackup
2012-09-22 14:26:53 Got fatal error during xfer (Child exited prematurely)
2012-09-22 14:26:58 Backup aborted (Child exited prematurely)
2012-09-22 15:00:10 incr backup started back to 2012-09-19 05:00:02 (backup #282) for directory rootBackup
2012-09-22 15:22:22 Got fatal error during xfer (Child exited prematurely)
2012-09-22 15:22:30 Backup aborted (Child exited prematurely)
2012-09-22 16:00:38 incr backup started back to 2012-09-19 05:00:02 (backup #282) for directory rootBackup
2012-09-22 16:25:32 Got fatal error during xfer (Child exited prematurely)
2012-09-22 16:25:38 Backup aborted (Child exited prematurely)
2012-09-22 17:00:11 incr backup started back to 2012-09-19 05:00:02 (backup #282) for directory rootBackup
2012-09-22 17:24:36 Got fatal error during xfer (Child exited prematurely)
2012-09-22 17:24:41 Backup aborted (Child exited prematurely)
2012-09-22 18:00:21 incr backup started back to 2012-09-19 05:00:02 (backup #282) for directory rootBackup
2012-09-22 18:24:42 Got fatal error during xfer (Child exited prematurely)
2012-09-22 18:24:50 Backup aborted (Child exited prematurely)
2012-09-22 19:00:20 incr backup started back to 2012-09-19 05:00:02 (backup #282) for directory rootBackup
2012-09-22 19:24:47 Got fatal error during xfer (Child exited prematurely)
2012-09-22 19:24:53 Backup aborted (Child exited prematurely)
2012-09-22 20:00:20 incr backup started back to 2012-09-19 05:00:02 (backup #282) for directory rootBackup
2012-09-22 20:23:21 Got fatal error during xfer (Child exited prematurely)
2012-09-22 20:23:26 Backup aborted (Child exited prematurely)
2012-09-22 21:00:14 incr backup started back to 2012-09-19 05:00:02 (backup #282) for directory rootBackup
2012-09-22 21:43:12 Aborting backup up after signal PIPE
2012-09-22 21:43:13 Got fatal error during xfer (aborted by signal=PIPE)
2012-09-22 22:00:10 incr backup started back to 2012-09-19 05:00:02 (backup #282) for directory rootBackup
2012-09-22 22:14:58 Got fatal error during xfer (Child exited prematurely)
2012-09-22 22:15:04 Backup aborted (Child exited prematurely)
2012-09-22 23:00:15 incr backup started back to 2012-09-19 05:00:02 (backup 

Note: I am using backuppc for backup. Backup server is able to take backup for other servers but for one server its experiencing error. So I believe there must be some problem in that particular client server so that its prematurely exiting .

Update:

Network seams to be OK(netstat -in) on client side.

Kernel Interface table
Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
eth0 1500 0 1297053361 0 126811 0 2321209324 0 0 0 BMRU
lo 16436 0 35169163 0 0 0 35169163 0 0 0 LRU

The bellow is the contents of: file /var/lib/backuppc/pc/ns381613.ovh.net-daily/XferLOG.bad.z, modified 2012-11-21 17:25:14 (Extracting only Errors)

full backup started for directory www (baseline backup #300)
Connected to ns381613.ovh.net:873, remote version 30
Negotiated protocol version 28
Connected to module www
Sending args: --server --sender --numeric-ids --perms --owner --group -D --links --hard-links --times --block-size=2048 --recursive --ignore-times . .
Xfer PIDs are now 4845
[ skipped 120129 lines ]
Remote[2]: file has vanished: "/mywebsite.com/sessions/sess_017p907qdbm3rn5vv8gdin2ab7" (in www)
Remote[2]: file has vanished: "/mywebsite.com/sessions Remote[2]: file has vanished: "/mywebsite.com/sessions/sess_vkn369qbj5dhoj4no0bei1sen3" (in www)
Remote[2]: file has vanished: "/mywebsite.com/sessions/sess_vkn7demudpv0othe6e98s2v1v4" (in www)
Remote[2]: file has vanished: "/mywebsite.com/sessions/sess_voseo6s018c8tmocgthj87irj1" (in www)
Remote[2]: file has vanished: "/mywebsite.com/sessions/sess_vqpqhv16urbrh99acecmujj8j1" (in www)
Remote[2]: file has vanished: "/mywebsite.com/sessions/sess_vs0j1bsfina3f4913lorb7j681" (in www)
Remote[2]: file has vanished: "/mywebsite.com/sessions/sess_vu3jupeug0qpls9f6ikub544t0" (in www)
Remote[2]: file has vanished: "/mywebsite.com/sessions/sess_vu3pp00ko7uip62jf7vp8o8la1" (in www)
[ skipped 55244 lines ]
Read EOF: Connection reset by peer
Can't write 32780 bytes to socket
Tried again: got 0 bytes
finish: removing in-process file mywebsite.com/www-bk-17-oct-2012/lms/archive/LAWYER_1350329888/document/screencasts/TechtutorTv.flv
Child is aborting
Done: 160949 files, 2244371209 bytes
Got fatal error during xfer (aborted by signal=PIPE)
Backup aborted by user signal
Not saving this as a partial backup since it has fewer files than the prior one (got 160949 and 160949 files versus 5985910)

Then I tried:

sudo -u backuppc /usr/share/backuppc/bin/BackupPC_dump -v -f
ns381613.ovh.net

Output

Sending args: --server --sender --numeric-ids --perms --owner --group -D --links --hard-links --times --block-size=2048 --recursive --ignore-times . .

Xfer PIDs are now 22310
xferPids 22310
Remote[1]: rsync error: timeout in data send/receive (code 30) at io.c(137) [sender=3.0.7]
Read EOF: 
Tried again: got 0 bytes
Child is aborting
Parent read EOF from child: fatal error!
Done: 0 files, 0 bytes
Got fatal error during xfer (Child exited prematurely)
cmdSystemOrEval: about to system /bin/ping -c 1 ns381613.ovh.net
cmdSystemOrEval: finished: got output PING ns381613.ovh.net (188.165.247.43) 56(84) bytes of data.
64 bytes from ns381613.ovh.net (188.165.247.43): icmp_req=1 ttl=60 time=0.591 ms

--- ns381613.ovh.net ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.591/0.591/0.591/0.000 ms

cmdSystemOrEval: about to system /bin/ping -c 1 ns381613.ovh.net
cmdSystemOrEval: finished: got output PING ns381613.ovh.net (188.165.247.43) 56(84) bytes of data.
64 bytes from ns381613.ovh.net (188.165.247.43): icmp_req=1 ttl=60 time=0.366 ms

--- ns381613.ovh.net ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.366/0.366/0.366/0.000 ms

CheckHostAlive: returning 0.366
Backup aborted (Child exited prematurely)
Not saving this as a partial backup since it has fewer files than the prior one (got 0 and 0 files versus 1)
dump failed: Child exited prematurely

In the above output I see it says timeout, But I have increased the timeout variable in configuration.

just a shot in the dark, but did the number of files in a backed up directory on the client end, increased significantly around the time this problem started exhibiting itself ? The message

(aborted by signal=PIPE)

makes me think of a Unix pipe. In the backup nomenclature, this pipe maybe a pipe of file names. And when the number of files exceed a certain buffer size (unable elaborate on that without knowing the system architectures of both the server and the client), you may encounter errors as such. Since this error is coming from a backup application that I am not familiar with, I can not attest to correlation between the message and the number of files, but it doesn't hurt to check.

How you can get around it, if it is the case that is, is totally up to you. Trim the directory by eliminating older files or create sub-directories and push certain number of files into these directories. Or a combination of the two.

Good luck