rsync --delete not removing all deleted files
Clash Royale CLAN TAG#URR8PPP
up vote
0
down vote
favorite
I am using rsync for backing up several systems, either to external hard drives or over the network to another machine. The idea is to maintain a replica of the original filesystem, with a few things like /dev, /proc, /mnt, and what not excluded, which is then snapshotted at the destination (i.e. with btrfs or zfs). To that end, I use a command along the lines of
rsync -avP --delete --exclude /dev/* --exclude /proc/* --exclude /mnt/* etc. etc. / backup@fileserver:/mnt/array/backup/machine/
Which works reasonably well. However, I just noticed that rsync is not reliably deleting things on the destination. As a result, there are many more files on the destination than exist at any time on the source, in one case the back up was 100 GB larger than the source. The leftover files seem to be mostly old versions of various software and libraries. For example, on the original machine, /usr/include/qt/QtWidgets/ contains
5.9.2
QAbstractButton
qabstractbutton.h
....
But on the backup target, the same folder contains:
5.6.0
5.6.1
5.7.0
5.7.1
5.8.0
5.9.0
5.9.1
5.9.2
QAbstractButton
qabstractbutton.h
rsync seems to be copying over the new files just fine, but forgetting to remove all the old ones. The odd part is that some files are in fact removed (rsync always reports deleting a lot of files when synchronizing) but it seems that many files are not actually deleted. As a result, my backups are taking up a lot more space than they should, and they can't be directly restored because they can take up more space than the original partition to be due to all the extra files, and things like flat-file package databases are all screwed up, containing many duplicate entries for old versions packages.
Any ideas on what might be going on here? Is there some other flag I need to pass to rsync so that it actually deletes everything it's supposed to?
rsync
add a comment |Â
up vote
0
down vote
favorite
I am using rsync for backing up several systems, either to external hard drives or over the network to another machine. The idea is to maintain a replica of the original filesystem, with a few things like /dev, /proc, /mnt, and what not excluded, which is then snapshotted at the destination (i.e. with btrfs or zfs). To that end, I use a command along the lines of
rsync -avP --delete --exclude /dev/* --exclude /proc/* --exclude /mnt/* etc. etc. / backup@fileserver:/mnt/array/backup/machine/
Which works reasonably well. However, I just noticed that rsync is not reliably deleting things on the destination. As a result, there are many more files on the destination than exist at any time on the source, in one case the back up was 100 GB larger than the source. The leftover files seem to be mostly old versions of various software and libraries. For example, on the original machine, /usr/include/qt/QtWidgets/ contains
5.9.2
QAbstractButton
qabstractbutton.h
....
But on the backup target, the same folder contains:
5.6.0
5.6.1
5.7.0
5.7.1
5.8.0
5.9.0
5.9.1
5.9.2
QAbstractButton
qabstractbutton.h
rsync seems to be copying over the new files just fine, but forgetting to remove all the old ones. The odd part is that some files are in fact removed (rsync always reports deleting a lot of files when synchronizing) but it seems that many files are not actually deleted. As a result, my backups are taking up a lot more space than they should, and they can't be directly restored because they can take up more space than the original partition to be due to all the extra files, and things like flat-file package databases are all screwed up, containing many duplicate entries for old versions packages.
Any ideas on what might be going on here? Is there some other flag I need to pass to rsync so that it actually deletes everything it's supposed to?
rsync
add a comment |Â
up vote
0
down vote
favorite
up vote
0
down vote
favorite
I am using rsync for backing up several systems, either to external hard drives or over the network to another machine. The idea is to maintain a replica of the original filesystem, with a few things like /dev, /proc, /mnt, and what not excluded, which is then snapshotted at the destination (i.e. with btrfs or zfs). To that end, I use a command along the lines of
rsync -avP --delete --exclude /dev/* --exclude /proc/* --exclude /mnt/* etc. etc. / backup@fileserver:/mnt/array/backup/machine/
Which works reasonably well. However, I just noticed that rsync is not reliably deleting things on the destination. As a result, there are many more files on the destination than exist at any time on the source, in one case the back up was 100 GB larger than the source. The leftover files seem to be mostly old versions of various software and libraries. For example, on the original machine, /usr/include/qt/QtWidgets/ contains
5.9.2
QAbstractButton
qabstractbutton.h
....
But on the backup target, the same folder contains:
5.6.0
5.6.1
5.7.0
5.7.1
5.8.0
5.9.0
5.9.1
5.9.2
QAbstractButton
qabstractbutton.h
rsync seems to be copying over the new files just fine, but forgetting to remove all the old ones. The odd part is that some files are in fact removed (rsync always reports deleting a lot of files when synchronizing) but it seems that many files are not actually deleted. As a result, my backups are taking up a lot more space than they should, and they can't be directly restored because they can take up more space than the original partition to be due to all the extra files, and things like flat-file package databases are all screwed up, containing many duplicate entries for old versions packages.
Any ideas on what might be going on here? Is there some other flag I need to pass to rsync so that it actually deletes everything it's supposed to?
rsync
I am using rsync for backing up several systems, either to external hard drives or over the network to another machine. The idea is to maintain a replica of the original filesystem, with a few things like /dev, /proc, /mnt, and what not excluded, which is then snapshotted at the destination (i.e. with btrfs or zfs). To that end, I use a command along the lines of
rsync -avP --delete --exclude /dev/* --exclude /proc/* --exclude /mnt/* etc. etc. / backup@fileserver:/mnt/array/backup/machine/
Which works reasonably well. However, I just noticed that rsync is not reliably deleting things on the destination. As a result, there are many more files on the destination than exist at any time on the source, in one case the back up was 100 GB larger than the source. The leftover files seem to be mostly old versions of various software and libraries. For example, on the original machine, /usr/include/qt/QtWidgets/ contains
5.9.2
QAbstractButton
qabstractbutton.h
....
But on the backup target, the same folder contains:
5.6.0
5.6.1
5.7.0
5.7.1
5.8.0
5.9.0
5.9.1
5.9.2
QAbstractButton
qabstractbutton.h
rsync seems to be copying over the new files just fine, but forgetting to remove all the old ones. The odd part is that some files are in fact removed (rsync always reports deleting a lot of files when synchronizing) but it seems that many files are not actually deleted. As a result, my backups are taking up a lot more space than they should, and they can't be directly restored because they can take up more space than the original partition to be due to all the extra files, and things like flat-file package databases are all screwed up, containing many duplicate entries for old versions packages.
Any ideas on what might be going on here? Is there some other flag I need to pass to rsync so that it actually deletes everything it's supposed to?
rsync
asked Oct 28 '17 at 1:25
alex.forencich
205128
205128
add a comment |Â
add a comment |Â
1 Answer
1
active
oldest
votes
up vote
2
down vote
accepted
Some suggestions for things to try come from the contents of the man page (the section about --delete
). There are a lot of caveats and warnings. Have you considered all of these?
Works only "for the directories that are being synchronized". I guess this means they must have a parent/ancestor directory being copied.
Similarly, it doesn't apply to anything being explicitly excluded:
Files that are excluded from the transfer are also excluded from
being deleted unless you use the--delete-excluded
option or mark
the rules as only matching on the sending side.
If the sending side detects any I/O errors, then the deletion of any files at the destination will be automatically disabled. Override with
--ignore-errors
.Try with
--dry-run
first? Combine with--verbose
and closely examine everything that's happening with problem directories.
This option can be dangerous if used incorrectly! It is a very good
idea to first try a run using the --dry-run option (-n) to see
what files are going to be deleted.
So consider trying --delete-excluded
, --ignore-errors
, and/or --dry-run
/--verbose
...not necessarily as a "fix" but at the very least there may be helpful data to be gleaned.
Also, take a look at the --delete-before
flag. This is supposed to do the receive-side deletes before the actual transfer. Again, this might not fix everything but could be revealing. It'd be interesting to see if it helps reduce disk space usage, too.
Hmm, maybe --ignore-errors is the ticket. Invariably, rsync runs in to some sort of error with a file disappearing before it can be copied or a gvfs mount point that can't be read by root. I'll try adding that flag and see what happens.
â alex.forencich
Oct 28 '17 at 19:49
Yep, --ignore-errors was the ticket. Thanks!
â alex.forencich
Oct 28 '17 at 19:54
add a comment |Â
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
2
down vote
accepted
Some suggestions for things to try come from the contents of the man page (the section about --delete
). There are a lot of caveats and warnings. Have you considered all of these?
Works only "for the directories that are being synchronized". I guess this means they must have a parent/ancestor directory being copied.
Similarly, it doesn't apply to anything being explicitly excluded:
Files that are excluded from the transfer are also excluded from
being deleted unless you use the--delete-excluded
option or mark
the rules as only matching on the sending side.
If the sending side detects any I/O errors, then the deletion of any files at the destination will be automatically disabled. Override with
--ignore-errors
.Try with
--dry-run
first? Combine with--verbose
and closely examine everything that's happening with problem directories.
This option can be dangerous if used incorrectly! It is a very good
idea to first try a run using the --dry-run option (-n) to see
what files are going to be deleted.
So consider trying --delete-excluded
, --ignore-errors
, and/or --dry-run
/--verbose
...not necessarily as a "fix" but at the very least there may be helpful data to be gleaned.
Also, take a look at the --delete-before
flag. This is supposed to do the receive-side deletes before the actual transfer. Again, this might not fix everything but could be revealing. It'd be interesting to see if it helps reduce disk space usage, too.
Hmm, maybe --ignore-errors is the ticket. Invariably, rsync runs in to some sort of error with a file disappearing before it can be copied or a gvfs mount point that can't be read by root. I'll try adding that flag and see what happens.
â alex.forencich
Oct 28 '17 at 19:49
Yep, --ignore-errors was the ticket. Thanks!
â alex.forencich
Oct 28 '17 at 19:54
add a comment |Â
up vote
2
down vote
accepted
Some suggestions for things to try come from the contents of the man page (the section about --delete
). There are a lot of caveats and warnings. Have you considered all of these?
Works only "for the directories that are being synchronized". I guess this means they must have a parent/ancestor directory being copied.
Similarly, it doesn't apply to anything being explicitly excluded:
Files that are excluded from the transfer are also excluded from
being deleted unless you use the--delete-excluded
option or mark
the rules as only matching on the sending side.
If the sending side detects any I/O errors, then the deletion of any files at the destination will be automatically disabled. Override with
--ignore-errors
.Try with
--dry-run
first? Combine with--verbose
and closely examine everything that's happening with problem directories.
This option can be dangerous if used incorrectly! It is a very good
idea to first try a run using the --dry-run option (-n) to see
what files are going to be deleted.
So consider trying --delete-excluded
, --ignore-errors
, and/or --dry-run
/--verbose
...not necessarily as a "fix" but at the very least there may be helpful data to be gleaned.
Also, take a look at the --delete-before
flag. This is supposed to do the receive-side deletes before the actual transfer. Again, this might not fix everything but could be revealing. It'd be interesting to see if it helps reduce disk space usage, too.
Hmm, maybe --ignore-errors is the ticket. Invariably, rsync runs in to some sort of error with a file disappearing before it can be copied or a gvfs mount point that can't be read by root. I'll try adding that flag and see what happens.
â alex.forencich
Oct 28 '17 at 19:49
Yep, --ignore-errors was the ticket. Thanks!
â alex.forencich
Oct 28 '17 at 19:54
add a comment |Â
up vote
2
down vote
accepted
up vote
2
down vote
accepted
Some suggestions for things to try come from the contents of the man page (the section about --delete
). There are a lot of caveats and warnings. Have you considered all of these?
Works only "for the directories that are being synchronized". I guess this means they must have a parent/ancestor directory being copied.
Similarly, it doesn't apply to anything being explicitly excluded:
Files that are excluded from the transfer are also excluded from
being deleted unless you use the--delete-excluded
option or mark
the rules as only matching on the sending side.
If the sending side detects any I/O errors, then the deletion of any files at the destination will be automatically disabled. Override with
--ignore-errors
.Try with
--dry-run
first? Combine with--verbose
and closely examine everything that's happening with problem directories.
This option can be dangerous if used incorrectly! It is a very good
idea to first try a run using the --dry-run option (-n) to see
what files are going to be deleted.
So consider trying --delete-excluded
, --ignore-errors
, and/or --dry-run
/--verbose
...not necessarily as a "fix" but at the very least there may be helpful data to be gleaned.
Also, take a look at the --delete-before
flag. This is supposed to do the receive-side deletes before the actual transfer. Again, this might not fix everything but could be revealing. It'd be interesting to see if it helps reduce disk space usage, too.
Some suggestions for things to try come from the contents of the man page (the section about --delete
). There are a lot of caveats and warnings. Have you considered all of these?
Works only "for the directories that are being synchronized". I guess this means they must have a parent/ancestor directory being copied.
Similarly, it doesn't apply to anything being explicitly excluded:
Files that are excluded from the transfer are also excluded from
being deleted unless you use the--delete-excluded
option or mark
the rules as only matching on the sending side.
If the sending side detects any I/O errors, then the deletion of any files at the destination will be automatically disabled. Override with
--ignore-errors
.Try with
--dry-run
first? Combine with--verbose
and closely examine everything that's happening with problem directories.
This option can be dangerous if used incorrectly! It is a very good
idea to first try a run using the --dry-run option (-n) to see
what files are going to be deleted.
So consider trying --delete-excluded
, --ignore-errors
, and/or --dry-run
/--verbose
...not necessarily as a "fix" but at the very least there may be helpful data to be gleaned.
Also, take a look at the --delete-before
flag. This is supposed to do the receive-side deletes before the actual transfer. Again, this might not fix everything but could be revealing. It'd be interesting to see if it helps reduce disk space usage, too.
edited Oct 28 '17 at 3:32
answered Oct 28 '17 at 3:02
B Layer
3,9241525
3,9241525
Hmm, maybe --ignore-errors is the ticket. Invariably, rsync runs in to some sort of error with a file disappearing before it can be copied or a gvfs mount point that can't be read by root. I'll try adding that flag and see what happens.
â alex.forencich
Oct 28 '17 at 19:49
Yep, --ignore-errors was the ticket. Thanks!
â alex.forencich
Oct 28 '17 at 19:54
add a comment |Â
Hmm, maybe --ignore-errors is the ticket. Invariably, rsync runs in to some sort of error with a file disappearing before it can be copied or a gvfs mount point that can't be read by root. I'll try adding that flag and see what happens.
â alex.forencich
Oct 28 '17 at 19:49
Yep, --ignore-errors was the ticket. Thanks!
â alex.forencich
Oct 28 '17 at 19:54
Hmm, maybe --ignore-errors is the ticket. Invariably, rsync runs in to some sort of error with a file disappearing before it can be copied or a gvfs mount point that can't be read by root. I'll try adding that flag and see what happens.
â alex.forencich
Oct 28 '17 at 19:49
Hmm, maybe --ignore-errors is the ticket. Invariably, rsync runs in to some sort of error with a file disappearing before it can be copied or a gvfs mount point that can't be read by root. I'll try adding that flag and see what happens.
â alex.forencich
Oct 28 '17 at 19:49
Yep, --ignore-errors was the ticket. Thanks!
â alex.forencich
Oct 28 '17 at 19:54
Yep, --ignore-errors was the ticket. Thanks!
â alex.forencich
Oct 28 '17 at 19:54
add a comment |Â
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f400990%2frsync-delete-not-removing-all-deleted-files%23new-answer', 'question_page');
);
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password