How to copy multiple snapshots at once without duplicating data?
Clash Royale CLAN TAG#URR8PPP
I have a live btrfs filesystem of 3.7TiB that's >90% full including old snapshots and a fresh 4TB backup harddisk. How to copy all existing snapshots to the backup harddisk?
I tried
# btrfs send home_1 home_2 home_3 share_1 share_2 share_3 ...
but the backup harddisk was full before 2/3 of the snapshots were transmitted. So I did some research:
# ### create test image
# dd bs=1M count=1000 > btrfs_test.dd
# mkfs.btrfs btrfs_test.dd
# ### create snapshots
# btrfs subvol create testvol/
# btrfs subvol snapshot -r testvol/ testvol_0/
# ### (copy some ISO image)
# btrfs subvol snapshot -r testvol/ testvol_1/
# ### (proceed until testvol_3)
My test filesystem then was 91% full with 818MiB used.
# btrfs send testvol_* | wc -c
At subvol testvol_0
At subvol testvol_1
At subvol testvol_2
At subvol testvol_3
1466441978 # 1398MiB >> 1000MiB
When simply sending all the snapshots with 1 command, data is duplicated and the size of the stream as well as the size of the snapshots at the receiver's end exceed the original used space and the harddisk's capacity.
So the actual question became: How to copy multiple snapshots without duplicating data that is contained in 2 or more of them?
For this simple test case I successfully tried the incremental approach:
# ( btrfs send testvol_0; btrfs send -p testvol_0 testvol_1; btrfs send -p testvol_1 testvol_2; btrfs send -p testvol_2 testvol_3 ) | wc -c
At subvol testvol_0
At subvol testvol_1
At subvol testvol_2
At subvol testvol_3
838778546 # 800 MiB < 1000MiB
But on the real filesystem there are multiple subvolumes, each of them with multiple snapshots. I can't define any order to use with -p
. If a data block is shared among the subvolumes home_1
, home_2
and share_3
I want to transmit and store it only once, of course. Any ideas how to do this?
btrfs snapshot deduplication
add a comment |
I have a live btrfs filesystem of 3.7TiB that's >90% full including old snapshots and a fresh 4TB backup harddisk. How to copy all existing snapshots to the backup harddisk?
I tried
# btrfs send home_1 home_2 home_3 share_1 share_2 share_3 ...
but the backup harddisk was full before 2/3 of the snapshots were transmitted. So I did some research:
# ### create test image
# dd bs=1M count=1000 > btrfs_test.dd
# mkfs.btrfs btrfs_test.dd
# ### create snapshots
# btrfs subvol create testvol/
# btrfs subvol snapshot -r testvol/ testvol_0/
# ### (copy some ISO image)
# btrfs subvol snapshot -r testvol/ testvol_1/
# ### (proceed until testvol_3)
My test filesystem then was 91% full with 818MiB used.
# btrfs send testvol_* | wc -c
At subvol testvol_0
At subvol testvol_1
At subvol testvol_2
At subvol testvol_3
1466441978 # 1398MiB >> 1000MiB
When simply sending all the snapshots with 1 command, data is duplicated and the size of the stream as well as the size of the snapshots at the receiver's end exceed the original used space and the harddisk's capacity.
So the actual question became: How to copy multiple snapshots without duplicating data that is contained in 2 or more of them?
For this simple test case I successfully tried the incremental approach:
# ( btrfs send testvol_0; btrfs send -p testvol_0 testvol_1; btrfs send -p testvol_1 testvol_2; btrfs send -p testvol_2 testvol_3 ) | wc -c
At subvol testvol_0
At subvol testvol_1
At subvol testvol_2
At subvol testvol_3
838778546 # 800 MiB < 1000MiB
But on the real filesystem there are multiple subvolumes, each of them with multiple snapshots. I can't define any order to use with -p
. If a data block is shared among the subvolumes home_1
, home_2
and share_3
I want to transmit and store it only once, of course. Any ideas how to do this?
btrfs snapshot deduplication
add a comment |
I have a live btrfs filesystem of 3.7TiB that's >90% full including old snapshots and a fresh 4TB backup harddisk. How to copy all existing snapshots to the backup harddisk?
I tried
# btrfs send home_1 home_2 home_3 share_1 share_2 share_3 ...
but the backup harddisk was full before 2/3 of the snapshots were transmitted. So I did some research:
# ### create test image
# dd bs=1M count=1000 > btrfs_test.dd
# mkfs.btrfs btrfs_test.dd
# ### create snapshots
# btrfs subvol create testvol/
# btrfs subvol snapshot -r testvol/ testvol_0/
# ### (copy some ISO image)
# btrfs subvol snapshot -r testvol/ testvol_1/
# ### (proceed until testvol_3)
My test filesystem then was 91% full with 818MiB used.
# btrfs send testvol_* | wc -c
At subvol testvol_0
At subvol testvol_1
At subvol testvol_2
At subvol testvol_3
1466441978 # 1398MiB >> 1000MiB
When simply sending all the snapshots with 1 command, data is duplicated and the size of the stream as well as the size of the snapshots at the receiver's end exceed the original used space and the harddisk's capacity.
So the actual question became: How to copy multiple snapshots without duplicating data that is contained in 2 or more of them?
For this simple test case I successfully tried the incremental approach:
# ( btrfs send testvol_0; btrfs send -p testvol_0 testvol_1; btrfs send -p testvol_1 testvol_2; btrfs send -p testvol_2 testvol_3 ) | wc -c
At subvol testvol_0
At subvol testvol_1
At subvol testvol_2
At subvol testvol_3
838778546 # 800 MiB < 1000MiB
But on the real filesystem there are multiple subvolumes, each of them with multiple snapshots. I can't define any order to use with -p
. If a data block is shared among the subvolumes home_1
, home_2
and share_3
I want to transmit and store it only once, of course. Any ideas how to do this?
btrfs snapshot deduplication
I have a live btrfs filesystem of 3.7TiB that's >90% full including old snapshots and a fresh 4TB backup harddisk. How to copy all existing snapshots to the backup harddisk?
I tried
# btrfs send home_1 home_2 home_3 share_1 share_2 share_3 ...
but the backup harddisk was full before 2/3 of the snapshots were transmitted. So I did some research:
# ### create test image
# dd bs=1M count=1000 > btrfs_test.dd
# mkfs.btrfs btrfs_test.dd
# ### create snapshots
# btrfs subvol create testvol/
# btrfs subvol snapshot -r testvol/ testvol_0/
# ### (copy some ISO image)
# btrfs subvol snapshot -r testvol/ testvol_1/
# ### (proceed until testvol_3)
My test filesystem then was 91% full with 818MiB used.
# btrfs send testvol_* | wc -c
At subvol testvol_0
At subvol testvol_1
At subvol testvol_2
At subvol testvol_3
1466441978 # 1398MiB >> 1000MiB
When simply sending all the snapshots with 1 command, data is duplicated and the size of the stream as well as the size of the snapshots at the receiver's end exceed the original used space and the harddisk's capacity.
So the actual question became: How to copy multiple snapshots without duplicating data that is contained in 2 or more of them?
For this simple test case I successfully tried the incremental approach:
# ( btrfs send testvol_0; btrfs send -p testvol_0 testvol_1; btrfs send -p testvol_1 testvol_2; btrfs send -p testvol_2 testvol_3 ) | wc -c
At subvol testvol_0
At subvol testvol_1
At subvol testvol_2
At subvol testvol_3
838778546 # 800 MiB < 1000MiB
But on the real filesystem there are multiple subvolumes, each of them with multiple snapshots. I can't define any order to use with -p
. If a data block is shared among the subvolumes home_1
, home_2
and share_3
I want to transmit and store it only once, of course. Any ideas how to do this?
btrfs snapshot deduplication
btrfs snapshot deduplication
asked Jan 2 at 19:52
Daniel BöhmerDaniel Böhmer
22228
22228
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
TL;DR: Using the -c
parameter as described at the top works in general. When the snapshots contain hardlinks there's a bug in the Linux kernel that triggers an error when sending the snapshots—for details see conclusion at the end of the answer.
I am experimenting with the -c
parameter and it looks promising:
# for i in 0..3; do btrfs send testvol_$i $(ls -d testvol_* | head -n$i | sed 's/^/-c /'); done | wc -c
### btrfs send testvol_0
### btrfs send testvol_1 -c testvol_0
### btrfs send testvol_2 -c testvol_0 -c testvol_1
### btrfs send testvol_3 -c testvol_0 -c testvol_1 -c testvol_2
At subvol testvol_0
At subvol testvol_1
At subvol testvol_2
At subvol testvol_3
838778546 # also 800MiB
I am still not sure if this is what I need. Any comments on this solution?
Update: To test this with my real filesystem I wrote a Perl script to comfortably send a set of subvolumes (creating the lists for -c
quickly became tedious) and sent some data to /dev/null
:
#!/usr/bin/env perl
use strict;
use warnings;
my @subvols = @ARGV
or die "Usage: $0 SUBVOLUME ...n";
for(@subvols)
-d $_
or die "Not a directory: $_n";
for my $i (0 .. $#subvols)
my $subvol = $subvols[$i];
my @clones = map ('-c', $_) @subvols[ 0 .. $i-1 ];
print "btrfs send $subvol @clonesn";
Results:
btrfs send some-subvolme_* | pv > /dev/null
: 24GiB 0:04:17 [95.2MiB/s]perl btrfs-send-all.pl some-subvolume_* | bash | pv > /dev/null
: 12.7GiB 0:03:58 [54.7MiB/s]
That's not much of a performance gain but nearly a 50% decrease in storage space! I am now trying to run this in real …
Update: I successfully transferred 2 snapshots but for the 3rd one btrfs receive
failed with the following error message:
ERROR: unlink path/to/some/file/in/the/snapshot failed. No such file or directory
The named file is present in subvol_2
and subvol_3
but not yet in subvol_1
.
I tried to compare the sent and the received snapshots:
# ### sender
# du -s subvol_1,2,3
132472304 subvol_1
117069504 subvol_2
126015636 subvol_3
# ### receiver
# du -s subvol_*
132472304 subvol_1
117069504 subvol_2
132472304 subvol_3
The first two snapshots seem to be transferred correctly but subvol_3
to be a clone of subvol_1
. It's definitely a clone because the used space on the backup disk is only 39% of the snapshots' combined size—tightly above 1/3 as they share most files.
Why does btrfs send subvol_3 -c subvol_1 -c subvol_2 | btrfs receive
not correctly transfer the snapshot but clone subvol_1
and then fail to delete a file that should be kept in subvol_3
and is only present in subvol_2
??
Update: Could it be this bug? https://patchwork.kernel.org/patch/10073969/
I am running Debian 9 Stretch with Kernel 4.9 which seems to be older than the patch.
Update: Because I could not find any solution I simply copied the most recent snapshot of each subvolume. Then I had roughly 500GiB free space and tried to add the previous snapshot with the the already copied snapshot as -p
parameter. Then I got the same error message for the same snapshot as above.
Conclusion: I conclude that I hit the bug linked above. I had to reboot this machine with a newer Linux kernel or access the filesystem from another computer but this is not feasible because this is a production system.
I had no problems with btrfs so far but running rsnapshot—which creates a lot of hardlinks—and sending btrfs snapshots still might have problems on current Debian stable.
add a comment |
Your Answer
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "106"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f492078%2fhow-to-copy-multiple-snapshots-at-once-without-duplicating-data%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
TL;DR: Using the -c
parameter as described at the top works in general. When the snapshots contain hardlinks there's a bug in the Linux kernel that triggers an error when sending the snapshots—for details see conclusion at the end of the answer.
I am experimenting with the -c
parameter and it looks promising:
# for i in 0..3; do btrfs send testvol_$i $(ls -d testvol_* | head -n$i | sed 's/^/-c /'); done | wc -c
### btrfs send testvol_0
### btrfs send testvol_1 -c testvol_0
### btrfs send testvol_2 -c testvol_0 -c testvol_1
### btrfs send testvol_3 -c testvol_0 -c testvol_1 -c testvol_2
At subvol testvol_0
At subvol testvol_1
At subvol testvol_2
At subvol testvol_3
838778546 # also 800MiB
I am still not sure if this is what I need. Any comments on this solution?
Update: To test this with my real filesystem I wrote a Perl script to comfortably send a set of subvolumes (creating the lists for -c
quickly became tedious) and sent some data to /dev/null
:
#!/usr/bin/env perl
use strict;
use warnings;
my @subvols = @ARGV
or die "Usage: $0 SUBVOLUME ...n";
for(@subvols)
-d $_
or die "Not a directory: $_n";
for my $i (0 .. $#subvols)
my $subvol = $subvols[$i];
my @clones = map ('-c', $_) @subvols[ 0 .. $i-1 ];
print "btrfs send $subvol @clonesn";
Results:
btrfs send some-subvolme_* | pv > /dev/null
: 24GiB 0:04:17 [95.2MiB/s]perl btrfs-send-all.pl some-subvolume_* | bash | pv > /dev/null
: 12.7GiB 0:03:58 [54.7MiB/s]
That's not much of a performance gain but nearly a 50% decrease in storage space! I am now trying to run this in real …
Update: I successfully transferred 2 snapshots but for the 3rd one btrfs receive
failed with the following error message:
ERROR: unlink path/to/some/file/in/the/snapshot failed. No such file or directory
The named file is present in subvol_2
and subvol_3
but not yet in subvol_1
.
I tried to compare the sent and the received snapshots:
# ### sender
# du -s subvol_1,2,3
132472304 subvol_1
117069504 subvol_2
126015636 subvol_3
# ### receiver
# du -s subvol_*
132472304 subvol_1
117069504 subvol_2
132472304 subvol_3
The first two snapshots seem to be transferred correctly but subvol_3
to be a clone of subvol_1
. It's definitely a clone because the used space on the backup disk is only 39% of the snapshots' combined size—tightly above 1/3 as they share most files.
Why does btrfs send subvol_3 -c subvol_1 -c subvol_2 | btrfs receive
not correctly transfer the snapshot but clone subvol_1
and then fail to delete a file that should be kept in subvol_3
and is only present in subvol_2
??
Update: Could it be this bug? https://patchwork.kernel.org/patch/10073969/
I am running Debian 9 Stretch with Kernel 4.9 which seems to be older than the patch.
Update: Because I could not find any solution I simply copied the most recent snapshot of each subvolume. Then I had roughly 500GiB free space and tried to add the previous snapshot with the the already copied snapshot as -p
parameter. Then I got the same error message for the same snapshot as above.
Conclusion: I conclude that I hit the bug linked above. I had to reboot this machine with a newer Linux kernel or access the filesystem from another computer but this is not feasible because this is a production system.
I had no problems with btrfs so far but running rsnapshot—which creates a lot of hardlinks—and sending btrfs snapshots still might have problems on current Debian stable.
add a comment |
TL;DR: Using the -c
parameter as described at the top works in general. When the snapshots contain hardlinks there's a bug in the Linux kernel that triggers an error when sending the snapshots—for details see conclusion at the end of the answer.
I am experimenting with the -c
parameter and it looks promising:
# for i in 0..3; do btrfs send testvol_$i $(ls -d testvol_* | head -n$i | sed 's/^/-c /'); done | wc -c
### btrfs send testvol_0
### btrfs send testvol_1 -c testvol_0
### btrfs send testvol_2 -c testvol_0 -c testvol_1
### btrfs send testvol_3 -c testvol_0 -c testvol_1 -c testvol_2
At subvol testvol_0
At subvol testvol_1
At subvol testvol_2
At subvol testvol_3
838778546 # also 800MiB
I am still not sure if this is what I need. Any comments on this solution?
Update: To test this with my real filesystem I wrote a Perl script to comfortably send a set of subvolumes (creating the lists for -c
quickly became tedious) and sent some data to /dev/null
:
#!/usr/bin/env perl
use strict;
use warnings;
my @subvols = @ARGV
or die "Usage: $0 SUBVOLUME ...n";
for(@subvols)
-d $_
or die "Not a directory: $_n";
for my $i (0 .. $#subvols)
my $subvol = $subvols[$i];
my @clones = map ('-c', $_) @subvols[ 0 .. $i-1 ];
print "btrfs send $subvol @clonesn";
Results:
btrfs send some-subvolme_* | pv > /dev/null
: 24GiB 0:04:17 [95.2MiB/s]perl btrfs-send-all.pl some-subvolume_* | bash | pv > /dev/null
: 12.7GiB 0:03:58 [54.7MiB/s]
That's not much of a performance gain but nearly a 50% decrease in storage space! I am now trying to run this in real …
Update: I successfully transferred 2 snapshots but for the 3rd one btrfs receive
failed with the following error message:
ERROR: unlink path/to/some/file/in/the/snapshot failed. No such file or directory
The named file is present in subvol_2
and subvol_3
but not yet in subvol_1
.
I tried to compare the sent and the received snapshots:
# ### sender
# du -s subvol_1,2,3
132472304 subvol_1
117069504 subvol_2
126015636 subvol_3
# ### receiver
# du -s subvol_*
132472304 subvol_1
117069504 subvol_2
132472304 subvol_3
The first two snapshots seem to be transferred correctly but subvol_3
to be a clone of subvol_1
. It's definitely a clone because the used space on the backup disk is only 39% of the snapshots' combined size—tightly above 1/3 as they share most files.
Why does btrfs send subvol_3 -c subvol_1 -c subvol_2 | btrfs receive
not correctly transfer the snapshot but clone subvol_1
and then fail to delete a file that should be kept in subvol_3
and is only present in subvol_2
??
Update: Could it be this bug? https://patchwork.kernel.org/patch/10073969/
I am running Debian 9 Stretch with Kernel 4.9 which seems to be older than the patch.
Update: Because I could not find any solution I simply copied the most recent snapshot of each subvolume. Then I had roughly 500GiB free space and tried to add the previous snapshot with the the already copied snapshot as -p
parameter. Then I got the same error message for the same snapshot as above.
Conclusion: I conclude that I hit the bug linked above. I had to reboot this machine with a newer Linux kernel or access the filesystem from another computer but this is not feasible because this is a production system.
I had no problems with btrfs so far but running rsnapshot—which creates a lot of hardlinks—and sending btrfs snapshots still might have problems on current Debian stable.
add a comment |
TL;DR: Using the -c
parameter as described at the top works in general. When the snapshots contain hardlinks there's a bug in the Linux kernel that triggers an error when sending the snapshots—for details see conclusion at the end of the answer.
I am experimenting with the -c
parameter and it looks promising:
# for i in 0..3; do btrfs send testvol_$i $(ls -d testvol_* | head -n$i | sed 's/^/-c /'); done | wc -c
### btrfs send testvol_0
### btrfs send testvol_1 -c testvol_0
### btrfs send testvol_2 -c testvol_0 -c testvol_1
### btrfs send testvol_3 -c testvol_0 -c testvol_1 -c testvol_2
At subvol testvol_0
At subvol testvol_1
At subvol testvol_2
At subvol testvol_3
838778546 # also 800MiB
I am still not sure if this is what I need. Any comments on this solution?
Update: To test this with my real filesystem I wrote a Perl script to comfortably send a set of subvolumes (creating the lists for -c
quickly became tedious) and sent some data to /dev/null
:
#!/usr/bin/env perl
use strict;
use warnings;
my @subvols = @ARGV
or die "Usage: $0 SUBVOLUME ...n";
for(@subvols)
-d $_
or die "Not a directory: $_n";
for my $i (0 .. $#subvols)
my $subvol = $subvols[$i];
my @clones = map ('-c', $_) @subvols[ 0 .. $i-1 ];
print "btrfs send $subvol @clonesn";
Results:
btrfs send some-subvolme_* | pv > /dev/null
: 24GiB 0:04:17 [95.2MiB/s]perl btrfs-send-all.pl some-subvolume_* | bash | pv > /dev/null
: 12.7GiB 0:03:58 [54.7MiB/s]
That's not much of a performance gain but nearly a 50% decrease in storage space! I am now trying to run this in real …
Update: I successfully transferred 2 snapshots but for the 3rd one btrfs receive
failed with the following error message:
ERROR: unlink path/to/some/file/in/the/snapshot failed. No such file or directory
The named file is present in subvol_2
and subvol_3
but not yet in subvol_1
.
I tried to compare the sent and the received snapshots:
# ### sender
# du -s subvol_1,2,3
132472304 subvol_1
117069504 subvol_2
126015636 subvol_3
# ### receiver
# du -s subvol_*
132472304 subvol_1
117069504 subvol_2
132472304 subvol_3
The first two snapshots seem to be transferred correctly but subvol_3
to be a clone of subvol_1
. It's definitely a clone because the used space on the backup disk is only 39% of the snapshots' combined size—tightly above 1/3 as they share most files.
Why does btrfs send subvol_3 -c subvol_1 -c subvol_2 | btrfs receive
not correctly transfer the snapshot but clone subvol_1
and then fail to delete a file that should be kept in subvol_3
and is only present in subvol_2
??
Update: Could it be this bug? https://patchwork.kernel.org/patch/10073969/
I am running Debian 9 Stretch with Kernel 4.9 which seems to be older than the patch.
Update: Because I could not find any solution I simply copied the most recent snapshot of each subvolume. Then I had roughly 500GiB free space and tried to add the previous snapshot with the the already copied snapshot as -p
parameter. Then I got the same error message for the same snapshot as above.
Conclusion: I conclude that I hit the bug linked above. I had to reboot this machine with a newer Linux kernel or access the filesystem from another computer but this is not feasible because this is a production system.
I had no problems with btrfs so far but running rsnapshot—which creates a lot of hardlinks—and sending btrfs snapshots still might have problems on current Debian stable.
TL;DR: Using the -c
parameter as described at the top works in general. When the snapshots contain hardlinks there's a bug in the Linux kernel that triggers an error when sending the snapshots—for details see conclusion at the end of the answer.
I am experimenting with the -c
parameter and it looks promising:
# for i in 0..3; do btrfs send testvol_$i $(ls -d testvol_* | head -n$i | sed 's/^/-c /'); done | wc -c
### btrfs send testvol_0
### btrfs send testvol_1 -c testvol_0
### btrfs send testvol_2 -c testvol_0 -c testvol_1
### btrfs send testvol_3 -c testvol_0 -c testvol_1 -c testvol_2
At subvol testvol_0
At subvol testvol_1
At subvol testvol_2
At subvol testvol_3
838778546 # also 800MiB
I am still not sure if this is what I need. Any comments on this solution?
Update: To test this with my real filesystem I wrote a Perl script to comfortably send a set of subvolumes (creating the lists for -c
quickly became tedious) and sent some data to /dev/null
:
#!/usr/bin/env perl
use strict;
use warnings;
my @subvols = @ARGV
or die "Usage: $0 SUBVOLUME ...n";
for(@subvols)
-d $_
or die "Not a directory: $_n";
for my $i (0 .. $#subvols)
my $subvol = $subvols[$i];
my @clones = map ('-c', $_) @subvols[ 0 .. $i-1 ];
print "btrfs send $subvol @clonesn";
Results:
btrfs send some-subvolme_* | pv > /dev/null
: 24GiB 0:04:17 [95.2MiB/s]perl btrfs-send-all.pl some-subvolume_* | bash | pv > /dev/null
: 12.7GiB 0:03:58 [54.7MiB/s]
That's not much of a performance gain but nearly a 50% decrease in storage space! I am now trying to run this in real …
Update: I successfully transferred 2 snapshots but for the 3rd one btrfs receive
failed with the following error message:
ERROR: unlink path/to/some/file/in/the/snapshot failed. No such file or directory
The named file is present in subvol_2
and subvol_3
but not yet in subvol_1
.
I tried to compare the sent and the received snapshots:
# ### sender
# du -s subvol_1,2,3
132472304 subvol_1
117069504 subvol_2
126015636 subvol_3
# ### receiver
# du -s subvol_*
132472304 subvol_1
117069504 subvol_2
132472304 subvol_3
The first two snapshots seem to be transferred correctly but subvol_3
to be a clone of subvol_1
. It's definitely a clone because the used space on the backup disk is only 39% of the snapshots' combined size—tightly above 1/3 as they share most files.
Why does btrfs send subvol_3 -c subvol_1 -c subvol_2 | btrfs receive
not correctly transfer the snapshot but clone subvol_1
and then fail to delete a file that should be kept in subvol_3
and is only present in subvol_2
??
Update: Could it be this bug? https://patchwork.kernel.org/patch/10073969/
I am running Debian 9 Stretch with Kernel 4.9 which seems to be older than the patch.
Update: Because I could not find any solution I simply copied the most recent snapshot of each subvolume. Then I had roughly 500GiB free space and tried to add the previous snapshot with the the already copied snapshot as -p
parameter. Then I got the same error message for the same snapshot as above.
Conclusion: I conclude that I hit the bug linked above. I had to reboot this machine with a newer Linux kernel or access the filesystem from another computer but this is not feasible because this is a production system.
I had no problems with btrfs so far but running rsnapshot—which creates a lot of hardlinks—and sending btrfs snapshots still might have problems on current Debian stable.
edited Jan 7 at 0:41
answered Jan 2 at 20:08
Daniel BöhmerDaniel Böhmer
22228
22228
add a comment |
add a comment |
Thanks for contributing an answer to Unix & Linux Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f492078%2fhow-to-copy-multiple-snapshots-at-once-without-duplicating-data%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown