ZFS deduplication does not work
Clash Royale CLAN TAG#URR8PPP
up vote
1
down vote
favorite
I have created pool tank1 and fs tank/home.
[root@zfs]# zfs list
NAME USED AVAIL REFER MOUNTPOINT
tank1 3.22G 381G 32.0K /tank1
tank1/home 3.22G 16.8G 1.95G /tank1/home
I crated two 1G files inside tank1/home from /dev/zero.Deduplication is turned on, compression is turned off:
# zfs get all | egrep 'dedup|compression'
tank1 compression off default
tank1 dedup on local
tank1/home compression off default
tank1/home dedup on inherited from tank1
# zpool list
NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
tank1 596G 4.84G 591G - 0% 0% 1.00x ONLINE -
# zdb -D tank1
All DDTs are empty
# zdb -S tank1
Simulated DDT histogram:
bucket allocated referenced
______ ______________________________ ______________________________
refcnt blocks LSIZE PSIZE DSIZE blocks LSIZE PSIZE DSIZE
------ ------ ----- ----- ----- ------ ----- ----- -----
16K 1 128K 128K 128K 25.8K 3.22G 3.22G 3.22G
Total 1 128K 128K 128K 25.8K 3.22G 3.22G 3.22G
dedup = 26400.00, compress = 1.00, copies = 1.00, dedup * compress / copies = 26425.81
Why does zdb output and zpool list does not show any deduplication and simulation zdb -S tank1 shows 26400? How to turn deduplication on? I alsto tried zfs set dedup=verify tank1, the problem persist. What does output of zdb -DD tank1/home menas?
# zdb -DD tank1/home
Dataset tank1/home [ZPL], ID 165, cr_txg 963, 1.95G, 8 objects
linux zfs
asked Apr 20 at 14:08
dorinand
313213
add a comment |Â
up vote
1
down vote
favorite
I have created pool tank1 and fs tank/home.
[root@zfs]# zfs list
NAME USED AVAIL REFER MOUNTPOINT
tank1 3.22G 381G 32.0K /tank1
tank1/home 3.22G 16.8G 1.95G /tank1/home
I crated two 1G files inside tank1/home from /dev/zero.Deduplication is turned on, compression is turned off:
# zfs get all | egrep 'dedup|compression'
tank1 compression off default
tank1 dedup on local
tank1/home compression off default
tank1/home dedup on inherited from tank1
# zpool list
NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
tank1 596G 4.84G 591G - 0% 0% 1.00x ONLINE -
# zdb -D tank1
All DDTs are empty
# zdb -S tank1
Simulated DDT histogram:
bucket allocated referenced
______ ______________________________ ______________________________
refcnt blocks LSIZE PSIZE DSIZE blocks LSIZE PSIZE DSIZE
------ ------ ----- ----- ----- ------ ----- ----- -----
16K 1 128K 128K 128K 25.8K 3.22G 3.22G 3.22G
Total 1 128K 128K 128K 25.8K 3.22G 3.22G 3.22G
dedup = 26400.00, compress = 1.00, copies = 1.00, dedup * compress / copies = 26425.81
Why does zdb output and zpool list does not show any deduplication and simulation zdb -S tank1 shows 26400? How to turn deduplication on? I alsto tried zfs set dedup=verify tank1, the problem persist. What does output of zdb -DD tank1/home menas?
# zdb -DD tank1/home
Dataset tank1/home [ZPL], ID 165, cr_txg 963, 1.95G, 8 objects
linux zfs
asked Apr 20 at 14:08
dorinand
313213
1
Deduplication only starts being used after it is turned on. Older files wont be modified.
– Rui F Ribeiro
Apr 20 at 14:32
It is turned on, see the output ofzfs get all | egrep 'dedup|compression'
– dorinand
Apr 20 at 14:35
1
That is not what I was calling you attention. If the files have been deployed before it was turned on, they wont use dedup.
– Rui F Ribeiro
Apr 20 at 14:51
Thank you, I delete files and copy it and it works now. How can I check which data are deduped? May I perform dedup just on files added added before ?
– dorinand
Apr 20 at 15:12
add a comment |Â
up vote
1
down vote
favorite
up vote
1
down vote
favorite
I have created pool tank1 and fs tank/home.
[root@zfs]# zfs list
NAME USED AVAIL REFER MOUNTPOINT
tank1 3.22G 381G 32.0K /tank1
tank1/home 3.22G 16.8G 1.95G /tank1/home
I crated two 1G files inside tank1/home from /dev/zero.Deduplication is turned on, compression is turned off:
# zfs get all | egrep 'dedup|compression'
tank1 compression off default
tank1 dedup on local
tank1/home compression off default
tank1/home dedup on inherited from tank1
# zpool list
NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
tank1 596G 4.84G 591G - 0% 0% 1.00x ONLINE -
# zdb -D tank1
All DDTs are empty
# zdb -S tank1
Simulated DDT histogram:
bucket allocated referenced
______ ______________________________ ______________________________
refcnt blocks LSIZE PSIZE DSIZE blocks LSIZE PSIZE DSIZE
------ ------ ----- ----- ----- ------ ----- ----- -----
16K 1 128K 128K 128K 25.8K 3.22G 3.22G 3.22G
Total 1 128K 128K 128K 25.8K 3.22G 3.22G 3.22G
dedup = 26400.00, compress = 1.00, copies = 1.00, dedup * compress / copies = 26425.81
Why does zdb output and zpool list does not show any deduplication and simulation zdb -S tank1 shows 26400? How to turn deduplication on? I alsto tried zfs set dedup=verify tank1, the problem persist. What does output of zdb -DD tank1/home menas?
# zdb -DD tank1/home
Dataset tank1/home [ZPL], ID 165, cr_txg 963, 1.95G, 8 objects
linux zfs
asked Apr 20 at 14:08
dorinand
313213
I have created pool tank1 and fs tank/home.
[root@zfs]# zfs list
NAME USED AVAIL REFER MOUNTPOINT
tank1 3.22G 381G 32.0K /tank1
tank1/home 3.22G 16.8G 1.95G /tank1/home
I crated two 1G files inside tank1/home from /dev/zero.Deduplication is turned on, compression is turned off:
# zfs get all | egrep 'dedup|compression'
tank1 compression off default
tank1 dedup on local
tank1/home compression off default
tank1/home dedup on inherited from tank1
# zpool list
NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
tank1 596G 4.84G 591G - 0% 0% 1.00x ONLINE -
# zdb -D tank1
All DDTs are empty
# zdb -S tank1
Simulated DDT histogram:
bucket allocated referenced
______ ______________________________ ______________________________
refcnt blocks LSIZE PSIZE DSIZE blocks LSIZE PSIZE DSIZE
------ ------ ----- ----- ----- ------ ----- ----- -----
16K 1 128K 128K 128K 25.8K 3.22G 3.22G 3.22G
Total 1 128K 128K 128K 25.8K 3.22G 3.22G 3.22G
dedup = 26400.00, compress = 1.00, copies = 1.00, dedup * compress / copies = 26425.81
Why does zdb output and zpool list does not show any deduplication and simulation zdb -S tank1 shows 26400? How to turn deduplication on? I alsto tried zfs set dedup=verify tank1, the problem persist. What does output of zdb -DD tank1/home menas?
# zdb -DD tank1/home
Dataset tank1/home [ZPL], ID 165, cr_txg 963, 1.95G, 8 objects
linux zfs
asked Apr 20 at 14:08
dorinand
313213
edited Apr 20 at 14:36
asked Apr 20 at 14:08
dorinand
313213
asked Apr 20 at 14:08
dorinand
313213
asked Apr 20 at 14:08
dorinand
313213
313213
1
Deduplication only starts being used after it is turned on. Older files wont be modified.
– Rui F Ribeiro
Apr 20 at 14:32
It is turned on, see the output ofzfs get all | egrep 'dedup|compression'
– dorinand
Apr 20 at 14:35
1
That is not what I was calling you attention. If the files have been deployed before it was turned on, they wont use dedup.
– Rui F Ribeiro
Apr 20 at 14:51
Thank you, I delete files and copy it and it works now. How can I check which data are deduped? May I perform dedup just on files added added before ?
– dorinand
Apr 20 at 15:12
add a comment |Â
1
Deduplication only starts being used after it is turned on. Older files wont be modified.
– Rui F Ribeiro
Apr 20 at 14:32
It is turned on, see the output ofzfs get all | egrep 'dedup|compression'
– dorinand
Apr 20 at 14:35
1
That is not what I was calling you attention. If the files have been deployed before it was turned on, they wont use dedup.
– Rui F Ribeiro
Apr 20 at 14:51
Thank you, I delete files and copy it and it works now. How can I check which data are deduped? May I perform dedup just on files added added before ?
– dorinand
Apr 20 at 15:12
1
1
Deduplication only starts being used after it is turned on. Older files wont be modified.
– Rui F Ribeiro
Apr 20 at 14:32
Deduplication only starts being used after it is turned on. Older files wont be modified.
– Rui F Ribeiro
Apr 20 at 14:32
It is turned on, see the output of
zfs get all | egrep 'dedup|compression'– dorinand
Apr 20 at 14:35
It is turned on, see the output of
zfs get all | egrep 'dedup|compression'– dorinand
Apr 20 at 14:35
1
1
That is not what I was calling you attention. If the files have been deployed before it was turned on, they wont use dedup.
– Rui F Ribeiro
Apr 20 at 14:51
That is not what I was calling you attention. If the files have been deployed before it was turned on, they wont use dedup.
– Rui F Ribeiro
Apr 20 at 14:51
Thank you, I delete files and copy it and it works now. How can I check which data are deduped? May I perform dedup just on files added added before ?
– dorinand
Apr 20 at 15:12
Thank you, I delete files and copy it and it works now. How can I check which data are deduped? May I perform dedup just on files added added before ?
– dorinand
Apr 20 at 15:12
add a comment |Â
1 Answer
1
active
oldest
votes
up vote
3
down vote
accepted
ZFS deduplication only will take effect on files deployed after you enable deduplication.
It won't affect older files before deduplication was enabled.
To see how much dedup is saving, use the command zpool status -D poolname and pay attention to the Total allocated blocks versus Total referenced blocks.
edited Apr 29 at 22:24
Jeff Schaller
31.1k846105
answered Apr 20 at 15:22
Rui F Ribeiro
34.5k1269113
I deleted my old pool tank and create new one named nfspool. I turn on dedup and create 5 files from/dev/zeroand the problem occours again. Even when I set up dedup to verify. It looks like problem is somewhere else.
– dorinand
Apr 20 at 17:12
@dorinand Try it with meaningful data. I suspect something about trying it with /dev/null, will have to check it out.
– Rui F Ribeiro
Apr 20 at 18:47
add a comment |Â
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
3
down vote
accepted
ZFS deduplication only will take effect on files deployed after you enable deduplication.
It won't affect older files before deduplication was enabled.
To see how much dedup is saving, use the command zpool status -D poolname and pay attention to the Total allocated blocks versus Total referenced blocks.
edited Apr 29 at 22:24
Jeff Schaller
31.1k846105
answered Apr 20 at 15:22
Rui F Ribeiro
34.5k1269113
I deleted my old pool tank and create new one named nfspool. I turn on dedup and create 5 files from/dev/zeroand the problem occours again. Even when I set up dedup to verify. It looks like problem is somewhere else.
– dorinand
Apr 20 at 17:12
@dorinand Try it with meaningful data. I suspect something about trying it with /dev/null, will have to check it out.
– Rui F Ribeiro
Apr 20 at 18:47
add a comment |Â
up vote
3
down vote
accepted
ZFS deduplication only will take effect on files deployed after you enable deduplication.
It won't affect older files before deduplication was enabled.
To see how much dedup is saving, use the command zpool status -D poolname and pay attention to the Total allocated blocks versus Total referenced blocks.
edited Apr 29 at 22:24
Jeff Schaller
31.1k846105
answered Apr 20 at 15:22
Rui F Ribeiro
34.5k1269113
I deleted my old pool tank and create new one named nfspool. I turn on dedup and create 5 files from/dev/zeroand the problem occours again. Even when I set up dedup to verify. It looks like problem is somewhere else.
– dorinand
Apr 20 at 17:12
@dorinand Try it with meaningful data. I suspect something about trying it with /dev/null, will have to check it out.
– Rui F Ribeiro
Apr 20 at 18:47
add a comment |Â
up vote
3
down vote
accepted
up vote
3
down vote
accepted
ZFS deduplication only will take effect on files deployed after you enable deduplication.
It won't affect older files before deduplication was enabled.
To see how much dedup is saving, use the command zpool status -D poolname and pay attention to the Total allocated blocks versus Total referenced blocks.
edited Apr 29 at 22:24
Jeff Schaller
31.1k846105
answered Apr 20 at 15:22
Rui F Ribeiro
34.5k1269113
ZFS deduplication only will take effect on files deployed after you enable deduplication.
It won't affect older files before deduplication was enabled.
To see how much dedup is saving, use the command zpool status -D poolname and pay attention to the Total allocated blocks versus Total referenced blocks.
edited Apr 29 at 22:24
Jeff Schaller
31.1k846105
answered Apr 20 at 15:22
Rui F Ribeiro
34.5k1269113
edited Apr 29 at 22:24
Jeff Schaller
31.1k846105
edited Apr 29 at 22:24
Jeff Schaller
31.1k846105
edited Apr 29 at 22:24
Jeff Schaller
31.1k846105
31.1k846105
answered Apr 20 at 15:22
Rui F Ribeiro
34.5k1269113
answered Apr 20 at 15:22
Rui F Ribeiro
34.5k1269113
answered Apr 20 at 15:22
Rui F Ribeiro
34.5k1269113
34.5k1269113
I deleted my old pool tank and create new one named nfspool. I turn on dedup and create 5 files from/dev/zeroand the problem occours again. Even when I set up dedup to verify. It looks like problem is somewhere else.
– dorinand
Apr 20 at 17:12
@dorinand Try it with meaningful data. I suspect something about trying it with /dev/null, will have to check it out.
– Rui F Ribeiro
Apr 20 at 18:47
add a comment |Â
I deleted my old pool tank and create new one named nfspool. I turn on dedup and create 5 files from/dev/zeroand the problem occours again. Even when I set up dedup to verify. It looks like problem is somewhere else.
– dorinand
Apr 20 at 17:12
@dorinand Try it with meaningful data. I suspect something about trying it with /dev/null, will have to check it out.
– Rui F Ribeiro
Apr 20 at 18:47
I deleted my old pool tank and create new one named nfspool. I turn on dedup and create 5 files from
/dev/zero and the problem occours again. Even when I set up dedup to verify. It looks like problem is somewhere else.– dorinand
Apr 20 at 17:12
I deleted my old pool tank and create new one named nfspool. I turn on dedup and create 5 files from
/dev/zero and the problem occours again. Even when I set up dedup to verify. It looks like problem is somewhere else.– dorinand
Apr 20 at 17:12
@dorinand Try it with meaningful data. I suspect something about trying it with /dev/null, will have to check it out.
– Rui F Ribeiro
Apr 20 at 18:47
@dorinand Try it with meaningful data. I suspect something about trying it with /dev/null, will have to check it out.
– Rui F Ribeiro
Apr 20 at 18:47
add a comment |Â
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f438955%2fzfs-deduplication-does-not-work%23new-answer', 'question_page');
);
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
1
Deduplication only starts being used after it is turned on. Older files wont be modified.
– Rui F Ribeiro
Apr 20 at 14:32
It is turned on, see the output of
zfs get all | egrep 'dedup|compression'– dorinand
Apr 20 at 14:35
1
That is not what I was calling you attention. If the files have been deployed before it was turned on, they wont use dedup.
– Rui F Ribeiro
Apr 20 at 14:51
Thank you, I delete files and copy it and it works now. How can I check which data are deduped? May I perform dedup just on files added added before ?
– dorinand
Apr 20 at 15:12