VMware: Why is zero-filling ext4 free space needed to shrink *.vmdk files?
Clash Royale CLAN TAG#URR8PPP
On my Linux Mint 17.2 VMware guest, df -h
was reporting my total disk usage steady at about 10GB total. I'm using this machine for Ruby on Rails development inside a Windows host running Workstation 12.1.1 Pro.
The *.vmdk
files kept on steadily growing to approx 100GB. Trying to shrink with vmware-vdiskmanager -k
or vmware-toolbox-cmd disk shrinkonly
made no difference.
I have Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
. A more complete dumpe2fs
output is available.
Running e4defrag /
and e2fsck -E discard
didn't allow any more space to be reclaimed (I killed the shrink after the first few vmdks (which started at around 6GB) showed no sign of shrinking.)
What finally did the trick was filling all free space with NUL
bytes:
dd if=/dev/zero of=wipefile bs=1M; sync; rm wipefile
I could now shrink to a total *.vmdk
size of 15.6GB with vmware-toolbox-cmd disk shrinkonly
.
That's about 85GB space saving on a VM with 10GB actual data.
It seems that ext4
is not re-using previously used blocks when asked for an unused block, often preferring to give out never-before-used blocks.
Questions
Having old data hanging around longer seems less secure. Why would
ext4
not re-use recently-used blocks as soon as possible?Is there a way to force
ext4
to reuse just-used blocks?Is there a way to prevent a VMware guest's
*.vmdk
files from continually growing without needing to0
-fill free space on a regular basis?- How do you safely (eg not entirely filling the filesystem) automate this?
ext4 vmware
|
show 1 more comment
On my Linux Mint 17.2 VMware guest, df -h
was reporting my total disk usage steady at about 10GB total. I'm using this machine for Ruby on Rails development inside a Windows host running Workstation 12.1.1 Pro.
The *.vmdk
files kept on steadily growing to approx 100GB. Trying to shrink with vmware-vdiskmanager -k
or vmware-toolbox-cmd disk shrinkonly
made no difference.
I have Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
. A more complete dumpe2fs
output is available.
Running e4defrag /
and e2fsck -E discard
didn't allow any more space to be reclaimed (I killed the shrink after the first few vmdks (which started at around 6GB) showed no sign of shrinking.)
What finally did the trick was filling all free space with NUL
bytes:
dd if=/dev/zero of=wipefile bs=1M; sync; rm wipefile
I could now shrink to a total *.vmdk
size of 15.6GB with vmware-toolbox-cmd disk shrinkonly
.
That's about 85GB space saving on a VM with 10GB actual data.
It seems that ext4
is not re-using previously used blocks when asked for an unused block, often preferring to give out never-before-used blocks.
Questions
Having old data hanging around longer seems less secure. Why would
ext4
not re-use recently-used blocks as soon as possible?Is there a way to force
ext4
to reuse just-used blocks?Is there a way to prevent a VMware guest's
*.vmdk
files from continually growing without needing to0
-fill free space on a regular basis?- How do you safely (eg not entirely filling the filesystem) automate this?
ext4 vmware
Would you mind to add more technical details, and the reasons you are suspicious?
– Rui F Ribeiro
Aug 22 '16 at 7:07
How now? Anything you can think of that I could check out?
– Tom Hale
Aug 22 '16 at 7:23
Some things are not clear. From the context, are you mentioning the use of ext4 inside the VM? What is the VM doing? What you suspect you can be changed? How are the filesystems created and defined on the VM side? Can reorganising the partitions make a difference? The fs management is highly dependent on the kernel and layer 7 usage, which you have not described. As dding to zero the free space, it should not making a difference except when making tar.gzs of dd.
– Rui F Ribeiro
Aug 22 '16 at 9:41
1
I've overhauled the question. Thedd
definitely was the trick and it's quite a common hack in shrinking linux guest*.vmdk
files. The real question is: "why should it be necessary?"
– Tom Hale
Aug 23 '16 at 4:06
1
It seems like a limitation with VMWare workstation 12.5.1. Even withscsi0.virtualDev = "pvscsi"
in my.vmx
,lsscsi -l
givesscsi_level=3
which doesn't supportUNMAP
.
– Tom Hale
Nov 21 '16 at 12:24
|
show 1 more comment
On my Linux Mint 17.2 VMware guest, df -h
was reporting my total disk usage steady at about 10GB total. I'm using this machine for Ruby on Rails development inside a Windows host running Workstation 12.1.1 Pro.
The *.vmdk
files kept on steadily growing to approx 100GB. Trying to shrink with vmware-vdiskmanager -k
or vmware-toolbox-cmd disk shrinkonly
made no difference.
I have Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
. A more complete dumpe2fs
output is available.
Running e4defrag /
and e2fsck -E discard
didn't allow any more space to be reclaimed (I killed the shrink after the first few vmdks (which started at around 6GB) showed no sign of shrinking.)
What finally did the trick was filling all free space with NUL
bytes:
dd if=/dev/zero of=wipefile bs=1M; sync; rm wipefile
I could now shrink to a total *.vmdk
size of 15.6GB with vmware-toolbox-cmd disk shrinkonly
.
That's about 85GB space saving on a VM with 10GB actual data.
It seems that ext4
is not re-using previously used blocks when asked for an unused block, often preferring to give out never-before-used blocks.
Questions
Having old data hanging around longer seems less secure. Why would
ext4
not re-use recently-used blocks as soon as possible?Is there a way to force
ext4
to reuse just-used blocks?Is there a way to prevent a VMware guest's
*.vmdk
files from continually growing without needing to0
-fill free space on a regular basis?- How do you safely (eg not entirely filling the filesystem) automate this?
ext4 vmware
On my Linux Mint 17.2 VMware guest, df -h
was reporting my total disk usage steady at about 10GB total. I'm using this machine for Ruby on Rails development inside a Windows host running Workstation 12.1.1 Pro.
The *.vmdk
files kept on steadily growing to approx 100GB. Trying to shrink with vmware-vdiskmanager -k
or vmware-toolbox-cmd disk shrinkonly
made no difference.
I have Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
. A more complete dumpe2fs
output is available.
Running e4defrag /
and e2fsck -E discard
didn't allow any more space to be reclaimed (I killed the shrink after the first few vmdks (which started at around 6GB) showed no sign of shrinking.)
What finally did the trick was filling all free space with NUL
bytes:
dd if=/dev/zero of=wipefile bs=1M; sync; rm wipefile
I could now shrink to a total *.vmdk
size of 15.6GB with vmware-toolbox-cmd disk shrinkonly
.
That's about 85GB space saving on a VM with 10GB actual data.
It seems that ext4
is not re-using previously used blocks when asked for an unused block, often preferring to give out never-before-used blocks.
Questions
Having old data hanging around longer seems less secure. Why would
ext4
not re-use recently-used blocks as soon as possible?Is there a way to force
ext4
to reuse just-used blocks?Is there a way to prevent a VMware guest's
*.vmdk
files from continually growing without needing to0
-fill free space on a regular basis?- How do you safely (eg not entirely filling the filesystem) automate this?
ext4 vmware
ext4 vmware
edited Aug 23 '16 at 9:47
Tom Hale
asked Aug 22 '16 at 0:32
Tom HaleTom Hale
6,69033592
6,69033592
Would you mind to add more technical details, and the reasons you are suspicious?
– Rui F Ribeiro
Aug 22 '16 at 7:07
How now? Anything you can think of that I could check out?
– Tom Hale
Aug 22 '16 at 7:23
Some things are not clear. From the context, are you mentioning the use of ext4 inside the VM? What is the VM doing? What you suspect you can be changed? How are the filesystems created and defined on the VM side? Can reorganising the partitions make a difference? The fs management is highly dependent on the kernel and layer 7 usage, which you have not described. As dding to zero the free space, it should not making a difference except when making tar.gzs of dd.
– Rui F Ribeiro
Aug 22 '16 at 9:41
1
I've overhauled the question. Thedd
definitely was the trick and it's quite a common hack in shrinking linux guest*.vmdk
files. The real question is: "why should it be necessary?"
– Tom Hale
Aug 23 '16 at 4:06
1
It seems like a limitation with VMWare workstation 12.5.1. Even withscsi0.virtualDev = "pvscsi"
in my.vmx
,lsscsi -l
givesscsi_level=3
which doesn't supportUNMAP
.
– Tom Hale
Nov 21 '16 at 12:24
|
show 1 more comment
Would you mind to add more technical details, and the reasons you are suspicious?
– Rui F Ribeiro
Aug 22 '16 at 7:07
How now? Anything you can think of that I could check out?
– Tom Hale
Aug 22 '16 at 7:23
Some things are not clear. From the context, are you mentioning the use of ext4 inside the VM? What is the VM doing? What you suspect you can be changed? How are the filesystems created and defined on the VM side? Can reorganising the partitions make a difference? The fs management is highly dependent on the kernel and layer 7 usage, which you have not described. As dding to zero the free space, it should not making a difference except when making tar.gzs of dd.
– Rui F Ribeiro
Aug 22 '16 at 9:41
1
I've overhauled the question. Thedd
definitely was the trick and it's quite a common hack in shrinking linux guest*.vmdk
files. The real question is: "why should it be necessary?"
– Tom Hale
Aug 23 '16 at 4:06
1
It seems like a limitation with VMWare workstation 12.5.1. Even withscsi0.virtualDev = "pvscsi"
in my.vmx
,lsscsi -l
givesscsi_level=3
which doesn't supportUNMAP
.
– Tom Hale
Nov 21 '16 at 12:24
Would you mind to add more technical details, and the reasons you are suspicious?
– Rui F Ribeiro
Aug 22 '16 at 7:07
Would you mind to add more technical details, and the reasons you are suspicious?
– Rui F Ribeiro
Aug 22 '16 at 7:07
How now? Anything you can think of that I could check out?
– Tom Hale
Aug 22 '16 at 7:23
How now? Anything you can think of that I could check out?
– Tom Hale
Aug 22 '16 at 7:23
Some things are not clear. From the context, are you mentioning the use of ext4 inside the VM? What is the VM doing? What you suspect you can be changed? How are the filesystems created and defined on the VM side? Can reorganising the partitions make a difference? The fs management is highly dependent on the kernel and layer 7 usage, which you have not described. As dding to zero the free space, it should not making a difference except when making tar.gzs of dd.
– Rui F Ribeiro
Aug 22 '16 at 9:41
Some things are not clear. From the context, are you mentioning the use of ext4 inside the VM? What is the VM doing? What you suspect you can be changed? How are the filesystems created and defined on the VM side? Can reorganising the partitions make a difference? The fs management is highly dependent on the kernel and layer 7 usage, which you have not described. As dding to zero the free space, it should not making a difference except when making tar.gzs of dd.
– Rui F Ribeiro
Aug 22 '16 at 9:41
1
1
I've overhauled the question. The
dd
definitely was the trick and it's quite a common hack in shrinking linux guest *.vmdk
files. The real question is: "why should it be necessary?"– Tom Hale
Aug 23 '16 at 4:06
I've overhauled the question. The
dd
definitely was the trick and it's quite a common hack in shrinking linux guest *.vmdk
files. The real question is: "why should it be necessary?"– Tom Hale
Aug 23 '16 at 4:06
1
1
It seems like a limitation with VMWare workstation 12.5.1. Even with
scsi0.virtualDev = "pvscsi"
in my .vmx
, lsscsi -l
gives scsi_level=3
which doesn't support UNMAP
.– Tom Hale
Nov 21 '16 at 12:24
It seems like a limitation with VMWare workstation 12.5.1. Even with
scsi0.virtualDev = "pvscsi"
in my .vmx
, lsscsi -l
gives scsi_level=3
which doesn't support UNMAP
.– Tom Hale
Nov 21 '16 at 12:24
|
show 1 more comment
3 Answers
3
active
oldest
votes
Yes, from security standpoint it would be better to immedeately erase any unused blocks. The reason that is not done (ignoring
chattr
secure-delete flag and patches) is performance. The same is a reson for not using any recenty freed block - it would lead to heavy fragmentations which hurts performance.No, not really. You could make your whole image much smaller (formatting it to say
15G
and then growing only if/when that becomes necessary.) - then it will never grow bigger than 15Gb.You can try to mount filesystem with
discard
option - seefstab(5)
, but I'm not sure if your vmware will heed this
As mentioned,e2fsck -E discard
didn't do anything, so I doubt that making realtime discardsmount
ing with-o discard
will do anything. Also I get:fstrim: /: FITRIM ioctl failed: Operation not supported
- trim is not listed as a feature of the virtual SATA device inhdparm -I /dev/dm-0
output.
– Tom Hale
Aug 23 '16 at 10:45
@TomHale did you look at the link? There are changes in vmware config that need to be done too before it could work. But I can't vouch for that part, as said in answer...
– Matija Nalis
Aug 23 '16 at 10:54
I'm currently stuck gettinghdparm
to correctly see virtual SCSI disks as required by that link - see this question.
– Tom Hale
Aug 25 '16 at 14:18
lsscsi -lL
is seeing a virgin SCSI disk as SCSI level 3, so UNMAPwon't work as per your link. I ended up shrinking the filesystem, Logical Volume, Volume Group and Physical Volume... and VMware *.vmdk files are still twice the size the used partition after the shrink. Oh well.
– Tom Hale
Aug 30 '16 at 9:14
@TomHale you should zero-fill all remaining space first to get it smaller - growPV
andVG
to maximum size available, create new logical volume with all free space, and then clear it withdd if=/dev/zero of=/dev/mapper/unusedLV bs=1m
. Only after that you shouldlvremove
it and shrinkVG
andPV
, and then run vmware image shrinking tools.
– Matija Nalis
Aug 31 '16 at 8:55
|
show 1 more comment
No filesystems zero out blocks of deleted files, they just mark those clusters as available. That's why recovery tools can recover deleted files if they weren't overwritten by other files. If filesystem drivers reuse those blocks ASAP then you'll no longer have the ability to recover accidentally erased files and customers will cry, and if they must overwrite the clusters with zero then performance will suffer severely.
Files that are expected to be stored securely should be encrypted instead of leaving as-is on disk. If needed, use shredder tools like shred on Unix and ccleaner drive wiper, eraser, sdelete... on Windows to delete securely.
About VMDK, you should know that it stores sectors in a sparse format, just like VHD, VDI or any other VM's dynamically-sized virtual disk image format. Therefore zeroing out sectors mark them as not needed anymore and the compactor will leave them out, resulting in a smaller file. Any non-zero sectors have to be stored explicitly because the VM doesn't know whether the sector belongs to a deleted file or not
Using dd if=/dev/zero
is a bad way because
- it is slow;
- it makes the disk image (temporarily) grow to its maximal extent;
- it (temporarily) uses all free space on the disk, so other concurrent write actions may fail.
as mentioned in the zerofree
manpage. Use a tool specifically for zeroing out disks like zerofree
instead.
There's no way to prevent a virtual disk image from expanding, because what would you expect if more data need to be written on the disk? Even in case files are deleted, their data are still on disk and take space in the image file. If you don't want the file to grow, the only way is making it fixed-sized on creation.
In the case of deleted files,btrfs
andext4
pass down a discard for blocks which are no longer in use. However, Linux distributions check the SCSI version, and unless it is version 5 or greater, it does not send UNMAPs. VMware's driver says SCSI version 3 for me :(
– Tom Hale
Nov 16 '16 at 5:39
I think in this case it's easier to try Virtualbox, which supports almost all common virtual harddisk file types. Or simply write script to zero and compact the vmdk file automatically like what I'm using if you're allowed to shutdown the VM at night
– phuclv
Nov 22 '16 at 8:11
I've found VirtualBox to be buggy in the Hypervisor UI arena, which didn't inspire me to use it for the VM itself. Myzero-balloon
script is very rough... do you have one that nearly-but-doesn't-quite fill the partitions to avoid disk full conditions?
– Tom Hale
Nov 23 '16 at 10:16
Actually, I'm usingLUKS
now, so zero-filling would actually make things worse... perhaps usingLVM
with thin-provisioning with the provisioned size no greater than the actual size is actually the way to go as it processes discards sensibly
– Tom Hale
Nov 23 '16 at 10:17
add a comment |
Possibly because writing a lot of NULs to the VMDK made it reclaim all the deleted blocks and allocate them to wipefile
. However, why e4defrag
did not do the same is something I do not understand.
What do you mean by 'writing to the VMDK'? I'm confused as every filesystem change in the guest is reflected in the VMDK fikes.
– Tom Hale
Aug 23 '16 at 9:49
add a comment |
Your Answer
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "106"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f304903%2fvmware-why-is-zero-filling-ext4-free-space-needed-to-shrink-vmdk-files%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
3 Answers
3
active
oldest
votes
3 Answers
3
active
oldest
votes
active
oldest
votes
active
oldest
votes
Yes, from security standpoint it would be better to immedeately erase any unused blocks. The reason that is not done (ignoring
chattr
secure-delete flag and patches) is performance. The same is a reson for not using any recenty freed block - it would lead to heavy fragmentations which hurts performance.No, not really. You could make your whole image much smaller (formatting it to say
15G
and then growing only if/when that becomes necessary.) - then it will never grow bigger than 15Gb.You can try to mount filesystem with
discard
option - seefstab(5)
, but I'm not sure if your vmware will heed this
As mentioned,e2fsck -E discard
didn't do anything, so I doubt that making realtime discardsmount
ing with-o discard
will do anything. Also I get:fstrim: /: FITRIM ioctl failed: Operation not supported
- trim is not listed as a feature of the virtual SATA device inhdparm -I /dev/dm-0
output.
– Tom Hale
Aug 23 '16 at 10:45
@TomHale did you look at the link? There are changes in vmware config that need to be done too before it could work. But I can't vouch for that part, as said in answer...
– Matija Nalis
Aug 23 '16 at 10:54
I'm currently stuck gettinghdparm
to correctly see virtual SCSI disks as required by that link - see this question.
– Tom Hale
Aug 25 '16 at 14:18
lsscsi -lL
is seeing a virgin SCSI disk as SCSI level 3, so UNMAPwon't work as per your link. I ended up shrinking the filesystem, Logical Volume, Volume Group and Physical Volume... and VMware *.vmdk files are still twice the size the used partition after the shrink. Oh well.
– Tom Hale
Aug 30 '16 at 9:14
@TomHale you should zero-fill all remaining space first to get it smaller - growPV
andVG
to maximum size available, create new logical volume with all free space, and then clear it withdd if=/dev/zero of=/dev/mapper/unusedLV bs=1m
. Only after that you shouldlvremove
it and shrinkVG
andPV
, and then run vmware image shrinking tools.
– Matija Nalis
Aug 31 '16 at 8:55
|
show 1 more comment
Yes, from security standpoint it would be better to immedeately erase any unused blocks. The reason that is not done (ignoring
chattr
secure-delete flag and patches) is performance. The same is a reson for not using any recenty freed block - it would lead to heavy fragmentations which hurts performance.No, not really. You could make your whole image much smaller (formatting it to say
15G
and then growing only if/when that becomes necessary.) - then it will never grow bigger than 15Gb.You can try to mount filesystem with
discard
option - seefstab(5)
, but I'm not sure if your vmware will heed this
As mentioned,e2fsck -E discard
didn't do anything, so I doubt that making realtime discardsmount
ing with-o discard
will do anything. Also I get:fstrim: /: FITRIM ioctl failed: Operation not supported
- trim is not listed as a feature of the virtual SATA device inhdparm -I /dev/dm-0
output.
– Tom Hale
Aug 23 '16 at 10:45
@TomHale did you look at the link? There are changes in vmware config that need to be done too before it could work. But I can't vouch for that part, as said in answer...
– Matija Nalis
Aug 23 '16 at 10:54
I'm currently stuck gettinghdparm
to correctly see virtual SCSI disks as required by that link - see this question.
– Tom Hale
Aug 25 '16 at 14:18
lsscsi -lL
is seeing a virgin SCSI disk as SCSI level 3, so UNMAPwon't work as per your link. I ended up shrinking the filesystem, Logical Volume, Volume Group and Physical Volume... and VMware *.vmdk files are still twice the size the used partition after the shrink. Oh well.
– Tom Hale
Aug 30 '16 at 9:14
@TomHale you should zero-fill all remaining space first to get it smaller - growPV
andVG
to maximum size available, create new logical volume with all free space, and then clear it withdd if=/dev/zero of=/dev/mapper/unusedLV bs=1m
. Only after that you shouldlvremove
it and shrinkVG
andPV
, and then run vmware image shrinking tools.
– Matija Nalis
Aug 31 '16 at 8:55
|
show 1 more comment
Yes, from security standpoint it would be better to immedeately erase any unused blocks. The reason that is not done (ignoring
chattr
secure-delete flag and patches) is performance. The same is a reson for not using any recenty freed block - it would lead to heavy fragmentations which hurts performance.No, not really. You could make your whole image much smaller (formatting it to say
15G
and then growing only if/when that becomes necessary.) - then it will never grow bigger than 15Gb.You can try to mount filesystem with
discard
option - seefstab(5)
, but I'm not sure if your vmware will heed this
Yes, from security standpoint it would be better to immedeately erase any unused blocks. The reason that is not done (ignoring
chattr
secure-delete flag and patches) is performance. The same is a reson for not using any recenty freed block - it would lead to heavy fragmentations which hurts performance.No, not really. You could make your whole image much smaller (formatting it to say
15G
and then growing only if/when that becomes necessary.) - then it will never grow bigger than 15Gb.You can try to mount filesystem with
discard
option - seefstab(5)
, but I'm not sure if your vmware will heed this
answered Aug 23 '16 at 10:30
Matija NalisMatija Nalis
2,332617
2,332617
As mentioned,e2fsck -E discard
didn't do anything, so I doubt that making realtime discardsmount
ing with-o discard
will do anything. Also I get:fstrim: /: FITRIM ioctl failed: Operation not supported
- trim is not listed as a feature of the virtual SATA device inhdparm -I /dev/dm-0
output.
– Tom Hale
Aug 23 '16 at 10:45
@TomHale did you look at the link? There are changes in vmware config that need to be done too before it could work. But I can't vouch for that part, as said in answer...
– Matija Nalis
Aug 23 '16 at 10:54
I'm currently stuck gettinghdparm
to correctly see virtual SCSI disks as required by that link - see this question.
– Tom Hale
Aug 25 '16 at 14:18
lsscsi -lL
is seeing a virgin SCSI disk as SCSI level 3, so UNMAPwon't work as per your link. I ended up shrinking the filesystem, Logical Volume, Volume Group and Physical Volume... and VMware *.vmdk files are still twice the size the used partition after the shrink. Oh well.
– Tom Hale
Aug 30 '16 at 9:14
@TomHale you should zero-fill all remaining space first to get it smaller - growPV
andVG
to maximum size available, create new logical volume with all free space, and then clear it withdd if=/dev/zero of=/dev/mapper/unusedLV bs=1m
. Only after that you shouldlvremove
it and shrinkVG
andPV
, and then run vmware image shrinking tools.
– Matija Nalis
Aug 31 '16 at 8:55
|
show 1 more comment
As mentioned,e2fsck -E discard
didn't do anything, so I doubt that making realtime discardsmount
ing with-o discard
will do anything. Also I get:fstrim: /: FITRIM ioctl failed: Operation not supported
- trim is not listed as a feature of the virtual SATA device inhdparm -I /dev/dm-0
output.
– Tom Hale
Aug 23 '16 at 10:45
@TomHale did you look at the link? There are changes in vmware config that need to be done too before it could work. But I can't vouch for that part, as said in answer...
– Matija Nalis
Aug 23 '16 at 10:54
I'm currently stuck gettinghdparm
to correctly see virtual SCSI disks as required by that link - see this question.
– Tom Hale
Aug 25 '16 at 14:18
lsscsi -lL
is seeing a virgin SCSI disk as SCSI level 3, so UNMAPwon't work as per your link. I ended up shrinking the filesystem, Logical Volume, Volume Group and Physical Volume... and VMware *.vmdk files are still twice the size the used partition after the shrink. Oh well.
– Tom Hale
Aug 30 '16 at 9:14
@TomHale you should zero-fill all remaining space first to get it smaller - growPV
andVG
to maximum size available, create new logical volume with all free space, and then clear it withdd if=/dev/zero of=/dev/mapper/unusedLV bs=1m
. Only after that you shouldlvremove
it and shrinkVG
andPV
, and then run vmware image shrinking tools.
– Matija Nalis
Aug 31 '16 at 8:55
As mentioned,
e2fsck -E discard
didn't do anything, so I doubt that making realtime discards mount
ing with -o discard
will do anything. Also I get: fstrim: /: FITRIM ioctl failed: Operation not supported
- trim is not listed as a feature of the virtual SATA device in hdparm -I /dev/dm-0
output.– Tom Hale
Aug 23 '16 at 10:45
As mentioned,
e2fsck -E discard
didn't do anything, so I doubt that making realtime discards mount
ing with -o discard
will do anything. Also I get: fstrim: /: FITRIM ioctl failed: Operation not supported
- trim is not listed as a feature of the virtual SATA device in hdparm -I /dev/dm-0
output.– Tom Hale
Aug 23 '16 at 10:45
@TomHale did you look at the link? There are changes in vmware config that need to be done too before it could work. But I can't vouch for that part, as said in answer...
– Matija Nalis
Aug 23 '16 at 10:54
@TomHale did you look at the link? There are changes in vmware config that need to be done too before it could work. But I can't vouch for that part, as said in answer...
– Matija Nalis
Aug 23 '16 at 10:54
I'm currently stuck getting
hdparm
to correctly see virtual SCSI disks as required by that link - see this question.– Tom Hale
Aug 25 '16 at 14:18
I'm currently stuck getting
hdparm
to correctly see virtual SCSI disks as required by that link - see this question.– Tom Hale
Aug 25 '16 at 14:18
lsscsi -lL
is seeing a virgin SCSI disk as SCSI level 3, so UNMAPwon't work as per your link. I ended up shrinking the filesystem, Logical Volume, Volume Group and Physical Volume... and VMware *.vmdk files are still twice the size the used partition after the shrink. Oh well.– Tom Hale
Aug 30 '16 at 9:14
lsscsi -lL
is seeing a virgin SCSI disk as SCSI level 3, so UNMAPwon't work as per your link. I ended up shrinking the filesystem, Logical Volume, Volume Group and Physical Volume... and VMware *.vmdk files are still twice the size the used partition after the shrink. Oh well.– Tom Hale
Aug 30 '16 at 9:14
@TomHale you should zero-fill all remaining space first to get it smaller - grow
PV
and VG
to maximum size available, create new logical volume with all free space, and then clear it with dd if=/dev/zero of=/dev/mapper/unusedLV bs=1m
. Only after that you should lvremove
it and shrink VG
and PV
, and then run vmware image shrinking tools.– Matija Nalis
Aug 31 '16 at 8:55
@TomHale you should zero-fill all remaining space first to get it smaller - grow
PV
and VG
to maximum size available, create new logical volume with all free space, and then clear it with dd if=/dev/zero of=/dev/mapper/unusedLV bs=1m
. Only after that you should lvremove
it and shrink VG
and PV
, and then run vmware image shrinking tools.– Matija Nalis
Aug 31 '16 at 8:55
|
show 1 more comment
No filesystems zero out blocks of deleted files, they just mark those clusters as available. That's why recovery tools can recover deleted files if they weren't overwritten by other files. If filesystem drivers reuse those blocks ASAP then you'll no longer have the ability to recover accidentally erased files and customers will cry, and if they must overwrite the clusters with zero then performance will suffer severely.
Files that are expected to be stored securely should be encrypted instead of leaving as-is on disk. If needed, use shredder tools like shred on Unix and ccleaner drive wiper, eraser, sdelete... on Windows to delete securely.
About VMDK, you should know that it stores sectors in a sparse format, just like VHD, VDI or any other VM's dynamically-sized virtual disk image format. Therefore zeroing out sectors mark them as not needed anymore and the compactor will leave them out, resulting in a smaller file. Any non-zero sectors have to be stored explicitly because the VM doesn't know whether the sector belongs to a deleted file or not
Using dd if=/dev/zero
is a bad way because
- it is slow;
- it makes the disk image (temporarily) grow to its maximal extent;
- it (temporarily) uses all free space on the disk, so other concurrent write actions may fail.
as mentioned in the zerofree
manpage. Use a tool specifically for zeroing out disks like zerofree
instead.
There's no way to prevent a virtual disk image from expanding, because what would you expect if more data need to be written on the disk? Even in case files are deleted, their data are still on disk and take space in the image file. If you don't want the file to grow, the only way is making it fixed-sized on creation.
In the case of deleted files,btrfs
andext4
pass down a discard for blocks which are no longer in use. However, Linux distributions check the SCSI version, and unless it is version 5 or greater, it does not send UNMAPs. VMware's driver says SCSI version 3 for me :(
– Tom Hale
Nov 16 '16 at 5:39
I think in this case it's easier to try Virtualbox, which supports almost all common virtual harddisk file types. Or simply write script to zero and compact the vmdk file automatically like what I'm using if you're allowed to shutdown the VM at night
– phuclv
Nov 22 '16 at 8:11
I've found VirtualBox to be buggy in the Hypervisor UI arena, which didn't inspire me to use it for the VM itself. Myzero-balloon
script is very rough... do you have one that nearly-but-doesn't-quite fill the partitions to avoid disk full conditions?
– Tom Hale
Nov 23 '16 at 10:16
Actually, I'm usingLUKS
now, so zero-filling would actually make things worse... perhaps usingLVM
with thin-provisioning with the provisioned size no greater than the actual size is actually the way to go as it processes discards sensibly
– Tom Hale
Nov 23 '16 at 10:17
add a comment |
No filesystems zero out blocks of deleted files, they just mark those clusters as available. That's why recovery tools can recover deleted files if they weren't overwritten by other files. If filesystem drivers reuse those blocks ASAP then you'll no longer have the ability to recover accidentally erased files and customers will cry, and if they must overwrite the clusters with zero then performance will suffer severely.
Files that are expected to be stored securely should be encrypted instead of leaving as-is on disk. If needed, use shredder tools like shred on Unix and ccleaner drive wiper, eraser, sdelete... on Windows to delete securely.
About VMDK, you should know that it stores sectors in a sparse format, just like VHD, VDI or any other VM's dynamically-sized virtual disk image format. Therefore zeroing out sectors mark them as not needed anymore and the compactor will leave them out, resulting in a smaller file. Any non-zero sectors have to be stored explicitly because the VM doesn't know whether the sector belongs to a deleted file or not
Using dd if=/dev/zero
is a bad way because
- it is slow;
- it makes the disk image (temporarily) grow to its maximal extent;
- it (temporarily) uses all free space on the disk, so other concurrent write actions may fail.
as mentioned in the zerofree
manpage. Use a tool specifically for zeroing out disks like zerofree
instead.
There's no way to prevent a virtual disk image from expanding, because what would you expect if more data need to be written on the disk? Even in case files are deleted, their data are still on disk and take space in the image file. If you don't want the file to grow, the only way is making it fixed-sized on creation.
In the case of deleted files,btrfs
andext4
pass down a discard for blocks which are no longer in use. However, Linux distributions check the SCSI version, and unless it is version 5 or greater, it does not send UNMAPs. VMware's driver says SCSI version 3 for me :(
– Tom Hale
Nov 16 '16 at 5:39
I think in this case it's easier to try Virtualbox, which supports almost all common virtual harddisk file types. Or simply write script to zero and compact the vmdk file automatically like what I'm using if you're allowed to shutdown the VM at night
– phuclv
Nov 22 '16 at 8:11
I've found VirtualBox to be buggy in the Hypervisor UI arena, which didn't inspire me to use it for the VM itself. Myzero-balloon
script is very rough... do you have one that nearly-but-doesn't-quite fill the partitions to avoid disk full conditions?
– Tom Hale
Nov 23 '16 at 10:16
Actually, I'm usingLUKS
now, so zero-filling would actually make things worse... perhaps usingLVM
with thin-provisioning with the provisioned size no greater than the actual size is actually the way to go as it processes discards sensibly
– Tom Hale
Nov 23 '16 at 10:17
add a comment |
No filesystems zero out blocks of deleted files, they just mark those clusters as available. That's why recovery tools can recover deleted files if they weren't overwritten by other files. If filesystem drivers reuse those blocks ASAP then you'll no longer have the ability to recover accidentally erased files and customers will cry, and if they must overwrite the clusters with zero then performance will suffer severely.
Files that are expected to be stored securely should be encrypted instead of leaving as-is on disk. If needed, use shredder tools like shred on Unix and ccleaner drive wiper, eraser, sdelete... on Windows to delete securely.
About VMDK, you should know that it stores sectors in a sparse format, just like VHD, VDI or any other VM's dynamically-sized virtual disk image format. Therefore zeroing out sectors mark them as not needed anymore and the compactor will leave them out, resulting in a smaller file. Any non-zero sectors have to be stored explicitly because the VM doesn't know whether the sector belongs to a deleted file or not
Using dd if=/dev/zero
is a bad way because
- it is slow;
- it makes the disk image (temporarily) grow to its maximal extent;
- it (temporarily) uses all free space on the disk, so other concurrent write actions may fail.
as mentioned in the zerofree
manpage. Use a tool specifically for zeroing out disks like zerofree
instead.
There's no way to prevent a virtual disk image from expanding, because what would you expect if more data need to be written on the disk? Even in case files are deleted, their data are still on disk and take space in the image file. If you don't want the file to grow, the only way is making it fixed-sized on creation.
No filesystems zero out blocks of deleted files, they just mark those clusters as available. That's why recovery tools can recover deleted files if they weren't overwritten by other files. If filesystem drivers reuse those blocks ASAP then you'll no longer have the ability to recover accidentally erased files and customers will cry, and if they must overwrite the clusters with zero then performance will suffer severely.
Files that are expected to be stored securely should be encrypted instead of leaving as-is on disk. If needed, use shredder tools like shred on Unix and ccleaner drive wiper, eraser, sdelete... on Windows to delete securely.
About VMDK, you should know that it stores sectors in a sparse format, just like VHD, VDI or any other VM's dynamically-sized virtual disk image format. Therefore zeroing out sectors mark them as not needed anymore and the compactor will leave them out, resulting in a smaller file. Any non-zero sectors have to be stored explicitly because the VM doesn't know whether the sector belongs to a deleted file or not
Using dd if=/dev/zero
is a bad way because
- it is slow;
- it makes the disk image (temporarily) grow to its maximal extent;
- it (temporarily) uses all free space on the disk, so other concurrent write actions may fail.
as mentioned in the zerofree
manpage. Use a tool specifically for zeroing out disks like zerofree
instead.
There's no way to prevent a virtual disk image from expanding, because what would you expect if more data need to be written on the disk? Even in case files are deleted, their data are still on disk and take space in the image file. If you don't want the file to grow, the only way is making it fixed-sized on creation.
edited Jan 4 at 2:26
answered Nov 15 '16 at 15:36
phuclvphuclv
306121
306121
In the case of deleted files,btrfs
andext4
pass down a discard for blocks which are no longer in use. However, Linux distributions check the SCSI version, and unless it is version 5 or greater, it does not send UNMAPs. VMware's driver says SCSI version 3 for me :(
– Tom Hale
Nov 16 '16 at 5:39
I think in this case it's easier to try Virtualbox, which supports almost all common virtual harddisk file types. Or simply write script to zero and compact the vmdk file automatically like what I'm using if you're allowed to shutdown the VM at night
– phuclv
Nov 22 '16 at 8:11
I've found VirtualBox to be buggy in the Hypervisor UI arena, which didn't inspire me to use it for the VM itself. Myzero-balloon
script is very rough... do you have one that nearly-but-doesn't-quite fill the partitions to avoid disk full conditions?
– Tom Hale
Nov 23 '16 at 10:16
Actually, I'm usingLUKS
now, so zero-filling would actually make things worse... perhaps usingLVM
with thin-provisioning with the provisioned size no greater than the actual size is actually the way to go as it processes discards sensibly
– Tom Hale
Nov 23 '16 at 10:17
add a comment |
In the case of deleted files,btrfs
andext4
pass down a discard for blocks which are no longer in use. However, Linux distributions check the SCSI version, and unless it is version 5 or greater, it does not send UNMAPs. VMware's driver says SCSI version 3 for me :(
– Tom Hale
Nov 16 '16 at 5:39
I think in this case it's easier to try Virtualbox, which supports almost all common virtual harddisk file types. Or simply write script to zero and compact the vmdk file automatically like what I'm using if you're allowed to shutdown the VM at night
– phuclv
Nov 22 '16 at 8:11
I've found VirtualBox to be buggy in the Hypervisor UI arena, which didn't inspire me to use it for the VM itself. Myzero-balloon
script is very rough... do you have one that nearly-but-doesn't-quite fill the partitions to avoid disk full conditions?
– Tom Hale
Nov 23 '16 at 10:16
Actually, I'm usingLUKS
now, so zero-filling would actually make things worse... perhaps usingLVM
with thin-provisioning with the provisioned size no greater than the actual size is actually the way to go as it processes discards sensibly
– Tom Hale
Nov 23 '16 at 10:17
In the case of deleted files,
btrfs
and ext4
pass down a discard for blocks which are no longer in use. However, Linux distributions check the SCSI version, and unless it is version 5 or greater, it does not send UNMAPs. VMware's driver says SCSI version 3 for me :(– Tom Hale
Nov 16 '16 at 5:39
In the case of deleted files,
btrfs
and ext4
pass down a discard for blocks which are no longer in use. However, Linux distributions check the SCSI version, and unless it is version 5 or greater, it does not send UNMAPs. VMware's driver says SCSI version 3 for me :(– Tom Hale
Nov 16 '16 at 5:39
I think in this case it's easier to try Virtualbox, which supports almost all common virtual harddisk file types. Or simply write script to zero and compact the vmdk file automatically like what I'm using if you're allowed to shutdown the VM at night
– phuclv
Nov 22 '16 at 8:11
I think in this case it's easier to try Virtualbox, which supports almost all common virtual harddisk file types. Or simply write script to zero and compact the vmdk file automatically like what I'm using if you're allowed to shutdown the VM at night
– phuclv
Nov 22 '16 at 8:11
I've found VirtualBox to be buggy in the Hypervisor UI arena, which didn't inspire me to use it for the VM itself. My
zero-balloon
script is very rough... do you have one that nearly-but-doesn't-quite fill the partitions to avoid disk full conditions?– Tom Hale
Nov 23 '16 at 10:16
I've found VirtualBox to be buggy in the Hypervisor UI arena, which didn't inspire me to use it for the VM itself. My
zero-balloon
script is very rough... do you have one that nearly-but-doesn't-quite fill the partitions to avoid disk full conditions?– Tom Hale
Nov 23 '16 at 10:16
Actually, I'm using
LUKS
now, so zero-filling would actually make things worse... perhaps using LVM
with thin-provisioning with the provisioned size no greater than the actual size is actually the way to go as it processes discards sensibly– Tom Hale
Nov 23 '16 at 10:17
Actually, I'm using
LUKS
now, so zero-filling would actually make things worse... perhaps using LVM
with thin-provisioning with the provisioned size no greater than the actual size is actually the way to go as it processes discards sensibly– Tom Hale
Nov 23 '16 at 10:17
add a comment |
Possibly because writing a lot of NULs to the VMDK made it reclaim all the deleted blocks and allocate them to wipefile
. However, why e4defrag
did not do the same is something I do not understand.
What do you mean by 'writing to the VMDK'? I'm confused as every filesystem change in the guest is reflected in the VMDK fikes.
– Tom Hale
Aug 23 '16 at 9:49
add a comment |
Possibly because writing a lot of NULs to the VMDK made it reclaim all the deleted blocks and allocate them to wipefile
. However, why e4defrag
did not do the same is something I do not understand.
What do you mean by 'writing to the VMDK'? I'm confused as every filesystem change in the guest is reflected in the VMDK fikes.
– Tom Hale
Aug 23 '16 at 9:49
add a comment |
Possibly because writing a lot of NULs to the VMDK made it reclaim all the deleted blocks and allocate them to wipefile
. However, why e4defrag
did not do the same is something I do not understand.
Possibly because writing a lot of NULs to the VMDK made it reclaim all the deleted blocks and allocate them to wipefile
. However, why e4defrag
did not do the same is something I do not understand.
edited Aug 23 '16 at 19:09
answered Aug 23 '16 at 4:29
SACHIN GARGSACHIN GARG
1144
1144
What do you mean by 'writing to the VMDK'? I'm confused as every filesystem change in the guest is reflected in the VMDK fikes.
– Tom Hale
Aug 23 '16 at 9:49
add a comment |
What do you mean by 'writing to the VMDK'? I'm confused as every filesystem change in the guest is reflected in the VMDK fikes.
– Tom Hale
Aug 23 '16 at 9:49
What do you mean by 'writing to the VMDK'? I'm confused as every filesystem change in the guest is reflected in the VMDK fikes.
– Tom Hale
Aug 23 '16 at 9:49
What do you mean by 'writing to the VMDK'? I'm confused as every filesystem change in the guest is reflected in the VMDK fikes.
– Tom Hale
Aug 23 '16 at 9:49
add a comment |
Thanks for contributing an answer to Unix & Linux Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f304903%2fvmware-why-is-zero-filling-ext4-free-space-needed-to-shrink-vmdk-files%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Would you mind to add more technical details, and the reasons you are suspicious?
– Rui F Ribeiro
Aug 22 '16 at 7:07
How now? Anything you can think of that I could check out?
– Tom Hale
Aug 22 '16 at 7:23
Some things are not clear. From the context, are you mentioning the use of ext4 inside the VM? What is the VM doing? What you suspect you can be changed? How are the filesystems created and defined on the VM side? Can reorganising the partitions make a difference? The fs management is highly dependent on the kernel and layer 7 usage, which you have not described. As dding to zero the free space, it should not making a difference except when making tar.gzs of dd.
– Rui F Ribeiro
Aug 22 '16 at 9:41
1
I've overhauled the question. The
dd
definitely was the trick and it's quite a common hack in shrinking linux guest*.vmdk
files. The real question is: "why should it be necessary?"– Tom Hale
Aug 23 '16 at 4:06
1
It seems like a limitation with VMWare workstation 12.5.1. Even with
scsi0.virtualDev = "pvscsi"
in my.vmx
,lsscsi -l
givesscsi_level=3
which doesn't supportUNMAP
.– Tom Hale
Nov 21 '16 at 12:24