What are data integrity / bit-rot protection options on CentOS 7?
Clash Royale CLAN TAG#URR8PPP
I have a 2 disk CentOS 7 machine build that that I need data integrity / bitrot protection on. How can I achieve this?
Note from my reading btrfs,zfs and DM-Integrity does not seem to be options.
- Btrfs is not an option as btrfs will be deprecated by RHEL and CentOS.
- ZFS is not natively supported on RHEL/CentOS and RH has not intention of supporting it in the future. Also the data corruption bug withing ZFSonLinux in Apr 2018 does not bode well for that implementation.
- DM-Integrity is not an option as the kernel versions are older and as far as I know are not available on CentOS.
- It seem RAID6 using md (on 4 partitions) is not option due to the fact AFAIK it does not calculate the checksum on each read. According to this answer, a scrub may not correctly fix anyway.
Note CentOS was chosen for stability and long term support.
centos zfs data integrity md
asked Jan 11 at 9:48
MeDataMeData
162
add a comment |
I have a 2 disk CentOS 7 machine build that that I need data integrity / bitrot protection on. How can I achieve this?
Note from my reading btrfs,zfs and DM-Integrity does not seem to be options.
- Btrfs is not an option as btrfs will be deprecated by RHEL and CentOS.
- ZFS is not natively supported on RHEL/CentOS and RH has not intention of supporting it in the future. Also the data corruption bug withing ZFSonLinux in Apr 2018 does not bode well for that implementation.
- DM-Integrity is not an option as the kernel versions are older and as far as I know are not available on CentOS.
- It seem RAID6 using md (on 4 partitions) is not option due to the fact AFAIK it does not calculate the checksum on each read. According to this answer, a scrub may not correctly fix anyway.
Note CentOS was chosen for stability and long term support.
centos zfs data integrity md
asked Jan 11 at 9:48
MeDataMeData
162
2
If you take a look at ALL filesystem bugs that exists, you will never find one that is imune to data loss or problems. EXT4 had a big bug last year - phoronix.com/… - If you trust no one here, you will not trust harware raid too, since you will be tied to the hardware model you get for this purpose.
– nwildner
Jan 11 at 10:14
You probably right re file systems bugs, but as I alluded to in my question RHEL/CentOS will not support ZFS and using ZoL on CentOS/RHEL may affect stability. But at this stage ZFS may be the best option, but I would like to see other opinions.
– MeData
Jan 11 at 10:31
3
You could use SmartOS as the hypervisor layer. That's based on OpenSolaris and has native ZFS. Then you can use CentOS inside your VMs and not need to worry about disk integrity.
– Stephen Harris
Jan 11 at 15:33
"RH has no intention of supporting it [ZFS] in the future" - I didn't think there was any (commercial) support from RedHat for CentOS anyway.
– roaima
Jan 12 at 19:18
add a comment |
I have a 2 disk CentOS 7 machine build that that I need data integrity / bitrot protection on. How can I achieve this?
Note from my reading btrfs,zfs and DM-Integrity does not seem to be options.
- Btrfs is not an option as btrfs will be deprecated by RHEL and CentOS.
- ZFS is not natively supported on RHEL/CentOS and RH has not intention of supporting it in the future. Also the data corruption bug withing ZFSonLinux in Apr 2018 does not bode well for that implementation.
- DM-Integrity is not an option as the kernel versions are older and as far as I know are not available on CentOS.
- It seem RAID6 using md (on 4 partitions) is not option due to the fact AFAIK it does not calculate the checksum on each read. According to this answer, a scrub may not correctly fix anyway.
Note CentOS was chosen for stability and long term support.
centos zfs data integrity md
asked Jan 11 at 9:48
MeDataMeData
162
I have a 2 disk CentOS 7 machine build that that I need data integrity / bitrot protection on. How can I achieve this?
Note from my reading btrfs,zfs and DM-Integrity does not seem to be options.
- Btrfs is not an option as btrfs will be deprecated by RHEL and CentOS.
- ZFS is not natively supported on RHEL/CentOS and RH has not intention of supporting it in the future. Also the data corruption bug withing ZFSonLinux in Apr 2018 does not bode well for that implementation.
- DM-Integrity is not an option as the kernel versions are older and as far as I know are not available on CentOS.
- It seem RAID6 using md (on 4 partitions) is not option due to the fact AFAIK it does not calculate the checksum on each read. According to this answer, a scrub may not correctly fix anyway.
Note CentOS was chosen for stability and long term support.
centos zfs data integrity md
centos zfs data integrity md
asked Jan 11 at 9:48
MeDataMeData
162
asked Jan 11 at 9:48
MeDataMeData
162
edited Jan 11 at 10:17
MeData
asked Jan 11 at 9:48
MeDataMeData
162
asked Jan 11 at 9:48
MeDataMeData
162
asked Jan 11 at 9:48
MeDataMeData
162
162
2
If you take a look at ALL filesystem bugs that exists, you will never find one that is imune to data loss or problems. EXT4 had a big bug last year - phoronix.com/… - If you trust no one here, you will not trust harware raid too, since you will be tied to the hardware model you get for this purpose.
– nwildner
Jan 11 at 10:14
You probably right re file systems bugs, but as I alluded to in my question RHEL/CentOS will not support ZFS and using ZoL on CentOS/RHEL may affect stability. But at this stage ZFS may be the best option, but I would like to see other opinions.
– MeData
Jan 11 at 10:31
3
You could use SmartOS as the hypervisor layer. That's based on OpenSolaris and has native ZFS. Then you can use CentOS inside your VMs and not need to worry about disk integrity.
– Stephen Harris
Jan 11 at 15:33
"RH has no intention of supporting it [ZFS] in the future" - I didn't think there was any (commercial) support from RedHat for CentOS anyway.
– roaima
Jan 12 at 19:18
add a comment |
2
If you take a look at ALL filesystem bugs that exists, you will never find one that is imune to data loss or problems. EXT4 had a big bug last year - phoronix.com/… - If you trust no one here, you will not trust harware raid too, since you will be tied to the hardware model you get for this purpose.
– nwildner
Jan 11 at 10:14
You probably right re file systems bugs, but as I alluded to in my question RHEL/CentOS will not support ZFS and using ZoL on CentOS/RHEL may affect stability. But at this stage ZFS may be the best option, but I would like to see other opinions.
– MeData
Jan 11 at 10:31
3
You could use SmartOS as the hypervisor layer. That's based on OpenSolaris and has native ZFS. Then you can use CentOS inside your VMs and not need to worry about disk integrity.
– Stephen Harris
Jan 11 at 15:33
"RH has no intention of supporting it [ZFS] in the future" - I didn't think there was any (commercial) support from RedHat for CentOS anyway.
– roaima
Jan 12 at 19:18
2
2
If you take a look at ALL filesystem bugs that exists, you will never find one that is imune to data loss or problems. EXT4 had a big bug last year - phoronix.com/… - If you trust no one here, you will not trust harware raid too, since you will be tied to the hardware model you get for this purpose.
– nwildner
Jan 11 at 10:14
If you take a look at ALL filesystem bugs that exists, you will never find one that is imune to data loss or problems. EXT4 had a big bug last year - phoronix.com/… - If you trust no one here, you will not trust harware raid too, since you will be tied to the hardware model you get for this purpose.
– nwildner
Jan 11 at 10:14
You probably right re file systems bugs, but as I alluded to in my question RHEL/CentOS will not support ZFS and using ZoL on CentOS/RHEL may affect stability. But at this stage ZFS may be the best option, but I would like to see other opinions.
– MeData
Jan 11 at 10:31
You probably right re file systems bugs, but as I alluded to in my question RHEL/CentOS will not support ZFS and using ZoL on CentOS/RHEL may affect stability. But at this stage ZFS may be the best option, but I would like to see other opinions.
– MeData
Jan 11 at 10:31
3
3
You could use SmartOS as the hypervisor layer. That's based on OpenSolaris and has native ZFS. Then you can use CentOS inside your VMs and not need to worry about disk integrity.
– Stephen Harris
Jan 11 at 15:33
You could use SmartOS as the hypervisor layer. That's based on OpenSolaris and has native ZFS. Then you can use CentOS inside your VMs and not need to worry about disk integrity.
– Stephen Harris
Jan 11 at 15:33
"RH has no intention of supporting it [ZFS] in the future" - I didn't think there was any (commercial) support from RedHat for CentOS anyway.
– roaima
Jan 12 at 19:18
"RH has no intention of supporting it [ZFS] in the future" - I didn't think there was any (commercial) support from RedHat for CentOS anyway.
– roaima
Jan 12 at 19:18
add a comment |
1 Answer
1
active
oldest
votes
mdadm RAID does not calculate (slow) and does not correct (reliably) but it can be used to detect (mismatch_cnt != 0 after check). If you do use mdadm (for other reasons) and do run obligatory checks (for obvious reasons), make it include the mismatch_cnt in the mail report. (And don't forget about SMART monitoring, either, and don't wait to replace drives with bad or reallocated sectors...)
That way, if there is bitrot going on any individual disk, you would at least get some notification about it. I've monitored my RAID like that for years and it never happened (other than when provoked to test the functionality itself).
As a result, I don't think bitrot is a common issue (on a hardware level).
Each drive uses checksums internally, that's how it detects read errors. If the drive gets the wrong data on a read, it won't return it, it'll report an error instead. And usually that's good enough.
Then there's a special kind of bitrot that no filesystem will help you with, either. It's when software writes corrupt data in the first place. Like the braindead photo manager that changes exif data of every picture it finds. The files will be corrupted but the checksumming anti-bitrot filesystem will happily tell you: yeah that's the data I was told to write and the checksum checks out, what about it?
At that point, you need a backup system that checksums files and detects changed files and does not remove/replace intact data with new/corrupt versions of the same data, so you can go back to the intact ones. And it would be grand if it could send you a report about changed files and you would notice if your entire picture collection was in it even though you can't remember changing them.
Heck, some things should be mounted read-only by default... but no one does it because it's a hassle.
answered Jan 11 at 11:29
frostschutzfrostschutz
26.5k15583
add a comment |
Your Answer
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "106"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f493908%2fwhat-are-data-integrity-bit-rot-protection-options-on-centos-7%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
mdadm RAID does not calculate (slow) and does not correct (reliably) but it can be used to detect (mismatch_cnt != 0 after check). If you do use mdadm (for other reasons) and do run obligatory checks (for obvious reasons), make it include the mismatch_cnt in the mail report. (And don't forget about SMART monitoring, either, and don't wait to replace drives with bad or reallocated sectors...)
That way, if there is bitrot going on any individual disk, you would at least get some notification about it. I've monitored my RAID like that for years and it never happened (other than when provoked to test the functionality itself).
As a result, I don't think bitrot is a common issue (on a hardware level).
Each drive uses checksums internally, that's how it detects read errors. If the drive gets the wrong data on a read, it won't return it, it'll report an error instead. And usually that's good enough.
Then there's a special kind of bitrot that no filesystem will help you with, either. It's when software writes corrupt data in the first place. Like the braindead photo manager that changes exif data of every picture it finds. The files will be corrupted but the checksumming anti-bitrot filesystem will happily tell you: yeah that's the data I was told to write and the checksum checks out, what about it?
At that point, you need a backup system that checksums files and detects changed files and does not remove/replace intact data with new/corrupt versions of the same data, so you can go back to the intact ones. And it would be grand if it could send you a report about changed files and you would notice if your entire picture collection was in it even though you can't remember changing them.
Heck, some things should be mounted read-only by default... but no one does it because it's a hassle.
answered Jan 11 at 11:29
frostschutzfrostschutz
26.5k15583
add a comment |
mdadm RAID does not calculate (slow) and does not correct (reliably) but it can be used to detect (mismatch_cnt != 0 after check). If you do use mdadm (for other reasons) and do run obligatory checks (for obvious reasons), make it include the mismatch_cnt in the mail report. (And don't forget about SMART monitoring, either, and don't wait to replace drives with bad or reallocated sectors...)
That way, if there is bitrot going on any individual disk, you would at least get some notification about it. I've monitored my RAID like that for years and it never happened (other than when provoked to test the functionality itself).
As a result, I don't think bitrot is a common issue (on a hardware level).
Each drive uses checksums internally, that's how it detects read errors. If the drive gets the wrong data on a read, it won't return it, it'll report an error instead. And usually that's good enough.
Then there's a special kind of bitrot that no filesystem will help you with, either. It's when software writes corrupt data in the first place. Like the braindead photo manager that changes exif data of every picture it finds. The files will be corrupted but the checksumming anti-bitrot filesystem will happily tell you: yeah that's the data I was told to write and the checksum checks out, what about it?
At that point, you need a backup system that checksums files and detects changed files and does not remove/replace intact data with new/corrupt versions of the same data, so you can go back to the intact ones. And it would be grand if it could send you a report about changed files and you would notice if your entire picture collection was in it even though you can't remember changing them.
Heck, some things should be mounted read-only by default... but no one does it because it's a hassle.
answered Jan 11 at 11:29
frostschutzfrostschutz
26.5k15583
add a comment |
mdadm RAID does not calculate (slow) and does not correct (reliably) but it can be used to detect (mismatch_cnt != 0 after check). If you do use mdadm (for other reasons) and do run obligatory checks (for obvious reasons), make it include the mismatch_cnt in the mail report. (And don't forget about SMART monitoring, either, and don't wait to replace drives with bad or reallocated sectors...)
That way, if there is bitrot going on any individual disk, you would at least get some notification about it. I've monitored my RAID like that for years and it never happened (other than when provoked to test the functionality itself).
As a result, I don't think bitrot is a common issue (on a hardware level).
Each drive uses checksums internally, that's how it detects read errors. If the drive gets the wrong data on a read, it won't return it, it'll report an error instead. And usually that's good enough.
Then there's a special kind of bitrot that no filesystem will help you with, either. It's when software writes corrupt data in the first place. Like the braindead photo manager that changes exif data of every picture it finds. The files will be corrupted but the checksumming anti-bitrot filesystem will happily tell you: yeah that's the data I was told to write and the checksum checks out, what about it?
At that point, you need a backup system that checksums files and detects changed files and does not remove/replace intact data with new/corrupt versions of the same data, so you can go back to the intact ones. And it would be grand if it could send you a report about changed files and you would notice if your entire picture collection was in it even though you can't remember changing them.
Heck, some things should be mounted read-only by default... but no one does it because it's a hassle.
answered Jan 11 at 11:29
frostschutzfrostschutz
26.5k15583
mdadm RAID does not calculate (slow) and does not correct (reliably) but it can be used to detect (mismatch_cnt != 0 after check). If you do use mdadm (for other reasons) and do run obligatory checks (for obvious reasons), make it include the mismatch_cnt in the mail report. (And don't forget about SMART monitoring, either, and don't wait to replace drives with bad or reallocated sectors...)
That way, if there is bitrot going on any individual disk, you would at least get some notification about it. I've monitored my RAID like that for years and it never happened (other than when provoked to test the functionality itself).
As a result, I don't think bitrot is a common issue (on a hardware level).
Each drive uses checksums internally, that's how it detects read errors. If the drive gets the wrong data on a read, it won't return it, it'll report an error instead. And usually that's good enough.
Then there's a special kind of bitrot that no filesystem will help you with, either. It's when software writes corrupt data in the first place. Like the braindead photo manager that changes exif data of every picture it finds. The files will be corrupted but the checksumming anti-bitrot filesystem will happily tell you: yeah that's the data I was told to write and the checksum checks out, what about it?
At that point, you need a backup system that checksums files and detects changed files and does not remove/replace intact data with new/corrupt versions of the same data, so you can go back to the intact ones. And it would be grand if it could send you a report about changed files and you would notice if your entire picture collection was in it even though you can't remember changing them.
Heck, some things should be mounted read-only by default... but no one does it because it's a hassle.
answered Jan 11 at 11:29
frostschutzfrostschutz
26.5k15583
answered Jan 11 at 11:29
frostschutzfrostschutz
26.5k15583
answered Jan 11 at 11:29
frostschutzfrostschutz
26.5k15583
answered Jan 11 at 11:29
frostschutzfrostschutz
26.5k15583
26.5k15583
add a comment |
add a comment |
Thanks for contributing an answer to Unix & Linux Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f493908%2fwhat-are-data-integrity-bit-rot-protection-options-on-centos-7%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
2
If you take a look at ALL filesystem bugs that exists, you will never find one that is imune to data loss or problems. EXT4 had a big bug last year - phoronix.com/… - If you trust no one here, you will not trust harware raid too, since you will be tied to the hardware model you get for this purpose.
– nwildner
Jan 11 at 10:14
You probably right re file systems bugs, but as I alluded to in my question RHEL/CentOS will not support ZFS and using ZoL on CentOS/RHEL may affect stability. But at this stage ZFS may be the best option, but I would like to see other opinions.
– MeData
Jan 11 at 10:31
3
You could use SmartOS as the hypervisor layer. That's based on OpenSolaris and has native ZFS. Then you can use CentOS inside your VMs and not need to worry about disk integrity.
– Stephen Harris
Jan 11 at 15:33
"RH has no intention of supporting it [ZFS] in the future" - I didn't think there was any (commercial) support from RedHat for CentOS anyway.
– roaima
Jan 12 at 19:18