rsync vs mtime and ctime
Clash Royale CLAN TAG#URR8PPP
I've been using rsync for Android to backup my phone to a remote NTFS filesystem on a Linux system for a while.
Recently, the HDD containing the NTFS filesystem has started to fail (or throw "I/O Errors") so I took the opportunity to copy all the files onto a new HDD and new NTFS filesystem. In this instance I used the "FastCopy v2.11" tool for Windows.
My problem is that when I do an rsync "dry run" I can see that it wants to recopy files which already exist on the remote rsync folder. For example, when I run with "-iv" I get this kind of output:
Which, as I understand it means that rsync wants to copy this file to the remote rsync because of a timestamp difference.
The strange thing is that if I use "Astro" for Android to look at the local file properties, I can see that the file's size, modified time, and MD5 checksum are exactly the same as that of the remote file (using ls -l to check the modified time).
Given that I recently copied the remote rsync files from an old NTFS filesystem, the remote file's ctime is different (using ls -lc).
Does rsync look at the remote ctime, and if so is there any way I can use rsync, or ntfs-3g to get around this problem?
rsync android ntfs
edited Dec 11 '13 at 15:35
Anthon
61k17104166
asked Nov 10 '13 at 23:18
wombatwombat
62
add a comment |
I've been using rsync for Android to backup my phone to a remote NTFS filesystem on a Linux system for a while.
Recently, the HDD containing the NTFS filesystem has started to fail (or throw "I/O Errors") so I took the opportunity to copy all the files onto a new HDD and new NTFS filesystem. In this instance I used the "FastCopy v2.11" tool for Windows.
My problem is that when I do an rsync "dry run" I can see that it wants to recopy files which already exist on the remote rsync folder. For example, when I run with "-iv" I get this kind of output:
Which, as I understand it means that rsync wants to copy this file to the remote rsync because of a timestamp difference.
The strange thing is that if I use "Astro" for Android to look at the local file properties, I can see that the file's size, modified time, and MD5 checksum are exactly the same as that of the remote file (using ls -l to check the modified time).
Given that I recently copied the remote rsync files from an old NTFS filesystem, the remote file's ctime is different (using ls -lc).
Does rsync look at the remote ctime, and if so is there any way I can use rsync, or ntfs-3g to get around this problem?
rsync android ntfs
edited Dec 11 '13 at 15:35
Anthon
61k17104166
asked Nov 10 '13 at 23:18
wombatwombat
62
add a comment |
I've been using rsync for Android to backup my phone to a remote NTFS filesystem on a Linux system for a while.
Recently, the HDD containing the NTFS filesystem has started to fail (or throw "I/O Errors") so I took the opportunity to copy all the files onto a new HDD and new NTFS filesystem. In this instance I used the "FastCopy v2.11" tool for Windows.
My problem is that when I do an rsync "dry run" I can see that it wants to recopy files which already exist on the remote rsync folder. For example, when I run with "-iv" I get this kind of output:
Which, as I understand it means that rsync wants to copy this file to the remote rsync because of a timestamp difference.
The strange thing is that if I use "Astro" for Android to look at the local file properties, I can see that the file's size, modified time, and MD5 checksum are exactly the same as that of the remote file (using ls -l to check the modified time).
Given that I recently copied the remote rsync files from an old NTFS filesystem, the remote file's ctime is different (using ls -lc).
Does rsync look at the remote ctime, and if so is there any way I can use rsync, or ntfs-3g to get around this problem?
rsync android ntfs
edited Dec 11 '13 at 15:35
Anthon
61k17104166
asked Nov 10 '13 at 23:18
wombatwombat
62
I've been using rsync for Android to backup my phone to a remote NTFS filesystem on a Linux system for a while.
Recently, the HDD containing the NTFS filesystem has started to fail (or throw "I/O Errors") so I took the opportunity to copy all the files onto a new HDD and new NTFS filesystem. In this instance I used the "FastCopy v2.11" tool for Windows.
My problem is that when I do an rsync "dry run" I can see that it wants to recopy files which already exist on the remote rsync folder. For example, when I run with "-iv" I get this kind of output:
Which, as I understand it means that rsync wants to copy this file to the remote rsync because of a timestamp difference.
The strange thing is that if I use "Astro" for Android to look at the local file properties, I can see that the file's size, modified time, and MD5 checksum are exactly the same as that of the remote file (using ls -l to check the modified time).
Given that I recently copied the remote rsync files from an old NTFS filesystem, the remote file's ctime is different (using ls -lc).
Does rsync look at the remote ctime, and if so is there any way I can use rsync, or ntfs-3g to get around this problem?
rsync android ntfs
rsync android ntfs
edited Dec 11 '13 at 15:35
Anthon
61k17104166
asked Nov 10 '13 at 23:18
wombatwombat
62
edited Dec 11 '13 at 15:35
Anthon
61k17104166
asked Nov 10 '13 at 23:18
wombatwombat
62
edited Dec 11 '13 at 15:35
Anthon
61k17104166
edited Dec 11 '13 at 15:35
Anthon
61k17104166
edited Dec 11 '13 at 15:35
Anthon
61k17104166
61k17104166
asked Nov 10 '13 at 23:18
wombatwombat
62
asked Nov 10 '13 at 23:18
wombatwombat
62
asked Nov 10 '13 at 23:18
wombatwombat
62
62
add a comment |
add a comment |
2 Answers
2
active
oldest
votes
By default rsync relies on the mtime/ctime and size for file comparison, but if you use the -c flag it would ignore them and use content checksums.
The problem is this way can be much slower(those checksums might be really expensive to calculate on your mobile device) and you might need to always run it like this from now on, so it might make sense to just let it run once without checksums, let it do its thing so it copies everything once again based on the mtime/ctime/size comparison method it uses by default, but at least next times you won't spend time calculating checksums.
answered Nov 10 '13 at 23:29
Cristian Măgherușan-StanciuCristian Măgherușan-Stanciu
60946
Thanks Cristian. I had looked at the checksum option but wanted to explore other possibilities for the reason you mention. Especially given that the remote linux rsync host is a Raspberry Pi at the moment! Maybe I should temporarily mount both filesystems on a faster linux system (laptop/desktop) and then do a local rsync based on the checksum algorithm, and then return the filesystems to their respective places from there on...
– wombat
Nov 12 '13 at 10:37
I guess the other option is to find or write a utility which can set my file ctime to the same value as the file mtime. Or maybe get the rsync source code and build my own version which can be instructed to only consider "size and mtime" in its quick check algorithm?
– wombat
Nov 12 '13 at 10:42
add a comment |
I just did a quick experiment; rsync compares mtime and ignores ctime (on Mac, at least.) Unfortunately, Windows file systems only have ctime, which they return for atime and mtime as well.
This is why rsync is so aggressive about copying files from Windows file systems that don't need to be copied — the mtime on the Unix file is being compared to the "ctime" on the Windows file.
answered Jan 10 '17 at 22:03
Edward FalkEdward Falk
1,004712
add a comment |
Your Answer
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "106"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f100707%2frsync-vs-mtime-and-ctime%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
By default rsync relies on the mtime/ctime and size for file comparison, but if you use the -c flag it would ignore them and use content checksums.
The problem is this way can be much slower(those checksums might be really expensive to calculate on your mobile device) and you might need to always run it like this from now on, so it might make sense to just let it run once without checksums, let it do its thing so it copies everything once again based on the mtime/ctime/size comparison method it uses by default, but at least next times you won't spend time calculating checksums.
answered Nov 10 '13 at 23:29
Cristian Măgherușan-StanciuCristian Măgherușan-Stanciu
60946
Thanks Cristian. I had looked at the checksum option but wanted to explore other possibilities for the reason you mention. Especially given that the remote linux rsync host is a Raspberry Pi at the moment! Maybe I should temporarily mount both filesystems on a faster linux system (laptop/desktop) and then do a local rsync based on the checksum algorithm, and then return the filesystems to their respective places from there on...
– wombat
Nov 12 '13 at 10:37
I guess the other option is to find or write a utility which can set my file ctime to the same value as the file mtime. Or maybe get the rsync source code and build my own version which can be instructed to only consider "size and mtime" in its quick check algorithm?
– wombat
Nov 12 '13 at 10:42
add a comment |
By default rsync relies on the mtime/ctime and size for file comparison, but if you use the -c flag it would ignore them and use content checksums.
The problem is this way can be much slower(those checksums might be really expensive to calculate on your mobile device) and you might need to always run it like this from now on, so it might make sense to just let it run once without checksums, let it do its thing so it copies everything once again based on the mtime/ctime/size comparison method it uses by default, but at least next times you won't spend time calculating checksums.
answered Nov 10 '13 at 23:29
Cristian Măgherușan-StanciuCristian Măgherușan-Stanciu
60946
Thanks Cristian. I had looked at the checksum option but wanted to explore other possibilities for the reason you mention. Especially given that the remote linux rsync host is a Raspberry Pi at the moment! Maybe I should temporarily mount both filesystems on a faster linux system (laptop/desktop) and then do a local rsync based on the checksum algorithm, and then return the filesystems to their respective places from there on...
– wombat
Nov 12 '13 at 10:37
I guess the other option is to find or write a utility which can set my file ctime to the same value as the file mtime. Or maybe get the rsync source code and build my own version which can be instructed to only consider "size and mtime" in its quick check algorithm?
– wombat
Nov 12 '13 at 10:42
add a comment |
By default rsync relies on the mtime/ctime and size for file comparison, but if you use the -c flag it would ignore them and use content checksums.
The problem is this way can be much slower(those checksums might be really expensive to calculate on your mobile device) and you might need to always run it like this from now on, so it might make sense to just let it run once without checksums, let it do its thing so it copies everything once again based on the mtime/ctime/size comparison method it uses by default, but at least next times you won't spend time calculating checksums.
answered Nov 10 '13 at 23:29
Cristian Măgherușan-StanciuCristian Măgherușan-Stanciu
60946
By default rsync relies on the mtime/ctime and size for file comparison, but if you use the -c flag it would ignore them and use content checksums.
The problem is this way can be much slower(those checksums might be really expensive to calculate on your mobile device) and you might need to always run it like this from now on, so it might make sense to just let it run once without checksums, let it do its thing so it copies everything once again based on the mtime/ctime/size comparison method it uses by default, but at least next times you won't spend time calculating checksums.
answered Nov 10 '13 at 23:29
Cristian Măgherușan-StanciuCristian Măgherușan-Stanciu
60946
answered Nov 10 '13 at 23:29
Cristian Măgherușan-StanciuCristian Măgherușan-Stanciu
60946
answered Nov 10 '13 at 23:29
Cristian Măgherușan-StanciuCristian Măgherușan-Stanciu
60946
answered Nov 10 '13 at 23:29
Cristian Măgherușan-StanciuCristian Măgherușan-Stanciu
60946
60946
Thanks Cristian. I had looked at the checksum option but wanted to explore other possibilities for the reason you mention. Especially given that the remote linux rsync host is a Raspberry Pi at the moment! Maybe I should temporarily mount both filesystems on a faster linux system (laptop/desktop) and then do a local rsync based on the checksum algorithm, and then return the filesystems to their respective places from there on...
– wombat
Nov 12 '13 at 10:37
I guess the other option is to find or write a utility which can set my file ctime to the same value as the file mtime. Or maybe get the rsync source code and build my own version which can be instructed to only consider "size and mtime" in its quick check algorithm?
– wombat
Nov 12 '13 at 10:42
add a comment |
Thanks Cristian. I had looked at the checksum option but wanted to explore other possibilities for the reason you mention. Especially given that the remote linux rsync host is a Raspberry Pi at the moment! Maybe I should temporarily mount both filesystems on a faster linux system (laptop/desktop) and then do a local rsync based on the checksum algorithm, and then return the filesystems to their respective places from there on...
– wombat
Nov 12 '13 at 10:37
I guess the other option is to find or write a utility which can set my file ctime to the same value as the file mtime. Or maybe get the rsync source code and build my own version which can be instructed to only consider "size and mtime" in its quick check algorithm?
– wombat
Nov 12 '13 at 10:42
Thanks Cristian. I had looked at the checksum option but wanted to explore other possibilities for the reason you mention. Especially given that the remote linux rsync host is a Raspberry Pi at the moment! Maybe I should temporarily mount both filesystems on a faster linux system (laptop/desktop) and then do a local rsync based on the checksum algorithm, and then return the filesystems to their respective places from there on...
– wombat
Nov 12 '13 at 10:37
Thanks Cristian. I had looked at the checksum option but wanted to explore other possibilities for the reason you mention. Especially given that the remote linux rsync host is a Raspberry Pi at the moment! Maybe I should temporarily mount both filesystems on a faster linux system (laptop/desktop) and then do a local rsync based on the checksum algorithm, and then return the filesystems to their respective places from there on...
– wombat
Nov 12 '13 at 10:37
I guess the other option is to find or write a utility which can set my file ctime to the same value as the file mtime. Or maybe get the rsync source code and build my own version which can be instructed to only consider "size and mtime" in its quick check algorithm?
– wombat
Nov 12 '13 at 10:42
I guess the other option is to find or write a utility which can set my file ctime to the same value as the file mtime. Or maybe get the rsync source code and build my own version which can be instructed to only consider "size and mtime" in its quick check algorithm?
– wombat
Nov 12 '13 at 10:42
add a comment |
I just did a quick experiment; rsync compares mtime and ignores ctime (on Mac, at least.) Unfortunately, Windows file systems only have ctime, which they return for atime and mtime as well.
This is why rsync is so aggressive about copying files from Windows file systems that don't need to be copied — the mtime on the Unix file is being compared to the "ctime" on the Windows file.
answered Jan 10 '17 at 22:03
Edward FalkEdward Falk
1,004712
add a comment |
I just did a quick experiment; rsync compares mtime and ignores ctime (on Mac, at least.) Unfortunately, Windows file systems only have ctime, which they return for atime and mtime as well.
This is why rsync is so aggressive about copying files from Windows file systems that don't need to be copied — the mtime on the Unix file is being compared to the "ctime" on the Windows file.
answered Jan 10 '17 at 22:03
Edward FalkEdward Falk
1,004712
add a comment |
I just did a quick experiment; rsync compares mtime and ignores ctime (on Mac, at least.) Unfortunately, Windows file systems only have ctime, which they return for atime and mtime as well.
This is why rsync is so aggressive about copying files from Windows file systems that don't need to be copied — the mtime on the Unix file is being compared to the "ctime" on the Windows file.
answered Jan 10 '17 at 22:03
Edward FalkEdward Falk
1,004712
I just did a quick experiment; rsync compares mtime and ignores ctime (on Mac, at least.) Unfortunately, Windows file systems only have ctime, which they return for atime and mtime as well.
This is why rsync is so aggressive about copying files from Windows file systems that don't need to be copied — the mtime on the Unix file is being compared to the "ctime" on the Windows file.
answered Jan 10 '17 at 22:03
Edward FalkEdward Falk
1,004712
answered Jan 10 '17 at 22:03
Edward FalkEdward Falk
1,004712
answered Jan 10 '17 at 22:03
Edward FalkEdward Falk
1,004712
answered Jan 10 '17 at 22:03
Edward FalkEdward Falk
1,004712
1,004712
add a comment |
add a comment |
Thanks for contributing an answer to Unix & Linux Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f100707%2frsync-vs-mtime-and-ctime%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
