rsync to USB flash drive always transferring all data

The name of the pictureThe name of the pictureThe name of the pictureClash Royale CLAN TAG#URR8PPP











up vote
1
down vote

favorite












When rsyncing a directory to a freshly plugged-in external USB flash drive, via



rsync -av /source/ /dest/


all files get transferred (i.e. rewritten) despite no changes in the files.



Note that overwriting the files only takes place once the USB is un- and replugged. Doing the rsync command twice in a row without unplugging the drive in-between does successfully skip the whole directory contents.



Including the -u update option and explicitly adding the -t option did not change anything.



The mount point remains the same (i.e. /media/user/<UUID>, the drive is automouted by xfce, the /dev/sdxy obviously changes)
The hard drive source is ext4, while the USB is vfat with utf8 character encoding.



What could be the reason for this behaviour is it the change in the /dev/ name entry? How can I make rsync run with properly recognizing file changes? My backup should just take seconds without this, while it now is always minutes due to the large amount of data being overwritten repeatedly, nor is the massive writing the best for the flash drive's life time expectancy.










share|improve this question

























    up vote
    1
    down vote

    favorite












    When rsyncing a directory to a freshly plugged-in external USB flash drive, via



    rsync -av /source/ /dest/


    all files get transferred (i.e. rewritten) despite no changes in the files.



    Note that overwriting the files only takes place once the USB is un- and replugged. Doing the rsync command twice in a row without unplugging the drive in-between does successfully skip the whole directory contents.



    Including the -u update option and explicitly adding the -t option did not change anything.



    The mount point remains the same (i.e. /media/user/<UUID>, the drive is automouted by xfce, the /dev/sdxy obviously changes)
    The hard drive source is ext4, while the USB is vfat with utf8 character encoding.



    What could be the reason for this behaviour is it the change in the /dev/ name entry? How can I make rsync run with properly recognizing file changes? My backup should just take seconds without this, while it now is always minutes due to the large amount of data being overwritten repeatedly, nor is the massive writing the best for the flash drive's life time expectancy.










    share|improve this question























      up vote
      1
      down vote

      favorite









      up vote
      1
      down vote

      favorite











      When rsyncing a directory to a freshly plugged-in external USB flash drive, via



      rsync -av /source/ /dest/


      all files get transferred (i.e. rewritten) despite no changes in the files.



      Note that overwriting the files only takes place once the USB is un- and replugged. Doing the rsync command twice in a row without unplugging the drive in-between does successfully skip the whole directory contents.



      Including the -u update option and explicitly adding the -t option did not change anything.



      The mount point remains the same (i.e. /media/user/<UUID>, the drive is automouted by xfce, the /dev/sdxy obviously changes)
      The hard drive source is ext4, while the USB is vfat with utf8 character encoding.



      What could be the reason for this behaviour is it the change in the /dev/ name entry? How can I make rsync run with properly recognizing file changes? My backup should just take seconds without this, while it now is always minutes due to the large amount of data being overwritten repeatedly, nor is the massive writing the best for the flash drive's life time expectancy.










      share|improve this question













      When rsyncing a directory to a freshly plugged-in external USB flash drive, via



      rsync -av /source/ /dest/


      all files get transferred (i.e. rewritten) despite no changes in the files.



      Note that overwriting the files only takes place once the USB is un- and replugged. Doing the rsync command twice in a row without unplugging the drive in-between does successfully skip the whole directory contents.



      Including the -u update option and explicitly adding the -t option did not change anything.



      The mount point remains the same (i.e. /media/user/<UUID>, the drive is automouted by xfce, the /dev/sdxy obviously changes)
      The hard drive source is ext4, while the USB is vfat with utf8 character encoding.



      What could be the reason for this behaviour is it the change in the /dev/ name entry? How can I make rsync run with properly recognizing file changes? My backup should just take seconds without this, while it now is always minutes due to the large amount of data being overwritten repeatedly, nor is the massive writing the best for the flash drive's life time expectancy.







      mount rsync ext4 vfat






      share|improve this question













      share|improve this question











      share|improve this question




      share|improve this question










      asked Mar 2 '16 at 23:00









      Fiximan

      3,123524




      3,123524




















          3 Answers
          3






          active

          oldest

          votes

















          up vote
          6
          down vote



          accepted










          Your FAT drive can store timestamps only to two second accuracy. When you unplug and replug the drive you effectively break all the file times. See the --modify-window option for a workaround.



          Secondly, you're never going to see fast backups with rsync like this, because when copying locally it behaves much like cp.






          share|improve this answer




















          • Setting a value for --modify-window solved the issue, thank you.
            – Fiximan
            Mar 4 '16 at 21:35

















          up vote
          1
          down vote













          By default, rsync identifies changed files through their modification time and file size. A quick Google search for "vfat timestamp" reveals several problems related to the tz=UTC mount option.



          Could it be that the timestamps on external drive differ from the internal drive and rsync therefore always considers the files to be different? If that is the case, you should probably check your mount options.



          Alternatively (and actually regardless of whether my theory is right), you could try to run rsync with the -c option, which makes it check for changes through hashes of the file contents. This potentially slows down things, but I'm unsure about the practical impact.






          share|improve this answer





























            up vote
            0
            down vote













            My Environment

            Raspberry Pi3 B+,Raspbian Strech(Linux)



            I added next 1 line below into /etc/fstab
            dev/sda1 /media/pi/TOSHIBA vfat defaults 0 0



            Then
            sudo umount /media/pi/TOSHIBA



            To find device name you can use next command
            lsusb



            Then I re-inserted the USB MEMORY.

            Then USB MEMORY's owner was changed like next 1 line below.
            drwxr-xr-x 3 root root 8192 11月 9 23:22 TOSHIBA



            Then I have been able to backup with permission,owner and timestamps using next 1 line below.
            rsync -alvptgo /home/pi /media/pi/TOSHIBA/pi

            which I have really used as DIFF backup command.



            But PAY ATTENTION to that you must insert a USB MEMORY in the slot when you reboot

            OR

            Delete that 1 line in /etc/fstab before you reboot.






            share|improve this answer








            New contributor




            USAKU is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
            Check out our Code of Conduct.

















              Your Answer








              StackExchange.ready(function()
              var channelOptions =
              tags: "".split(" "),
              id: "106"
              ;
              initTagRenderer("".split(" "), "".split(" "), channelOptions);

              StackExchange.using("externalEditor", function()
              // Have to fire editor after snippets, if snippets enabled
              if (StackExchange.settings.snippets.snippetsEnabled)
              StackExchange.using("snippets", function()
              createEditor();
              );

              else
              createEditor();

              );

              function createEditor()
              StackExchange.prepareEditor(
              heartbeatType: 'answer',
              convertImagesToLinks: false,
              noModals: true,
              showLowRepImageUploadWarning: true,
              reputationToPostImages: null,
              bindNavPrevention: true,
              postfix: "",
              imageUploader:
              brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
              contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
              allowUrls: true
              ,
              onDemand: true,
              discardSelector: ".discard-answer"
              ,immediatelyShowMarkdownHelp:true
              );



              );













               

              draft saved


              draft discarded


















              StackExchange.ready(
              function ()
              StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f267204%2frsync-to-usb-flash-drive-always-transferring-all-data%23new-answer', 'question_page');

              );

              Post as a guest






























              3 Answers
              3






              active

              oldest

              votes








              3 Answers
              3






              active

              oldest

              votes









              active

              oldest

              votes






              active

              oldest

              votes








              up vote
              6
              down vote



              accepted










              Your FAT drive can store timestamps only to two second accuracy. When you unplug and replug the drive you effectively break all the file times. See the --modify-window option for a workaround.



              Secondly, you're never going to see fast backups with rsync like this, because when copying locally it behaves much like cp.






              share|improve this answer




















              • Setting a value for --modify-window solved the issue, thank you.
                – Fiximan
                Mar 4 '16 at 21:35














              up vote
              6
              down vote



              accepted










              Your FAT drive can store timestamps only to two second accuracy. When you unplug and replug the drive you effectively break all the file times. See the --modify-window option for a workaround.



              Secondly, you're never going to see fast backups with rsync like this, because when copying locally it behaves much like cp.






              share|improve this answer




















              • Setting a value for --modify-window solved the issue, thank you.
                – Fiximan
                Mar 4 '16 at 21:35












              up vote
              6
              down vote



              accepted







              up vote
              6
              down vote



              accepted






              Your FAT drive can store timestamps only to two second accuracy. When you unplug and replug the drive you effectively break all the file times. See the --modify-window option for a workaround.



              Secondly, you're never going to see fast backups with rsync like this, because when copying locally it behaves much like cp.






              share|improve this answer












              Your FAT drive can store timestamps only to two second accuracy. When you unplug and replug the drive you effectively break all the file times. See the --modify-window option for a workaround.



              Secondly, you're never going to see fast backups with rsync like this, because when copying locally it behaves much like cp.







              share|improve this answer












              share|improve this answer



              share|improve this answer










              answered Mar 2 '16 at 23:19









              roaima

              41.8k550114




              41.8k550114











              • Setting a value for --modify-window solved the issue, thank you.
                – Fiximan
                Mar 4 '16 at 21:35
















              • Setting a value for --modify-window solved the issue, thank you.
                – Fiximan
                Mar 4 '16 at 21:35















              Setting a value for --modify-window solved the issue, thank you.
              – Fiximan
              Mar 4 '16 at 21:35




              Setting a value for --modify-window solved the issue, thank you.
              – Fiximan
              Mar 4 '16 at 21:35












              up vote
              1
              down vote













              By default, rsync identifies changed files through their modification time and file size. A quick Google search for "vfat timestamp" reveals several problems related to the tz=UTC mount option.



              Could it be that the timestamps on external drive differ from the internal drive and rsync therefore always considers the files to be different? If that is the case, you should probably check your mount options.



              Alternatively (and actually regardless of whether my theory is right), you could try to run rsync with the -c option, which makes it check for changes through hashes of the file contents. This potentially slows down things, but I'm unsure about the practical impact.






              share|improve this answer


























                up vote
                1
                down vote













                By default, rsync identifies changed files through their modification time and file size. A quick Google search for "vfat timestamp" reveals several problems related to the tz=UTC mount option.



                Could it be that the timestamps on external drive differ from the internal drive and rsync therefore always considers the files to be different? If that is the case, you should probably check your mount options.



                Alternatively (and actually regardless of whether my theory is right), you could try to run rsync with the -c option, which makes it check for changes through hashes of the file contents. This potentially slows down things, but I'm unsure about the practical impact.






                share|improve this answer
























                  up vote
                  1
                  down vote










                  up vote
                  1
                  down vote









                  By default, rsync identifies changed files through their modification time and file size. A quick Google search for "vfat timestamp" reveals several problems related to the tz=UTC mount option.



                  Could it be that the timestamps on external drive differ from the internal drive and rsync therefore always considers the files to be different? If that is the case, you should probably check your mount options.



                  Alternatively (and actually regardless of whether my theory is right), you could try to run rsync with the -c option, which makes it check for changes through hashes of the file contents. This potentially slows down things, but I'm unsure about the practical impact.






                  share|improve this answer














                  By default, rsync identifies changed files through their modification time and file size. A quick Google search for "vfat timestamp" reveals several problems related to the tz=UTC mount option.



                  Could it be that the timestamps on external drive differ from the internal drive and rsync therefore always considers the files to be different? If that is the case, you should probably check your mount options.



                  Alternatively (and actually regardless of whether my theory is right), you could try to run rsync with the -c option, which makes it check for changes through hashes of the file contents. This potentially slows down things, but I'm unsure about the practical impact.







                  share|improve this answer














                  share|improve this answer



                  share|improve this answer








                  edited Mar 2 '16 at 23:29

























                  answered Mar 2 '16 at 23:18









                  F30

                  216112




                  216112




















                      up vote
                      0
                      down vote













                      My Environment

                      Raspberry Pi3 B+,Raspbian Strech(Linux)



                      I added next 1 line below into /etc/fstab
                      dev/sda1 /media/pi/TOSHIBA vfat defaults 0 0



                      Then
                      sudo umount /media/pi/TOSHIBA



                      To find device name you can use next command
                      lsusb



                      Then I re-inserted the USB MEMORY.

                      Then USB MEMORY's owner was changed like next 1 line below.
                      drwxr-xr-x 3 root root 8192 11月 9 23:22 TOSHIBA



                      Then I have been able to backup with permission,owner and timestamps using next 1 line below.
                      rsync -alvptgo /home/pi /media/pi/TOSHIBA/pi

                      which I have really used as DIFF backup command.



                      But PAY ATTENTION to that you must insert a USB MEMORY in the slot when you reboot

                      OR

                      Delete that 1 line in /etc/fstab before you reboot.






                      share|improve this answer








                      New contributor




                      USAKU is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
                      Check out our Code of Conduct.





















                        up vote
                        0
                        down vote













                        My Environment

                        Raspberry Pi3 B+,Raspbian Strech(Linux)



                        I added next 1 line below into /etc/fstab
                        dev/sda1 /media/pi/TOSHIBA vfat defaults 0 0



                        Then
                        sudo umount /media/pi/TOSHIBA



                        To find device name you can use next command
                        lsusb



                        Then I re-inserted the USB MEMORY.

                        Then USB MEMORY's owner was changed like next 1 line below.
                        drwxr-xr-x 3 root root 8192 11月 9 23:22 TOSHIBA



                        Then I have been able to backup with permission,owner and timestamps using next 1 line below.
                        rsync -alvptgo /home/pi /media/pi/TOSHIBA/pi

                        which I have really used as DIFF backup command.



                        But PAY ATTENTION to that you must insert a USB MEMORY in the slot when you reboot

                        OR

                        Delete that 1 line in /etc/fstab before you reboot.






                        share|improve this answer








                        New contributor




                        USAKU is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
                        Check out our Code of Conduct.



















                          up vote
                          0
                          down vote










                          up vote
                          0
                          down vote









                          My Environment

                          Raspberry Pi3 B+,Raspbian Strech(Linux)



                          I added next 1 line below into /etc/fstab
                          dev/sda1 /media/pi/TOSHIBA vfat defaults 0 0



                          Then
                          sudo umount /media/pi/TOSHIBA



                          To find device name you can use next command
                          lsusb



                          Then I re-inserted the USB MEMORY.

                          Then USB MEMORY's owner was changed like next 1 line below.
                          drwxr-xr-x 3 root root 8192 11月 9 23:22 TOSHIBA



                          Then I have been able to backup with permission,owner and timestamps using next 1 line below.
                          rsync -alvptgo /home/pi /media/pi/TOSHIBA/pi

                          which I have really used as DIFF backup command.



                          But PAY ATTENTION to that you must insert a USB MEMORY in the slot when you reboot

                          OR

                          Delete that 1 line in /etc/fstab before you reboot.






                          share|improve this answer








                          New contributor




                          USAKU is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
                          Check out our Code of Conduct.









                          My Environment

                          Raspberry Pi3 B+,Raspbian Strech(Linux)



                          I added next 1 line below into /etc/fstab
                          dev/sda1 /media/pi/TOSHIBA vfat defaults 0 0



                          Then
                          sudo umount /media/pi/TOSHIBA



                          To find device name you can use next command
                          lsusb



                          Then I re-inserted the USB MEMORY.

                          Then USB MEMORY's owner was changed like next 1 line below.
                          drwxr-xr-x 3 root root 8192 11月 9 23:22 TOSHIBA



                          Then I have been able to backup with permission,owner and timestamps using next 1 line below.
                          rsync -alvptgo /home/pi /media/pi/TOSHIBA/pi

                          which I have really used as DIFF backup command.



                          But PAY ATTENTION to that you must insert a USB MEMORY in the slot when you reboot

                          OR

                          Delete that 1 line in /etc/fstab before you reboot.







                          share|improve this answer








                          New contributor




                          USAKU is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
                          Check out our Code of Conduct.









                          share|improve this answer



                          share|improve this answer






                          New contributor




                          USAKU is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
                          Check out our Code of Conduct.









                          answered 4 hours ago









                          USAKU

                          1




                          1




                          New contributor




                          USAKU is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
                          Check out our Code of Conduct.





                          New contributor





                          USAKU is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
                          Check out our Code of Conduct.






                          USAKU is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
                          Check out our Code of Conduct.



























                               

                              draft saved


                              draft discarded















































                               


                              draft saved


                              draft discarded














                              StackExchange.ready(
                              function ()
                              StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f267204%2frsync-to-usb-flash-drive-always-transferring-all-data%23new-answer', 'question_page');

                              );

                              Post as a guest













































































                              Popular posts from this blog

                              How to check contact read email or not when send email to Individual?

                              Bahrain

                              Postfix configuration issue with fips on centos 7; mailgun relay