Tar produces different files each time
Clash Royale CLAN TAG#URR8PPP
up vote
4
down vote
favorite
I often have large directories that I want to transfer to a local computer from a server. Instead of using recursive scp or rsync on the directory itself, I'll often tar and gzip it first and then transfer it.
Recently, I've wanted to check that this is actually working so I ran md5sum on two independently generated tar and gzip archives of the same source directory. To my suprise, the MD5 hash was different. I did this two more times and it was always a new value. Why am I seeing this result? Are two tar and gzipped directories both generated with the same version of GNU tar in the exact same way not supposed to be exactly the same?
For clarity, I have a source directory and a destination directory. In the destination directory I have dir1 and dir2. I'm running:
tar -zcvf /destination/dir1/source.tar.gz source && md5sum /destination/dir1/source.tar.gz >> md5.txt
tar -zcvf /destination/dir2/source.tar.gz source && md5sum /destination/dir2/source.tar.gz >> md5.txt
Each time I do this, I get a different result from md5sum. Tar produces no errors or warnings.
files tar hashsum
edited Apr 19 at 21:25
Jesse_b
11.5k23063
asked Apr 17 at 14:55
Alon Gelber
232
add a comment |
up vote
4
down vote
favorite
I often have large directories that I want to transfer to a local computer from a server. Instead of using recursive scp or rsync on the directory itself, I'll often tar and gzip it first and then transfer it.
Recently, I've wanted to check that this is actually working so I ran md5sum on two independently generated tar and gzip archives of the same source directory. To my suprise, the MD5 hash was different. I did this two more times and it was always a new value. Why am I seeing this result? Are two tar and gzipped directories both generated with the same version of GNU tar in the exact same way not supposed to be exactly the same?
For clarity, I have a source directory and a destination directory. In the destination directory I have dir1 and dir2. I'm running:
tar -zcvf /destination/dir1/source.tar.gz source && md5sum /destination/dir1/source.tar.gz >> md5.txt
tar -zcvf /destination/dir2/source.tar.gz source && md5sum /destination/dir2/source.tar.gz >> md5.txt
Each time I do this, I get a different result from md5sum. Tar produces no errors or warnings.
files tar hashsum
edited Apr 19 at 21:25
Jesse_b
11.5k23063
asked Apr 17 at 14:55
Alon Gelber
232
As you mentioned large directories, there is no excuse to not substitutelbzipor7zforgzipthese days. This is not going to address your original question, but at least speed up compression by parallel threads.
– ajeh
Apr 19 at 20:51
add a comment |
up vote
4
down vote
favorite
up vote
4
down vote
favorite
I often have large directories that I want to transfer to a local computer from a server. Instead of using recursive scp or rsync on the directory itself, I'll often tar and gzip it first and then transfer it.
Recently, I've wanted to check that this is actually working so I ran md5sum on two independently generated tar and gzip archives of the same source directory. To my suprise, the MD5 hash was different. I did this two more times and it was always a new value. Why am I seeing this result? Are two tar and gzipped directories both generated with the same version of GNU tar in the exact same way not supposed to be exactly the same?
For clarity, I have a source directory and a destination directory. In the destination directory I have dir1 and dir2. I'm running:
tar -zcvf /destination/dir1/source.tar.gz source && md5sum /destination/dir1/source.tar.gz >> md5.txt
tar -zcvf /destination/dir2/source.tar.gz source && md5sum /destination/dir2/source.tar.gz >> md5.txt
Each time I do this, I get a different result from md5sum. Tar produces no errors or warnings.
files tar hashsum
edited Apr 19 at 21:25
Jesse_b
11.5k23063
asked Apr 17 at 14:55
Alon Gelber
232
I often have large directories that I want to transfer to a local computer from a server. Instead of using recursive scp or rsync on the directory itself, I'll often tar and gzip it first and then transfer it.
Recently, I've wanted to check that this is actually working so I ran md5sum on two independently generated tar and gzip archives of the same source directory. To my suprise, the MD5 hash was different. I did this two more times and it was always a new value. Why am I seeing this result? Are two tar and gzipped directories both generated with the same version of GNU tar in the exact same way not supposed to be exactly the same?
For clarity, I have a source directory and a destination directory. In the destination directory I have dir1 and dir2. I'm running:
tar -zcvf /destination/dir1/source.tar.gz source && md5sum /destination/dir1/source.tar.gz >> md5.txt
tar -zcvf /destination/dir2/source.tar.gz source && md5sum /destination/dir2/source.tar.gz >> md5.txt
Each time I do this, I get a different result from md5sum. Tar produces no errors or warnings.
files tar hashsum
files tar hashsum
edited Apr 19 at 21:25
Jesse_b
11.5k23063
asked Apr 17 at 14:55
Alon Gelber
232
edited Apr 19 at 21:25
Jesse_b
11.5k23063
asked Apr 17 at 14:55
Alon Gelber
232
edited Apr 19 at 21:25
Jesse_b
11.5k23063
edited Apr 19 at 21:25
Jesse_b
11.5k23063
edited Apr 19 at 21:25
Jesse_b
11.5k23063
11.5k23063
asked Apr 17 at 14:55
Alon Gelber
232
asked Apr 17 at 14:55
Alon Gelber
232
asked Apr 17 at 14:55
Alon Gelber
232
232
As you mentioned large directories, there is no excuse to not substitutelbzipor7zforgzipthese days. This is not going to address your original question, but at least speed up compression by parallel threads.
– ajeh
Apr 19 at 20:51
add a comment |
As you mentioned large directories, there is no excuse to not substitutelbzipor7zforgzipthese days. This is not going to address your original question, but at least speed up compression by parallel threads.
– ajeh
Apr 19 at 20:51
As you mentioned large directories, there is no excuse to not substitute
lbzip or 7z for gzip these days. This is not going to address your original question, but at least speed up compression by parallel threads.– ajeh
Apr 19 at 20:51
As you mentioned large directories, there is no excuse to not substitute
lbzip or 7z for gzip these days. This is not going to address your original question, but at least speed up compression by parallel threads.– ajeh
Apr 19 at 20:51
add a comment |
1 Answer
1
active
oldest
votes
up vote
7
down vote
accepted
From the looks of things you’re probably being bitten by gzip timestamps; to avoid those, run
GZIP=-n tar -zcvf ...
Note that to get fully reproducible tarballs, you should also impose the sort order used by tar:
GZIP=-n tar --sort=name -zcvf ...
If your version of tar doesn’t support --sort, use this instead:
find source -print0 | LC_ALL=C sort -z | GZIP=-n tar --no-recursion --null -T - -zcvf ...
answered Apr 17 at 15:07
Stephen Kitt
160k24357432
I was trying to reproduce this withtar -czf - dir | md5sumbut failed to get varying checksums. Was that because I was writing to a pipe? (No it wasn't, it turns out, but because of something else relating to not using Linux presumably)
– Kusalananda
Apr 17 at 15:19
@Kusalananda perhaps OpenBSDtarbehaves differently... On Debian I get different sums when piping too.
– Stephen Kitt
Apr 17 at 15:22
I used GNUtar1.29, butgzipcomes from my base system... hmm... but that got the-noption as well. Oh well.
– Kusalananda
Apr 17 at 15:25
add a comment |
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
7
down vote
accepted
From the looks of things you’re probably being bitten by gzip timestamps; to avoid those, run
GZIP=-n tar -zcvf ...
Note that to get fully reproducible tarballs, you should also impose the sort order used by tar:
GZIP=-n tar --sort=name -zcvf ...
If your version of tar doesn’t support --sort, use this instead:
find source -print0 | LC_ALL=C sort -z | GZIP=-n tar --no-recursion --null -T - -zcvf ...
answered Apr 17 at 15:07
Stephen Kitt
160k24357432
I was trying to reproduce this withtar -czf - dir | md5sumbut failed to get varying checksums. Was that because I was writing to a pipe? (No it wasn't, it turns out, but because of something else relating to not using Linux presumably)
– Kusalananda
Apr 17 at 15:19
@Kusalananda perhaps OpenBSDtarbehaves differently... On Debian I get different sums when piping too.
– Stephen Kitt
Apr 17 at 15:22
I used GNUtar1.29, butgzipcomes from my base system... hmm... but that got the-noption as well. Oh well.
– Kusalananda
Apr 17 at 15:25
add a comment |
up vote
7
down vote
accepted
From the looks of things you’re probably being bitten by gzip timestamps; to avoid those, run
GZIP=-n tar -zcvf ...
Note that to get fully reproducible tarballs, you should also impose the sort order used by tar:
GZIP=-n tar --sort=name -zcvf ...
If your version of tar doesn’t support --sort, use this instead:
find source -print0 | LC_ALL=C sort -z | GZIP=-n tar --no-recursion --null -T - -zcvf ...
answered Apr 17 at 15:07
Stephen Kitt
160k24357432
I was trying to reproduce this withtar -czf - dir | md5sumbut failed to get varying checksums. Was that because I was writing to a pipe? (No it wasn't, it turns out, but because of something else relating to not using Linux presumably)
– Kusalananda
Apr 17 at 15:19
@Kusalananda perhaps OpenBSDtarbehaves differently... On Debian I get different sums when piping too.
– Stephen Kitt
Apr 17 at 15:22
I used GNUtar1.29, butgzipcomes from my base system... hmm... but that got the-noption as well. Oh well.
– Kusalananda
Apr 17 at 15:25
add a comment |
up vote
7
down vote
accepted
up vote
7
down vote
accepted
From the looks of things you’re probably being bitten by gzip timestamps; to avoid those, run
GZIP=-n tar -zcvf ...
Note that to get fully reproducible tarballs, you should also impose the sort order used by tar:
GZIP=-n tar --sort=name -zcvf ...
If your version of tar doesn’t support --sort, use this instead:
find source -print0 | LC_ALL=C sort -z | GZIP=-n tar --no-recursion --null -T - -zcvf ...
answered Apr 17 at 15:07
Stephen Kitt
160k24357432
From the looks of things you’re probably being bitten by gzip timestamps; to avoid those, run
GZIP=-n tar -zcvf ...
Note that to get fully reproducible tarballs, you should also impose the sort order used by tar:
GZIP=-n tar --sort=name -zcvf ...
If your version of tar doesn’t support --sort, use this instead:
find source -print0 | LC_ALL=C sort -z | GZIP=-n tar --no-recursion --null -T - -zcvf ...
answered Apr 17 at 15:07
Stephen Kitt
160k24357432
edited Apr 17 at 15:28
answered Apr 17 at 15:07
Stephen Kitt
160k24357432
answered Apr 17 at 15:07
Stephen Kitt
160k24357432
answered Apr 17 at 15:07
Stephen Kitt
160k24357432
160k24357432
I was trying to reproduce this withtar -czf - dir | md5sumbut failed to get varying checksums. Was that because I was writing to a pipe? (No it wasn't, it turns out, but because of something else relating to not using Linux presumably)
– Kusalananda
Apr 17 at 15:19
@Kusalananda perhaps OpenBSDtarbehaves differently... On Debian I get different sums when piping too.
– Stephen Kitt
Apr 17 at 15:22
I used GNUtar1.29, butgzipcomes from my base system... hmm... but that got the-noption as well. Oh well.
– Kusalananda
Apr 17 at 15:25
add a comment |
I was trying to reproduce this withtar -czf - dir | md5sumbut failed to get varying checksums. Was that because I was writing to a pipe? (No it wasn't, it turns out, but because of something else relating to not using Linux presumably)
– Kusalananda
Apr 17 at 15:19
@Kusalananda perhaps OpenBSDtarbehaves differently... On Debian I get different sums when piping too.
– Stephen Kitt
Apr 17 at 15:22
I used GNUtar1.29, butgzipcomes from my base system... hmm... but that got the-noption as well. Oh well.
– Kusalananda
Apr 17 at 15:25
I was trying to reproduce this with
tar -czf - dir | md5sum but failed to get varying checksums. Was that because I was writing to a pipe? (No it wasn't, it turns out, but because of something else relating to not using Linux presumably)– Kusalananda
Apr 17 at 15:19
I was trying to reproduce this with
tar -czf - dir | md5sum but failed to get varying checksums. Was that because I was writing to a pipe? (No it wasn't, it turns out, but because of something else relating to not using Linux presumably)– Kusalananda
Apr 17 at 15:19
@Kusalananda perhaps OpenBSD
tar behaves differently... On Debian I get different sums when piping too.– Stephen Kitt
Apr 17 at 15:22
@Kusalananda perhaps OpenBSD
tar behaves differently... On Debian I get different sums when piping too.– Stephen Kitt
Apr 17 at 15:22
I used GNU
tar 1.29, but gzip comes from my base system... hmm... but that got the -n option as well. Oh well.– Kusalananda
Apr 17 at 15:25
I used GNU
tar 1.29, but gzip comes from my base system... hmm... but that got the -n option as well. Oh well.– Kusalananda
Apr 17 at 15:25
add a comment |
Thanks for contributing an answer to Unix & Linux Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f438329%2ftar-produces-different-files-each-time%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
As you mentioned large directories, there is no excuse to not substitute
lbzipor7zforgzipthese days. This is not going to address your original question, but at least speed up compression by parallel threads.– ajeh
Apr 19 at 20:51