Find in a given directory, if there exist 2 files that have exactly the same content irrespective of their names [duplicate]

This question already has an answer here:

Can I get the opposite of `diff -q` — matching identical files without printing their contents

4 answers

Find duplicate files

7 answers

if the problem could be solved using a series of commands on the command line, it would be better for me than writing a script

edited Jan 28 at 18:06

Rui F Ribeiro

40.3k1479136

asked Jan 28 at 18:00

jospin deek

marked as duplicate by Rui F Ribeiro, Christopher, RalfFriedl, Jeff Schaller, G-Man Jan 28 at 19:01

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

Hi and welcome to Unix Stack Exchange. I think to do this, you would need to use the diff command. You would probably have to construct a for loop to compare each file in the directory to every other. Could do it with a fairly simple script.

– Time4Tea
Jan 28 at 18:05

add a comment |

This question already has an answer here:

Can I get the opposite of `diff -q` — matching identical files without printing their contents

4 answers

Find duplicate files

7 answers

if the problem could be solved using a series of commands on the command line, it would be better for me than writing a script

edited Jan 28 at 18:06

Rui F Ribeiro

40.3k1479136

asked Jan 28 at 18:00

jospin deek

marked as duplicate by Rui F Ribeiro, Christopher, RalfFriedl, Jeff Schaller, G-Man Jan 28 at 19:01

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

Hi and welcome to Unix Stack Exchange. I think to do this, you would need to use the diff command. You would probably have to construct a for loop to compare each file in the directory to every other. Could do it with a fairly simple script.

– Time4Tea
Jan 28 at 18:05

add a comment |

This question already has an answer here:

Can I get the opposite of `diff -q` — matching identical files without printing their contents

4 answers

Find duplicate files

7 answers

if the problem could be solved using a series of commands on the command line, it would be better for me than writing a script

edited Jan 28 at 18:06

Rui F Ribeiro

40.3k1479136

asked Jan 28 at 18:00

jospin deek

This question already has an answer here:

Can I get the opposite of `diff -q` — matching identical files without printing their contents

4 answers

Find duplicate files

7 answers

if the problem could be solved using a series of commands on the command line, it would be better for me than writing a script

This question already has an answer here:

Can I get the opposite of `diff -q` — matching identical files without printing their contents

4 answers

Find duplicate files

7 answers

bash shell-script command-line

edited Jan 28 at 18:06

Rui F Ribeiro

40.3k1479136

asked Jan 28 at 18:00

jospin deek

edited Jan 28 at 18:06

Rui F Ribeiro

40.3k1479136

asked Jan 28 at 18:00

jospin deek

edited Jan 28 at 18:06

Rui F Ribeiro

40.3k1479136

edited Jan 28 at 18:06

Rui F Ribeiro

40.3k1479136

edited Jan 28 at 18:06

Rui F Ribeiro

40.3k1479136

asked Jan 28 at 18:00

jospin deek

asked Jan 28 at 18:00

jospin deek

asked Jan 28 at 18:00

jospin deek

marked as duplicate by Rui F Ribeiro, Christopher, RalfFriedl, Jeff Schaller, G-Man Jan 28 at 19:01

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

marked as duplicate by Rui F Ribeiro, Christopher, RalfFriedl, Jeff Schaller, G-Man Jan 28 at 19:01

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

Hi and welcome to Unix Stack Exchange. I think to do this, you would need to use the diff command. You would probably have to construct a for loop to compare each file in the directory to every other. Could do it with a fairly simple script.

– Time4Tea
Jan 28 at 18:05

add a comment |

Hi and welcome to Unix Stack Exchange. I think to do this, you would need to use the diff command. You would probably have to construct a for loop to compare each file in the directory to every other. Could do it with a fairly simple script.

– Time4Tea
Jan 28 at 18:05

Hi and welcome to Unix Stack Exchange. I think to do this, you would need to use the diff command. You would probably have to construct a for loop to compare each file in the directory to every other. Could do it with a fairly simple script.

– Time4Tea
Jan 28 at 18:05

add a comment |

2 Answers
2

active

oldest

votes

for x in *; do for y in *; do [ "$x" = "$y" ] && continue; cmp -s "$x" "$y" && echo Same: "$x" and "$y"; done; done|head -1

Or, broken up a bit for readability:

for x in *
do 
 for y in *
 do 
 [ "$x" = "$y" ] && continue
 cmp -s "$x" "$y" && echo Same: "$x" and "$y"
 done
done | head -1

The head is just to keep the mirror reporting down ("a = b" and "b = a").

answered Jan 28 at 18:11

Jeff Schaller

41.5k1056132

thank you, that was very helpful

– jospin deek
Jan 28 at 18:21

add a comment |

find . -type f -exec md5sum "" ; | awk 'seen[$1] print "Duplicate file "$2" with hash "$1" at "seen[$1]" ! seen[$1] seen[$1]=$2'

edited Jan 28 at 20:17

answered Jan 28 at 18:22

DopeGhoti

45.6k55988

It says "Duplicate file xxx with hash yyy". That's right. But duplicate to what file? But nicely scripted!

– Freddy
Jan 28 at 18:55

Tweaked to store the first filename found in the array for reference in the output.

– DopeGhoti
Jan 28 at 20:17

add a comment |

2 Answers
2

active

oldest

votes

2 Answers
2

active

oldest

votes

for x in *; do for y in *; do [ "$x" = "$y" ] && continue; cmp -s "$x" "$y" && echo Same: "$x" and "$y"; done; done|head -1

Or, broken up a bit for readability:

for x in *
do 
 for y in *
 do 
 [ "$x" = "$y" ] && continue
 cmp -s "$x" "$y" && echo Same: "$x" and "$y"
 done
done | head -1

The head is just to keep the mirror reporting down ("a = b" and "b = a").

answered Jan 28 at 18:11

Jeff Schaller

41.5k1056132

thank you, that was very helpful

– jospin deek
Jan 28 at 18:21

add a comment |

for x in *; do for y in *; do [ "$x" = "$y" ] && continue; cmp -s "$x" "$y" && echo Same: "$x" and "$y"; done; done|head -1

Or, broken up a bit for readability:

for x in *
do 
 for y in *
 do 
 [ "$x" = "$y" ] && continue
 cmp -s "$x" "$y" && echo Same: "$x" and "$y"
 done
done | head -1

The head is just to keep the mirror reporting down ("a = b" and "b = a").

answered Jan 28 at 18:11

Jeff Schaller

41.5k1056132

thank you, that was very helpful

– jospin deek
Jan 28 at 18:21

add a comment |

for x in *; do for y in *; do [ "$x" = "$y" ] && continue; cmp -s "$x" "$y" && echo Same: "$x" and "$y"; done; done|head -1

Or, broken up a bit for readability:

for x in *
do 
 for y in *
 do 
 [ "$x" = "$y" ] && continue
 cmp -s "$x" "$y" && echo Same: "$x" and "$y"
 done
done | head -1

The head is just to keep the mirror reporting down ("a = b" and "b = a").

answered Jan 28 at 18:11

Jeff Schaller

41.5k1056132

for x in *; do for y in *; do [ "$x" = "$y" ] && continue; cmp -s "$x" "$y" && echo Same: "$x" and "$y"; done; done|head -1

Or, broken up a bit for readability:

for x in *
do 
 for y in *
 do 
 [ "$x" = "$y" ] && continue
 cmp -s "$x" "$y" && echo Same: "$x" and "$y"
 done
done | head -1

The head is just to keep the mirror reporting down ("a = b" and "b = a").

answered Jan 28 at 18:11

Jeff Schaller

41.5k1056132

answered Jan 28 at 18:11

Jeff Schaller

41.5k1056132

answered Jan 28 at 18:11

Jeff Schaller

41.5k1056132

answered Jan 28 at 18:11

Jeff Schaller

41.5k1056132

thank you, that was very helpful

– jospin deek
Jan 28 at 18:21

add a comment |

thank you, that was very helpful

– jospin deek
Jan 28 at 18:21

thank you, that was very helpful

– jospin deek
Jan 28 at 18:21

add a comment |

find . -type f -exec md5sum "" ; | awk 'seen[$1] print "Duplicate file "$2" with hash "$1" at "seen[$1]" ! seen[$1] seen[$1]=$2'

edited Jan 28 at 20:17

answered Jan 28 at 18:22

DopeGhoti

45.6k55988

It says "Duplicate file xxx with hash yyy". That's right. But duplicate to what file? But nicely scripted!

– Freddy
Jan 28 at 18:55

Tweaked to store the first filename found in the array for reference in the output.

– DopeGhoti
Jan 28 at 20:17

add a comment |

find . -type f -exec md5sum "" ; | awk 'seen[$1] print "Duplicate file "$2" with hash "$1" at "seen[$1]" ! seen[$1] seen[$1]=$2'

edited Jan 28 at 20:17

answered Jan 28 at 18:22

DopeGhoti

45.6k55988

It says "Duplicate file xxx with hash yyy". That's right. But duplicate to what file? But nicely scripted!

– Freddy
Jan 28 at 18:55

Tweaked to store the first filename found in the array for reference in the output.

– DopeGhoti
Jan 28 at 20:17

add a comment |

find . -type f -exec md5sum "" ; | awk 'seen[$1] print "Duplicate file "$2" with hash "$1" at "seen[$1]" ! seen[$1] seen[$1]=$2'

edited Jan 28 at 20:17

answered Jan 28 at 18:22

DopeGhoti

45.6k55988

find . -type f -exec md5sum "" ; | awk 'seen[$1] print "Duplicate file "$2" with hash "$1" at "seen[$1]" ! seen[$1] seen[$1]=$2'

edited Jan 28 at 20:17

answered Jan 28 at 18:22

DopeGhoti

45.6k55988

edited Jan 28 at 20:17

answered Jan 28 at 18:22

DopeGhoti

45.6k55988

answered Jan 28 at 18:22

DopeGhoti

45.6k55988

answered Jan 28 at 18:22

DopeGhoti

45.6k55988

It says "Duplicate file xxx with hash yyy". That's right. But duplicate to what file? But nicely scripted!

– Freddy
Jan 28 at 18:55

Tweaked to store the first filename found in the array for reference in the output.

– DopeGhoti
Jan 28 at 20:17

add a comment |

It says "Duplicate file xxx with hash yyy". That's right. But duplicate to what file? But nicely scripted!

– Freddy
Jan 28 at 18:55

Tweaked to store the first filename found in the array for reference in the output.

– DopeGhoti
Jan 28 at 20:17

It says "Duplicate file xxx with hash yyy". That's right. But duplicate to what file? But nicely scripted!

– Freddy
Jan 28 at 18:55

Tweaked to store the first filename found in the array for reference in the output.

– DopeGhoti
Jan 28 at 20:17

add a comment |

搜尋此網誌

mjhjmtu