Command to delete duplicate files from current directory [duplicate]
Clash Royale CLAN TAG#URR8PPP
up vote
-3
down vote
favorite
This question already has an answer here:
How to find and delete duplicate files within the same directory?
7 answers
I have a directory with several duplicate files, created by a program. The duplicates have the same name (except for a number), but not all files with the same name are duplicates.
What's a simple command to delete the duplicates (ideally a single line limited to GNU coreutils, unlike the question about scripts)?
Example filename:
parra1998.pdf
parra1998(1).pdf
parra1998(2).pdf
command-line files rm deduplication
asked Apr 29 at 10:18
Nemo
15516
marked as duplicate by αғsýιη, jimmij, Jeff Schaller, meuh, GAD3R Apr 29 at 16:14
This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.
 |Â
show 5 more comments
up vote
-3
down vote
favorite
This question already has an answer here:
How to find and delete duplicate files within the same directory?
7 answers
I have a directory with several duplicate files, created by a program. The duplicates have the same name (except for a number), but not all files with the same name are duplicates.
What's a simple command to delete the duplicates (ideally a single line limited to GNU coreutils, unlike the question about scripts)?
Example filename:
parra1998.pdf
parra1998(1).pdf
parra1998(2).pdf
command-line files rm deduplication
asked Apr 29 at 10:18
Nemo
15516
marked as duplicate by αғsýιη, jimmij, Jeff Schaller, meuh, GAD3R Apr 29 at 16:14
This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.
2
"but not all files with the same name are duplicates", you cannot have two files with same name. Then how you want to detect fileparra1998.pdfis not duplicated of fileparra1998(1).pdfor it is? Based on their contents? if yes, then your question is duplicated of How to find and delete duplicate files within the same directory?
– Î±Ò“sýιη
Apr 29 at 10:32
@αғsýιη "same name (except for a number)"
– Nemo
Apr 29 at 11:56
@dsstorefile1 No, this question asks for a simple command while that question is more generic (answers include entire bash scripts, GUI programs etc.)
– Nemo
Apr 29 at 11:57
1
@Nemo The answers in that question solve the same problem in your question, which is why yours is a duplicate.
– dsstorefile1
Apr 29 at 12:07
1
Indeed, I can't parseThe duplicates have the same name (except for a number), but not all files with the same name are duplicates-- how do we know if a numbered suffix file is a duplicate of the base name?
– Jeff Schaller
Apr 29 at 13:21
 |Â
show 5 more comments
up vote
-3
down vote
favorite
up vote
-3
down vote
favorite
This question already has an answer here:
How to find and delete duplicate files within the same directory?
7 answers
I have a directory with several duplicate files, created by a program. The duplicates have the same name (except for a number), but not all files with the same name are duplicates.
What's a simple command to delete the duplicates (ideally a single line limited to GNU coreutils, unlike the question about scripts)?
Example filename:
parra1998.pdf
parra1998(1).pdf
parra1998(2).pdf
command-line files rm deduplication
asked Apr 29 at 10:18
Nemo
15516
This question already has an answer here:
How to find and delete duplicate files within the same directory?
7 answers
I have a directory with several duplicate files, created by a program. The duplicates have the same name (except for a number), but not all files with the same name are duplicates.
What's a simple command to delete the duplicates (ideally a single line limited to GNU coreutils, unlike the question about scripts)?
Example filename:
parra1998.pdf
parra1998(1).pdf
parra1998(2).pdf
This question already has an answer here:
How to find and delete duplicate files within the same directory?
7 answers
command-line files rm deduplication
asked Apr 29 at 10:18
Nemo
15516
edited Apr 29 at 12:07
asked Apr 29 at 10:18
Nemo
15516
asked Apr 29 at 10:18
Nemo
15516
asked Apr 29 at 10:18
Nemo
15516
15516
marked as duplicate by αғsýιη, jimmij, Jeff Schaller, meuh, GAD3R Apr 29 at 16:14
This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.
marked as duplicate by αғsýιη, jimmij, Jeff Schaller, meuh, GAD3R Apr 29 at 16:14
This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.
2
"but not all files with the same name are duplicates", you cannot have two files with same name. Then how you want to detect fileparra1998.pdfis not duplicated of fileparra1998(1).pdfor it is? Based on their contents? if yes, then your question is duplicated of How to find and delete duplicate files within the same directory?
– Î±Ò“sýιη
Apr 29 at 10:32
@αғsýιη "same name (except for a number)"
– Nemo
Apr 29 at 11:56
@dsstorefile1 No, this question asks for a simple command while that question is more generic (answers include entire bash scripts, GUI programs etc.)
– Nemo
Apr 29 at 11:57
1
@Nemo The answers in that question solve the same problem in your question, which is why yours is a duplicate.
– dsstorefile1
Apr 29 at 12:07
1
Indeed, I can't parseThe duplicates have the same name (except for a number), but not all files with the same name are duplicates-- how do we know if a numbered suffix file is a duplicate of the base name?
– Jeff Schaller
Apr 29 at 13:21
 |Â
show 5 more comments
2
"but not all files with the same name are duplicates", you cannot have two files with same name. Then how you want to detect fileparra1998.pdfis not duplicated of fileparra1998(1).pdfor it is? Based on their contents? if yes, then your question is duplicated of How to find and delete duplicate files within the same directory?
– Î±Ò“sýιη
Apr 29 at 10:32
@αғsýιη "same name (except for a number)"
– Nemo
Apr 29 at 11:56
@dsstorefile1 No, this question asks for a simple command while that question is more generic (answers include entire bash scripts, GUI programs etc.)
– Nemo
Apr 29 at 11:57
1
@Nemo The answers in that question solve the same problem in your question, which is why yours is a duplicate.
– dsstorefile1
Apr 29 at 12:07
1
Indeed, I can't parseThe duplicates have the same name (except for a number), but not all files with the same name are duplicates-- how do we know if a numbered suffix file is a duplicate of the base name?
– Jeff Schaller
Apr 29 at 13:21
2
2
"
but not all files with the same name are duplicates", you cannot have two files with same name. Then how you want to detect file parra1998.pdf is not duplicated of file parra1998(1).pdf or it is? Based on their contents? if yes, then your question is duplicated of How to find and delete duplicate files within the same directory?– Î±Ò“sýιη
Apr 29 at 10:32
"
but not all files with the same name are duplicates", you cannot have two files with same name. Then how you want to detect file parra1998.pdf is not duplicated of file parra1998(1).pdf or it is? Based on their contents? if yes, then your question is duplicated of How to find and delete duplicate files within the same directory?– Î±Ò“sýιη
Apr 29 at 10:32
@αғsýιη "same name (except for a number)"
– Nemo
Apr 29 at 11:56
@αғsýιη "same name (except for a number)"
– Nemo
Apr 29 at 11:56
@dsstorefile1 No, this question asks for a simple command while that question is more generic (answers include entire bash scripts, GUI programs etc.)
– Nemo
Apr 29 at 11:57
@dsstorefile1 No, this question asks for a simple command while that question is more generic (answers include entire bash scripts, GUI programs etc.)
– Nemo
Apr 29 at 11:57
1
1
@Nemo The answers in that question solve the same problem in your question, which is why yours is a duplicate.
– dsstorefile1
Apr 29 at 12:07
@Nemo The answers in that question solve the same problem in your question, which is why yours is a duplicate.
– dsstorefile1
Apr 29 at 12:07
1
1
Indeed, I can't parse
The duplicates have the same name (except for a number), but not all files with the same name are duplicates -- how do we know if a numbered suffix file is a duplicate of the base name?– Jeff Schaller
Apr 29 at 13:21
Indeed, I can't parse
The duplicates have the same name (except for a number), but not all files with the same name are duplicates -- how do we know if a numbered suffix file is a duplicate of the base name?– Jeff Schaller
Apr 29 at 13:21
 |Â
show 5 more comments
1 Answer
1
active
oldest
votes
up vote
-2
down vote
accepted
A quick and dirty solution is to hash the files, then search the hashes which appear more than once and delete those whose filename is numbered.
For instance:
sha1sum * > files.sha1sum
cat files.sha1sum | cut -f1 -d" " | sort | uniq -c | grep -v " 1 " | sed --regexp-extended 's/^[^0-9]+[0-9] //g' | xargs -n1 -I§ grep § files.sha1sum | sed --regexp-extended 's/^[^ ]+ +//g' | grep -v '(' | xargs -n1 -I§ rm "§"
answered Apr 29 at 10:18
Nemo
15516
The line is a bit long and convoluted, but it relies on commands which I use almost daily so that it's easier to remember and adapt. Depending on your habits, using[:blank:]etc. in patterns may be easier.
– Nemo
May 2 at 7:07
add a comment |Â
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
-2
down vote
accepted
A quick and dirty solution is to hash the files, then search the hashes which appear more than once and delete those whose filename is numbered.
For instance:
sha1sum * > files.sha1sum
cat files.sha1sum | cut -f1 -d" " | sort | uniq -c | grep -v " 1 " | sed --regexp-extended 's/^[^0-9]+[0-9] //g' | xargs -n1 -I§ grep § files.sha1sum | sed --regexp-extended 's/^[^ ]+ +//g' | grep -v '(' | xargs -n1 -I§ rm "§"
answered Apr 29 at 10:18
Nemo
15516
The line is a bit long and convoluted, but it relies on commands which I use almost daily so that it's easier to remember and adapt. Depending on your habits, using[:blank:]etc. in patterns may be easier.
– Nemo
May 2 at 7:07
add a comment |Â
up vote
-2
down vote
accepted
A quick and dirty solution is to hash the files, then search the hashes which appear more than once and delete those whose filename is numbered.
For instance:
sha1sum * > files.sha1sum
cat files.sha1sum | cut -f1 -d" " | sort | uniq -c | grep -v " 1 " | sed --regexp-extended 's/^[^0-9]+[0-9] //g' | xargs -n1 -I§ grep § files.sha1sum | sed --regexp-extended 's/^[^ ]+ +//g' | grep -v '(' | xargs -n1 -I§ rm "§"
answered Apr 29 at 10:18
Nemo
15516
The line is a bit long and convoluted, but it relies on commands which I use almost daily so that it's easier to remember and adapt. Depending on your habits, using[:blank:]etc. in patterns may be easier.
– Nemo
May 2 at 7:07
add a comment |Â
up vote
-2
down vote
accepted
up vote
-2
down vote
accepted
A quick and dirty solution is to hash the files, then search the hashes which appear more than once and delete those whose filename is numbered.
For instance:
sha1sum * > files.sha1sum
cat files.sha1sum | cut -f1 -d" " | sort | uniq -c | grep -v " 1 " | sed --regexp-extended 's/^[^0-9]+[0-9] //g' | xargs -n1 -I§ grep § files.sha1sum | sed --regexp-extended 's/^[^ ]+ +//g' | grep -v '(' | xargs -n1 -I§ rm "§"
answered Apr 29 at 10:18
Nemo
15516
A quick and dirty solution is to hash the files, then search the hashes which appear more than once and delete those whose filename is numbered.
For instance:
sha1sum * > files.sha1sum
cat files.sha1sum | cut -f1 -d" " | sort | uniq -c | grep -v " 1 " | sed --regexp-extended 's/^[^0-9]+[0-9] //g' | xargs -n1 -I§ grep § files.sha1sum | sed --regexp-extended 's/^[^ ]+ +//g' | grep -v '(' | xargs -n1 -I§ rm "§"
answered Apr 29 at 10:18
Nemo
15516
edited Apr 29 at 12:01
answered Apr 29 at 10:18
Nemo
15516
answered Apr 29 at 10:18
Nemo
15516
answered Apr 29 at 10:18
Nemo
15516
15516
The line is a bit long and convoluted, but it relies on commands which I use almost daily so that it's easier to remember and adapt. Depending on your habits, using[:blank:]etc. in patterns may be easier.
– Nemo
May 2 at 7:07
add a comment |Â
The line is a bit long and convoluted, but it relies on commands which I use almost daily so that it's easier to remember and adapt. Depending on your habits, using[:blank:]etc. in patterns may be easier.
– Nemo
May 2 at 7:07
The line is a bit long and convoluted, but it relies on commands which I use almost daily so that it's easier to remember and adapt. Depending on your habits, using
[:blank:] etc. in patterns may be easier.– Nemo
May 2 at 7:07
The line is a bit long and convoluted, but it relies on commands which I use almost daily so that it's easier to remember and adapt. Depending on your habits, using
[:blank:] etc. in patterns may be easier.– Nemo
May 2 at 7:07
add a comment |Â
2
"
but not all files with the same name are duplicates", you cannot have two files with same name. Then how you want to detect fileparra1998.pdfis not duplicated of fileparra1998(1).pdfor it is? Based on their contents? if yes, then your question is duplicated of How to find and delete duplicate files within the same directory?– Î±Ò“sýιη
Apr 29 at 10:32
@αғsýιη "same name (except for a number)"
– Nemo
Apr 29 at 11:56
@dsstorefile1 No, this question asks for a simple command while that question is more generic (answers include entire bash scripts, GUI programs etc.)
– Nemo
Apr 29 at 11:57
1
@Nemo The answers in that question solve the same problem in your question, which is why yours is a duplicate.
– dsstorefile1
Apr 29 at 12:07
1
Indeed, I can't parse
The duplicates have the same name (except for a number), but not all files with the same name are duplicates-- how do we know if a numbered suffix file is a duplicate of the base name?– Jeff Schaller
Apr 29 at 13:21