awk delete duplicate [duplicate]
Clash Royale CLAN TAG#URR8PPP
This question already has an answer here:
How to use awk to print nth column and remove duplicates?
1 answer
Remove lines based on duplicates within one column without sort
2 answers
I want to determine if a record contains a duplicate value for a specific field and then delete the duplicate record and save the new file.
abc|123|def|456
abc|456|ghi|789
def|123|def|456
I want to save a new file with any record that duplicates field 1 removed.
abc|123|def|456
def|123|def|456
This awk code comes close, but actually does the opposite. It creates a new duplicate row and then saves it to the new file.
awk -F'|' 'myv=a[$1] !/^myv++/' file.txt > newFile.txt
awk
asked Dec 11 at 20:00
user3439308
62
marked as duplicate by Stephen Kitt, Sparhawk, roaima, RalfFriedl, jimmij Dec 12 at 2:08
This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.
|
show 1 more comment
This question already has an answer here:
How to use awk to print nth column and remove duplicates?
1 answer
Remove lines based on duplicates within one column without sort
2 answers
I want to determine if a record contains a duplicate value for a specific field and then delete the duplicate record and save the new file.
abc|123|def|456
abc|456|ghi|789
def|123|def|456
I want to save a new file with any record that duplicates field 1 removed.
abc|123|def|456
def|123|def|456
This awk code comes close, but actually does the opposite. It creates a new duplicate row and then saves it to the new file.
awk -F'|' 'myv=a[$1] !/^myv++/' file.txt > newFile.txt
awk
asked Dec 11 at 20:00
user3439308
62
marked as duplicate by Stephen Kitt, Sparhawk, roaima, RalfFriedl, jimmij Dec 12 at 2:08
This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.
1
awk -F'|' '!a[$1]++' file.txt > newFile.txt
– steeldriver
Dec 11 at 20:04
oops, this is the code that comes close -- awk -F'|' 'myv=a[$1]++ !/^myv/' file.txt > newFile.txt ------ with the ++ to catch duplicates. but I do not want to then add the duplicate lines, i want to remove them.
– user3439308
Dec 11 at 20:04
SOLVED - thank you steeldriver -- why did you submit that as a comment when it is the answer? I want to give you credit.
– user3439308
Dec 11 at 20:07
Thanks - I posted it as a comment because I'm convinced it must be a duplicate of a previous question (although I can't quite find it)
– steeldriver
Dec 11 at 20:36
1
@Sparhawk I reckoned the important part was the array incrementing technique; but your suggestion is better, thanks!
– Stephen Kitt
Dec 11 at 21:11
|
show 1 more comment
This question already has an answer here:
How to use awk to print nth column and remove duplicates?
1 answer
Remove lines based on duplicates within one column without sort
2 answers
I want to determine if a record contains a duplicate value for a specific field and then delete the duplicate record and save the new file.
abc|123|def|456
abc|456|ghi|789
def|123|def|456
I want to save a new file with any record that duplicates field 1 removed.
abc|123|def|456
def|123|def|456
This awk code comes close, but actually does the opposite. It creates a new duplicate row and then saves it to the new file.
awk -F'|' 'myv=a[$1] !/^myv++/' file.txt > newFile.txt
awk
asked Dec 11 at 20:00
user3439308
62
This question already has an answer here:
How to use awk to print nth column and remove duplicates?
1 answer
Remove lines based on duplicates within one column without sort
2 answers
I want to determine if a record contains a duplicate value for a specific field and then delete the duplicate record and save the new file.
abc|123|def|456
abc|456|ghi|789
def|123|def|456
I want to save a new file with any record that duplicates field 1 removed.
abc|123|def|456
def|123|def|456
This awk code comes close, but actually does the opposite. It creates a new duplicate row and then saves it to the new file.
awk -F'|' 'myv=a[$1] !/^myv++/' file.txt > newFile.txt
This question already has an answer here:
How to use awk to print nth column and remove duplicates?
1 answer
Remove lines based on duplicates within one column without sort
2 answers
awk
awk
asked Dec 11 at 20:00
user3439308
62
asked Dec 11 at 20:00
user3439308
62
edited Dec 11 at 20:06
asked Dec 11 at 20:00
user3439308
62
asked Dec 11 at 20:00
user3439308
62
asked Dec 11 at 20:00
user3439308
62
62
marked as duplicate by Stephen Kitt, Sparhawk, roaima, RalfFriedl, jimmij Dec 12 at 2:08
This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.
marked as duplicate by Stephen Kitt, Sparhawk, roaima, RalfFriedl, jimmij Dec 12 at 2:08
This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.
1
awk -F'|' '!a[$1]++' file.txt > newFile.txt
– steeldriver
Dec 11 at 20:04
oops, this is the code that comes close -- awk -F'|' 'myv=a[$1]++ !/^myv/' file.txt > newFile.txt ------ with the ++ to catch duplicates. but I do not want to then add the duplicate lines, i want to remove them.
– user3439308
Dec 11 at 20:04
SOLVED - thank you steeldriver -- why did you submit that as a comment when it is the answer? I want to give you credit.
– user3439308
Dec 11 at 20:07
Thanks - I posted it as a comment because I'm convinced it must be a duplicate of a previous question (although I can't quite find it)
– steeldriver
Dec 11 at 20:36
1
@Sparhawk I reckoned the important part was the array incrementing technique; but your suggestion is better, thanks!
– Stephen Kitt
Dec 11 at 21:11
|
show 1 more comment
1
awk -F'|' '!a[$1]++' file.txt > newFile.txt
– steeldriver
Dec 11 at 20:04
oops, this is the code that comes close -- awk -F'|' 'myv=a[$1]++ !/^myv/' file.txt > newFile.txt ------ with the ++ to catch duplicates. but I do not want to then add the duplicate lines, i want to remove them.
– user3439308
Dec 11 at 20:04
SOLVED - thank you steeldriver -- why did you submit that as a comment when it is the answer? I want to give you credit.
– user3439308
Dec 11 at 20:07
Thanks - I posted it as a comment because I'm convinced it must be a duplicate of a previous question (although I can't quite find it)
– steeldriver
Dec 11 at 20:36
1
@Sparhawk I reckoned the important part was the array incrementing technique; but your suggestion is better, thanks!
– Stephen Kitt
Dec 11 at 21:11
1
1
awk -F'|' '!a[$1]++' file.txt > newFile.txt– steeldriver
Dec 11 at 20:04
awk -F'|' '!a[$1]++' file.txt > newFile.txt– steeldriver
Dec 11 at 20:04
oops, this is the code that comes close -- awk -F'|' 'myv=a[$1]++ !/^myv/' file.txt > newFile.txt ------ with the ++ to catch duplicates. but I do not want to then add the duplicate lines, i want to remove them.
– user3439308
Dec 11 at 20:04
oops, this is the code that comes close -- awk -F'|' 'myv=a[$1]++ !/^myv/' file.txt > newFile.txt ------ with the ++ to catch duplicates. but I do not want to then add the duplicate lines, i want to remove them.
– user3439308
Dec 11 at 20:04
SOLVED - thank you steeldriver -- why did you submit that as a comment when it is the answer? I want to give you credit.
– user3439308
Dec 11 at 20:07
SOLVED - thank you steeldriver -- why did you submit that as a comment when it is the answer? I want to give you credit.
– user3439308
Dec 11 at 20:07
Thanks - I posted it as a comment because I'm convinced it must be a duplicate of a previous question (although I can't quite find it)
– steeldriver
Dec 11 at 20:36
Thanks - I posted it as a comment because I'm convinced it must be a duplicate of a previous question (although I can't quite find it)
– steeldriver
Dec 11 at 20:36
1
1
@Sparhawk I reckoned the important part was the array incrementing technique; but your suggestion is better, thanks!
– Stephen Kitt
Dec 11 at 21:11
@Sparhawk I reckoned the important part was the array incrementing technique; but your suggestion is better, thanks!
– Stephen Kitt
Dec 11 at 21:11
|
show 1 more comment
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
1
awk -F'|' '!a[$1]++' file.txt > newFile.txt– steeldriver
Dec 11 at 20:04
oops, this is the code that comes close -- awk -F'|' 'myv=a[$1]++ !/^myv/' file.txt > newFile.txt ------ with the ++ to catch duplicates. but I do not want to then add the duplicate lines, i want to remove them.
– user3439308
Dec 11 at 20:04
SOLVED - thank you steeldriver -- why did you submit that as a comment when it is the answer? I want to give you credit.
– user3439308
Dec 11 at 20:07
Thanks - I posted it as a comment because I'm convinced it must be a duplicate of a previous question (although I can't quite find it)
– steeldriver
Dec 11 at 20:36
1
@Sparhawk I reckoned the important part was the array incrementing technique; but your suggestion is better, thanks!
– Stephen Kitt
Dec 11 at 21:11