awk delete duplicate [duplicate]

This question already has an answer here:

How to use awk to print nth column and remove duplicates?

1 answer

Remove lines based on duplicates within one column without sort

2 answers

I want to determine if a record contains a duplicate value for a specific field and then delete the duplicate record and save the new file.

abc|123|def|456
abc|456|ghi|789
def|123|def|456

I want to save a new file with any record that duplicates field 1 removed.

abc|123|def|456
def|123|def|456

This awk code comes close, but actually does the opposite. It creates a new duplicate row and then saves it to the new file.

awk -F'|' 'myv=a[$1] !/^myv++/' file.txt > newFile.txt

edited Dec 11 at 20:06

asked Dec 11 at 20:00

user3439308

marked as duplicate by Stephen Kitt, Sparhawk, roaima, RalfFriedl, jimmij Dec 12 at 2:08

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

1

awk -F'|' '!a[$1]++' file.txt > newFile.txt
– steeldriver
Dec 11 at 20:04

oops, this is the code that comes close -- awk -F'|' 'myv=a[$1]++ !/^myv/' file.txt > newFile.txt ------ with the ++ to catch duplicates. but I do not want to then add the duplicate lines, i want to remove them.
– user3439308
Dec 11 at 20:04

SOLVED - thank you steeldriver -- why did you submit that as a comment when it is the answer? I want to give you credit.
– user3439308
Dec 11 at 20:07

Thanks - I posted it as a comment because I'm convinced it must be a duplicate of a previous question (although I can't quite find it)
– steeldriver
Dec 11 at 20:36

1

@Sparhawk I reckoned the important part was the array incrementing technique; but your suggestion is better, thanks!
– Stephen Kitt
Dec 11 at 21:11

|
show 1 more comment

This question already has an answer here:

How to use awk to print nth column and remove duplicates?

1 answer

Remove lines based on duplicates within one column without sort

2 answers

I want to determine if a record contains a duplicate value for a specific field and then delete the duplicate record and save the new file.

abc|123|def|456
abc|456|ghi|789
def|123|def|456

I want to save a new file with any record that duplicates field 1 removed.

abc|123|def|456
def|123|def|456

This awk code comes close, but actually does the opposite. It creates a new duplicate row and then saves it to the new file.

awk -F'|' 'myv=a[$1] !/^myv++/' file.txt > newFile.txt

edited Dec 11 at 20:06

asked Dec 11 at 20:00

user3439308

marked as duplicate by Stephen Kitt, Sparhawk, roaima, RalfFriedl, jimmij Dec 12 at 2:08

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

1

awk -F'|' '!a[$1]++' file.txt > newFile.txt
– steeldriver
Dec 11 at 20:04

oops, this is the code that comes close -- awk -F'|' 'myv=a[$1]++ !/^myv/' file.txt > newFile.txt ------ with the ++ to catch duplicates. but I do not want to then add the duplicate lines, i want to remove them.
– user3439308
Dec 11 at 20:04

SOLVED - thank you steeldriver -- why did you submit that as a comment when it is the answer? I want to give you credit.
– user3439308
Dec 11 at 20:07

Thanks - I posted it as a comment because I'm convinced it must be a duplicate of a previous question (although I can't quite find it)
– steeldriver
Dec 11 at 20:36

1

@Sparhawk I reckoned the important part was the array incrementing technique; but your suggestion is better, thanks!
– Stephen Kitt
Dec 11 at 21:11

|
show 1 more comment

This question already has an answer here:

How to use awk to print nth column and remove duplicates?

1 answer

Remove lines based on duplicates within one column without sort

2 answers

I want to determine if a record contains a duplicate value for a specific field and then delete the duplicate record and save the new file.

abc|123|def|456
abc|456|ghi|789
def|123|def|456

I want to save a new file with any record that duplicates field 1 removed.

abc|123|def|456
def|123|def|456

This awk code comes close, but actually does the opposite. It creates a new duplicate row and then saves it to the new file.

awk -F'|' 'myv=a[$1] !/^myv++/' file.txt > newFile.txt

edited Dec 11 at 20:06

asked Dec 11 at 20:00

user3439308

This question already has an answer here:

How to use awk to print nth column and remove duplicates?

1 answer

Remove lines based on duplicates within one column without sort

2 answers

I want to determine if a record contains a duplicate value for a specific field and then delete the duplicate record and save the new file.

abc|123|def|456
abc|456|ghi|789
def|123|def|456

I want to save a new file with any record that duplicates field 1 removed.

abc|123|def|456
def|123|def|456

This awk code comes close, but actually does the opposite. It creates a new duplicate row and then saves it to the new file.

awk -F'|' 'myv=a[$1] !/^myv++/' file.txt > newFile.txt

This question already has an answer here:

How to use awk to print nth column and remove duplicates?

1 answer

Remove lines based on duplicates within one column without sort

2 answers

awk

edited Dec 11 at 20:06

asked Dec 11 at 20:00

user3439308

edited Dec 11 at 20:06

asked Dec 11 at 20:00

user3439308

edited Dec 11 at 20:06

asked Dec 11 at 20:00

user3439308

asked Dec 11 at 20:00

user3439308

asked Dec 11 at 20:00

user3439308

marked as duplicate by Stephen Kitt, Sparhawk, roaima, RalfFriedl, jimmij Dec 12 at 2:08

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

marked as duplicate by Stephen Kitt, Sparhawk, roaima, RalfFriedl, jimmij Dec 12 at 2:08

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

1

awk -F'|' '!a[$1]++' file.txt > newFile.txt
– steeldriver
Dec 11 at 20:04

oops, this is the code that comes close -- awk -F'|' 'myv=a[$1]++ !/^myv/' file.txt > newFile.txt ------ with the ++ to catch duplicates. but I do not want to then add the duplicate lines, i want to remove them.
– user3439308
Dec 11 at 20:04

SOLVED - thank you steeldriver -- why did you submit that as a comment when it is the answer? I want to give you credit.
– user3439308
Dec 11 at 20:07

Thanks - I posted it as a comment because I'm convinced it must be a duplicate of a previous question (although I can't quite find it)
– steeldriver
Dec 11 at 20:36

1

@Sparhawk I reckoned the important part was the array incrementing technique; but your suggestion is better, thanks!
– Stephen Kitt
Dec 11 at 21:11

|
show 1 more comment

1

awk -F'|' '!a[$1]++' file.txt > newFile.txt
– steeldriver
Dec 11 at 20:04

oops, this is the code that comes close -- awk -F'|' 'myv=a[$1]++ !/^myv/' file.txt > newFile.txt ------ with the ++ to catch duplicates. but I do not want to then add the duplicate lines, i want to remove them.
– user3439308
Dec 11 at 20:04

SOLVED - thank you steeldriver -- why did you submit that as a comment when it is the answer? I want to give you credit.
– user3439308
Dec 11 at 20:07

Thanks - I posted it as a comment because I'm convinced it must be a duplicate of a previous question (although I can't quite find it)
– steeldriver
Dec 11 at 20:36

1

@Sparhawk I reckoned the important part was the array incrementing technique; but your suggestion is better, thanks!
– Stephen Kitt
Dec 11 at 21:11

awk -F'|' '!a[$1]++' file.txt > newFile.txt
– steeldriver
Dec 11 at 20:04

oops, this is the code that comes close -- awk -F'|' 'myv=a[$1]++ !/^myv/' file.txt > newFile.txt ------ with the ++ to catch duplicates. but I do not want to then add the duplicate lines, i want to remove them.
– user3439308
Dec 11 at 20:04

SOLVED - thank you steeldriver -- why did you submit that as a comment when it is the answer? I want to give you credit.
– user3439308
Dec 11 at 20:07

Thanks - I posted it as a comment because I'm convinced it must be a duplicate of a previous question (although I can't quite find it)
– steeldriver
Dec 11 at 20:36

@Sparhawk I reckoned the important part was the array incrementing technique; but your suggestion is better, thanks!
– Stephen Kitt
Dec 11 at 21:11

|
show 1 more comment

active

oldest

votes

搜尋此網誌

mjhjmtu