Remove row of first duplicated column 1 from csv

The name of the pictureThe name of the pictureThe name of the pictureClash Royale CLAN TAG#URR8PPP











up vote
0
down vote

favorite












I need some help removing first repeated line (for field 1) from a csv file.



I have a large csv file with a structure similar to this:



334050049049426,2018-11-06T20:21:56.591Z,xxx,gdl-qns28-1540279057144
334050049049426,2018-11-06T21:32:47.431Z,xxx,gdl-qns19-1540278993723
334090015032064,2018-11-06T22:22:31.247Z,xxx,gdl-qns15-1540279009813
334090015032064,2018-11-07T01:44:11.442Z,xxx,gdl-qns25-1540279437614
334090015032064,2018-11-07T03:57:18.911Z,xxx,gdl-qns28-1540279710160
334050069888299,2018-11-07T03:32:12.899Z,xxx,gdl-qns29-1540279367769
334050069888299,2018-11-07T03:58:15.475Z,xxx,mgc-qns20-1540281468455


I need to be able to remove the first line found of duplicated values from the first column so for example line # 1,3 and 6 needs to be remove.



any help is greatly appreciated.






shell-script shell







share|improve this question









New contributor




Ian is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.



















  • Better use a tool designed to process csv, not the shell directly.
    – Isaac
    2 mins ago














up vote
0
down vote

favorite












I need some help removing first repeated line (for field 1) from a csv file.



I have a large csv file with a structure similar to this:



334050049049426,2018-11-06T20:21:56.591Z,xxx,gdl-qns28-1540279057144
334050049049426,2018-11-06T21:32:47.431Z,xxx,gdl-qns19-1540278993723
334090015032064,2018-11-06T22:22:31.247Z,xxx,gdl-qns15-1540279009813
334090015032064,2018-11-07T01:44:11.442Z,xxx,gdl-qns25-1540279437614
334090015032064,2018-11-07T03:57:18.911Z,xxx,gdl-qns28-1540279710160
334050069888299,2018-11-07T03:32:12.899Z,xxx,gdl-qns29-1540279367769
334050069888299,2018-11-07T03:58:15.475Z,xxx,mgc-qns20-1540281468455


I need to be able to remove the first line found of duplicated values from the first column so for example line # 1,3 and 6 needs to be remove.



any help is greatly appreciated.






shell-script shell







share|improve this question









New contributor




Ian is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.



















  • Better use a tool designed to process csv, not the shell directly.
    – Isaac
    2 mins ago












up vote
0
down vote

favorite









up vote
0
down vote

favorite











I need some help removing first repeated line (for field 1) from a csv file.



I have a large csv file with a structure similar to this:



334050049049426,2018-11-06T20:21:56.591Z,xxx,gdl-qns28-1540279057144
334050049049426,2018-11-06T21:32:47.431Z,xxx,gdl-qns19-1540278993723
334090015032064,2018-11-06T22:22:31.247Z,xxx,gdl-qns15-1540279009813
334090015032064,2018-11-07T01:44:11.442Z,xxx,gdl-qns25-1540279437614
334090015032064,2018-11-07T03:57:18.911Z,xxx,gdl-qns28-1540279710160
334050069888299,2018-11-07T03:32:12.899Z,xxx,gdl-qns29-1540279367769
334050069888299,2018-11-07T03:58:15.475Z,xxx,mgc-qns20-1540281468455


I need to be able to remove the first line found of duplicated values from the first column so for example line # 1,3 and 6 needs to be remove.



any help is greatly appreciated.






shell-script shell







share|improve this question









New contributor




Ian is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.











I need some help removing first repeated line (for field 1) from a csv file.



I have a large csv file with a structure similar to this:



334050049049426,2018-11-06T20:21:56.591Z,xxx,gdl-qns28-1540279057144
334050049049426,2018-11-06T21:32:47.431Z,xxx,gdl-qns19-1540278993723
334090015032064,2018-11-06T22:22:31.247Z,xxx,gdl-qns15-1540279009813
334090015032064,2018-11-07T01:44:11.442Z,xxx,gdl-qns25-1540279437614
334090015032064,2018-11-07T03:57:18.911Z,xxx,gdl-qns28-1540279710160
334050069888299,2018-11-07T03:32:12.899Z,xxx,gdl-qns29-1540279367769
334050069888299,2018-11-07T03:58:15.475Z,xxx,mgc-qns20-1540281468455


I need to be able to remove the first line found of duplicated values from the first column so for example line # 1,3 and 6 needs to be remove.



any help is greatly appreciated.






shell-script shell




shell-script shell






share|improve this question









New contributor




Ian is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.











share|improve this question









New contributor




Ian is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.









share|improve this question




share|improve this question








edited 4 mins ago









Isaac

8,92211342




8,92211342






New contributor




Ian is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.









asked 25 mins ago









Ian

1




1




New contributor




Ian is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.





New contributor





Ian is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.






Ian is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.











  • Better use a tool designed to process csv, not the shell directly.
    – Isaac
    2 mins ago
















  • Better use a tool designed to process csv, not the shell directly.
    – Isaac
    2 mins ago















Better use a tool designed to process csv, not the shell directly.
– Isaac
2 mins ago




Better use a tool designed to process csv, not the shell directly.
– Isaac
2 mins ago










1 Answer
1






active

oldest

votes

















up vote
0
down vote













Try the below command,



awk -F',' '!seen[$1]++' file.csv





share|improve this answer




















  • This does the opposite of what was asked. That is: it will print the first line and remove all following repeats.
    – Isaac
    2 mins ago










Your Answer








StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "106"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);






Ian is a new contributor. Be nice, and check out our Code of Conduct.









 

draft saved


draft discarded


















StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f479304%2fremove-row-of-first-duplicated-column-1-from-csv%23new-answer', 'question_page');

);

Post as a guest

















1 Answer
1






active

oldest

votes








1 Answer
1






active

oldest

votes









active

oldest

votes






active

oldest

votes








up vote
0
down vote













Try the below command,



awk -F',' '!seen[$1]++' file.csv





share|improve this answer




















  • This does the opposite of what was asked. That is: it will print the first line and remove all following repeats.
    – Isaac
    2 mins ago














up vote
0
down vote













Try the below command,



awk -F',' '!seen[$1]++' file.csv





share|improve this answer




















  • This does the opposite of what was asked. That is: it will print the first line and remove all following repeats.
    – Isaac
    2 mins ago












up vote
0
down vote










up vote
0
down vote









Try the below command,



awk -F',' '!seen[$1]++' file.csv





share|improve this answer












Try the below command,



awk -F',' '!seen[$1]++' file.csv






share|improve this answer












share|improve this answer



share|improve this answer










answered 19 mins ago









EBIN GLADSON

4156




4156











  • This does the opposite of what was asked. That is: it will print the first line and remove all following repeats.
    – Isaac
    2 mins ago
















  • This does the opposite of what was asked. That is: it will print the first line and remove all following repeats.
    – Isaac
    2 mins ago















This does the opposite of what was asked. That is: it will print the first line and remove all following repeats.
– Isaac
2 mins ago




This does the opposite of what was asked. That is: it will print the first line and remove all following repeats.
– Isaac
2 mins ago










Ian is a new contributor. Be nice, and check out our Code of Conduct.









 

draft saved


draft discarded


















Ian is a new contributor. Be nice, and check out our Code of Conduct.












Ian is a new contributor. Be nice, and check out our Code of Conduct.











Ian is a new contributor. Be nice, and check out our Code of Conduct.













 


draft saved


draft discarded














StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f479304%2fremove-row-of-first-duplicated-column-1-from-csv%23new-answer', 'question_page');

);

Post as a guest
































































-

Popular posts from this blog

How to check contact read email or not when send email to Individual?

Image
Clash Royale CLAN TAG #URR8PPP up vote 1 down vote favorite I'm using WordPress 4.9.8, CiviCRM to 5.5.1, I usually send email to contact by Search> Find contacts View contact details Action> Send email Send email ok, Contact received mail ok like picture But status only Email sent though contact read email or not. So, can CiviCRM can change status to Email read when contact read email? wordpress email share | improve this question asked Sep 26 at 0:12 ToanLuong 49 9 add a comment  |  up vote 1 down vote favorite I'm using WordPress 4.9.8, CiviCRM to 5.5.1, I usually send email to contact by Search> Find contacts View contact details Action> Send email Send email ok, Contact received mail ok like picture But status only Email sent though contact read email or not. So, can CiviCRM can change status to Email read when contact read email? wordpress email share | improve this question asked Se
Read more

Bahrain

Image
For other uses, see Bahrain (disambiguation). Sovereign island state in the Persian Gulf Kingdom of Bahrain مملكة البحرين   (Arabic) Mamlakat al-Baḥrayn Flag Coat of arms Anthem:  نشيد البحرين الوطني Bahrainona Our Bahrain Location of   Bahrain    (circled in red) Capital and largest city Manama 26°13′N 50°35′E  /  26.217°N 50.583°E  / 26.217; 50.583 Official languages Arabic Ethnic groups (2010 [1] ) 50.7% Arab (46% Bahrani, 4.7% other Arabs) 45.5% Asian 1% European 1.2% Others Religion Islam Demonym(s) Bahraini Government Unitary constitutional monarchy • King Hamad bin Isa Al Khalifa • Crown Prince Salman bin Hamad bin Isa Al Khalifa • Prime Minister Khalifa bin Salman Al Khalifa Legislature National Assembly • Upper house Consultative Council • Lower house Council of Representatives Independence • Declared Independence [2] 14 August 1971 • from United Kingdom [1] 15 August 1971 • Admitted to the United Nations 21 September 1971 • Kingdom of Bahrain 14 February 2002
Read more

Postfix configuration issue with fips on centos 7; mailgun relay

Image
Clash Royale CLAN TAG #URR8PPP up vote 0 down vote favorite I am trying to setup postfix to relay all mail generated on the local machine via SMTP to a mailgun relay. I have used the mailgun relay before with success on an ubuntu server, but I am migrating to a Centos 7 server which I will be running in FIPS mode. There error log is below, slightly sanitized. I have a small enough network that I choose to have each machine reach out to mailgun individually (this the loopback-only, 127.0.0.0/8 restrictions) and no firewall open port allowing smtp in to the machine. I assume the FIPS mode (and with it disabling of MD5) is causing problems, but I don't know how to overcome it or if it is even possible for tls_fprint to use some supported hash such as sha256 or sha512. However, the relay=none is slightly concerning since I have relayhost set, but perhaps that is because the smtp process is failing? Any help would be appreciated! postconf -n: alias_database = hash:/etc/alia
Read more