Removing duplicate entries and replacing it with comma | Bash

The name of the pictureThe name of the pictureThe name of the pictureClash Royale CLAN TAG#URR8PPP












0















I have a file which contains ip address and port number in this order:



ipaddress : port



1.1.1.1:21

1.1.1.1:22

2.2.2.2:443

3.3.3.3:80

3.3.3.3:443



I need Result in this below format


ipaddress : port, port



1.1.1.1:21,22

2.2.2.2:443

3.3.3.3:80,443






text-processing awk sed







share|improve this question




























    0















    I have a file which contains ip address and port number in this order:



    ipaddress : port



    1.1.1.1:21

    1.1.1.1:22

    2.2.2.2:443

    3.3.3.3:80

    3.3.3.3:443



    I need Result in this below format


    ipaddress : port, port



    1.1.1.1:21,22

    2.2.2.2:443

    3.3.3.3:80,443






    text-processing awk sed







    share|improve this question


























      0












      0








      0








      I have a file which contains ip address and port number in this order:



      ipaddress : port



      1.1.1.1:21

      1.1.1.1:22

      2.2.2.2:443

      3.3.3.3:80

      3.3.3.3:443



      I need Result in this below format


      ipaddress : port, port



      1.1.1.1:21,22

      2.2.2.2:443

      3.3.3.3:80,443






      text-processing awk sed







      share|improve this question
















      I have a file which contains ip address and port number in this order:



      ipaddress : port



      1.1.1.1:21

      1.1.1.1:22

      2.2.2.2:443

      3.3.3.3:80

      3.3.3.3:443



      I need Result in this below format


      ipaddress : port, port



      1.1.1.1:21,22

      2.2.2.2:443

      3.3.3.3:80,443






      text-processing awk sed




      text-processing awk sed






      share|improve this question















      share|improve this question













      share|improve this question




      share|improve this question








      edited Feb 1 at 15:44









      Jeff Schaller

      42k1156133




      42k1156133










      asked Feb 1 at 13:33









      user334662user334662

      32




      32




















          5 Answers
          5






          active

          oldest

          votes


















          2














          Assuming there are no trailing spaces on the lines in the input file:



          $ awk -F ':' 'BEGIN OFS=FS $1 in ports ports[$1] = ports[$1] "," $2; next ports[$1] = $2 END for (ip in ports) print ip, ports[ip] ' file
          3.3.3.3:80,443
          1.1.1.1:21,22
          2.2.2.2:443


          The awk script,



          BEGIN OFS=FS 
          $1 in ports ports[$1] = ports[$1] "," $2; next 
           ports[$1] = $2 
          END for (ip in ports) print ip, ports[ip]


          would first set the output field separator to be the same as the input field separator, which is a : character (this is given on the command line with -F ':'), then it would test whether the current first field (the IP address) is a key in the ports array. If it is, the port number (the second field) is added with a comma as a delimiter to that array entry. If it's not, the entry in the array is simply set to the port number for that IP address.



          At the end, all stored IP addresses are printed with their collected port numbers.






          share|improve this answer























          • Thank you soo much it worked :)

            – user334662
            Feb 1 at 13:53


















          2














          With GNU Datamash



          datamash -t: -s groupby 1 collapse 2 < file


          If your data are already sorted, you can omit the -s .



          Or using an anonymous array inside a hash in Perl:



          $ perl -F: -lne '
           push @ $h$F[0] , $F[1] 
           } sort  











          2














          With GNU Datamash



          datamash -t: -s groupby 1 collapse 2 < file


          If your data are already sorted, you can omit the -s .



          Or using an anonymous array inside a hash in Perl:



          $ perl -F: -lne '
           push @ $h$F[0] , $F[1] 
            










          2














          With GNU Datamash



          datamash -t: -s groupby 1 collapse 2 < file


          If your data are already sorted, you can omit the -s .



          Or using an anonymous array inside a hash in Perl:



          $ perl -F: -lne '
           push @ $h$F[0] , $F[1] 
           
           for $k (sort keys %h) print "$k:", join ",", @ $h$k 
          ' file
          1.1.1.1:21,22
          2.2.2.2:443
          3.3.3.3:80,443





          share
          for $k (sort keys %h) print "$k:", join ",", @ $h$k
          ' file
          1.1.1.1:21,22
          2.2.2.2:443
          3.3.3.3:80,443





          share uniq`; do awk -F ":" -v i="$i" '$1 == iprint i,$2' l.txt 











          0














          You can do using the sed editor. There we maintain 2 lines at any time in the pattern space and look for changes in the IP number. So long as we continue getting the same IP, we remove from the 2nd portion the IP and join it with the 1st portion with a comma. If not, then that means an IP change has been detected and we promptly print the first portion only, remove it from the pattern space, and go back and read in the next IP line into the pattern space and repeat the same checks. 



          $ sed -e '
           :loop
           $!N
           s/^(([^:]*:).*[^[:space:]]).*n2/1,/
           tloop
           P;D
           ' input-file.txt

           1.1.1.1:21,22
           2.2.2.2:443 
           3.3.3.3:80,443

           $ perl -lne '
           my($ip, $port) = /(H+):(H+)/;
           push @seen, $ip if ! exists $h$ip;
           push @$h$ip, $port; 








          0












          0








          0







          You can do using the sed editor. There we maintain 2 lines at any time in the pattern space and look for changes in the IP number. So long as we continue getting the same IP, we remove from the 2nd portion the IP and join it with the 1st portion with a comma. If not, then that means an IP change has been detected and we promptly print the first portion only, remove it from the pattern space, and go back and read in the next IP line into the pattern space and repeat the same checks. 



          $ sed -e '
           :loop
           $!N
           s/^(([^:]*:).*[^[:space:]]).*n2/1,/
           tloop
           P;D
           ' input-file.txt

           1.1.1.1:21,22
           2.2.2.2:443 
           3.3.3.3:80,443

           $ perl -lne '
           my($ip, $port) = /(H+):(H+)/;
           push @seen, $ip if ! exists $h$ip;
           push @$h$ip, $port;improve this answer















          You can do using the sed editor. There we maintain 2 lines at any time in the pattern space and look for changes in the IP number. So long as we continue getting the same IP, we remove from the 2nd portion the IP and join it with the 1st portion with a comma. If not, then that means an IP change has been detected and we promptly print the first portion only, remove it from the pattern space, and go back and read in the next IP line into the pattern space and repeat the same checks. 



          $ sed -e '
           :loop
           $!N
           s/^(([^:]*:).*[^[:space:]]).*n2/1,/
           tloop
           P;D
           ' input-file.txt

           1.1.1.1:21,22
           2.2.2.2:443 
           3.3.3.3:80,443

           $ perl -lne '
           my($ip, $port) = /(H+):(H+)/;
           push @seen, $ip if ! exists $h$ip;
           push @$h$ip, $port;{
           print $_, ":", join ",", @$h$_ for @seen;
           ' input-file.txt


          With Perl we can do the same by means of a hash which will maintain the IPs as it's keys and an array ref as the values comprising the ports. Also, we ensure to not consider any trailing blanks. The array @seen maintains the IPs in the order they were seen.







          share|improve this answer














          share|improve this answer



          share|improve this answer








          edited Feb 2 at 7:33

























          answered Feb 2 at 6:01









          Rakesh SharmaRakesh Sharma

          312113




          312113



























              draft saved

              draft discarded
















































              Thanks for contributing an answer to Unix & Linux Stack Exchange!


              • Please be sure to answer the question. Provide details and share your research!

              But avoid …


              • Asking for help, clarification, or responding to other answers.

              • Making statements based on opinion; back them up with references or personal experience.

              To learn more, see our tips on writing great answers.




              draft saved


              draft discarded














              StackExchange.ready(
              function ()
              StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f498132%2fremoving-duplicate-entries-and-replacing-it-with-comma-bash%23new-answer', 'question_page');

              );

              Post as a guest















              Required, but never shown





















































              Required, but never shown














              Required, but never shown












              Required, but never shown







              Required, but never shown

































              Required, but never shown














              Required, but never shown












              Required, but never shown







              Required, but never shown






              -

              Popular posts from this blog

              How to check contact read email or not when send email to Individual?

              Image
              Clash Royale CLAN TAG #URR8PPP up vote 1 down vote favorite I'm using WordPress 4.9.8, CiviCRM to 5.5.1, I usually send email to contact by Search> Find contacts View contact details Action> Send email Send email ok, Contact received mail ok like picture But status only Email sent though contact read email or not. So, can CiviCRM can change status to Email read when contact read email? wordpress email share | improve this question asked Sep 26 at 0:12 ToanLuong 49 9 add a comment  |  up vote 1 down vote favorite I'm using WordPress 4.9.8, CiviCRM to 5.5.1, I usually send email to contact by Search> Find contacts View contact details Action> Send email Send email ok, Contact received mail ok like picture But status only Email sent though contact read email or not. So, can CiviCRM can change status to Email read when contact read email? wordpress email share | improve this questi...
              Read more

              Displaying single band from multi-band raster using QGIS

              Image
              Clash Royale CLAN TAG #URR8PPP 1 How can I extract a single band from multi-band raster in QGIS? I have an remote sensed image which has 6 bands (including NDVI band), I want to display each band separately, but have no idea how to do. I have seen some questions similar here but none worked for me. The original image (has 6 bands) is: I want to display the band 6 which should be like this: But I tried gdal_translate, and couldn't get the correct result. What I have got is: qgis raster multi-band share | improve this question edited Mar 5 at 0:53 Summer asked Mar 4 at 6:42 Summer Summer 23 6 Is this any help gis.stackexchange.com/questions/220658/… ? if not gis.stackexchange.com/questions/62133/… might help. – Michael Stimson Mar 4 at 6:46 Thanks for answering but when I used gdal_translate, qgis showed that 'Error 4: Kayena.tif: No such file or directory". Would you know how to fi...
              Read more

              How many registers does an x86_64 CPU actually have?

              Image
              Clash Royale CLAN TAG #URR8PPP up vote 2 down vote favorite I am currently learning reverse engineering and am studying the flags register. I had in my mind that rflags was just another name for one of the 16 general purpose registers, for example rax or rbx . But it looks like rflags is actually an additional register. So that makes 17 registers in total... how many more could there be? I have spent at least an hour on this and found numerous different answers. The best answer so far is this, which says that there are 40 registers in total. 16 General Purpose Registers 2 Status Registers 6 Code Segment Registers 16 SSE Registers 8 FPU/MMX Registers But if I add that up, I get 48. Could anybody provide an official answer on how many registers an x86_64 CPU has (e.g. an Intel i7). Additionally, I have seen references to 'hardware' and 'architectural' registers. What are those registers and how many are there? register x86-64 share | improve this...
              Read more