Trying to perform two operations on a single huge source file to gain performance

Multi tool use
Multi tool use

The name of the pictureThe name of the pictureThe name of the pictureClash Royale CLAN TAG#URR8PPP











up vote
0
down vote

favorite












I am performing two operations in a single Linux command. The operations are:



  1. Sending data from source file into a new target file.


  2. Counting the number of records in the source file.


For example: source file: a.txt, target file: b.txt



cat a.txt > b.txt; cat a.txt | wc -l


In the above example, I extract data from a.txt twice to perform the two operations. However, my source file is very huge. So for better performance, I am trying to find a way to extract data from source file just once and perform both the operations.



How can I accomplish this?










share|improve this question









New contributor




Puneeth is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.























    up vote
    0
    down vote

    favorite












    I am performing two operations in a single Linux command. The operations are:



    1. Sending data from source file into a new target file.


    2. Counting the number of records in the source file.


    For example: source file: a.txt, target file: b.txt



    cat a.txt > b.txt; cat a.txt | wc -l


    In the above example, I extract data from a.txt twice to perform the two operations. However, my source file is very huge. So for better performance, I am trying to find a way to extract data from source file just once and perform both the operations.



    How can I accomplish this?










    share|improve this question









    New contributor




    Puneeth is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
    Check out our Code of Conduct.





















      up vote
      0
      down vote

      favorite









      up vote
      0
      down vote

      favorite











      I am performing two operations in a single Linux command. The operations are:



      1. Sending data from source file into a new target file.


      2. Counting the number of records in the source file.


      For example: source file: a.txt, target file: b.txt



      cat a.txt > b.txt; cat a.txt | wc -l


      In the above example, I extract data from a.txt twice to perform the two operations. However, my source file is very huge. So for better performance, I am trying to find a way to extract data from source file just once and perform both the operations.



      How can I accomplish this?










      share|improve this question









      New contributor




      Puneeth is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.











      I am performing two operations in a single Linux command. The operations are:



      1. Sending data from source file into a new target file.


      2. Counting the number of records in the source file.


      For example: source file: a.txt, target file: b.txt



      cat a.txt > b.txt; cat a.txt | wc -l


      In the above example, I extract data from a.txt twice to perform the two operations. However, my source file is very huge. So for better performance, I am trying to find a way to extract data from source file just once and perform both the operations.



      How can I accomplish this?







      shell-script pipe






      share|improve this question









      New contributor




      Puneeth is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.











      share|improve this question









      New contributor




      Puneeth is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.









      share|improve this question




      share|improve this question








      edited 12 hours ago









      Rui F Ribeiro

      38.1k1475123




      38.1k1475123






      New contributor




      Puneeth is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.









      asked 12 hours ago









      Puneeth

      1




      1




      New contributor




      Puneeth is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.





      New contributor





      Puneeth is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.






      Puneeth is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.




















          2 Answers
          2






          active

          oldest

          votes

















          up vote
          4
          down vote













          tee is the command you're looking for:



          cat a.txt | tee b.txt | wc -l


          Also, as suggested in the comments, if you want something shorter you can avoid using cat entirely and redirect a.txt straight into tee's stdin:



          tee b.txt < a.txt | wc -l


          From the man page:




          tee - read from standard input and write to standard output and files







          share|improve this answer


















          • 1




            tee b.txt <a.txt | wc -l. No need for the cat.
            – Kusalananda
            12 hours ago










          • @Kusalananda thank you, added to the answer
            – Marco Bonelli
            12 hours ago

















          up vote
          0
          down vote













          To get both actions in one file read you can do:



          awk '++c;ENDprint c >"/dev/stderr"' <a.txt >b.txt 


          Understand that the count will come from stderr.






          share|improve this answer




















            Your Answer








            StackExchange.ready(function()
            var channelOptions =
            tags: "".split(" "),
            id: "106"
            ;
            initTagRenderer("".split(" "), "".split(" "), channelOptions);

            StackExchange.using("externalEditor", function()
            // Have to fire editor after snippets, if snippets enabled
            if (StackExchange.settings.snippets.snippetsEnabled)
            StackExchange.using("snippets", function()
            createEditor();
            );

            else
            createEditor();

            );

            function createEditor()
            StackExchange.prepareEditor(
            heartbeatType: 'answer',
            convertImagesToLinks: false,
            noModals: true,
            showLowRepImageUploadWarning: true,
            reputationToPostImages: null,
            bindNavPrevention: true,
            postfix: "",
            imageUploader:
            brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
            contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
            allowUrls: true
            ,
            onDemand: true,
            discardSelector: ".discard-answer"
            ,immediatelyShowMarkdownHelp:true
            );



            );






            Puneeth is a new contributor. Be nice, and check out our Code of Conduct.









             

            draft saved


            draft discarded


















            StackExchange.ready(
            function ()
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f481265%2ftrying-to-perform-two-operations-on-a-single-huge-source-file-to-gain-performanc%23new-answer', 'question_page');

            );

            Post as a guest






























            2 Answers
            2






            active

            oldest

            votes








            2 Answers
            2






            active

            oldest

            votes









            active

            oldest

            votes






            active

            oldest

            votes








            up vote
            4
            down vote













            tee is the command you're looking for:



            cat a.txt | tee b.txt | wc -l


            Also, as suggested in the comments, if you want something shorter you can avoid using cat entirely and redirect a.txt straight into tee's stdin:



            tee b.txt < a.txt | wc -l


            From the man page:




            tee - read from standard input and write to standard output and files







            share|improve this answer


















            • 1




              tee b.txt <a.txt | wc -l. No need for the cat.
              – Kusalananda
              12 hours ago










            • @Kusalananda thank you, added to the answer
              – Marco Bonelli
              12 hours ago














            up vote
            4
            down vote













            tee is the command you're looking for:



            cat a.txt | tee b.txt | wc -l


            Also, as suggested in the comments, if you want something shorter you can avoid using cat entirely and redirect a.txt straight into tee's stdin:



            tee b.txt < a.txt | wc -l


            From the man page:




            tee - read from standard input and write to standard output and files







            share|improve this answer


















            • 1




              tee b.txt <a.txt | wc -l. No need for the cat.
              – Kusalananda
              12 hours ago










            • @Kusalananda thank you, added to the answer
              – Marco Bonelli
              12 hours ago












            up vote
            4
            down vote










            up vote
            4
            down vote









            tee is the command you're looking for:



            cat a.txt | tee b.txt | wc -l


            Also, as suggested in the comments, if you want something shorter you can avoid using cat entirely and redirect a.txt straight into tee's stdin:



            tee b.txt < a.txt | wc -l


            From the man page:




            tee - read from standard input and write to standard output and files







            share|improve this answer














            tee is the command you're looking for:



            cat a.txt | tee b.txt | wc -l


            Also, as suggested in the comments, if you want something shorter you can avoid using cat entirely and redirect a.txt straight into tee's stdin:



            tee b.txt < a.txt | wc -l


            From the man page:




            tee - read from standard input and write to standard output and files








            share|improve this answer














            share|improve this answer



            share|improve this answer








            edited 11 hours ago

























            answered 12 hours ago









            Marco Bonelli

            22112




            22112







            • 1




              tee b.txt <a.txt | wc -l. No need for the cat.
              – Kusalananda
              12 hours ago










            • @Kusalananda thank you, added to the answer
              – Marco Bonelli
              12 hours ago












            • 1




              tee b.txt <a.txt | wc -l. No need for the cat.
              – Kusalananda
              12 hours ago










            • @Kusalananda thank you, added to the answer
              – Marco Bonelli
              12 hours ago







            1




            1




            tee b.txt <a.txt | wc -l. No need for the cat.
            – Kusalananda
            12 hours ago




            tee b.txt <a.txt | wc -l. No need for the cat.
            – Kusalananda
            12 hours ago












            @Kusalananda thank you, added to the answer
            – Marco Bonelli
            12 hours ago




            @Kusalananda thank you, added to the answer
            – Marco Bonelli
            12 hours ago












            up vote
            0
            down vote













            To get both actions in one file read you can do:



            awk '++c;ENDprint c >"/dev/stderr"' <a.txt >b.txt 


            Understand that the count will come from stderr.






            share|improve this answer
























              up vote
              0
              down vote













              To get both actions in one file read you can do:



              awk '++c;ENDprint c >"/dev/stderr"' <a.txt >b.txt 


              Understand that the count will come from stderr.






              share|improve this answer






















                up vote
                0
                down vote










                up vote
                0
                down vote









                To get both actions in one file read you can do:



                awk '++c;ENDprint c >"/dev/stderr"' <a.txt >b.txt 


                Understand that the count will come from stderr.






                share|improve this answer












                To get both actions in one file read you can do:



                awk '++c;ENDprint c >"/dev/stderr"' <a.txt >b.txt 


                Understand that the count will come from stderr.







                share|improve this answer












                share|improve this answer



                share|improve this answer










                answered 12 hours ago









                Isaac

                9,34911442




                9,34911442




















                    Puneeth is a new contributor. Be nice, and check out our Code of Conduct.









                     

                    draft saved


                    draft discarded


















                    Puneeth is a new contributor. Be nice, and check out our Code of Conduct.












                    Puneeth is a new contributor. Be nice, and check out our Code of Conduct.











                    Puneeth is a new contributor. Be nice, and check out our Code of Conduct.













                     


                    draft saved


                    draft discarded














                    StackExchange.ready(
                    function ()
                    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f481265%2ftrying-to-perform-two-operations-on-a-single-huge-source-file-to-gain-performanc%23new-answer', 'question_page');

                    );

                    Post as a guest













































































                    uh3Hhf,p Nj7rxrwzU,E24K7Ddk49H5sHd4ckQTP8Zza73yZoqkER8Ma4K ZtIgCSzr,51,Ib u8QDKHo5EvGhAbJKKULyd7FFt0 AZMLZf
                    EdK8ONErlH Qio0PSjPntkY6kBkYBip,5LM7oS,xRBkKvg5GRWMz GH LM72g5wREs

                    Popular posts from this blog

                    How to check contact read email or not when send email to Individual?

                    How many registers does an x86_64 CPU actually have?

                    Displaying single band from multi-band raster using QGIS