Trying to perform two operations on a single huge source file to gain performance

The name of the pictureThe name of the pictureThe name of the pictureClash Royale CLAN TAG#URR8PPP











up vote
0
down vote

favorite












I am performing two operations in a single Linux command. The operations are:



  1. Sending data from source file into a new target file.


  2. Counting the number of records in the source file.


For example: source file: a.txt, target file: b.txt



cat a.txt > b.txt; cat a.txt | wc -l


In the above example, I extract data from a.txt twice to perform the two operations. However, my source file is very huge. So for better performance, I am trying to find a way to extract data from source file just once and perform both the operations.



How can I accomplish this?










share|improve this question









New contributor




Puneeth is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.























    up vote
    0
    down vote

    favorite












    I am performing two operations in a single Linux command. The operations are:



    1. Sending data from source file into a new target file.


    2. Counting the number of records in the source file.


    For example: source file: a.txt, target file: b.txt



    cat a.txt > b.txt; cat a.txt | wc -l


    In the above example, I extract data from a.txt twice to perform the two operations. However, my source file is very huge. So for better performance, I am trying to find a way to extract data from source file just once and perform both the operations.



    How can I accomplish this?










    share|improve this question









    New contributor




    Puneeth is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
    Check out our Code of Conduct.





















      up vote
      0
      down vote

      favorite









      up vote
      0
      down vote

      favorite











      I am performing two operations in a single Linux command. The operations are:



      1. Sending data from source file into a new target file.


      2. Counting the number of records in the source file.


      For example: source file: a.txt, target file: b.txt



      cat a.txt > b.txt; cat a.txt | wc -l


      In the above example, I extract data from a.txt twice to perform the two operations. However, my source file is very huge. So for better performance, I am trying to find a way to extract data from source file just once and perform both the operations.



      How can I accomplish this?










      share|improve this question









      New contributor




      Puneeth is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.











      I am performing two operations in a single Linux command. The operations are:



      1. Sending data from source file into a new target file.


      2. Counting the number of records in the source file.


      For example: source file: a.txt, target file: b.txt



      cat a.txt > b.txt; cat a.txt | wc -l


      In the above example, I extract data from a.txt twice to perform the two operations. However, my source file is very huge. So for better performance, I am trying to find a way to extract data from source file just once and perform both the operations.



      How can I accomplish this?







      shell-script pipe






      share|improve this question









      New contributor




      Puneeth is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.











      share|improve this question









      New contributor




      Puneeth is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.









      share|improve this question




      share|improve this question








      edited 12 hours ago









      Rui F Ribeiro

      38.1k1475123




      38.1k1475123






      New contributor




      Puneeth is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.









      asked 12 hours ago









      Puneeth

      1




      1




      New contributor




      Puneeth is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.





      New contributor





      Puneeth is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.






      Puneeth is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.




















          2 Answers
          2






          active

          oldest

          votes

















          up vote
          4
          down vote













          tee is the command you're looking for:



          cat a.txt | tee b.txt | wc -l


          Also, as suggested in the comments, if you want something shorter you can avoid using cat entirely and redirect a.txt straight into tee's stdin:



          tee b.txt < a.txt | wc -l


          From the man page:




          tee - read from standard input and write to standard output and files







          share|improve this answer


















          • 1




            tee b.txt <a.txt | wc -l. No need for the cat.
            – Kusalananda
            12 hours ago










          • @Kusalananda thank you, added to the answer
            – Marco Bonelli
            12 hours ago

















          up vote
          0
          down vote













          To get both actions in one file read you can do:



          awk '++c;ENDprint c >"/dev/stderr"' <a.txt >b.txt 


          Understand that the count will come from stderr.






          share|improve this answer




















            Your Answer








            StackExchange.ready(function()
            var channelOptions =
            tags: "".split(" "),
            id: "106"
            ;
            initTagRenderer("".split(" "), "".split(" "), channelOptions);

            StackExchange.using("externalEditor", function()
            // Have to fire editor after snippets, if snippets enabled
            if (StackExchange.settings.snippets.snippetsEnabled)
            StackExchange.using("snippets", function()
            createEditor();
            );

            else
            createEditor();

            );

            function createEditor()
            StackExchange.prepareEditor(
            heartbeatType: 'answer',
            convertImagesToLinks: false,
            noModals: true,
            showLowRepImageUploadWarning: true,
            reputationToPostImages: null,
            bindNavPrevention: true,
            postfix: "",
            imageUploader:
            brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
            contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
            allowUrls: true
            ,
            onDemand: true,
            discardSelector: ".discard-answer"
            ,immediatelyShowMarkdownHelp:true
            );



            );






            Puneeth is a new contributor. Be nice, and check out our Code of Conduct.









             

            draft saved


            draft discarded


















            StackExchange.ready(
            function ()
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f481265%2ftrying-to-perform-two-operations-on-a-single-huge-source-file-to-gain-performanc%23new-answer', 'question_page');

            );

            Post as a guest






























            2 Answers
            2






            active

            oldest

            votes








            2 Answers
            2






            active

            oldest

            votes









            active

            oldest

            votes






            active

            oldest

            votes








            up vote
            4
            down vote













            tee is the command you're looking for:



            cat a.txt | tee b.txt | wc -l


            Also, as suggested in the comments, if you want something shorter you can avoid using cat entirely and redirect a.txt straight into tee's stdin:



            tee b.txt < a.txt | wc -l


            From the man page:




            tee - read from standard input and write to standard output and files







            share|improve this answer


















            • 1




              tee b.txt <a.txt | wc -l. No need for the cat.
              – Kusalananda
              12 hours ago










            • @Kusalananda thank you, added to the answer
              – Marco Bonelli
              12 hours ago














            up vote
            4
            down vote













            tee is the command you're looking for:



            cat a.txt | tee b.txt | wc -l


            Also, as suggested in the comments, if you want something shorter you can avoid using cat entirely and redirect a.txt straight into tee's stdin:



            tee b.txt < a.txt | wc -l


            From the man page:




            tee - read from standard input and write to standard output and files







            share|improve this answer


















            • 1




              tee b.txt <a.txt | wc -l. No need for the cat.
              – Kusalananda
              12 hours ago










            • @Kusalananda thank you, added to the answer
              – Marco Bonelli
              12 hours ago












            up vote
            4
            down vote










            up vote
            4
            down vote









            tee is the command you're looking for:



            cat a.txt | tee b.txt | wc -l


            Also, as suggested in the comments, if you want something shorter you can avoid using cat entirely and redirect a.txt straight into tee's stdin:



            tee b.txt < a.txt | wc -l


            From the man page:




            tee - read from standard input and write to standard output and files







            share|improve this answer














            tee is the command you're looking for:



            cat a.txt | tee b.txt | wc -l


            Also, as suggested in the comments, if you want something shorter you can avoid using cat entirely and redirect a.txt straight into tee's stdin:



            tee b.txt < a.txt | wc -l


            From the man page:




            tee - read from standard input and write to standard output and files








            share|improve this answer














            share|improve this answer



            share|improve this answer








            edited 11 hours ago

























            answered 12 hours ago









            Marco Bonelli

            22112




            22112







            • 1




              tee b.txt <a.txt | wc -l. No need for the cat.
              – Kusalananda
              12 hours ago










            • @Kusalananda thank you, added to the answer
              – Marco Bonelli
              12 hours ago












            • 1




              tee b.txt <a.txt | wc -l. No need for the cat.
              – Kusalananda
              12 hours ago










            • @Kusalananda thank you, added to the answer
              – Marco Bonelli
              12 hours ago







            1




            1




            tee b.txt <a.txt | wc -l. No need for the cat.
            – Kusalananda
            12 hours ago




            tee b.txt <a.txt | wc -l. No need for the cat.
            – Kusalananda
            12 hours ago












            @Kusalananda thank you, added to the answer
            – Marco Bonelli
            12 hours ago




            @Kusalananda thank you, added to the answer
            – Marco Bonelli
            12 hours ago












            up vote
            0
            down vote













            To get both actions in one file read you can do:



            awk '++c;ENDprint c >"/dev/stderr"' <a.txt >b.txt 


            Understand that the count will come from stderr.






            share|improve this answer
























              up vote
              0
              down vote













              To get both actions in one file read you can do:



              awk '++c;ENDprint c >"/dev/stderr"' <a.txt >b.txt 


              Understand that the count will come from stderr.






              share|improve this answer






















                up vote
                0
                down vote










                up vote
                0
                down vote









                To get both actions in one file read you can do:



                awk '++c;ENDprint c >"/dev/stderr"' <a.txt >b.txt 


                Understand that the count will come from stderr.






                share|improve this answer












                To get both actions in one file read you can do:



                awk '++c;ENDprint c >"/dev/stderr"' <a.txt >b.txt 


                Understand that the count will come from stderr.







                share|improve this answer












                share|improve this answer



                share|improve this answer










                answered 12 hours ago









                Isaac

                9,34911442




                9,34911442




















                    Puneeth is a new contributor. Be nice, and check out our Code of Conduct.









                     

                    draft saved


                    draft discarded


















                    Puneeth is a new contributor. Be nice, and check out our Code of Conduct.












                    Puneeth is a new contributor. Be nice, and check out our Code of Conduct.











                    Puneeth is a new contributor. Be nice, and check out our Code of Conduct.













                     


                    draft saved


                    draft discarded














                    StackExchange.ready(
                    function ()
                    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f481265%2ftrying-to-perform-two-operations-on-a-single-huge-source-file-to-gain-performanc%23new-answer', 'question_page');

                    );

                    Post as a guest













































































                    Popular posts from this blog

                    How to check contact read email or not when send email to Individual?

                    Bahrain

                    Postfix configuration issue with fips on centos 7; mailgun relay