Concatenating a list of files using For loop

The name of the pictureThe name of the pictureThe name of the pictureClash Royale CLAN TAG#URR8PPP











up vote
0
down vote

favorite












I'm new to unix/bash/code in general, and I'm trying to merge multiple sample files with the same name in different directories into a new file in its own directory (the outputs of multiple data collections).



To do this, I'm trying to create a script which uses two arrays - listA, the name of each sample file, and list B, the names for the newly merged sample files.



My code looks a little like this:



#!/bin/sh

listA=( old1 old2 old3 etc.)
listB=( new1 new2 new3 etc.)

i=0

for i in $listA $listB

do

cp ./folder1/$listA ./merged/$listB
cat ./folder2/$listA >> ./merged/$listB
cat ./folder3/$listA >> ./merged/$listB
cat ./folder4/$listA >> ./merged/$listB

((i=+1))

done
echo "Done stitching"


As is, it seems to merge the files for the first entry in listA into the first file in listB perfectly, but it won't repeat the process for the subsequent entries on the list.



Any advice to make this work as intended? Apologies for my ignorance, I'm very new to all of this and enjoying the learning process immensely - just a bit stumped.







share|improve this question




















  • Are the files with consecutive names? Like file1 file2 and so on. Is it only the base name a difference between source and target filenames?
    – Romeo Ninov
    Feb 27 at 8:42










  • Do you have control over the format of listA and listB? At the moment, they are two independent arrays. It'd be simpler to use a dictionary, if you are happy to use Bash 4.
    – Sparhawk
    Feb 27 at 8:50










  • @RomeoNinov unfortunately no, it would make my job easier if they were, but they all have nonconsecutive sample IDs
    – bellicosemagpie
    Feb 27 at 11:56










  • @Sparhawk I do & I have Bash 4, I'll have to do some reading on dictionaries.
    – bellicosemagpie
    Feb 27 at 12:04











  • Having said that, even though dictionaries are probably the simplest way to parse, it'd be fiddly to enter the lists. What about feeding the script a text file of tab-delimited pairs of files instead? Would that be easiest to create? How many files are we talking about?
    – Sparhawk
    Feb 27 at 20:37














up vote
0
down vote

favorite












I'm new to unix/bash/code in general, and I'm trying to merge multiple sample files with the same name in different directories into a new file in its own directory (the outputs of multiple data collections).



To do this, I'm trying to create a script which uses two arrays - listA, the name of each sample file, and list B, the names for the newly merged sample files.



My code looks a little like this:



#!/bin/sh

listA=( old1 old2 old3 etc.)
listB=( new1 new2 new3 etc.)

i=0

for i in $listA $listB

do

cp ./folder1/$listA ./merged/$listB
cat ./folder2/$listA >> ./merged/$listB
cat ./folder3/$listA >> ./merged/$listB
cat ./folder4/$listA >> ./merged/$listB

((i=+1))

done
echo "Done stitching"


As is, it seems to merge the files for the first entry in listA into the first file in listB perfectly, but it won't repeat the process for the subsequent entries on the list.



Any advice to make this work as intended? Apologies for my ignorance, I'm very new to all of this and enjoying the learning process immensely - just a bit stumped.







share|improve this question




















  • Are the files with consecutive names? Like file1 file2 and so on. Is it only the base name a difference between source and target filenames?
    – Romeo Ninov
    Feb 27 at 8:42










  • Do you have control over the format of listA and listB? At the moment, they are two independent arrays. It'd be simpler to use a dictionary, if you are happy to use Bash 4.
    – Sparhawk
    Feb 27 at 8:50










  • @RomeoNinov unfortunately no, it would make my job easier if they were, but they all have nonconsecutive sample IDs
    – bellicosemagpie
    Feb 27 at 11:56










  • @Sparhawk I do & I have Bash 4, I'll have to do some reading on dictionaries.
    – bellicosemagpie
    Feb 27 at 12:04











  • Having said that, even though dictionaries are probably the simplest way to parse, it'd be fiddly to enter the lists. What about feeding the script a text file of tab-delimited pairs of files instead? Would that be easiest to create? How many files are we talking about?
    – Sparhawk
    Feb 27 at 20:37












up vote
0
down vote

favorite









up vote
0
down vote

favorite











I'm new to unix/bash/code in general, and I'm trying to merge multiple sample files with the same name in different directories into a new file in its own directory (the outputs of multiple data collections).



To do this, I'm trying to create a script which uses two arrays - listA, the name of each sample file, and list B, the names for the newly merged sample files.



My code looks a little like this:



#!/bin/sh

listA=( old1 old2 old3 etc.)
listB=( new1 new2 new3 etc.)

i=0

for i in $listA $listB

do

cp ./folder1/$listA ./merged/$listB
cat ./folder2/$listA >> ./merged/$listB
cat ./folder3/$listA >> ./merged/$listB
cat ./folder4/$listA >> ./merged/$listB

((i=+1))

done
echo "Done stitching"


As is, it seems to merge the files for the first entry in listA into the first file in listB perfectly, but it won't repeat the process for the subsequent entries on the list.



Any advice to make this work as intended? Apologies for my ignorance, I'm very new to all of this and enjoying the learning process immensely - just a bit stumped.







share|improve this question












I'm new to unix/bash/code in general, and I'm trying to merge multiple sample files with the same name in different directories into a new file in its own directory (the outputs of multiple data collections).



To do this, I'm trying to create a script which uses two arrays - listA, the name of each sample file, and list B, the names for the newly merged sample files.



My code looks a little like this:



#!/bin/sh

listA=( old1 old2 old3 etc.)
listB=( new1 new2 new3 etc.)

i=0

for i in $listA $listB

do

cp ./folder1/$listA ./merged/$listB
cat ./folder2/$listA >> ./merged/$listB
cat ./folder3/$listA >> ./merged/$listB
cat ./folder4/$listA >> ./merged/$listB

((i=+1))

done
echo "Done stitching"


As is, it seems to merge the files for the first entry in listA into the first file in listB perfectly, but it won't repeat the process for the subsequent entries on the list.



Any advice to make this work as intended? Apologies for my ignorance, I'm very new to all of this and enjoying the learning process immensely - just a bit stumped.









share|improve this question











share|improve this question




share|improve this question










asked Feb 27 at 8:29









bellicosemagpie

32




32











  • Are the files with consecutive names? Like file1 file2 and so on. Is it only the base name a difference between source and target filenames?
    – Romeo Ninov
    Feb 27 at 8:42










  • Do you have control over the format of listA and listB? At the moment, they are two independent arrays. It'd be simpler to use a dictionary, if you are happy to use Bash 4.
    – Sparhawk
    Feb 27 at 8:50










  • @RomeoNinov unfortunately no, it would make my job easier if they were, but they all have nonconsecutive sample IDs
    – bellicosemagpie
    Feb 27 at 11:56










  • @Sparhawk I do & I have Bash 4, I'll have to do some reading on dictionaries.
    – bellicosemagpie
    Feb 27 at 12:04











  • Having said that, even though dictionaries are probably the simplest way to parse, it'd be fiddly to enter the lists. What about feeding the script a text file of tab-delimited pairs of files instead? Would that be easiest to create? How many files are we talking about?
    – Sparhawk
    Feb 27 at 20:37
















  • Are the files with consecutive names? Like file1 file2 and so on. Is it only the base name a difference between source and target filenames?
    – Romeo Ninov
    Feb 27 at 8:42










  • Do you have control over the format of listA and listB? At the moment, they are two independent arrays. It'd be simpler to use a dictionary, if you are happy to use Bash 4.
    – Sparhawk
    Feb 27 at 8:50










  • @RomeoNinov unfortunately no, it would make my job easier if they were, but they all have nonconsecutive sample IDs
    – bellicosemagpie
    Feb 27 at 11:56










  • @Sparhawk I do & I have Bash 4, I'll have to do some reading on dictionaries.
    – bellicosemagpie
    Feb 27 at 12:04











  • Having said that, even though dictionaries are probably the simplest way to parse, it'd be fiddly to enter the lists. What about feeding the script a text file of tab-delimited pairs of files instead? Would that be easiest to create? How many files are we talking about?
    – Sparhawk
    Feb 27 at 20:37















Are the files with consecutive names? Like file1 file2 and so on. Is it only the base name a difference between source and target filenames?
– Romeo Ninov
Feb 27 at 8:42




Are the files with consecutive names? Like file1 file2 and so on. Is it only the base name a difference between source and target filenames?
– Romeo Ninov
Feb 27 at 8:42












Do you have control over the format of listA and listB? At the moment, they are two independent arrays. It'd be simpler to use a dictionary, if you are happy to use Bash 4.
– Sparhawk
Feb 27 at 8:50




Do you have control over the format of listA and listB? At the moment, they are two independent arrays. It'd be simpler to use a dictionary, if you are happy to use Bash 4.
– Sparhawk
Feb 27 at 8:50












@RomeoNinov unfortunately no, it would make my job easier if they were, but they all have nonconsecutive sample IDs
– bellicosemagpie
Feb 27 at 11:56




@RomeoNinov unfortunately no, it would make my job easier if they were, but they all have nonconsecutive sample IDs
– bellicosemagpie
Feb 27 at 11:56












@Sparhawk I do & I have Bash 4, I'll have to do some reading on dictionaries.
– bellicosemagpie
Feb 27 at 12:04





@Sparhawk I do & I have Bash 4, I'll have to do some reading on dictionaries.
– bellicosemagpie
Feb 27 at 12:04













Having said that, even though dictionaries are probably the simplest way to parse, it'd be fiddly to enter the lists. What about feeding the script a text file of tab-delimited pairs of files instead? Would that be easiest to create? How many files are we talking about?
– Sparhawk
Feb 27 at 20:37




Having said that, even though dictionaries are probably the simplest way to parse, it'd be fiddly to enter the lists. What about feeding the script a text file of tab-delimited pairs of files instead? Would that be easiest to create? How many files are we talking about?
– Sparhawk
Feb 27 at 20:37










1 Answer
1






active

oldest

votes

















up vote
0
down vote



accepted










Short answer



while IFS=$'t' read -r old new; do cat folder*/"$old" > merged/"$new"; done <list.tsv


This assumes that list.tsv contains a tab-delimited list of source (old) and destination (new) files.



Explanation




  • while IFS=$'t' read -r old new; FOO; done <list.tsv: this is a common shell contruct to read each line of list.tsv, then do FOO on each line. In this case, the input file separator IFS is a tab $'t', i.e. the first field of the line is saved to $old, and the second field (after the tab) is saved to $new.


  • cat folder*/"$old" > merged/"$new": concatenate the files in folder*/"$old", and write this to merged/"$new". This assumes that the source files are all in folder*, e.g. named folder1, folder2, etc.

  • As with any script, I would certainly test this before running it. You can do that by copying a subset of your files to another directory, then using a short version of your input list list.tsv. Check the results, and if okay, then run it for real (but keep a backup!).

Notes on your suggested solution



I just thought I'd write a few comments on your proposed solution, in order to clarify how the various commands work. Hopefully this is helpful to your understanding!




  • listA=( old1 old2 old3 etc.). This creates an array. The way to access parts of it is shown here. Using for i in $listA… will just access the first entry, as you have (kinda) observed.


  • for i in a b c. This iterates over a, b, and c, putting each into $i. For example, try running for i in a b c; do echo $i; done. N.B. it makes no sense to refer to $listA within the do…done block. Similarly, i=0 and ((i=+1)) are inconsistent with for usage.


  • ./folder1/$listA. ./ is unnecessary. . is the current folder, so it doesn't do anything here.


  • cat FOO >>. cat will concatenate by itself (hence the name). There is no need to write separate cat statements that append >> to a file. You can just list multiple files after cat, e.g. cat file1 file2 file3 > output_file.





share|improve this answer




















    Your Answer







    StackExchange.ready(function()
    var channelOptions =
    tags: "".split(" "),
    id: "106"
    ;
    initTagRenderer("".split(" "), "".split(" "), channelOptions);

    StackExchange.using("externalEditor", function()
    // Have to fire editor after snippets, if snippets enabled
    if (StackExchange.settings.snippets.snippetsEnabled)
    StackExchange.using("snippets", function()
    createEditor();
    );

    else
    createEditor();

    );

    function createEditor()
    StackExchange.prepareEditor(
    heartbeatType: 'answer',
    convertImagesToLinks: false,
    noModals: false,
    showLowRepImageUploadWarning: true,
    reputationToPostImages: null,
    bindNavPrevention: true,
    postfix: "",
    onDemand: true,
    discardSelector: ".discard-answer"
    ,immediatelyShowMarkdownHelp:true
    );



    );








     

    draft saved


    draft discarded


















    StackExchange.ready(
    function ()
    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f426874%2fconcatenating-a-list-of-files-using-for-loop%23new-answer', 'question_page');

    );

    Post as a guest






























    1 Answer
    1






    active

    oldest

    votes








    1 Answer
    1






    active

    oldest

    votes









    active

    oldest

    votes






    active

    oldest

    votes








    up vote
    0
    down vote



    accepted










    Short answer



    while IFS=$'t' read -r old new; do cat folder*/"$old" > merged/"$new"; done <list.tsv


    This assumes that list.tsv contains a tab-delimited list of source (old) and destination (new) files.



    Explanation




    • while IFS=$'t' read -r old new; FOO; done <list.tsv: this is a common shell contruct to read each line of list.tsv, then do FOO on each line. In this case, the input file separator IFS is a tab $'t', i.e. the first field of the line is saved to $old, and the second field (after the tab) is saved to $new.


    • cat folder*/"$old" > merged/"$new": concatenate the files in folder*/"$old", and write this to merged/"$new". This assumes that the source files are all in folder*, e.g. named folder1, folder2, etc.

    • As with any script, I would certainly test this before running it. You can do that by copying a subset of your files to another directory, then using a short version of your input list list.tsv. Check the results, and if okay, then run it for real (but keep a backup!).

    Notes on your suggested solution



    I just thought I'd write a few comments on your proposed solution, in order to clarify how the various commands work. Hopefully this is helpful to your understanding!




    • listA=( old1 old2 old3 etc.). This creates an array. The way to access parts of it is shown here. Using for i in $listA… will just access the first entry, as you have (kinda) observed.


    • for i in a b c. This iterates over a, b, and c, putting each into $i. For example, try running for i in a b c; do echo $i; done. N.B. it makes no sense to refer to $listA within the do…done block. Similarly, i=0 and ((i=+1)) are inconsistent with for usage.


    • ./folder1/$listA. ./ is unnecessary. . is the current folder, so it doesn't do anything here.


    • cat FOO >>. cat will concatenate by itself (hence the name). There is no need to write separate cat statements that append >> to a file. You can just list multiple files after cat, e.g. cat file1 file2 file3 > output_file.





    share|improve this answer
























      up vote
      0
      down vote



      accepted










      Short answer



      while IFS=$'t' read -r old new; do cat folder*/"$old" > merged/"$new"; done <list.tsv


      This assumes that list.tsv contains a tab-delimited list of source (old) and destination (new) files.



      Explanation




      • while IFS=$'t' read -r old new; FOO; done <list.tsv: this is a common shell contruct to read each line of list.tsv, then do FOO on each line. In this case, the input file separator IFS is a tab $'t', i.e. the first field of the line is saved to $old, and the second field (after the tab) is saved to $new.


      • cat folder*/"$old" > merged/"$new": concatenate the files in folder*/"$old", and write this to merged/"$new". This assumes that the source files are all in folder*, e.g. named folder1, folder2, etc.

      • As with any script, I would certainly test this before running it. You can do that by copying a subset of your files to another directory, then using a short version of your input list list.tsv. Check the results, and if okay, then run it for real (but keep a backup!).

      Notes on your suggested solution



      I just thought I'd write a few comments on your proposed solution, in order to clarify how the various commands work. Hopefully this is helpful to your understanding!




      • listA=( old1 old2 old3 etc.). This creates an array. The way to access parts of it is shown here. Using for i in $listA… will just access the first entry, as you have (kinda) observed.


      • for i in a b c. This iterates over a, b, and c, putting each into $i. For example, try running for i in a b c; do echo $i; done. N.B. it makes no sense to refer to $listA within the do…done block. Similarly, i=0 and ((i=+1)) are inconsistent with for usage.


      • ./folder1/$listA. ./ is unnecessary. . is the current folder, so it doesn't do anything here.


      • cat FOO >>. cat will concatenate by itself (hence the name). There is no need to write separate cat statements that append >> to a file. You can just list multiple files after cat, e.g. cat file1 file2 file3 > output_file.





      share|improve this answer






















        up vote
        0
        down vote



        accepted







        up vote
        0
        down vote



        accepted






        Short answer



        while IFS=$'t' read -r old new; do cat folder*/"$old" > merged/"$new"; done <list.tsv


        This assumes that list.tsv contains a tab-delimited list of source (old) and destination (new) files.



        Explanation




        • while IFS=$'t' read -r old new; FOO; done <list.tsv: this is a common shell contruct to read each line of list.tsv, then do FOO on each line. In this case, the input file separator IFS is a tab $'t', i.e. the first field of the line is saved to $old, and the second field (after the tab) is saved to $new.


        • cat folder*/"$old" > merged/"$new": concatenate the files in folder*/"$old", and write this to merged/"$new". This assumes that the source files are all in folder*, e.g. named folder1, folder2, etc.

        • As with any script, I would certainly test this before running it. You can do that by copying a subset of your files to another directory, then using a short version of your input list list.tsv. Check the results, and if okay, then run it for real (but keep a backup!).

        Notes on your suggested solution



        I just thought I'd write a few comments on your proposed solution, in order to clarify how the various commands work. Hopefully this is helpful to your understanding!




        • listA=( old1 old2 old3 etc.). This creates an array. The way to access parts of it is shown here. Using for i in $listA… will just access the first entry, as you have (kinda) observed.


        • for i in a b c. This iterates over a, b, and c, putting each into $i. For example, try running for i in a b c; do echo $i; done. N.B. it makes no sense to refer to $listA within the do…done block. Similarly, i=0 and ((i=+1)) are inconsistent with for usage.


        • ./folder1/$listA. ./ is unnecessary. . is the current folder, so it doesn't do anything here.


        • cat FOO >>. cat will concatenate by itself (hence the name). There is no need to write separate cat statements that append >> to a file. You can just list multiple files after cat, e.g. cat file1 file2 file3 > output_file.





        share|improve this answer












        Short answer



        while IFS=$'t' read -r old new; do cat folder*/"$old" > merged/"$new"; done <list.tsv


        This assumes that list.tsv contains a tab-delimited list of source (old) and destination (new) files.



        Explanation




        • while IFS=$'t' read -r old new; FOO; done <list.tsv: this is a common shell contruct to read each line of list.tsv, then do FOO on each line. In this case, the input file separator IFS is a tab $'t', i.e. the first field of the line is saved to $old, and the second field (after the tab) is saved to $new.


        • cat folder*/"$old" > merged/"$new": concatenate the files in folder*/"$old", and write this to merged/"$new". This assumes that the source files are all in folder*, e.g. named folder1, folder2, etc.

        • As with any script, I would certainly test this before running it. You can do that by copying a subset of your files to another directory, then using a short version of your input list list.tsv. Check the results, and if okay, then run it for real (but keep a backup!).

        Notes on your suggested solution



        I just thought I'd write a few comments on your proposed solution, in order to clarify how the various commands work. Hopefully this is helpful to your understanding!




        • listA=( old1 old2 old3 etc.). This creates an array. The way to access parts of it is shown here. Using for i in $listA… will just access the first entry, as you have (kinda) observed.


        • for i in a b c. This iterates over a, b, and c, putting each into $i. For example, try running for i in a b c; do echo $i; done. N.B. it makes no sense to refer to $listA within the do…done block. Similarly, i=0 and ((i=+1)) are inconsistent with for usage.


        • ./folder1/$listA. ./ is unnecessary. . is the current folder, so it doesn't do anything here.


        • cat FOO >>. cat will concatenate by itself (hence the name). There is no need to write separate cat statements that append >> to a file. You can just list multiple files after cat, e.g. cat file1 file2 file3 > output_file.






        share|improve this answer












        share|improve this answer



        share|improve this answer










        answered Feb 28 at 9:15









        Sparhawk

        8,33363487




        8,33363487






















             

            draft saved


            draft discarded


























             


            draft saved


            draft discarded














            StackExchange.ready(
            function ()
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f426874%2fconcatenating-a-list-of-files-using-for-loop%23new-answer', 'question_page');

            );

            Post as a guest













































































            Popular posts from this blog

            How to check contact read email or not when send email to Individual?

            Displaying single band from multi-band raster using QGIS

            How many registers does an x86_64 CPU actually have?