Need to iterate through subdirectories, concatenating files, with an iterative number

The name of the pictureThe name of the pictureThe name of the pictureClash Royale CLAN TAG#URR8PPP












2














I'm trying to concatenate files from three different subdirectories into one file. The names of the files in each subdirectory are exactly the same. I'd like to use a loop to iterate through the subdirectories, and then place an iteration number into the newly named concatenated file, in a new directory.
For example a directory structure like:



Foo
|||
||Bar3
|Bar2
Bar1


Inside each Bar(?) folder are files named:
File1, File2, File3



I'd like to concatenate the Files with the same names to a larger file with a new name including a number:



cat Foo/Bar1/File1 Foo/Bar2/File1 Foo/Bar3/File1 > /combined_files/all_file1

cat Foo/Bar1/File2 Foo/Bar2/File2 Foo/Bar3/File2 > /combined_files/all_file2

cat Foo/Bar1/File3 Foo/Bar2/File3 Foo/Bar3/File3 > /combined_files/all_file3


From the Foo directory I can use:



for number in 1..3
do
cat Bar1/File$number_* Bar2/File$number_* Bar3/File$number_* > combined_files/'all_files'$number
done
exit


But I need to have a more universal script, for larger numbers of Bar directories, and Files.
I want something like



files=`ls ./Run1/ | wc -l` #to count the number of files and assign a number
For n in 1..$files
do
cat Bar1/File$n_* Bar2/File$n_* Bar3/File$n_* > combined_files/'all_files'$n
done


But I'm stuck.










share|improve this question



















  • 1




    Do you want to use all of Barx? Will x always be a single digit?
    – drewbenn
    Dec 10 at 21:00










  • Yes I want all the files in BarX, but no X will go from 1 to a large number.
    – hoytpr
    Dec 10 at 21:29










  • Does order need to be preserved between each run? Eg should Bar1/File2 always appear in the concatenated output before Bar2/File1? This will affect whether it can be accomplished using a simple one-liner, or whether a more complicated (perhaps recursive) result will be required.
    – cryptarch
    Dec 10 at 21:32










  • Unfortunately yes. The files are ordered so the concatenation needs to be ordered.
    – hoytpr
    Dec 10 at 21:34










  • This is similar to a question asked a few months ago, but the answers didn't seem to work for me, or maybe I don't interpret them correctly
    – hoytpr
    Dec 10 at 21:49















2














I'm trying to concatenate files from three different subdirectories into one file. The names of the files in each subdirectory are exactly the same. I'd like to use a loop to iterate through the subdirectories, and then place an iteration number into the newly named concatenated file, in a new directory.
For example a directory structure like:



Foo
|||
||Bar3
|Bar2
Bar1


Inside each Bar(?) folder are files named:
File1, File2, File3



I'd like to concatenate the Files with the same names to a larger file with a new name including a number:



cat Foo/Bar1/File1 Foo/Bar2/File1 Foo/Bar3/File1 > /combined_files/all_file1

cat Foo/Bar1/File2 Foo/Bar2/File2 Foo/Bar3/File2 > /combined_files/all_file2

cat Foo/Bar1/File3 Foo/Bar2/File3 Foo/Bar3/File3 > /combined_files/all_file3


From the Foo directory I can use:



for number in 1..3
do
cat Bar1/File$number_* Bar2/File$number_* Bar3/File$number_* > combined_files/'all_files'$number
done
exit


But I need to have a more universal script, for larger numbers of Bar directories, and Files.
I want something like



files=`ls ./Run1/ | wc -l` #to count the number of files and assign a number
For n in 1..$files
do
cat Bar1/File$n_* Bar2/File$n_* Bar3/File$n_* > combined_files/'all_files'$n
done


But I'm stuck.










share|improve this question



















  • 1




    Do you want to use all of Barx? Will x always be a single digit?
    – drewbenn
    Dec 10 at 21:00










  • Yes I want all the files in BarX, but no X will go from 1 to a large number.
    – hoytpr
    Dec 10 at 21:29










  • Does order need to be preserved between each run? Eg should Bar1/File2 always appear in the concatenated output before Bar2/File1? This will affect whether it can be accomplished using a simple one-liner, or whether a more complicated (perhaps recursive) result will be required.
    – cryptarch
    Dec 10 at 21:32










  • Unfortunately yes. The files are ordered so the concatenation needs to be ordered.
    – hoytpr
    Dec 10 at 21:34










  • This is similar to a question asked a few months ago, but the answers didn't seem to work for me, or maybe I don't interpret them correctly
    – hoytpr
    Dec 10 at 21:49













2












2








2







I'm trying to concatenate files from three different subdirectories into one file. The names of the files in each subdirectory are exactly the same. I'd like to use a loop to iterate through the subdirectories, and then place an iteration number into the newly named concatenated file, in a new directory.
For example a directory structure like:



Foo
|||
||Bar3
|Bar2
Bar1


Inside each Bar(?) folder are files named:
File1, File2, File3



I'd like to concatenate the Files with the same names to a larger file with a new name including a number:



cat Foo/Bar1/File1 Foo/Bar2/File1 Foo/Bar3/File1 > /combined_files/all_file1

cat Foo/Bar1/File2 Foo/Bar2/File2 Foo/Bar3/File2 > /combined_files/all_file2

cat Foo/Bar1/File3 Foo/Bar2/File3 Foo/Bar3/File3 > /combined_files/all_file3


From the Foo directory I can use:



for number in 1..3
do
cat Bar1/File$number_* Bar2/File$number_* Bar3/File$number_* > combined_files/'all_files'$number
done
exit


But I need to have a more universal script, for larger numbers of Bar directories, and Files.
I want something like



files=`ls ./Run1/ | wc -l` #to count the number of files and assign a number
For n in 1..$files
do
cat Bar1/File$n_* Bar2/File$n_* Bar3/File$n_* > combined_files/'all_files'$n
done


But I'm stuck.










share|improve this question















I'm trying to concatenate files from three different subdirectories into one file. The names of the files in each subdirectory are exactly the same. I'd like to use a loop to iterate through the subdirectories, and then place an iteration number into the newly named concatenated file, in a new directory.
For example a directory structure like:



Foo
|||
||Bar3
|Bar2
Bar1


Inside each Bar(?) folder are files named:
File1, File2, File3



I'd like to concatenate the Files with the same names to a larger file with a new name including a number:



cat Foo/Bar1/File1 Foo/Bar2/File1 Foo/Bar3/File1 > /combined_files/all_file1

cat Foo/Bar1/File2 Foo/Bar2/File2 Foo/Bar3/File2 > /combined_files/all_file2

cat Foo/Bar1/File3 Foo/Bar2/File3 Foo/Bar3/File3 > /combined_files/all_file3


From the Foo directory I can use:



for number in 1..3
do
cat Bar1/File$number_* Bar2/File$number_* Bar3/File$number_* > combined_files/'all_files'$number
done
exit


But I need to have a more universal script, for larger numbers of Bar directories, and Files.
I want something like



files=`ls ./Run1/ | wc -l` #to count the number of files and assign a number
For n in 1..$files
do
cat Bar1/File$n_* Bar2/File$n_* Bar3/File$n_* > combined_files/'all_files'$n
done


But I'm stuck.







shell cat for






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Dec 10 at 22:16









Kusalananda

121k16227372




121k16227372










asked Dec 10 at 20:56









hoytpr

165




165







  • 1




    Do you want to use all of Barx? Will x always be a single digit?
    – drewbenn
    Dec 10 at 21:00










  • Yes I want all the files in BarX, but no X will go from 1 to a large number.
    – hoytpr
    Dec 10 at 21:29










  • Does order need to be preserved between each run? Eg should Bar1/File2 always appear in the concatenated output before Bar2/File1? This will affect whether it can be accomplished using a simple one-liner, or whether a more complicated (perhaps recursive) result will be required.
    – cryptarch
    Dec 10 at 21:32










  • Unfortunately yes. The files are ordered so the concatenation needs to be ordered.
    – hoytpr
    Dec 10 at 21:34










  • This is similar to a question asked a few months ago, but the answers didn't seem to work for me, or maybe I don't interpret them correctly
    – hoytpr
    Dec 10 at 21:49












  • 1




    Do you want to use all of Barx? Will x always be a single digit?
    – drewbenn
    Dec 10 at 21:00










  • Yes I want all the files in BarX, but no X will go from 1 to a large number.
    – hoytpr
    Dec 10 at 21:29










  • Does order need to be preserved between each run? Eg should Bar1/File2 always appear in the concatenated output before Bar2/File1? This will affect whether it can be accomplished using a simple one-liner, or whether a more complicated (perhaps recursive) result will be required.
    – cryptarch
    Dec 10 at 21:32










  • Unfortunately yes. The files are ordered so the concatenation needs to be ordered.
    – hoytpr
    Dec 10 at 21:34










  • This is similar to a question asked a few months ago, but the answers didn't seem to work for me, or maybe I don't interpret them correctly
    – hoytpr
    Dec 10 at 21:49







1




1




Do you want to use all of Barx? Will x always be a single digit?
– drewbenn
Dec 10 at 21:00




Do you want to use all of Barx? Will x always be a single digit?
– drewbenn
Dec 10 at 21:00












Yes I want all the files in BarX, but no X will go from 1 to a large number.
– hoytpr
Dec 10 at 21:29




Yes I want all the files in BarX, but no X will go from 1 to a large number.
– hoytpr
Dec 10 at 21:29












Does order need to be preserved between each run? Eg should Bar1/File2 always appear in the concatenated output before Bar2/File1? This will affect whether it can be accomplished using a simple one-liner, or whether a more complicated (perhaps recursive) result will be required.
– cryptarch
Dec 10 at 21:32




Does order need to be preserved between each run? Eg should Bar1/File2 always appear in the concatenated output before Bar2/File1? This will affect whether it can be accomplished using a simple one-liner, or whether a more complicated (perhaps recursive) result will be required.
– cryptarch
Dec 10 at 21:32












Unfortunately yes. The files are ordered so the concatenation needs to be ordered.
– hoytpr
Dec 10 at 21:34




Unfortunately yes. The files are ordered so the concatenation needs to be ordered.
– hoytpr
Dec 10 at 21:34












This is similar to a question asked a few months ago, but the answers didn't seem to work for me, or maybe I don't interpret them correctly
– hoytpr
Dec 10 at 21:49




This is similar to a question asked a few months ago, but the answers didn't seem to work for me, or maybe I don't interpret them correctly
– hoytpr
Dec 10 at 21:49










2 Answers
2






active

oldest

votes


















1














#!/bin/sh

for pathname in Foo/Bar1/File*; do
filename=$pathname##*/
cat "$pathname"
"Foo/Bar2/$filename"
"Foo/Bar3/$filename" >"combined/all_$filename"
done


This would loop over all files whose names matches File* under Foo/Bar1 (we assume that the pattern matches exactly the names we are actually interested in).



For each such file, we extract the filename portion of the pathname, yielding $filename (this could also have been done with filename=$(basename "$pathname")). We then concatenate the original file with the corresponding files in the Foo/Bar2 and Foo/Bar3 directories, writing the result to a new all_$filename file in some other directory.




With a bit of error checking:



#!/bin/sh

for pathname in Foo/Bar1/File*; do
if [ ! -f "$pathname" ]; then
printf '% is not a regular file, skippingn' "$pathname" >&2
continue
fi

filename=$pathname##*/

if [ -f "Foo/Bar2/$filename" ] &&
[ -f "Foo/Bar3/$filename" ]
then
cat "$pathname"
"Foo/Bar2/$filename"
"Foo/Bar3/$filename" >"combined/all_$filename"
else
printf 'Missing %s or %sn' "Foo/Bar2/$filename" "Foo/Bar3/$filename" >&2
fi
done



A variation that also allows for a varied number of BarN subdirectories. It is assumed that each BarN directory is numbered sequentially from 1 to some large number.



#!/bin/sh

# This is just used to count the number of BarN subdirectories.
# The number of these will be $#.
set -- Foo/Bar*/

for pathname in Foo/Bar1/File*; do
filename=$pathname##*/

n=1
while [ "$n" -le "$#" ]; do
if [ ! -f "Foo/Bar$n/$filename" ]; then
printf '%s missing, %s will be incompleten'
"Foo/Bar$n/$filename" "combined/all_$filename" >&2
break
fi

cat "Foo/Bar$n/$filename"
n=$(( n + 1 ))
done >"combined/all_$filename"
done





share|improve this answer






















  • Just to make shorter, in your last code what if write cat "Foo/Bar1/$filename" >"combined/all_$filename" above n=1 and then make n=2 instead of 1. So it will save one comparison.
    – Debian_yadav
    Dec 10 at 23:07










  • @Debian_yadav Or just remove the first if statement (which I just did because it looks nicer). In either case, the timing difference will be minuscule.
    – Kusalananda
    Dec 10 at 23:15











  • Thanks @Kusalananda very much for the great explanation and code. My directory structure is not as perfect as I described. The BarX directories will be sequentially numbered, but the Filenames can vary from my original post. Using File* doesn't always capture the filename correctly. Would it be possible to use something from the other post, like: $FILE#*/ to capture the entire filename? I will take the code you have written and work with it tonight. Thank-you.
    – hoytpr
    Dec 10 at 23:28











  • @hoytpr If you want to match all files in Bar1 (that are not hidden filenames), just use Foo/Bar1/* instead of Foo/Bar1/File*, or construct some other pattern that will match the names that you are interested in.
    – Kusalananda
    Dec 12 at 22:35










  • @Kusalananda; yes, that's what I should have been doing, but was overthinking when it was the directory structure using Foo/BarX/Filex when I should have been using just BarX/Filex or as you say; BarX/* when running the script from inside the Foo directory. Thanks!
    – hoytpr
    Dec 13 at 19:56


















0














Thank you again @Kusalananda and @Debian_yadav. I was able to make your script work on my system. My actual directory names are now:



Joes
|||
||Run3
|Run2
Run1



Inside each RunX directory I created Files named the same but with different contents



Run1File1

Run2File1

Run3File1



First I ran the simple script you showed, slightly modified to my directory structure:



ANSWER



#!/bin/bash
for pathname in Run1/File*; do
filename=$pathname##*/
cat "$pathname"
"Run2/$filename"
"Run3/$filename" > "RunCat/all_$filename"
one


The script output was a file "allFile1" with contents:
123



Your longer (final) script (I named K2script.sh) also worked on my system.

The output exactly the same again after slightly modifying the directory structure:



ANSWER



#!/bin/sh
# This is just used to count the number of RunN subdirectories.
# The number of these will be $#.
set -- Joes/Run*/

for pathname in Run1/File*; do
filename=$pathname##*/

n=1
while [ "$n" -le "$#" ]; do
if [ ! -f "Run$n/$filename" ]; then
printf '%s missing, %s will be incompleten'
"Run$n/$filename" "RunCat/all_$filename" >&2
break
fi

cat "Run$n/$filename"
n=$(( n + 1 ))
done >"RunCat/all_$filename"
done


Also, using the script from
an other stackexchange discussion



ANSWER

Changing the folder names made it work on my system.



#!/bin/bash
for FILE in Run1/* ; do
FILE2=Run2/$FILE#*/
FILE3=Run3/$FILE#*/
if [ -f $FILE2 ] ; then
cat $FILE $FILE2 $FILE3 > RunCat/$FILE#*/
fi
done


I've learned a lot about while [ "$n" -le "$#" ]; do and how $pathname##*/ works,
but wasn't able to fully grasp why $FILE#*/ worked when other wildcard symbols or regular expressions would not.






share|improve this answer






















    Your Answer








    StackExchange.ready(function()
    var channelOptions =
    tags: "".split(" "),
    id: "106"
    ;
    initTagRenderer("".split(" "), "".split(" "), channelOptions);

    StackExchange.using("externalEditor", function()
    // Have to fire editor after snippets, if snippets enabled
    if (StackExchange.settings.snippets.snippetsEnabled)
    StackExchange.using("snippets", function()
    createEditor();
    );

    else
    createEditor();

    );

    function createEditor()
    StackExchange.prepareEditor(
    heartbeatType: 'answer',
    autoActivateHeartbeat: false,
    convertImagesToLinks: false,
    noModals: true,
    showLowRepImageUploadWarning: true,
    reputationToPostImages: null,
    bindNavPrevention: true,
    postfix: "",
    imageUploader:
    brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
    contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
    allowUrls: true
    ,
    onDemand: true,
    discardSelector: ".discard-answer"
    ,immediatelyShowMarkdownHelp:true
    );



    );













    draft saved

    draft discarded


















    StackExchange.ready(
    function ()
    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f487207%2fneed-to-iterate-through-subdirectories-concatenating-files-with-an-iterative-n%23new-answer', 'question_page');

    );

    Post as a guest















    Required, but never shown

























    2 Answers
    2






    active

    oldest

    votes








    2 Answers
    2






    active

    oldest

    votes









    active

    oldest

    votes






    active

    oldest

    votes









    1














    #!/bin/sh

    for pathname in Foo/Bar1/File*; do
    filename=$pathname##*/
    cat "$pathname"
    "Foo/Bar2/$filename"
    "Foo/Bar3/$filename" >"combined/all_$filename"
    done


    This would loop over all files whose names matches File* under Foo/Bar1 (we assume that the pattern matches exactly the names we are actually interested in).



    For each such file, we extract the filename portion of the pathname, yielding $filename (this could also have been done with filename=$(basename "$pathname")). We then concatenate the original file with the corresponding files in the Foo/Bar2 and Foo/Bar3 directories, writing the result to a new all_$filename file in some other directory.




    With a bit of error checking:



    #!/bin/sh

    for pathname in Foo/Bar1/File*; do
    if [ ! -f "$pathname" ]; then
    printf '% is not a regular file, skippingn' "$pathname" >&2
    continue
    fi

    filename=$pathname##*/

    if [ -f "Foo/Bar2/$filename" ] &&
    [ -f "Foo/Bar3/$filename" ]
    then
    cat "$pathname"
    "Foo/Bar2/$filename"
    "Foo/Bar3/$filename" >"combined/all_$filename"
    else
    printf 'Missing %s or %sn' "Foo/Bar2/$filename" "Foo/Bar3/$filename" >&2
    fi
    done



    A variation that also allows for a varied number of BarN subdirectories. It is assumed that each BarN directory is numbered sequentially from 1 to some large number.



    #!/bin/sh

    # This is just used to count the number of BarN subdirectories.
    # The number of these will be $#.
    set -- Foo/Bar*/

    for pathname in Foo/Bar1/File*; do
    filename=$pathname##*/

    n=1
    while [ "$n" -le "$#" ]; do
    if [ ! -f "Foo/Bar$n/$filename" ]; then
    printf '%s missing, %s will be incompleten'
    "Foo/Bar$n/$filename" "combined/all_$filename" >&2
    break
    fi

    cat "Foo/Bar$n/$filename"
    n=$(( n + 1 ))
    done >"combined/all_$filename"
    done





    share|improve this answer






















    • Just to make shorter, in your last code what if write cat "Foo/Bar1/$filename" >"combined/all_$filename" above n=1 and then make n=2 instead of 1. So it will save one comparison.
      – Debian_yadav
      Dec 10 at 23:07










    • @Debian_yadav Or just remove the first if statement (which I just did because it looks nicer). In either case, the timing difference will be minuscule.
      – Kusalananda
      Dec 10 at 23:15











    • Thanks @Kusalananda very much for the great explanation and code. My directory structure is not as perfect as I described. The BarX directories will be sequentially numbered, but the Filenames can vary from my original post. Using File* doesn't always capture the filename correctly. Would it be possible to use something from the other post, like: $FILE#*/ to capture the entire filename? I will take the code you have written and work with it tonight. Thank-you.
      – hoytpr
      Dec 10 at 23:28











    • @hoytpr If you want to match all files in Bar1 (that are not hidden filenames), just use Foo/Bar1/* instead of Foo/Bar1/File*, or construct some other pattern that will match the names that you are interested in.
      – Kusalananda
      Dec 12 at 22:35










    • @Kusalananda; yes, that's what I should have been doing, but was overthinking when it was the directory structure using Foo/BarX/Filex when I should have been using just BarX/Filex or as you say; BarX/* when running the script from inside the Foo directory. Thanks!
      – hoytpr
      Dec 13 at 19:56















    1














    #!/bin/sh

    for pathname in Foo/Bar1/File*; do
    filename=$pathname##*/
    cat "$pathname"
    "Foo/Bar2/$filename"
    "Foo/Bar3/$filename" >"combined/all_$filename"
    done


    This would loop over all files whose names matches File* under Foo/Bar1 (we assume that the pattern matches exactly the names we are actually interested in).



    For each such file, we extract the filename portion of the pathname, yielding $filename (this could also have been done with filename=$(basename "$pathname")). We then concatenate the original file with the corresponding files in the Foo/Bar2 and Foo/Bar3 directories, writing the result to a new all_$filename file in some other directory.




    With a bit of error checking:



    #!/bin/sh

    for pathname in Foo/Bar1/File*; do
    if [ ! -f "$pathname" ]; then
    printf '% is not a regular file, skippingn' "$pathname" >&2
    continue
    fi

    filename=$pathname##*/

    if [ -f "Foo/Bar2/$filename" ] &&
    [ -f "Foo/Bar3/$filename" ]
    then
    cat "$pathname"
    "Foo/Bar2/$filename"
    "Foo/Bar3/$filename" >"combined/all_$filename"
    else
    printf 'Missing %s or %sn' "Foo/Bar2/$filename" "Foo/Bar3/$filename" >&2
    fi
    done



    A variation that also allows for a varied number of BarN subdirectories. It is assumed that each BarN directory is numbered sequentially from 1 to some large number.



    #!/bin/sh

    # This is just used to count the number of BarN subdirectories.
    # The number of these will be $#.
    set -- Foo/Bar*/

    for pathname in Foo/Bar1/File*; do
    filename=$pathname##*/

    n=1
    while [ "$n" -le "$#" ]; do
    if [ ! -f "Foo/Bar$n/$filename" ]; then
    printf '%s missing, %s will be incompleten'
    "Foo/Bar$n/$filename" "combined/all_$filename" >&2
    break
    fi

    cat "Foo/Bar$n/$filename"
    n=$(( n + 1 ))
    done >"combined/all_$filename"
    done





    share|improve this answer






















    • Just to make shorter, in your last code what if write cat "Foo/Bar1/$filename" >"combined/all_$filename" above n=1 and then make n=2 instead of 1. So it will save one comparison.
      – Debian_yadav
      Dec 10 at 23:07










    • @Debian_yadav Or just remove the first if statement (which I just did because it looks nicer). In either case, the timing difference will be minuscule.
      – Kusalananda
      Dec 10 at 23:15











    • Thanks @Kusalananda very much for the great explanation and code. My directory structure is not as perfect as I described. The BarX directories will be sequentially numbered, but the Filenames can vary from my original post. Using File* doesn't always capture the filename correctly. Would it be possible to use something from the other post, like: $FILE#*/ to capture the entire filename? I will take the code you have written and work with it tonight. Thank-you.
      – hoytpr
      Dec 10 at 23:28











    • @hoytpr If you want to match all files in Bar1 (that are not hidden filenames), just use Foo/Bar1/* instead of Foo/Bar1/File*, or construct some other pattern that will match the names that you are interested in.
      – Kusalananda
      Dec 12 at 22:35










    • @Kusalananda; yes, that's what I should have been doing, but was overthinking when it was the directory structure using Foo/BarX/Filex when I should have been using just BarX/Filex or as you say; BarX/* when running the script from inside the Foo directory. Thanks!
      – hoytpr
      Dec 13 at 19:56













    1












    1








    1






    #!/bin/sh

    for pathname in Foo/Bar1/File*; do
    filename=$pathname##*/
    cat "$pathname"
    "Foo/Bar2/$filename"
    "Foo/Bar3/$filename" >"combined/all_$filename"
    done


    This would loop over all files whose names matches File* under Foo/Bar1 (we assume that the pattern matches exactly the names we are actually interested in).



    For each such file, we extract the filename portion of the pathname, yielding $filename (this could also have been done with filename=$(basename "$pathname")). We then concatenate the original file with the corresponding files in the Foo/Bar2 and Foo/Bar3 directories, writing the result to a new all_$filename file in some other directory.




    With a bit of error checking:



    #!/bin/sh

    for pathname in Foo/Bar1/File*; do
    if [ ! -f "$pathname" ]; then
    printf '% is not a regular file, skippingn' "$pathname" >&2
    continue
    fi

    filename=$pathname##*/

    if [ -f "Foo/Bar2/$filename" ] &&
    [ -f "Foo/Bar3/$filename" ]
    then
    cat "$pathname"
    "Foo/Bar2/$filename"
    "Foo/Bar3/$filename" >"combined/all_$filename"
    else
    printf 'Missing %s or %sn' "Foo/Bar2/$filename" "Foo/Bar3/$filename" >&2
    fi
    done



    A variation that also allows for a varied number of BarN subdirectories. It is assumed that each BarN directory is numbered sequentially from 1 to some large number.



    #!/bin/sh

    # This is just used to count the number of BarN subdirectories.
    # The number of these will be $#.
    set -- Foo/Bar*/

    for pathname in Foo/Bar1/File*; do
    filename=$pathname##*/

    n=1
    while [ "$n" -le "$#" ]; do
    if [ ! -f "Foo/Bar$n/$filename" ]; then
    printf '%s missing, %s will be incompleten'
    "Foo/Bar$n/$filename" "combined/all_$filename" >&2
    break
    fi

    cat "Foo/Bar$n/$filename"
    n=$(( n + 1 ))
    done >"combined/all_$filename"
    done





    share|improve this answer














    #!/bin/sh

    for pathname in Foo/Bar1/File*; do
    filename=$pathname##*/
    cat "$pathname"
    "Foo/Bar2/$filename"
    "Foo/Bar3/$filename" >"combined/all_$filename"
    done


    This would loop over all files whose names matches File* under Foo/Bar1 (we assume that the pattern matches exactly the names we are actually interested in).



    For each such file, we extract the filename portion of the pathname, yielding $filename (this could also have been done with filename=$(basename "$pathname")). We then concatenate the original file with the corresponding files in the Foo/Bar2 and Foo/Bar3 directories, writing the result to a new all_$filename file in some other directory.




    With a bit of error checking:



    #!/bin/sh

    for pathname in Foo/Bar1/File*; do
    if [ ! -f "$pathname" ]; then
    printf '% is not a regular file, skippingn' "$pathname" >&2
    continue
    fi

    filename=$pathname##*/

    if [ -f "Foo/Bar2/$filename" ] &&
    [ -f "Foo/Bar3/$filename" ]
    then
    cat "$pathname"
    "Foo/Bar2/$filename"
    "Foo/Bar3/$filename" >"combined/all_$filename"
    else
    printf 'Missing %s or %sn' "Foo/Bar2/$filename" "Foo/Bar3/$filename" >&2
    fi
    done



    A variation that also allows for a varied number of BarN subdirectories. It is assumed that each BarN directory is numbered sequentially from 1 to some large number.



    #!/bin/sh

    # This is just used to count the number of BarN subdirectories.
    # The number of these will be $#.
    set -- Foo/Bar*/

    for pathname in Foo/Bar1/File*; do
    filename=$pathname##*/

    n=1
    while [ "$n" -le "$#" ]; do
    if [ ! -f "Foo/Bar$n/$filename" ]; then
    printf '%s missing, %s will be incompleten'
    "Foo/Bar$n/$filename" "combined/all_$filename" >&2
    break
    fi

    cat "Foo/Bar$n/$filename"
    n=$(( n + 1 ))
    done >"combined/all_$filename"
    done






    share|improve this answer














    share|improve this answer



    share|improve this answer








    edited Dec 10 at 23:18

























    answered Dec 10 at 22:26









    Kusalananda

    121k16227372




    121k16227372











    • Just to make shorter, in your last code what if write cat "Foo/Bar1/$filename" >"combined/all_$filename" above n=1 and then make n=2 instead of 1. So it will save one comparison.
      – Debian_yadav
      Dec 10 at 23:07










    • @Debian_yadav Or just remove the first if statement (which I just did because it looks nicer). In either case, the timing difference will be minuscule.
      – Kusalananda
      Dec 10 at 23:15











    • Thanks @Kusalananda very much for the great explanation and code. My directory structure is not as perfect as I described. The BarX directories will be sequentially numbered, but the Filenames can vary from my original post. Using File* doesn't always capture the filename correctly. Would it be possible to use something from the other post, like: $FILE#*/ to capture the entire filename? I will take the code you have written and work with it tonight. Thank-you.
      – hoytpr
      Dec 10 at 23:28











    • @hoytpr If you want to match all files in Bar1 (that are not hidden filenames), just use Foo/Bar1/* instead of Foo/Bar1/File*, or construct some other pattern that will match the names that you are interested in.
      – Kusalananda
      Dec 12 at 22:35










    • @Kusalananda; yes, that's what I should have been doing, but was overthinking when it was the directory structure using Foo/BarX/Filex when I should have been using just BarX/Filex or as you say; BarX/* when running the script from inside the Foo directory. Thanks!
      – hoytpr
      Dec 13 at 19:56
















    • Just to make shorter, in your last code what if write cat "Foo/Bar1/$filename" >"combined/all_$filename" above n=1 and then make n=2 instead of 1. So it will save one comparison.
      – Debian_yadav
      Dec 10 at 23:07










    • @Debian_yadav Or just remove the first if statement (which I just did because it looks nicer). In either case, the timing difference will be minuscule.
      – Kusalananda
      Dec 10 at 23:15











    • Thanks @Kusalananda very much for the great explanation and code. My directory structure is not as perfect as I described. The BarX directories will be sequentially numbered, but the Filenames can vary from my original post. Using File* doesn't always capture the filename correctly. Would it be possible to use something from the other post, like: $FILE#*/ to capture the entire filename? I will take the code you have written and work with it tonight. Thank-you.
      – hoytpr
      Dec 10 at 23:28











    • @hoytpr If you want to match all files in Bar1 (that are not hidden filenames), just use Foo/Bar1/* instead of Foo/Bar1/File*, or construct some other pattern that will match the names that you are interested in.
      – Kusalananda
      Dec 12 at 22:35










    • @Kusalananda; yes, that's what I should have been doing, but was overthinking when it was the directory structure using Foo/BarX/Filex when I should have been using just BarX/Filex or as you say; BarX/* when running the script from inside the Foo directory. Thanks!
      – hoytpr
      Dec 13 at 19:56















    Just to make shorter, in your last code what if write cat "Foo/Bar1/$filename" >"combined/all_$filename" above n=1 and then make n=2 instead of 1. So it will save one comparison.
    – Debian_yadav
    Dec 10 at 23:07




    Just to make shorter, in your last code what if write cat "Foo/Bar1/$filename" >"combined/all_$filename" above n=1 and then make n=2 instead of 1. So it will save one comparison.
    – Debian_yadav
    Dec 10 at 23:07












    @Debian_yadav Or just remove the first if statement (which I just did because it looks nicer). In either case, the timing difference will be minuscule.
    – Kusalananda
    Dec 10 at 23:15





    @Debian_yadav Or just remove the first if statement (which I just did because it looks nicer). In either case, the timing difference will be minuscule.
    – Kusalananda
    Dec 10 at 23:15













    Thanks @Kusalananda very much for the great explanation and code. My directory structure is not as perfect as I described. The BarX directories will be sequentially numbered, but the Filenames can vary from my original post. Using File* doesn't always capture the filename correctly. Would it be possible to use something from the other post, like: $FILE#*/ to capture the entire filename? I will take the code you have written and work with it tonight. Thank-you.
    – hoytpr
    Dec 10 at 23:28





    Thanks @Kusalananda very much for the great explanation and code. My directory structure is not as perfect as I described. The BarX directories will be sequentially numbered, but the Filenames can vary from my original post. Using File* doesn't always capture the filename correctly. Would it be possible to use something from the other post, like: $FILE#*/ to capture the entire filename? I will take the code you have written and work with it tonight. Thank-you.
    – hoytpr
    Dec 10 at 23:28













    @hoytpr If you want to match all files in Bar1 (that are not hidden filenames), just use Foo/Bar1/* instead of Foo/Bar1/File*, or construct some other pattern that will match the names that you are interested in.
    – Kusalananda
    Dec 12 at 22:35




    @hoytpr If you want to match all files in Bar1 (that are not hidden filenames), just use Foo/Bar1/* instead of Foo/Bar1/File*, or construct some other pattern that will match the names that you are interested in.
    – Kusalananda
    Dec 12 at 22:35












    @Kusalananda; yes, that's what I should have been doing, but was overthinking when it was the directory structure using Foo/BarX/Filex when I should have been using just BarX/Filex or as you say; BarX/* when running the script from inside the Foo directory. Thanks!
    – hoytpr
    Dec 13 at 19:56




    @Kusalananda; yes, that's what I should have been doing, but was overthinking when it was the directory structure using Foo/BarX/Filex when I should have been using just BarX/Filex or as you say; BarX/* when running the script from inside the Foo directory. Thanks!
    – hoytpr
    Dec 13 at 19:56













    0














    Thank you again @Kusalananda and @Debian_yadav. I was able to make your script work on my system. My actual directory names are now:



    Joes
    |||
    ||Run3
    |Run2
    Run1



    Inside each RunX directory I created Files named the same but with different contents



    Run1File1

    Run2File1

    Run3File1



    First I ran the simple script you showed, slightly modified to my directory structure:



    ANSWER



    #!/bin/bash
    for pathname in Run1/File*; do
    filename=$pathname##*/
    cat "$pathname"
    "Run2/$filename"
    "Run3/$filename" > "RunCat/all_$filename"
    one


    The script output was a file "allFile1" with contents:
    123



    Your longer (final) script (I named K2script.sh) also worked on my system.

    The output exactly the same again after slightly modifying the directory structure:



    ANSWER



    #!/bin/sh
    # This is just used to count the number of RunN subdirectories.
    # The number of these will be $#.
    set -- Joes/Run*/

    for pathname in Run1/File*; do
    filename=$pathname##*/

    n=1
    while [ "$n" -le "$#" ]; do
    if [ ! -f "Run$n/$filename" ]; then
    printf '%s missing, %s will be incompleten'
    "Run$n/$filename" "RunCat/all_$filename" >&2
    break
    fi

    cat "Run$n/$filename"
    n=$(( n + 1 ))
    done >"RunCat/all_$filename"
    done


    Also, using the script from
    an other stackexchange discussion



    ANSWER

    Changing the folder names made it work on my system.



    #!/bin/bash
    for FILE in Run1/* ; do
    FILE2=Run2/$FILE#*/
    FILE3=Run3/$FILE#*/
    if [ -f $FILE2 ] ; then
    cat $FILE $FILE2 $FILE3 > RunCat/$FILE#*/
    fi
    done


    I've learned a lot about while [ "$n" -le "$#" ]; do and how $pathname##*/ works,
    but wasn't able to fully grasp why $FILE#*/ worked when other wildcard symbols or regular expressions would not.






    share|improve this answer



























      0














      Thank you again @Kusalananda and @Debian_yadav. I was able to make your script work on my system. My actual directory names are now:



      Joes
      |||
      ||Run3
      |Run2
      Run1



      Inside each RunX directory I created Files named the same but with different contents



      Run1File1

      Run2File1

      Run3File1



      First I ran the simple script you showed, slightly modified to my directory structure:



      ANSWER



      #!/bin/bash
      for pathname in Run1/File*; do
      filename=$pathname##*/
      cat "$pathname"
      "Run2/$filename"
      "Run3/$filename" > "RunCat/all_$filename"
      one


      The script output was a file "allFile1" with contents:
      123



      Your longer (final) script (I named K2script.sh) also worked on my system.

      The output exactly the same again after slightly modifying the directory structure:



      ANSWER



      #!/bin/sh
      # This is just used to count the number of RunN subdirectories.
      # The number of these will be $#.
      set -- Joes/Run*/

      for pathname in Run1/File*; do
      filename=$pathname##*/

      n=1
      while [ "$n" -le "$#" ]; do
      if [ ! -f "Run$n/$filename" ]; then
      printf '%s missing, %s will be incompleten'
      "Run$n/$filename" "RunCat/all_$filename" >&2
      break
      fi

      cat "Run$n/$filename"
      n=$(( n + 1 ))
      done >"RunCat/all_$filename"
      done


      Also, using the script from
      an other stackexchange discussion



      ANSWER

      Changing the folder names made it work on my system.



      #!/bin/bash
      for FILE in Run1/* ; do
      FILE2=Run2/$FILE#*/
      FILE3=Run3/$FILE#*/
      if [ -f $FILE2 ] ; then
      cat $FILE $FILE2 $FILE3 > RunCat/$FILE#*/
      fi
      done


      I've learned a lot about while [ "$n" -le "$#" ]; do and how $pathname##*/ works,
      but wasn't able to fully grasp why $FILE#*/ worked when other wildcard symbols or regular expressions would not.






      share|improve this answer

























        0












        0








        0






        Thank you again @Kusalananda and @Debian_yadav. I was able to make your script work on my system. My actual directory names are now:



        Joes
        |||
        ||Run3
        |Run2
        Run1



        Inside each RunX directory I created Files named the same but with different contents



        Run1File1

        Run2File1

        Run3File1



        First I ran the simple script you showed, slightly modified to my directory structure:



        ANSWER



        #!/bin/bash
        for pathname in Run1/File*; do
        filename=$pathname##*/
        cat "$pathname"
        "Run2/$filename"
        "Run3/$filename" > "RunCat/all_$filename"
        one


        The script output was a file "allFile1" with contents:
        123



        Your longer (final) script (I named K2script.sh) also worked on my system.

        The output exactly the same again after slightly modifying the directory structure:



        ANSWER



        #!/bin/sh
        # This is just used to count the number of RunN subdirectories.
        # The number of these will be $#.
        set -- Joes/Run*/

        for pathname in Run1/File*; do
        filename=$pathname##*/

        n=1
        while [ "$n" -le "$#" ]; do
        if [ ! -f "Run$n/$filename" ]; then
        printf '%s missing, %s will be incompleten'
        "Run$n/$filename" "RunCat/all_$filename" >&2
        break
        fi

        cat "Run$n/$filename"
        n=$(( n + 1 ))
        done >"RunCat/all_$filename"
        done


        Also, using the script from
        an other stackexchange discussion



        ANSWER

        Changing the folder names made it work on my system.



        #!/bin/bash
        for FILE in Run1/* ; do
        FILE2=Run2/$FILE#*/
        FILE3=Run3/$FILE#*/
        if [ -f $FILE2 ] ; then
        cat $FILE $FILE2 $FILE3 > RunCat/$FILE#*/
        fi
        done


        I've learned a lot about while [ "$n" -le "$#" ]; do and how $pathname##*/ works,
        but wasn't able to fully grasp why $FILE#*/ worked when other wildcard symbols or regular expressions would not.






        share|improve this answer














        Thank you again @Kusalananda and @Debian_yadav. I was able to make your script work on my system. My actual directory names are now:



        Joes
        |||
        ||Run3
        |Run2
        Run1



        Inside each RunX directory I created Files named the same but with different contents



        Run1File1

        Run2File1

        Run3File1



        First I ran the simple script you showed, slightly modified to my directory structure:



        ANSWER



        #!/bin/bash
        for pathname in Run1/File*; do
        filename=$pathname##*/
        cat "$pathname"
        "Run2/$filename"
        "Run3/$filename" > "RunCat/all_$filename"
        one


        The script output was a file "allFile1" with contents:
        123



        Your longer (final) script (I named K2script.sh) also worked on my system.

        The output exactly the same again after slightly modifying the directory structure:



        ANSWER



        #!/bin/sh
        # This is just used to count the number of RunN subdirectories.
        # The number of these will be $#.
        set -- Joes/Run*/

        for pathname in Run1/File*; do
        filename=$pathname##*/

        n=1
        while [ "$n" -le "$#" ]; do
        if [ ! -f "Run$n/$filename" ]; then
        printf '%s missing, %s will be incompleten'
        "Run$n/$filename" "RunCat/all_$filename" >&2
        break
        fi

        cat "Run$n/$filename"
        n=$(( n + 1 ))
        done >"RunCat/all_$filename"
        done


        Also, using the script from
        an other stackexchange discussion



        ANSWER

        Changing the folder names made it work on my system.



        #!/bin/bash
        for FILE in Run1/* ; do
        FILE2=Run2/$FILE#*/
        FILE3=Run3/$FILE#*/
        if [ -f $FILE2 ] ; then
        cat $FILE $FILE2 $FILE3 > RunCat/$FILE#*/
        fi
        done


        I've learned a lot about while [ "$n" -le "$#" ]; do and how $pathname##*/ works,
        but wasn't able to fully grasp why $FILE#*/ worked when other wildcard symbols or regular expressions would not.







        share|improve this answer














        share|improve this answer



        share|improve this answer








        edited Dec 12 at 22:27









        Rui F Ribeiro

        38.8k1479128




        38.8k1479128










        answered Dec 11 at 23:26









        hoytpr

        165




        165



























            draft saved

            draft discarded
















































            Thanks for contributing an answer to Unix & Linux Stack Exchange!


            • Please be sure to answer the question. Provide details and share your research!

            But avoid


            • Asking for help, clarification, or responding to other answers.

            • Making statements based on opinion; back them up with references or personal experience.

            To learn more, see our tips on writing great answers.





            Some of your past answers have not been well-received, and you're in danger of being blocked from answering.


            Please pay close attention to the following guidance:


            • Please be sure to answer the question. Provide details and share your research!

            But avoid


            • Asking for help, clarification, or responding to other answers.

            • Making statements based on opinion; back them up with references or personal experience.

            To learn more, see our tips on writing great answers.




            draft saved


            draft discarded














            StackExchange.ready(
            function ()
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f487207%2fneed-to-iterate-through-subdirectories-concatenating-files-with-an-iterative-n%23new-answer', 'question_page');

            );

            Post as a guest















            Required, but never shown





















































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown

































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown






            Popular posts from this blog

            How to check contact read email or not when send email to Individual?

            Bahrain

            Postfix configuration issue with fips on centos 7; mailgun relay