mjhjmtu

I have lots of very large files that take a long time to join manually. They look something like this:

• file_a1.txt


• file_a2.txt


• file_a3.txt


• file_b1.txt


• file_b2.txt


• file_b3.txt


• file_c1.txt


• file_c2.txt


• file_c3.txt
  and so on. How do I join all the a's together, all the b's together and then all the c's using a loop ?

I'm assuming that by "joining" you mean concatenating, and not "performing a relational JOIN operation between them using the join command". Furthermore I assume that the filenames follow the pattern file_XN.txt where X is some single letter and N is some number and that all files are located in the current directory.

Concatenation of files is done using the cat command, so you would have to call cat with all the "a-files", then the "b-files", etc., while writing the output to some appropriately named file.

for filename in ./file_[a-z]*.txt; do
 # extract the letter
 letter=$filename#*_ # ./file_XN.txt --> XN.txt
 letter=$letter%[0-9]* # XN.txt --> X

 cat "$filename" >>"combined_$letter.txt"
done

This would loop over all files and extract the letter from the filename. The letter would then be used to construct the output filename and the contents of the current file would be appended to that output file.

The data would be appended in the same order as the files would be listed in the directory by default.

Another shorter way of doing it if you're using bash and know exactly the letters you want to go through (using a through to t here):

for letter in a..t; do
 cat "file_$letter"*.txt >"combined_$letter.txt"
done

Here, instead of looping over the filenames, we loop over the letters and for each letter we concatenate all files for that letter in one go.