mjhjmtu

My Linux directory contains a dump of files and they look like:

EDW_Infile_ABC_Daily_Activity_20190204.csv
EDW_Infile_ABC_Daily_Activity.zip
EDW_Infile_PQRInc_Daily_Activity_20190204.csv
EDW_Infile_PQRInc_Daily_Activity_zip
EDW_Infile_ABC_Daily_Payment_20190204.csv
EDW_Infile_PQRInc_Daily_Payment_20190204.csv
EDW_Infile_ABC_Daily_Status_20190204.csv
EDW_Infile_PQRInc_Daily_Status_20190204.csv

These files follow few common name patterns such as

EDW_Infile_*<3 to 8 bytes company name>*_Daily_Activity_*YYYYMMDD*.csv
EDW_Infile_*<3 to 8 bytes company name>*_Daily_Payment_*YYYYMMDD*.csv
EDW_Infile_*<3 to 8 bytes company name>*_Daily_Status_*YYYYMMDD*.csv

How can I -

1) Find all files for all customers, for all dates, which follow the pattern EDW_Infile_3 to 8 bytes any name_Daily_Activity_Any Date.csv

2) Each file contains a header. How can I combine all of them into one file and have only one header

I pushed my zsh knowledge a bit in order to answer more specifically, in case you weren't in control of the filenames and had files named like EDQ_Infile_some uninteresting stuff here_Daily_Activity_junk here.csv and so didn't want to use a * wildcard.

To gather the list of filenames ...

which follow the pattern EDW_Infile_3 to 8 bytes any name_Daily_Activity_Any Date.csv

I would set up this extended_glob pattern in zsh (don't type the $ -- that's the shell prompt):

$ set -o extended_glob
$ files=(EDW_Infile_?(#c3,8)_Daily_Activity_[[:digit:]](#c8).csv)

The pattern, apart from the plain text, is:

• 
  ? -- any (single) character


• 
  (#c3,8) -- require between three and eight characters, inclusive


• 
  [[:digit:]] -- require a digit


• 
  (#c8) -- require eight of them

See the list with:

$ print -l $files
EDW_Infile_ABC_Daily_Activity_20190204.csv
EDW_Infile_PQRInc_Daily_Activity_20190204.csv

To then ...

combine all of them into one file and have only one header

 head -1 "$files[1]"; for f in $files; do sed 1d "$f"; done; > output.csv

This groups two commands and redirects their output to output.csv. The first command, head, takes the first line from the first file in the array; the second command then loops through all of the files and deletes the first line (default-printing the remainder to stdout).

You might want something like this

# collect all the "EDW_Infile_ABC" prefixes
declare -A prefix
for f in EDQ_Infile_*_Daily_Activity_*.csv; do
 p=$f%_*.csv
 prefix[$p]=1
done

for p in "$!prefixes[@]"; do
 awk 'NR==1 print FNR==1next print' "$p"_*.csv > "$p"_all.csv
 zip "$p".zip "$p"_all.csv
 rm "$p"_all.csv
done

For bash, requires version 4 for associative arrays. Otherwise, we can work with positional parameters.