mjhjmtu

I have several folders each containing several .nc files. I want to loop through all folders and for each folder do some calculations (mergetime) using cdo on files that have similar pattern. So far I have written the following:

for dir in /mnt/meteo_a/africa_cordex/historical/0.44/pr/*/ 
do 
 dir=$dir%*/
 echo $dir##*/ 
 export dir
 cd $dir
 pwd
 for f in `find . -type f -regex /(.*?day)/*`
 cdo mergetiem io
 done
done

The files in each folder have names like the following:

pr_AFR-44_CNRM-CERFACS-CNRM-CM5_historical_r1i1p1_CLMcom-CCLM4-8-17_v1_day_19500101-19501231.nc
pr_AFR-44_CNRM-CERFACS-CNRM-CM5_historical_r1i1p1_CLMcom-CCLM4-8-17_v1_day_19510101-19551231.nc
pr_AFR-44_ICHEC-EC-EARTH_historical_r12i1p1_CLMcom-CCLM4-8-17_v1_day_19491201-19501231.nc
pr_AFR-44_ICHEC-EC-EARTH_historical_r12i1p1_CLMcom-CCLM4-8-17_v1_day_19510101-19551231.nc

I want to apply the mergetime CDO command on all files that have similar pattern before _day_.

I want to merge the first two files and the last two files. More specifically, merge files that start with

"pr_AFR-44_CNRM-CERFACS-CNRM-CM5_historical_r1i1p1_CLMcom-CCLM4-8-17_v1_" 

and likewise merge files that start with

"pr_AFR-44_ICHEC-EC-EARTH_historical_r12i1p1_CLMcom-CCLM4-8-17_v1"

The output of the mergetime can be something like:

 pattern_mergetime.nc 

This script snippet would work with your example files

for file in *.nc
do 
 echo "$file"
done | sed 's/_day_.*//' | sort -u | while read -r pattern
do
 cdo mergetime "$pattern"* "$pattern_mergetime.nc"
done

Explanation:

• The for loop with echo prints one file name per line.


• The sed command removes _day_ and all following characters.


• 
  sort -u sorts the partial file names and removes duplicates.


• 
  while read -r pattern reads one pattern per line and loops over the patterns


• 
  "$pattern"* is expanded by the shell to all file names starting with the pattern

Better than the for loop with echo might be

find . -maxdepth 1 -type f -name '*.nc'

This will print all file names matching *.nc in the current directory without subdirectories.

You can combine this with a loop over all subdirectories similar to the script in the question

for dir in /mnt/meteo_a/africa_cordex/historical/0.44/pr/*/ 
do 
 dir=$dir%*/
 echo $dir##*/ 
 export dir
 pushd $dir
 pwd

 find . -maxdepth 1 -type f -name '*.nc' | sed 's/_day_.*//' | sort -u | while read -r pattern
 do
 cdo mergetime "$pattern"* "$pattern_mergetime.nc"
 done

 popd
done

Instead of cd I propose pushd to allow going back with popd later.

You could also replace the for loop over the directories with an additional find

find /mnt/meteo_a/africa_cordex/historical/0.44/pr -maxdepth 1 -mindepth 1 -type d | while read dir
do
 pushd "$dir"

 find . -maxdepth 1 -type f -name '*.nc' | sed 's/_day_.*//' | sort -u | while read -r pattern
 do
 cdo mergetime "$pattern"* "$pattern_mergetime.nc"
 done

 popd
done