I want to split sourcefile.txt which contains 10000 lines, (increasing everyday) into 30 equal files. I have directories called prog1 to prog30 and I would like to save split the file into these directories with the same filename. For example /prog1/myfile.txt, /prog2/myfile.txt to /prog30/myfile.txt.

Here is my bash script called divide.sh runs in prog directory

#!/bin/bash
programpath=/home/mywebsite/project/a1/
array=/prog1/
totalline=$(wc -l < ./sourcefile.txt) 
divide="$(( $totalline / 30 ))" 
split --lines=$divide $./prog1/myfile.txt 
exit 1
fi

Sed version for fun:

lines=$(wc -l <sourcefile.txt)
perfile=$(( (lines+29)/30 )) # see https://www.rfc-editor.org/rfc/rfc968.txt
last=0
sed -nf- sourcefile.txt <<EOD
$(while let $((last<lines)); do 
 mkdir -p prog$((last/perfile+1))
 echo $((last+1)),$((last+perfile)) w prog$((last/perfile+1))/myfile.txt
 : $((last+=perfile))
 done)
EOD

#!/bin/bash

# assuming the file is in the same folder as the script
INPUT=large_file.txt
# assuming the folder called "output" is in the same folder
# as the script and there are folders that have the patter
# prog01 prog02 ... prog30
# create that with mkdir output/prog01..30 
OUTPUT_FOLDER=output

OUTPUT_FILE_FORMAT=myfile

# split 
# -n -> 30 files
# $OUTPUT_FILE_FORMAT -> should start with this pattern
# --numeric-suffixes=1 -> end of file name should start from 01 
split -n 30 $INPUT $OUTPUT_FILE_FORMAT --numeric-suffixes=1

# move all files to their repective directories
for i in 01..30 
do
 mv $OUTPUT_FILE_FORMAT$i $OUTPUT_FOLDER/prog$i/myfile.txt
done

echo "done :)"

exit

The split command is more than enough for this task. However the solution here requires you to make your folder names start from prog01 and not prog1

The awk only solution (N here equals 30 files):

awk 'BEGIN cmd="wc -l <sourcefile.txt"; cmd 
 NR%l==1trgt=sprintf("prog%d",((++c)))print >trgt"/myfile.txt"' sourcefile.txt

Or let shell run and return the number of lines in sourcefile.txt and pass to awk as suggested by jthill.

awk 'NR%l==1trgt=sprintf("prog%d",((++c)))print >trgt"/myfile.txt"' 
 l=$(( ($(wc -l <sourcefile.txt)+29)/30 )) sourcefile.txt

split + bash solution:

lines=$(echo "t=$(wc -l ./sourcefile.txt | cut -d' ' -f1); d=30; if(t%d) t/d+1 else t/d" | bc)
split -l $lines ./sourcefile.txt "myfile.txt" --numeric-suffixes=1

for f in myfile.txt[0-9]*; do 
 dir_n="prog"$(printf "%d" "$f#*txt") # constructing directory name
 mv "$f" "$dir_n/myfile.txt"
done

Assuming that you already have folders called prog1 to prog30 (as you mentioned)

• 
  

  lines - contains the integer number of lines per output file

  
  
  
  • 
    t - total number of lines of file ./sourcefile.txt
    
  
  
  • 
    d=30 is a divider
  
  



• --numeric-suffixes=1 - split's option, tells to use numeric suffixes starting at 1

Steps

1. count the lines in file and divide by 30
   lines = cat $file | wc -l




2. get the amount of files you need (bash will round it up to an integer)
   numOfFiles = $lines / 30




3. use split to divide the file
   split -l $lines -d --additional-suffix=-filename.extension $file

Expected result

x01-filename.extension, x02-filename.extension... xN-filename.extension

Wrap it into a for loop to process more than one file at a time

#!/bin/bash 
for FILE in $(find $pathToWorkingDir -type f -name "filename.extension")
do
 split -l $lines -d --additional-suffix=-filename.extension $file
 if [ $? -eq 0 ]; then
 echo "$file splitted file correctly"
 else
 echo "there was a problem splitting $file"
 exit 1 #we exit with an error code
 fi
done
exit 0 #if all processed fine we exit with a success code

Parallelized with GNU Parallel:

parallel -j30 -a sourcefile.txt --pipepart --block -1 cat '>'prog#/myfile.txt

This will run 30 jobs in parallel, splitting sourcefile.txt into one part per job (i.e. 30) and give the parts to cat that saves into progjobnumber/myfile.txt.