mjhjmtu

I am writing a PHP script to parse a large text file to do database inserts from it. However on my host, the file is too large, and I hit the memory limit for PHP.

The file has about 16,000 lines; I want to split it up into four separate files (at first) to see if I can load those.

The first part I can get with head -4000 file.txt. The middle sections are slightly trickier -- I was thinking about piping tail output into head ( tail -4001 file.txt | head -4000 > section2.txt ), but is there another/better way?

Actually my logic is messed up -- for section two, I would need to so something like tail -12001 file.txt | head - 4000, and then lower the tail argument for the next sections. I'm getting mixed up already! :P

If you want not to get messed up but still do it using tail and head, there is a useful way of invoking tail using a line-count from the beginning, not the end:

tail -n +4001 yourfile | head -4000

... But a better, automatic tool made just for splitting files is called... split! It's also a part of GNU coreutils, so any normal Linux system should have it. Here's how you can use it:

split -l 4000 yourInputFile thePrefixForOutputFiles

(See man split if in doubt.)

Combining head and tail as you did will work, but for this I would use sed

sed -n '1,4000p' input_file # print lines 1-4000 of input_file

This lets you solve your problem with a quick shell function

chunk_it()
 step=4
 start=1
 end=$step
 for n in 1..4 ; do
 sed -n "$start,$endp" "$1" > "$1".$start-$end
 let start+=$step
 let end+=$step
 done


chunk_it your_file

Now you have your_file.1-4000 and yuor_file.4001-8000 and so on.

Note: requires bash