mjhjmtu

I have a list of directories and subdirectories that contain large csv files. There are about 500 million lines in these files, each is a record. I would like to know

1. How many lines are in each file.


2. How many lines are in directory.


3. How many lines in total

Most importantly, I need this in 'human readable format' eg. 12,345,678 rather than 12345678

It would be nice to learn how to do this in 3 ways. Plain vanilla bash tools, awk etc., and perl (or python).

How many lines are in each file.

Use wc, originally for word count, I believe, but it can do lines, words, characters, bytes, and the longest line length. The -l option tells it to count lines.

wc -l <filename>

This will output the number of lines in :

$ wc -l /dir/file.txt
32724 /dir/file.txt

You can also pipe data to wc as well:

$ cat /dir/file.txt | wc -l
32724
$ curl google.com --silent | wc -l
63

How many lines are in directory.

Try:

find . -name '*.pl' | xargs wc -l

another one-liner:

( find ./ -name '*.pl' -print0 | xargs -0 cat ) | wc -l

BTW, wc command counts new lines codes, not lines. When last line in the file does not end with new line code, this will not counted.

You may use grep -c ^ , full example:

#this example prints line count for all found files
total=0
find /path -type f -name "*.php" | while read FILE; do
 #you see use grep instead wc ! for properly counting
 count=$(grep -c ^ < "$FILE")
 echo "$FILE has $count lines"
 let total=total+count #in bash, you can convert this for another shell
done
echo TOTAL LINES COUNTED: $total

How many lines in total

Not sure that I understood you request correctly. e.g. this will output results in the following format, showing the number of lines for each file:

# wc -l `find /path/to/directory/ -type f`
 103 /dir/a.php
 378 /dir/b/c.xml
 132 /dir/d/e.xml
 613 total

Alternatively, to output just the total number of new line characters without the file by file counts to following command can prove useful:

# find /path/to/directory/ -type f -exec wc -l ; | awk 'total += $1 ENDprint total'
 613

Most importantly, I need this in 'human readable format' eg.
12,345,678 rather than 12345678

Bash has a printf function built in:

printf "%0.2fn" $T

As always, there are many different methods that could be used to achieve the same results mentioned here.

In many cases combining the wc command and the wildcard * may be enough.

If all your files are in a single directory you can call:

wc -l src/*

You can also list several files and directories:

wc -l file.txt readme src/* include/*

This command will show a list of the files and their number of lines.

The last line will be the sum of the lines from all files.

To count all files in a directory recursively:

First, enable globstar by adding shopt -s globstar to your .bash_profile. Support for globstar requires Bash ≥ 4.x which can be installed with brew install bash if needed. You can check your version with bash --version.

Then run:

wc -l **/*

Note that this output will be incorrect if globstar is not enabled.

a bit late to the game, but I got a bunch of argument errors with the above due to the size of the dir. This worked for me:

for i in $(find . -type f); do wc -l $i; done >> /home/counts.txt

This command will give list of lines code in each directory:

find . -name '*.*' -type f | xargs wc -l