mjhjmtu

I have 100 million rows in my file.

Each row has only one column.

e.g.

aaaaa
bb
cc
ddddddd
ee

I would like to list the character count

Like this

2 character words - 3
5 character words - 1
7 character words - 1

etc.

Is there any easy way to do this in terminal?

Another way to do it all with awk alone

$ awk 'words[length()]++ ENDfor(k in words)print k " character words - " words[k]' ip.txt 
2 character words - 3
5 character words - 1
7 character words - 1

• 
  words[length()]++ use length of input line as key to save count


• 
  ENDfor(k in words)print k " character words - " words[k] after all lines are processed, print contents of array in desired format

Performance comparison, numbers selected are best of two runs

$ wc words.txt
 71813 71813 655873 words.txt
$ perl -0777 -ne 'print $_ x 1000' words.txt > long_file.txt
$ du -h --apparent-size long_file.txt
626M long_file.txt

$ time awk 'words[length()]++ ENDfor(k in words)print k " character words - " words[k]' long_file.txt > t1

real 0m20.632s
user 0m20.464s
sys 0m0.108s

$ time perl -lne '$hlength($_)++  
 show 5 more comments
 

Here's a perl equivalent (with - optional - sort):

$ perl -lne '
 $hlength($_)++ { for $n (sort keys %h) print "$n character words - $h$n"
' file
2 character words - 3
5 character words - 1
7 character words - 1

An alternative one call to GNU awk, using printf:

$ awk 'BEGIN PROCINFO["sorted_in"] = "@ind_str_asc"
 c[length($0)]++
 END
 for(i in c)printf("%s character words - %sn",i,c[i])
 ' infile
2 character words - 3
5 character words - 1
7 character words - 1

The core algorithm just collects character counts in an array.
The end part prints the collected counts formatted with printf.

Fast, simple, one single call to awk.

To be precise: some more memory is used to keep the array.

But no sort is called (numeric arrays indexes are set to be always traversed sorted upward with PROCINFO), and only one external program: awk, instead of several.