mjhjmtu

To capture a particular pattern, awk and grep can be used. Why should we use one over the other? Which is faster and why?

If I had a log file and I wanted to grab a certain pattern, I could do one of the following

awk '/pattern/' /var/log/messages

or

grep 'pattern' /var/log/messages

I haven't done any benchmarking, so I wouldn't know. Can someone elaborate this? It is great to know the inner workings of these two tools.

grep will most likely be faster:

# time awk '/USAGE/' imapd.log.1 | wc -l
73832

real 0m2.756s
user 0m2.740s
sys 0m0.020s

# time grep 'USAGE' imapd.log.1 | wc -l
73832

real 0m0.110s
user 0m0.100s
sys 0m0.030s

awk is a interpreted programming language, where as grep is a compiled c-code program (which is additionally optimized towards finding patterns in files).

(Note - I ran both commands twice so that caching would not potentially skew the results)

More details about interpreted languages on wikipedia.

As Stephane has rightly pointed out in comments, your mileage may vary due to the implementation of the grep and awk you use, the operating system it is on and the character set you are processing.

Use the most specific and expressive tool. The tool that best fits your use case is likely to be the fastest.

As a rough guide:

• searching for lines matching a substring or regexp? Use grep.


• selecting certain columns from a simply-delimited file? Use cut.


• performing pattern-based substitutions or ... other stuff sed can reasonably do? Use sed.


• need some combination of the above 3, or printf formatting, or general purpose loops and branches? Use awk.

When only searching for strings, and speed matters, you should almost always use grep. It's orders of magnitude faster than awk when it comes to just gross searching.

source The functional and performance differences of sed, awk and other Unix parsing utilities

UTILITY OPERATION TYPE EXECUTION TIME CHARACTERS PROCESSED PER SECOND
 (10 ITERATIONS)
------- -------------- --------------- -------------------------------
grep search only 41 sec. 489.3 million
sed search & replace 4 min. 4 sec. 82.1 million
awk search & replace 4 min. 46 sec. 69.8 million
Python search & replace 4 min. 50 sec. 69.0 million
PHP search & replace 15 min. 44 sec. 21.2 million

While I agree that in theory grep should be faster than awk, in practice, YMMV as that depends a lot on the implementation you use.

here comparing busybox 1.20.0's grep and awk, GNU grep 2.14, mawk 1.3.3, GNU awk 4.0.1 on Debian/Linux 7.0 amd64 (with glibc 2.17) in a UTF-8 locale on a 240MB file of 2.5M lines of ASCII-only characters.

$ time busybox grep error error | wc -l
331003
busybox grep error error 8.31s user 0.12s system 99% cpu 8.450 total
wc -l 0.07s user 0.11s system 2% cpu 8.448 total
$ time busybox awk /error/ error | wc -l
331003
busybox awk /error/ error 2.39s user 0.84s system 98% cpu 3.265 total
wc -l 0.12s user 1.23s system 41% cpu 3.264 total
$ time grep error error | wc -l
331003
grep error error 0.80s user 0.10s system 99% cpu 0.914 total
wc -l 0.00s user 0.11s system 12% cpu 0.913 total
$ time mawk /error/ error | wc -l
330803
mawk /error/ error 0.54s user 0.13s system 91% cpu 0.732 total
wc -l 0.03s user 0.08s system 14% cpu 0.731 total
$ time gawk /error/ error | wc -l
331003
gawk /error/ error 1.37s user 0.12s system 99% cpu 1.494 total
wc -l 0.04s user 0.07s system 7% cpu 1.492 total
$ time 

In the C locale, only GNU grep gets a significant boost and becomes faster than mawk.

The dataset, the type of the regexp may also make a big difference. For regexps, awk should be compared to grep -E as awk's regexps are extended REs.

For this dataset, awk could be faster than grep on busybox based systems or systems where mawk is the default awk and the default locale is UTF-8 based (IIRC, it used to be the case in Ubuntu).

In a nutshell, grep does one thing only as many other UNIX tools and that's matching a line to the given pattern and it does it well. On the other hand, awk is more sophisticated tool as it is a complete programming language defined by POSIX standard with typical features like variables, arrays, expressions, functions or control statements for pattern scanning and processing.

In my opinion, it depends on the implementation how both tools perform in case of pattern matching and on the size of some input you want to process. I would expect that grep is usually more efficient than awk as it does matching only. But you can't write with grep a simple code to perform more complex tasks like further processing of matched records, computation or printing results without using other tools.