mjhjmtu

This one-liner removes duplicate lines from text input without pre-sorting.

For example:

$ cat >f
q
w
e
w
r
$ awk '!a[$0]++' <f
q
w
e
r
$ 

The original code I have found on the internets read:

awk '!_[$0]++'

This was even more perplexing to me as I took _ to have a special meaning in awk, like in Perl, but it turned out to be just a name of an array.

Now, I understand the logic behind the one-liner:
each input line is used as a key in a hash array, thus, upon completion, the hash contains unique lines in the order of arrival.

What I would like to learn is how exactly this notation is interpreted by awk. E.g. what the bang sign (!) means and the other elements of this code snippet.

How does it work?

Let's see,

 !a[$0]++

first

 a[$0]

we look at the value of a[$0] (array a with whole input line ($0) as key).

If it does not exist ( ! is negation in test will eval to true)

 !a[$0]

we print the input line $0 (default action).

Also, we add one ( ++ ) to a[$0], so next time !a[$0] will evaluate to false.

Nice, find!! You should have a look at code golf!

Here is the processing:

• a[$0]: look at the value of key $0, in associative array a. If it does not exist, create it.




• a[$0]++: increment the value of a[$0], return the old value as value of expression. If a[$0] does not exist, return 0 and increment a[$0] to 1 (++ operator returns numeric value).




• !a[$0]++: negate the value of expression. If a[$0]++ return 0, the whole expression is evaluated to true, make awk performed default action print $0. Otherwise, the whole expression is evaluated to false, causes awk do nothing.

References:

• Expression in awk


• gawk - Increment and Decrement Operators

With gawk, we can use dgawk (or awk --debug with newer version) to debug a gawk script. First, create a gawk script, named test.awk:

BEGIN 
 a = 0; 
 !a++; 

Then run:

dgawk -f test.awk

or:

gawk --debug -f test.awk

In debugger console:

$ dgawk -f test.awk
dgawk> trace on
dgawk> watch a
Watchpoint 1: a
dgawk> run
Starting program: 
[ 1:0x7fe59154cfe0] Op_rule : [in_rule = BEGIN] [source_file = test.awk]
[ 2:0x7fe59154bf80] Op_push_i : 0 [PERM|NUMCUR|NUMBER]
[ 2:0x7fe59154bf20] Op_store_var : a [do_reference = FALSE]
[ 3:0x7fe59154bf60] Op_push_lhs : a [do_reference = TRUE]
Stopping in BEGIN ...
Watchpoint 1: a
 Old value: untyped variable
 New value: 0
main() at `test.awk':3
3 !a++;
dgawk> step
[ 3:0x7fe59154bfc0] Op_postincrement : 
[ 3:0x7fe59154bf40] Op_not : 
Watchpoint 1: a
 Old value: 0
 New value: 1
main() at `test.awk':3
3 !a++;
dgawk>

You can see, Op_postincrement was executed before Op_not.

You can also use si or stepi instead of s or step to see more clearly:

dgawk> si
[ 3:0x7ff061ac1fc0] Op_postincrement : 
3 !a++;
dgawk> si
[ 3:0x7ff061ac1f40] Op_not : 
Watchpoint 1: a
 Old value: 0
 New value: 1
main() at `test.awk':3
3 !a++;