mjhjmtu

How do I get perl to properly replace UTF-8 character from a shell?

The examples use stdin, but I need something that works for perl ... file too.

This is what I expect:

$ echo ABCæøåDEF | perl -CS -pe "s/([æøå])/[\1]/g"
ABC[æ][ø][å]DEF

This is what I get:

$ echo ABCæøåDEF | perl -CS -pe "s/([æøå])/[\1]/g"
ABCæøåDEF

Replacing the Unicode characters with ASCII works instantly:

$ echo ABC123DEF | perl -CS -pe "s/([123])/[\1]/g"
ABC[1][2][3]DEF

My environment:

perl 5.18.2
Bash 3.2.57
LC_ALL=en_US.UTF-8
LANG=en_US.UTF-8

Use this :

 $ echo 'ABCæøåDEF' |
 perl -CSD -Mutf8 -pe 's/([æøå])/[$1]/g'

Works also for files

Output :

ABC[æ][ø][å]DEF

Note :

• substitutions: \1 is for awk, 1 is for sed and in perl we use $1
  


• check perldoc perlrun for -CSD utf8 tricks

Your input:

$ cat input.txt 
ABCæøåDEF
$ hexdump -C input.txt 
00000000 41 42 43 c3 a6 c3 b8 c3 a5 44 45 46 0a |ABC......DEF.|
0000000d

One good way IMO is the -C option plus utf8:

$ perl -CSD -Mutf8 -pe 's/([æøå])/[$1]/g' input.txt 
ABC[æ][ø][å]DEF
$ cat input.txt | perl -CSD -Mutf8 -pe 's/([æøå])/[$1]/g'
ABC[æ][ø][å]DEF

If you don't want to use UTF-8 on the command line, you can always write your Perl code in plain ASCII and use escapes such as xAB, xABCD, or in newer Perls NU+ABCD or NCHARNAME:

$ perl -CSD -pe 's/([xE6xF8xE5])/[$1]/g' input.txt 
ABC[æ][ø][å]DEF
$ cat input.txt | perl -CSD -pe 's/([xE6xF8xE5])/[$1]/g'
ABC[æ][ø][å]DEF

This one is getting a little creative: @ARGV will be interpreted as UTF-8, so you can keep your source code as ASCII and pass the UTF-8 characters via a command line argument (not necessarily the nicest solution, just showing how you could make use of the the -CA option):

$ perl -CSDA -pe 'BEGIN$p=shift; s/($p)/[$1]/g' '[æøå]' input.txt 
ABC[æ][ø][å]DEF
$ cat input.txt | perl -CSDA -pe 'BEGIN$p=shift; s/($p)/[$1]/g' '[æøå]'
ABC[æ][ø][å]DEF

Or, of course you can always turn the oneliner into an actual script, where you can

use warnings;
use 5.012;
use utf8;
use open qw/:std :encoding(UTF-8)/;
use charnames qw/:full :short/;

Further reading: perlunitut, perlunifaq, perluniintro, perlunicode, perlunicook.

$ echo 'ABCæøåDEF' |
perl -CS -Mutf8 -pe 's/([æøå])/[$1]/g'