Bash convert xC3x89 to Ã?
Clash Royale CLAN TAG#URR8PPP
up vote
3
down vote
favorite
I'm running iwlist wlo1 scan | grep ESSID
inside a script.
It displays French characters in the following format
xC3x89
for ÃÂ and xC3xA9
for é.
I'm not sure what this format is called. I tried using an answer for converting unicode echo -ne 'xC3xA9' | iconv -f utf-16be
but it converted to ì©
.
What is the official name for this format and how can I convert it in bash?
character-encoding escape-characters conversion
add a comment |Â
up vote
3
down vote
favorite
I'm running iwlist wlo1 scan | grep ESSID
inside a script.
It displays French characters in the following format
xC3x89
for ÃÂ and xC3xA9
for é.
I'm not sure what this format is called. I tried using an answer for converting unicode echo -ne 'xC3xA9' | iconv -f utf-16be
but it converted to ì©
.
What is the official name for this format and how can I convert it in bash?
character-encoding escape-characters conversion
echo -ne "xC3x89"
givesÃ
â RomanPerekhrest
Oct 28 '17 at 14:10
@RomanPerekhrest thank you, that fixes my issue.
â Philip Kirkbride
Oct 28 '17 at 14:12
@RomanPerekhrest do you want to post as an answer?
â Philip Kirkbride
Oct 28 '17 at 14:14
Philip Kirkbride, ok, posted ...
â RomanPerekhrest
Oct 28 '17 at 14:39
add a comment |Â
up vote
3
down vote
favorite
up vote
3
down vote
favorite
I'm running iwlist wlo1 scan | grep ESSID
inside a script.
It displays French characters in the following format
xC3x89
for ÃÂ and xC3xA9
for é.
I'm not sure what this format is called. I tried using an answer for converting unicode echo -ne 'xC3xA9' | iconv -f utf-16be
but it converted to ì©
.
What is the official name for this format and how can I convert it in bash?
character-encoding escape-characters conversion
I'm running iwlist wlo1 scan | grep ESSID
inside a script.
It displays French characters in the following format
xC3x89
for ÃÂ and xC3xA9
for é.
I'm not sure what this format is called. I tried using an answer for converting unicode echo -ne 'xC3xA9' | iconv -f utf-16be
but it converted to ì©
.
What is the official name for this format and how can I convert it in bash?
character-encoding escape-characters conversion
asked Oct 28 '17 at 13:36
Philip Kirkbride
2,2922470
2,2922470
echo -ne "xC3x89"
givesÃ
â RomanPerekhrest
Oct 28 '17 at 14:10
@RomanPerekhrest thank you, that fixes my issue.
â Philip Kirkbride
Oct 28 '17 at 14:12
@RomanPerekhrest do you want to post as an answer?
â Philip Kirkbride
Oct 28 '17 at 14:14
Philip Kirkbride, ok, posted ...
â RomanPerekhrest
Oct 28 '17 at 14:39
add a comment |Â
echo -ne "xC3x89"
givesÃ
â RomanPerekhrest
Oct 28 '17 at 14:10
@RomanPerekhrest thank you, that fixes my issue.
â Philip Kirkbride
Oct 28 '17 at 14:12
@RomanPerekhrest do you want to post as an answer?
â Philip Kirkbride
Oct 28 '17 at 14:14
Philip Kirkbride, ok, posted ...
â RomanPerekhrest
Oct 28 '17 at 14:39
echo -ne "xC3x89"
gives Ã
â RomanPerekhrest
Oct 28 '17 at 14:10
echo -ne "xC3x89"
gives Ã
â RomanPerekhrest
Oct 28 '17 at 14:10
@RomanPerekhrest thank you, that fixes my issue.
â Philip Kirkbride
Oct 28 '17 at 14:12
@RomanPerekhrest thank you, that fixes my issue.
â Philip Kirkbride
Oct 28 '17 at 14:12
@RomanPerekhrest do you want to post as an answer?
â Philip Kirkbride
Oct 28 '17 at 14:14
@RomanPerekhrest do you want to post as an answer?
â Philip Kirkbride
Oct 28 '17 at 14:14
Philip Kirkbride, ok, posted ...
â RomanPerekhrest
Oct 28 '17 at 14:39
Philip Kirkbride, ok, posted ...
â RomanPerekhrest
Oct 28 '17 at 14:39
add a comment |Â
3 Answers
3
active
oldest
votes
up vote
5
down vote
accepted
Hexdecimal numeric constants are usually represented with 0x
prefix.
Character and string constants may express character codes in hexadecimal with the prefix x
followed by two hex digits.
echo -ne 'xC3x89'
should give you ÃÂ
.
-e
- enable interpretation of backslash escapes(includingxHH
- byte with hexadecimal valueHH
(1 to 2 digits))
To deal with better portability use printf
function:
printf "%b" 'xC3x89'
ÃÂ
echo -e
is nonportable -- even bash doesn't always offer it, since when bothposix
andxpg_echo
options are enabledecho -e
just prints-e
on output, as standard compliance requires. Better to useprintf '%bn' 'xC3x89'
â Charles Duffy
Oct 28 '17 at 17:29
...to demonstrate this, considerbash -c 'set -o posix; shopt -s xpg_echo; printf "%bn" "$1"; echo -e "$1"' _ 'xC3x89'
(and note thatxpg_echo
can be set on-by-default at compile time).
â Charles Duffy
Oct 28 '17 at 17:38
@CharlesDuffy, you have my update
â RomanPerekhrest
Oct 28 '17 at 20:05
add a comment |Â
up vote
3
down vote
Try
% echo -n éà| hd
00000000 c3 a9 c3 89 |....|
00000004
where hd
is the hexdump utility.
So your thing is just UTF-8 encoding (and your observed output xC3x89
is the hexadecimal representation of UTF-8). Today we have UTF-8 everywhere.
My environment has both LANG=en_US.UTF-8
and LC_ALL=en_US.UTF-8
(even if I am French).
Also,
echo 'xC3x89xC3xA9'
produces
ÃÂé
as expected.
And so does
echo "\xC3\x89\xC3\xA9" | iconv -t utf-8
Read also locale(7).
Notice that some character encodings (e.g. old plain ASCII) don't have any é
or ÃÂ
characters, and others (e.g. ISO-8859-1) encode them differently.
When I runecho "\xC3\x89\xC3\xA9" | iconv -t utf-8
my output is the same as the input, same for normalecho
.LANG=en_CA.UTF-8
andLC_ALL
not set. I'm hoping to have something that will work on a variety of machines.
â Philip Kirkbride
Oct 28 '17 at 14:10
1
You can't displayé
on an ASCII system, because that letter does not exist in ASCII encoding.
â Basile Starynkevitch
Oct 28 '17 at 14:12
add a comment |Â
up vote
1
down vote
iwlist
displays all the bytes that don't correspond to ASCII non-control characters as the xHH
where HH
is the byte number in hexadecimal.
To undo that encoding, you can pipe the output of iwlist
to
perl -pe 's/\x([da-f]2)/chr(hex($1))/gie'
Byte 0xc3
followed by 0xa9
make up the UTF-8 encoding of the é
character.
If your locale also uses the UTF-8 encoding, then the output of perl
will show that é
character. If not (unlikely as UTF-8 is the norm), you can always pipe the output of perl
to iconv -f utf-8
to convert it f
rom UTF-8 to the character encoding used in your locale.
add a comment |Â
3 Answers
3
active
oldest
votes
3 Answers
3
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
5
down vote
accepted
Hexdecimal numeric constants are usually represented with 0x
prefix.
Character and string constants may express character codes in hexadecimal with the prefix x
followed by two hex digits.
echo -ne 'xC3x89'
should give you ÃÂ
.
-e
- enable interpretation of backslash escapes(includingxHH
- byte with hexadecimal valueHH
(1 to 2 digits))
To deal with better portability use printf
function:
printf "%b" 'xC3x89'
ÃÂ
echo -e
is nonportable -- even bash doesn't always offer it, since when bothposix
andxpg_echo
options are enabledecho -e
just prints-e
on output, as standard compliance requires. Better to useprintf '%bn' 'xC3x89'
â Charles Duffy
Oct 28 '17 at 17:29
...to demonstrate this, considerbash -c 'set -o posix; shopt -s xpg_echo; printf "%bn" "$1"; echo -e "$1"' _ 'xC3x89'
(and note thatxpg_echo
can be set on-by-default at compile time).
â Charles Duffy
Oct 28 '17 at 17:38
@CharlesDuffy, you have my update
â RomanPerekhrest
Oct 28 '17 at 20:05
add a comment |Â
up vote
5
down vote
accepted
Hexdecimal numeric constants are usually represented with 0x
prefix.
Character and string constants may express character codes in hexadecimal with the prefix x
followed by two hex digits.
echo -ne 'xC3x89'
should give you ÃÂ
.
-e
- enable interpretation of backslash escapes(includingxHH
- byte with hexadecimal valueHH
(1 to 2 digits))
To deal with better portability use printf
function:
printf "%b" 'xC3x89'
ÃÂ
echo -e
is nonportable -- even bash doesn't always offer it, since when bothposix
andxpg_echo
options are enabledecho -e
just prints-e
on output, as standard compliance requires. Better to useprintf '%bn' 'xC3x89'
â Charles Duffy
Oct 28 '17 at 17:29
...to demonstrate this, considerbash -c 'set -o posix; shopt -s xpg_echo; printf "%bn" "$1"; echo -e "$1"' _ 'xC3x89'
(and note thatxpg_echo
can be set on-by-default at compile time).
â Charles Duffy
Oct 28 '17 at 17:38
@CharlesDuffy, you have my update
â RomanPerekhrest
Oct 28 '17 at 20:05
add a comment |Â
up vote
5
down vote
accepted
up vote
5
down vote
accepted
Hexdecimal numeric constants are usually represented with 0x
prefix.
Character and string constants may express character codes in hexadecimal with the prefix x
followed by two hex digits.
echo -ne 'xC3x89'
should give you ÃÂ
.
-e
- enable interpretation of backslash escapes(includingxHH
- byte with hexadecimal valueHH
(1 to 2 digits))
To deal with better portability use printf
function:
printf "%b" 'xC3x89'
ÃÂ
Hexdecimal numeric constants are usually represented with 0x
prefix.
Character and string constants may express character codes in hexadecimal with the prefix x
followed by two hex digits.
echo -ne 'xC3x89'
should give you ÃÂ
.
-e
- enable interpretation of backslash escapes(includingxHH
- byte with hexadecimal valueHH
(1 to 2 digits))
To deal with better portability use printf
function:
printf "%b" 'xC3x89'
ÃÂ
edited Oct 28 '17 at 20:05
answered Oct 28 '17 at 14:38
RomanPerekhrest
22.5k12145
22.5k12145
echo -e
is nonportable -- even bash doesn't always offer it, since when bothposix
andxpg_echo
options are enabledecho -e
just prints-e
on output, as standard compliance requires. Better to useprintf '%bn' 'xC3x89'
â Charles Duffy
Oct 28 '17 at 17:29
...to demonstrate this, considerbash -c 'set -o posix; shopt -s xpg_echo; printf "%bn" "$1"; echo -e "$1"' _ 'xC3x89'
(and note thatxpg_echo
can be set on-by-default at compile time).
â Charles Duffy
Oct 28 '17 at 17:38
@CharlesDuffy, you have my update
â RomanPerekhrest
Oct 28 '17 at 20:05
add a comment |Â
echo -e
is nonportable -- even bash doesn't always offer it, since when bothposix
andxpg_echo
options are enabledecho -e
just prints-e
on output, as standard compliance requires. Better to useprintf '%bn' 'xC3x89'
â Charles Duffy
Oct 28 '17 at 17:29
...to demonstrate this, considerbash -c 'set -o posix; shopt -s xpg_echo; printf "%bn" "$1"; echo -e "$1"' _ 'xC3x89'
(and note thatxpg_echo
can be set on-by-default at compile time).
â Charles Duffy
Oct 28 '17 at 17:38
@CharlesDuffy, you have my update
â RomanPerekhrest
Oct 28 '17 at 20:05
echo -e
is nonportable -- even bash doesn't always offer it, since when both posix
and xpg_echo
options are enabled echo -e
just prints -e
on output, as standard compliance requires. Better to use printf '%bn' 'xC3x89'
â Charles Duffy
Oct 28 '17 at 17:29
echo -e
is nonportable -- even bash doesn't always offer it, since when both posix
and xpg_echo
options are enabled echo -e
just prints -e
on output, as standard compliance requires. Better to use printf '%bn' 'xC3x89'
â Charles Duffy
Oct 28 '17 at 17:29
...to demonstrate this, consider
bash -c 'set -o posix; shopt -s xpg_echo; printf "%bn" "$1"; echo -e "$1"' _ 'xC3x89'
(and note that xpg_echo
can be set on-by-default at compile time).â Charles Duffy
Oct 28 '17 at 17:38
...to demonstrate this, consider
bash -c 'set -o posix; shopt -s xpg_echo; printf "%bn" "$1"; echo -e "$1"' _ 'xC3x89'
(and note that xpg_echo
can be set on-by-default at compile time).â Charles Duffy
Oct 28 '17 at 17:38
@CharlesDuffy, you have my update
â RomanPerekhrest
Oct 28 '17 at 20:05
@CharlesDuffy, you have my update
â RomanPerekhrest
Oct 28 '17 at 20:05
add a comment |Â
up vote
3
down vote
Try
% echo -n éà| hd
00000000 c3 a9 c3 89 |....|
00000004
where hd
is the hexdump utility.
So your thing is just UTF-8 encoding (and your observed output xC3x89
is the hexadecimal representation of UTF-8). Today we have UTF-8 everywhere.
My environment has both LANG=en_US.UTF-8
and LC_ALL=en_US.UTF-8
(even if I am French).
Also,
echo 'xC3x89xC3xA9'
produces
ÃÂé
as expected.
And so does
echo "\xC3\x89\xC3\xA9" | iconv -t utf-8
Read also locale(7).
Notice that some character encodings (e.g. old plain ASCII) don't have any é
or ÃÂ
characters, and others (e.g. ISO-8859-1) encode them differently.
When I runecho "\xC3\x89\xC3\xA9" | iconv -t utf-8
my output is the same as the input, same for normalecho
.LANG=en_CA.UTF-8
andLC_ALL
not set. I'm hoping to have something that will work on a variety of machines.
â Philip Kirkbride
Oct 28 '17 at 14:10
1
You can't displayé
on an ASCII system, because that letter does not exist in ASCII encoding.
â Basile Starynkevitch
Oct 28 '17 at 14:12
add a comment |Â
up vote
3
down vote
Try
% echo -n éà| hd
00000000 c3 a9 c3 89 |....|
00000004
where hd
is the hexdump utility.
So your thing is just UTF-8 encoding (and your observed output xC3x89
is the hexadecimal representation of UTF-8). Today we have UTF-8 everywhere.
My environment has both LANG=en_US.UTF-8
and LC_ALL=en_US.UTF-8
(even if I am French).
Also,
echo 'xC3x89xC3xA9'
produces
ÃÂé
as expected.
And so does
echo "\xC3\x89\xC3\xA9" | iconv -t utf-8
Read also locale(7).
Notice that some character encodings (e.g. old plain ASCII) don't have any é
or ÃÂ
characters, and others (e.g. ISO-8859-1) encode them differently.
When I runecho "\xC3\x89\xC3\xA9" | iconv -t utf-8
my output is the same as the input, same for normalecho
.LANG=en_CA.UTF-8
andLC_ALL
not set. I'm hoping to have something that will work on a variety of machines.
â Philip Kirkbride
Oct 28 '17 at 14:10
1
You can't displayé
on an ASCII system, because that letter does not exist in ASCII encoding.
â Basile Starynkevitch
Oct 28 '17 at 14:12
add a comment |Â
up vote
3
down vote
up vote
3
down vote
Try
% echo -n éà| hd
00000000 c3 a9 c3 89 |....|
00000004
where hd
is the hexdump utility.
So your thing is just UTF-8 encoding (and your observed output xC3x89
is the hexadecimal representation of UTF-8). Today we have UTF-8 everywhere.
My environment has both LANG=en_US.UTF-8
and LC_ALL=en_US.UTF-8
(even if I am French).
Also,
echo 'xC3x89xC3xA9'
produces
ÃÂé
as expected.
And so does
echo "\xC3\x89\xC3\xA9" | iconv -t utf-8
Read also locale(7).
Notice that some character encodings (e.g. old plain ASCII) don't have any é
or ÃÂ
characters, and others (e.g. ISO-8859-1) encode them differently.
Try
% echo -n éà| hd
00000000 c3 a9 c3 89 |....|
00000004
where hd
is the hexdump utility.
So your thing is just UTF-8 encoding (and your observed output xC3x89
is the hexadecimal representation of UTF-8). Today we have UTF-8 everywhere.
My environment has both LANG=en_US.UTF-8
and LC_ALL=en_US.UTF-8
(even if I am French).
Also,
echo 'xC3x89xC3xA9'
produces
ÃÂé
as expected.
And so does
echo "\xC3\x89\xC3\xA9" | iconv -t utf-8
Read also locale(7).
Notice that some character encodings (e.g. old plain ASCII) don't have any é
or ÃÂ
characters, and others (e.g. ISO-8859-1) encode them differently.
edited Oct 28 '17 at 14:43
answered Oct 28 '17 at 13:49
Basile Starynkevitch
7,9231940
7,9231940
When I runecho "\xC3\x89\xC3\xA9" | iconv -t utf-8
my output is the same as the input, same for normalecho
.LANG=en_CA.UTF-8
andLC_ALL
not set. I'm hoping to have something that will work on a variety of machines.
â Philip Kirkbride
Oct 28 '17 at 14:10
1
You can't displayé
on an ASCII system, because that letter does not exist in ASCII encoding.
â Basile Starynkevitch
Oct 28 '17 at 14:12
add a comment |Â
When I runecho "\xC3\x89\xC3\xA9" | iconv -t utf-8
my output is the same as the input, same for normalecho
.LANG=en_CA.UTF-8
andLC_ALL
not set. I'm hoping to have something that will work on a variety of machines.
â Philip Kirkbride
Oct 28 '17 at 14:10
1
You can't displayé
on an ASCII system, because that letter does not exist in ASCII encoding.
â Basile Starynkevitch
Oct 28 '17 at 14:12
When I run
echo "\xC3\x89\xC3\xA9" | iconv -t utf-8
my output is the same as the input, same for normal echo
. LANG=en_CA.UTF-8
and LC_ALL
not set. I'm hoping to have something that will work on a variety of machines.â Philip Kirkbride
Oct 28 '17 at 14:10
When I run
echo "\xC3\x89\xC3\xA9" | iconv -t utf-8
my output is the same as the input, same for normal echo
. LANG=en_CA.UTF-8
and LC_ALL
not set. I'm hoping to have something that will work on a variety of machines.â Philip Kirkbride
Oct 28 '17 at 14:10
1
1
You can't display
é
on an ASCII system, because that letter does not exist in ASCII encoding.â Basile Starynkevitch
Oct 28 '17 at 14:12
You can't display
é
on an ASCII system, because that letter does not exist in ASCII encoding.â Basile Starynkevitch
Oct 28 '17 at 14:12
add a comment |Â
up vote
1
down vote
iwlist
displays all the bytes that don't correspond to ASCII non-control characters as the xHH
where HH
is the byte number in hexadecimal.
To undo that encoding, you can pipe the output of iwlist
to
perl -pe 's/\x([da-f]2)/chr(hex($1))/gie'
Byte 0xc3
followed by 0xa9
make up the UTF-8 encoding of the é
character.
If your locale also uses the UTF-8 encoding, then the output of perl
will show that é
character. If not (unlikely as UTF-8 is the norm), you can always pipe the output of perl
to iconv -f utf-8
to convert it f
rom UTF-8 to the character encoding used in your locale.
add a comment |Â
up vote
1
down vote
iwlist
displays all the bytes that don't correspond to ASCII non-control characters as the xHH
where HH
is the byte number in hexadecimal.
To undo that encoding, you can pipe the output of iwlist
to
perl -pe 's/\x([da-f]2)/chr(hex($1))/gie'
Byte 0xc3
followed by 0xa9
make up the UTF-8 encoding of the é
character.
If your locale also uses the UTF-8 encoding, then the output of perl
will show that é
character. If not (unlikely as UTF-8 is the norm), you can always pipe the output of perl
to iconv -f utf-8
to convert it f
rom UTF-8 to the character encoding used in your locale.
add a comment |Â
up vote
1
down vote
up vote
1
down vote
iwlist
displays all the bytes that don't correspond to ASCII non-control characters as the xHH
where HH
is the byte number in hexadecimal.
To undo that encoding, you can pipe the output of iwlist
to
perl -pe 's/\x([da-f]2)/chr(hex($1))/gie'
Byte 0xc3
followed by 0xa9
make up the UTF-8 encoding of the é
character.
If your locale also uses the UTF-8 encoding, then the output of perl
will show that é
character. If not (unlikely as UTF-8 is the norm), you can always pipe the output of perl
to iconv -f utf-8
to convert it f
rom UTF-8 to the character encoding used in your locale.
iwlist
displays all the bytes that don't correspond to ASCII non-control characters as the xHH
where HH
is the byte number in hexadecimal.
To undo that encoding, you can pipe the output of iwlist
to
perl -pe 's/\x([da-f]2)/chr(hex($1))/gie'
Byte 0xc3
followed by 0xa9
make up the UTF-8 encoding of the é
character.
If your locale also uses the UTF-8 encoding, then the output of perl
will show that é
character. If not (unlikely as UTF-8 is the norm), you can always pipe the output of perl
to iconv -f utf-8
to convert it f
rom UTF-8 to the character encoding used in your locale.
answered Oct 28 '17 at 21:11
Stéphane Chazelas
283k53521855
283k53521855
add a comment |Â
add a comment |Â
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f401064%2fbash-convert-xc3-x89-to-%25c3%2589%23new-answer', 'question_page');
);
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
echo -ne "xC3x89"
givesÃ
â RomanPerekhrest
Oct 28 '17 at 14:10
@RomanPerekhrest thank you, that fixes my issue.
â Philip Kirkbride
Oct 28 '17 at 14:12
@RomanPerekhrest do you want to post as an answer?
â Philip Kirkbride
Oct 28 '17 at 14:14
Philip Kirkbride, ok, posted ...
â RomanPerekhrest
Oct 28 '17 at 14:39