mjhjmtu

I want to filter a file by character (for the purpose of removing invalid xml characters which I cannot control the generation of) but I cannot seem to even be able to copy individual characters from one file to another. I used printf to copy literal sections including carriage returns before, but now it does not copy a carriage return as one, but as some empty length string. My code:

infile=$1
outfile=$2
touch $outfile
while IFS= read -r -n1 char
do
 # display one character at a time
 printf "%s" "$char" >> $outfile
done < "$infile"
diff $infile $outfile

I don't mind using sed or awk, but I would have to encode the allowed characters.

Char ::= #x9 | #xA | #xD | [#x20-#xD7FF] | [#xE000-#xFFFD] | [#x10000-#x10FFFF] /* any Unicode character, excluding the surrogate blocks, FFFE, and FFFF. */


Carriage return shouldn't be a problem, read should read it just fine. The newline (linefeed) is, since it's the default delimiter for read. You could use the read -d '' trick to make it work.

echo $'r' | xxd; # CR
echo $'n' | xxd; # LF fails
echo $'n' | IFS= read -d '' -r -n1 x; echo "$x" # LF ok

But, like they say, you probably don't want to do stuff like this in the shell. tr would be just what you need for deleting a fixed set of characters, but at least GNU tr works on bytes, not characters, so it's not much use for Unicode.

I think this Perl should work, for UTF-8 data, if your locales are correctly set to UTF-8:

perl -C -pe 'tr/x09x0ax0dx20-xd7ffxe000-xfffdx10000-x10ffff//cd' < in > out

But better test it, I'm not that used to Unicode quirks.

tr/abc//cd deletes the characters that are not listed in abc (tr/// is actually meant to transform characters to others, see perlop). It takes lists of characters, as well as ranges, and xHH means the character with hex value HH, and xHHHH one with value HHHH. So the above accepts 0x09, 0x0a, 0x0d, everything from 0x20 to 0xd7ff etc.

The list above is taken directly from the list presented in the question. I'll leave it to the end user to evaluate if it should be changed.