I have large (~180MB) xml file with some wrong characters in it, for example

<Data ss:Type="String">7402953^@</Data>

The ^@ part should by removed. The job supposed to be done with

sed -i 's/^@//g' /tmp/large.xml

but for some unknown reason it doesn't work as expected if string is located in my large xml file. If the file has only few KB in size, sed works perfectly.

It looks like a bug but I think it can't be because the task is quite obvious. I'm doing something wrong?

Judging by your question (because there are no examples), I would say that ^@ in the big file are not actually the two characters (^ and @), but one of those unprintable characters.

You can input that unprintable character in the terminal with Ctrl + v + Ctrl + 2.

Use that in sed instead of the characters ^ and @ and it should be fine.

Also remove the escape sequence because it is not needed for the unprintable character.

awk

If a solution using awk is acceptable, this will remove all non-printable characters.

This works in GNU awk (Linux) and BSD awk (Mac).

awk ' gsub(/[^[:print:][:blank:]]/,"",$0) ; print $0 ' input.xml > output.xml

• 
  gsub(/[^[:print:][:blank:]]/,"",$0)
  From each line of input, remove any unwanted characters.
  
  
  • 
    [:print:]
    Any printable character.
  
  
  • 
    [:blank:]
    Space or tab.
  
  
  • 
    [^[:print:][:blank:]]
    Any character not included in those two classes.
  
  


• 
  print $0
  Print each line of input.


• 
  > output.xml
  Save the output to a file instead of printing it to the screen.

Do the same thing with fewer keystrokes (it's just a little harder to read):

awk 'gsub(/[^[:print:][:blank:]]/,"")1' input.xml > output.xml

• You don't need to specify ,$0 (the entire line of input) in gsub if you're examining the entire line.


• The 1 at the end means "now do the default action (ie, print) for every line".