mjhjmtu

I have a script that reads a text stream and generates a file of sed commands that is later run with sed -f. The generated sed commands are like:

s/cid:image002.gif@01CC3D46.926E77E0/https://mysite.com/files/1922/g
s/cid:image003.gif@01CC3D46.926E77E0/https://mysite.com/files/1923/g
s/cid:image004.jpg@01CC3D46.926E77E0/https://mysite.com/files/1924/g

Assume the script which generates the sed commands is something like:

while read cid fileid
do
 cidpat="$(echo $cid | sed -e s/\./\\./g)"
 echo 's/'"$cidpat"'/https://mysite.com/files/'"$fileid"'/g' >> sedscr
done

How can I improve the script to ensure all regex metacharacters in the cid string are escaped and interpolated properly?

To escape variables to be used on the left hand side and right hand side of a s command in sed (here $lhs and $rhs respectively), you'd do:

escaped_lhs=$(printf '%sn' "$lhs" | sed 's:[/.^$*]:\&:g')
escaped_rhs=$(printf '%sn' "$rhs" | sed 's:[/&]:\&:g;$!s/$/\/')

sed "s/$escaped_lhs/$escaped_rhs/"

Note that $lhs cannot contain a newline character.

That is, on the LHS, escape all the regexp operators (][.^$*), the escaping character itself (), and the separator (/).

On the RHS, you only need to escape &, the separator, backslash and the newline character (which you do by inserting a backslash at the end of each line except the last one ($!s/$/\/)).

That assumes you use / as a separator in your sed s commands and that you don't enable Extended REs with -r (GNU sed/ssed/ast/busybox sed) or -E (BSDs, ast, recent GNU, recent busybox) or PCREs with -R (ssed) or Augmented REs with -A/-X (ast) which all have extra RE operators.

A few ground rules when dealing with arbitrary data:

• Don't use echo
  


• quote your variables


• consider the impact of the locale (especially its character set: it's important that the escaping sed commands are run in the same locale as the sed command using the escaped strings (and with the same sed command) for instance)


• don't forget about the newline character (here you may want to check if $lhs contains any and take action).

Another option is to use perl instead of sed and pass the strings in the environment and use the Q/E perl regexp operators for taking strings literally:

A=lhs B=rhs perl -pe 's/Q$ENVAE/$ENVB/g'

perl (by default) will not be affected by the locale's character set as, in the above, it only considers the strings as arrays of bytes without caring about what characters (if any) they may represent for the user. With sed, you could achieve the same by fixing the locale to C with LC_ALL=C for all sed commands (though that will also affect the language of error messages, if any).