mjhjmtu

I need to parse large amounts text in command line and replace all (possibly nested) text quotes with spaces. Quotes are marked with specific syntax: [quote=username]quoted text[/quote].

Example input with nested quotes could be something like:

text part 1 [quote=foo] outer quote 1 [quote=bar] inner quote [/quote] outer quote 2 [/quote] text part 2 [quote=foo-bar] next quote [/quote] text part 3

And expected output would be:

text part 1 text part 2 text part 3

With help of this question I got it somehow work (got output above) with sed ':b; s/[quote=[^]]*][^[/]*[/quote]/ /g; t b' but middle part ([^[/]] is problematic since quotes can contain characters like [ or ].

That being said, my sed command doesn't work if input is eg.

text part 1 [quote=foo] outer quote 1 [quote=bar] inner quote [foo] [/quote] outer quote 2 [/quote] text part 2 [quote=foo-bar] next quote [/quote] text part 3

One problem is that sed doesn't seem to support non-greedy qualifier and thus catches always longest possible match from the input. That makes it hard to deal with a) usernames and b) quoted texts in general.

I also guess that sed is not the best tool to solve this and it might not be even capable of doing things like that. Maybe eg. perl or awk could work better?

Now the final question is that what would be the best and most efficient way to solve this?

If you know the input doesn't contain < or > characters, you could do:

sed '
 # replace opening quote with <
 s|[quote=[^]]*]|<|g
 # and closing quotes with >
 s|[/quote]|>|g
 :1
 # work our way from the inner quotes
 s|<[^<>]*>||g
 t1'

If it may contain < or > characters, you can escape them using a scheme like:

sed '
 # escape < and > (and the escaping character _ itself)
 s/_/_u/g; s/</_l/g; s/>/_r/g

 <code-above>

 # undo escaping after the work has been done
 s/_r/>/g; s/_l/</g; s/_u/_/g'

With perl, using recursive regexps:

perl -pe 's@([quote=[^]]*](?:(?1)|.)*?[/quote])@@g'

Or even, as you mention:

perl -pe 's@([quote=.*?](?:(?1)|.)*?[/quote])@@g'

With perl, you can handle multiline input by adding the -0777 option. With sed, you'd need to prefix the code with:

:0
$!
 N;b0

So as to load the whole input into the pattern space.

I checked this one and it worked for me. You might want to choose another temporary pattern instead of foobar. Without it sed deleted everything between tags leaving just text part 1 text part 3

sed -e 's//quote]/foobar]/3' -e 's/[.*/quote]//' -e 's/[.*foobar]//' testfile

instead if testfile you may just pipe it with cat

A little script that increments a counter variable on each start-quote and decrements it on each end-quote. If the counter variable is greater 0, then text snippets are skipped.

#!/bin/bash

# disable pathname expansion
set -f 
cnt=0
for i in $(<$1); do
 # start quote
 if [ "$i##[quote=" != "$i" ] && [ "$i: -1" = "]" ]; then
 ((++cnt))
 elif [ "$i" = "[/quote]" ]; then
 ((--cnt))
 elif [ $cnt -eq 0 ]; then
 echo -n "$i "
 fi
done
echo

Output:

$ cat q1
text part 1 [quote=foo] outer quote 1 [quote=bar] inner quote [/quote] outer quote 2 [/quote] text part 2 [quote=foo-bar] next quote [/quote] text part 3
$ ./parse.sh q1
text part 1 text part 2 text part 3
$ cat q2
text part 1 [quote=foo] outer quote 1 [quote=bar] inner quote [foo] [/quote] outer quote 2 [/quote] text part 2 [quote=foo-bar] next quote [/quote] text part 3
$ ./parse.sh q2
text part 1 text part 2 text part 3

You can do this with POSIX sed as detailed here. Note this solution applies to both kind of inputs shown by you. The limitations the input is not mulitiline, as we make use of newlines as markers to effect
transformation required.

$ sed -e '
 :top
 /[/quote]/!b
 s//
&/
 s/[quote=/

&/

 :loop
 s/(nn)([quote=.*)([quote=.*n)/213/
 tloop

 s/nn.*n[/quote]//
 btop
 ' input.txt