How can I add message delimiters to a UDP stream socat is piping?
Multi tool use
Clash Royale CLAN TAG#URR8PPP
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty margin-bottom:0;
up vote
5
down vote
favorite
I am using socat to intercept UDP messages and send them to a named pipe:
socat UDP-LISTEN:9999,fork PIPE:/tmp/mypipe,append
I am able to tail this pipe and see all the messages it receives.
I would like to pipe the output of tail -f /tmp/mypipe to sed to do some post-processing of the messages, but unfortunately some of them are not newline-terminated. This is a problem because it means multiple distinct UDP messages could be on the same line, and also because tail -f /tmp/mypipe | sed ... does not pass the last line if it is unterminated.
Ideally I would like to be able to add a custom message delimiter as they are sent to my pipe, so that I can easily find the message boundaries. If that's not possible then is there some way I can follow this file and pipe the final (potentially unterminated) line to another program for post-processing?
socat
asked Jul 27 at 20:43
abc
284
add a comment |Â
up vote
5
down vote
favorite
I am using socat to intercept UDP messages and send them to a named pipe:
socat UDP-LISTEN:9999,fork PIPE:/tmp/mypipe,append
I am able to tail this pipe and see all the messages it receives.
I would like to pipe the output of tail -f /tmp/mypipe to sed to do some post-processing of the messages, but unfortunately some of them are not newline-terminated. This is a problem because it means multiple distinct UDP messages could be on the same line, and also because tail -f /tmp/mypipe | sed ... does not pass the last line if it is unterminated.
Ideally I would like to be able to add a custom message delimiter as they are sent to my pipe, so that I can easily find the message boundaries. If that's not possible then is there some way I can follow this file and pipe the final (potentially unterminated) line to another program for post-processing?
socat
asked Jul 27 at 20:43
abc
284
I was able to get closer to my goal, though it's not perfect. NeitherUDP-LISTENnorUDP-RECVseemed to work -- when I tried to execute a system command to echo a trailing newline it appears they would hang oncat;. UsingUDP-RECVFROMlike so seems to work:socat -u UDP-RECVFROM:9999,fork SYSTEM:"cat; echo "n""
– abc
Jul 27 at 22:21
The solution in my comment above appears to suffer from a race condition -- since the receives are forked (as they have to be, elsesocatexits after the first UDP message) theSYSTEMcommands are not executed atomically. I noticed thecatandechogetting interleaved when multiple messages came in at the same time. I serialized it by wrapping it in a lock like this:socat -u UDP-RECVFROM:9999,fork SYSTEM:"(flock -x 200; cat; echo "n";) 200>./udplockfile"
– abc
Jul 27 at 22:43
add a comment |Â
up vote
5
down vote
favorite
up vote
5
down vote
favorite
I am using socat to intercept UDP messages and send them to a named pipe:
socat UDP-LISTEN:9999,fork PIPE:/tmp/mypipe,append
I am able to tail this pipe and see all the messages it receives.
I would like to pipe the output of tail -f /tmp/mypipe to sed to do some post-processing of the messages, but unfortunately some of them are not newline-terminated. This is a problem because it means multiple distinct UDP messages could be on the same line, and also because tail -f /tmp/mypipe | sed ... does not pass the last line if it is unterminated.
Ideally I would like to be able to add a custom message delimiter as they are sent to my pipe, so that I can easily find the message boundaries. If that's not possible then is there some way I can follow this file and pipe the final (potentially unterminated) line to another program for post-processing?
socat
asked Jul 27 at 20:43
abc
284
I am using socat to intercept UDP messages and send them to a named pipe:
socat UDP-LISTEN:9999,fork PIPE:/tmp/mypipe,append
I am able to tail this pipe and see all the messages it receives.
I would like to pipe the output of tail -f /tmp/mypipe to sed to do some post-processing of the messages, but unfortunately some of them are not newline-terminated. This is a problem because it means multiple distinct UDP messages could be on the same line, and also because tail -f /tmp/mypipe | sed ... does not pass the last line if it is unterminated.
Ideally I would like to be able to add a custom message delimiter as they are sent to my pipe, so that I can easily find the message boundaries. If that's not possible then is there some way I can follow this file and pipe the final (potentially unterminated) line to another program for post-processing?
socat
asked Jul 27 at 20:43
abc
284
edited Jul 27 at 20:56
asked Jul 27 at 20:43
abc
284
asked Jul 27 at 20:43
abc
284
asked Jul 27 at 20:43
abc
284
284
I was able to get closer to my goal, though it's not perfect. NeitherUDP-LISTENnorUDP-RECVseemed to work -- when I tried to execute a system command to echo a trailing newline it appears they would hang oncat;. UsingUDP-RECVFROMlike so seems to work:socat -u UDP-RECVFROM:9999,fork SYSTEM:"cat; echo "n""
– abc
Jul 27 at 22:21
The solution in my comment above appears to suffer from a race condition -- since the receives are forked (as they have to be, elsesocatexits after the first UDP message) theSYSTEMcommands are not executed atomically. I noticed thecatandechogetting interleaved when multiple messages came in at the same time. I serialized it by wrapping it in a lock like this:socat -u UDP-RECVFROM:9999,fork SYSTEM:"(flock -x 200; cat; echo "n";) 200>./udplockfile"
– abc
Jul 27 at 22:43
add a comment |Â
I was able to get closer to my goal, though it's not perfect. NeitherUDP-LISTENnorUDP-RECVseemed to work -- when I tried to execute a system command to echo a trailing newline it appears they would hang oncat;. UsingUDP-RECVFROMlike so seems to work:socat -u UDP-RECVFROM:9999,fork SYSTEM:"cat; echo "n""
– abc
Jul 27 at 22:21
The solution in my comment above appears to suffer from a race condition -- since the receives are forked (as they have to be, elsesocatexits after the first UDP message) theSYSTEMcommands are not executed atomically. I noticed thecatandechogetting interleaved when multiple messages came in at the same time. I serialized it by wrapping it in a lock like this:socat -u UDP-RECVFROM:9999,fork SYSTEM:"(flock -x 200; cat; echo "n";) 200>./udplockfile"
– abc
Jul 27 at 22:43
I was able to get closer to my goal, though it's not perfect. Neither
UDP-LISTEN nor UDP-RECV seemed to work -- when I tried to execute a system command to echo a trailing newline it appears they would hang on cat;. Using UDP-RECVFROM like so seems to work: socat -u UDP-RECVFROM:9999,fork SYSTEM:"cat; echo "n""– abc
Jul 27 at 22:21
I was able to get closer to my goal, though it's not perfect. Neither
UDP-LISTEN nor UDP-RECV seemed to work -- when I tried to execute a system command to echo a trailing newline it appears they would hang on cat;. Using UDP-RECVFROM like so seems to work: socat -u UDP-RECVFROM:9999,fork SYSTEM:"cat; echo "n""– abc
Jul 27 at 22:21
The solution in my comment above appears to suffer from a race condition -- since the receives are forked (as they have to be, else
socat exits after the first UDP message) the SYSTEM commands are not executed atomically. I noticed the cat and echo getting interleaved when multiple messages came in at the same time. I serialized it by wrapping it in a lock like this: socat -u UDP-RECVFROM:9999,fork SYSTEM:"(flock -x 200; cat; echo "n";) 200>./udplockfile"– abc
Jul 27 at 22:43
The solution in my comment above appears to suffer from a race condition -- since the receives are forked (as they have to be, else
socat exits after the first UDP message) the SYSTEM commands are not executed atomically. I noticed the cat and echo getting interleaved when multiple messages came in at the same time. I serialized it by wrapping it in a lock like this: socat -u UDP-RECVFROM:9999,fork SYSTEM:"(flock -x 200; cat; echo "n";) 200>./udplockfile"– abc
Jul 27 at 22:43
add a comment |Â
1 Answer
1
active
oldest
votes
up vote
3
down vote
accepted
One possibility that does not involve forking is to use the socat verbose output rather than the data. My version of socat -v includes the length of the data in the verbose output, so you know where it ends. For example,
mkfifo mypipe
while sleep 3
do printf "%sNONEWLINE" $RANDOM
done |
socat -u - UDP4:localhost:9999 &
socat -u -v UDP-RECV:9999 - >/dev/null 2>mypipe &
cat -uv mypipe
will output before each data item (eg 9430NONEWLINE) a header starting > with the date and a length.
> 2018/07/28 10:29:33.965222 length=13 from=0 to=12
9430NONEWLINE> 2018/07/28 10:29:36.968335 length=14 from=13 to=26
26947NONEWLINE> 2018/07/28 10:29:39.971025 length=14 from=27 to=40
15126NONEWLINE
answered Jul 28 at 8:34
meuh
29k11648
add a comment |Â
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
3
down vote
accepted
One possibility that does not involve forking is to use the socat verbose output rather than the data. My version of socat -v includes the length of the data in the verbose output, so you know where it ends. For example,
mkfifo mypipe
while sleep 3
do printf "%sNONEWLINE" $RANDOM
done |
socat -u - UDP4:localhost:9999 &
socat -u -v UDP-RECV:9999 - >/dev/null 2>mypipe &
cat -uv mypipe
will output before each data item (eg 9430NONEWLINE) a header starting > with the date and a length.
> 2018/07/28 10:29:33.965222 length=13 from=0 to=12
9430NONEWLINE> 2018/07/28 10:29:36.968335 length=14 from=13 to=26
26947NONEWLINE> 2018/07/28 10:29:39.971025 length=14 from=27 to=40
15126NONEWLINE
answered Jul 28 at 8:34
meuh
29k11648
add a comment |Â
up vote
3
down vote
accepted
One possibility that does not involve forking is to use the socat verbose output rather than the data. My version of socat -v includes the length of the data in the verbose output, so you know where it ends. For example,
mkfifo mypipe
while sleep 3
do printf "%sNONEWLINE" $RANDOM
done |
socat -u - UDP4:localhost:9999 &
socat -u -v UDP-RECV:9999 - >/dev/null 2>mypipe &
cat -uv mypipe
will output before each data item (eg 9430NONEWLINE) a header starting > with the date and a length.
> 2018/07/28 10:29:33.965222 length=13 from=0 to=12
9430NONEWLINE> 2018/07/28 10:29:36.968335 length=14 from=13 to=26
26947NONEWLINE> 2018/07/28 10:29:39.971025 length=14 from=27 to=40
15126NONEWLINE
answered Jul 28 at 8:34
meuh
29k11648
add a comment |Â
up vote
3
down vote
accepted
up vote
3
down vote
accepted
One possibility that does not involve forking is to use the socat verbose output rather than the data. My version of socat -v includes the length of the data in the verbose output, so you know where it ends. For example,
mkfifo mypipe
while sleep 3
do printf "%sNONEWLINE" $RANDOM
done |
socat -u - UDP4:localhost:9999 &
socat -u -v UDP-RECV:9999 - >/dev/null 2>mypipe &
cat -uv mypipe
will output before each data item (eg 9430NONEWLINE) a header starting > with the date and a length.
> 2018/07/28 10:29:33.965222 length=13 from=0 to=12
9430NONEWLINE> 2018/07/28 10:29:36.968335 length=14 from=13 to=26
26947NONEWLINE> 2018/07/28 10:29:39.971025 length=14 from=27 to=40
15126NONEWLINE
answered Jul 28 at 8:34
meuh
29k11648
One possibility that does not involve forking is to use the socat verbose output rather than the data. My version of socat -v includes the length of the data in the verbose output, so you know where it ends. For example,
mkfifo mypipe
while sleep 3
do printf "%sNONEWLINE" $RANDOM
done |
socat -u - UDP4:localhost:9999 &
socat -u -v UDP-RECV:9999 - >/dev/null 2>mypipe &
cat -uv mypipe
will output before each data item (eg 9430NONEWLINE) a header starting > with the date and a length.
> 2018/07/28 10:29:33.965222 length=13 from=0 to=12
9430NONEWLINE> 2018/07/28 10:29:36.968335 length=14 from=13 to=26
26947NONEWLINE> 2018/07/28 10:29:39.971025 length=14 from=27 to=40
15126NONEWLINE
answered Jul 28 at 8:34
meuh
29k11648
answered Jul 28 at 8:34
meuh
29k11648
answered Jul 28 at 8:34
meuh
29k11648
answered Jul 28 at 8:34
meuh
29k11648
29k11648
add a comment |Â
add a comment |Â
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f458951%2fhow-can-i-add-message-delimiters-to-a-udp-stream-socat-is-piping%23new-answer', 'question_page');
);
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
I was able to get closer to my goal, though it's not perfect. Neither
UDP-LISTENnorUDP-RECVseemed to work -- when I tried to execute a system command to echo a trailing newline it appears they would hang oncat;. UsingUDP-RECVFROMlike so seems to work:socat -u UDP-RECVFROM:9999,fork SYSTEM:"cat; echo "n""– abc
Jul 27 at 22:21
The solution in my comment above appears to suffer from a race condition -- since the receives are forked (as they have to be, else
socatexits after the first UDP message) theSYSTEMcommands are not executed atomically. I noticed thecatandechogetting interleaved when multiple messages came in at the same time. I serialized it by wrapping it in a lock like this:socat -u UDP-RECVFROM:9999,fork SYSTEM:"(flock -x 200; cat; echo "n";) 200>./udplockfile"– abc
Jul 27 at 22:43