How to prevent I/O buffering for piped commands
Clash Royale CLAN TAG#URR8PPP
up vote
0
down vote
favorite
The following chain of commands are to
ping with datestamp (UNIX),
convert the UNIX datestamp to more human-readable format, and
output to the terminal and a log file.
ping -D localhost 2>&1 | sed 's/^[([0-9]*.[0-9]*)](.*$)/echo "[`date -d @1 +"%Y-%m-%d %H:%M:%S"`] 2"/e' | tee -a ping.log
Problem is that as written, the output seems to be buffered in chunks of almost a minute or ~50 lines, unlike the usual second-by-second and line-by-line output from ping.
What is causing the buffering and how can it be avoided?
bash
asked Sep 11 at 0:55
adatum
577
add a comment |Â
up vote
0
down vote
favorite
The following chain of commands are to
ping with datestamp (UNIX),
convert the UNIX datestamp to more human-readable format, and
output to the terminal and a log file.
ping -D localhost 2>&1 | sed 's/^[([0-9]*.[0-9]*)](.*$)/echo "[`date -d @1 +"%Y-%m-%d %H:%M:%S"`] 2"/e' | tee -a ping.log
Problem is that as written, the output seems to be buffered in chunks of almost a minute or ~50 lines, unlike the usual second-by-second and line-by-line output from ping.
What is causing the buffering and how can it be avoided?
bash
asked Sep 11 at 0:55
adatum
577
1
Apparently related: Process each line of output frompingimmediately in pipeline. If you have GNU sed you can try adding the-uor--unbufferedoption
– steeldriver
Sep 11 at 1:03
Great! From one of the answers in the link above, adding-uflag tosedavoids the buffering.
– adatum
Sep 11 at 1:07
It's the same question, and the same answer, as at unix.stackexchange.com/questions/467039 but withsedinstead ofgrep. The specific C program being invoked makes little difference.
– JdeBP
Sep 11 at 8:57
add a comment |Â
up vote
0
down vote
favorite
up vote
0
down vote
favorite
The following chain of commands are to
ping with datestamp (UNIX),
convert the UNIX datestamp to more human-readable format, and
output to the terminal and a log file.
ping -D localhost 2>&1 | sed 's/^[([0-9]*.[0-9]*)](.*$)/echo "[`date -d @1 +"%Y-%m-%d %H:%M:%S"`] 2"/e' | tee -a ping.log
Problem is that as written, the output seems to be buffered in chunks of almost a minute or ~50 lines, unlike the usual second-by-second and line-by-line output from ping.
What is causing the buffering and how can it be avoided?
bash
asked Sep 11 at 0:55
adatum
577
The following chain of commands are to
ping with datestamp (UNIX),
convert the UNIX datestamp to more human-readable format, and
output to the terminal and a log file.
ping -D localhost 2>&1 | sed 's/^[([0-9]*.[0-9]*)](.*$)/echo "[`date -d @1 +"%Y-%m-%d %H:%M:%S"`] 2"/e' | tee -a ping.log
Problem is that as written, the output seems to be buffered in chunks of almost a minute or ~50 lines, unlike the usual second-by-second and line-by-line output from ping.
What is causing the buffering and how can it be avoided?
bash
bash
asked Sep 11 at 0:55
adatum
577
asked Sep 11 at 0:55
adatum
577
asked Sep 11 at 0:55
adatum
577
asked Sep 11 at 0:55
adatum
577
asked Sep 11 at 0:55
adatum
577
577
1
Apparently related: Process each line of output frompingimmediately in pipeline. If you have GNU sed you can try adding the-uor--unbufferedoption
– steeldriver
Sep 11 at 1:03
Great! From one of the answers in the link above, adding-uflag tosedavoids the buffering.
– adatum
Sep 11 at 1:07
It's the same question, and the same answer, as at unix.stackexchange.com/questions/467039 but withsedinstead ofgrep. The specific C program being invoked makes little difference.
– JdeBP
Sep 11 at 8:57
add a comment |Â
1
Apparently related: Process each line of output frompingimmediately in pipeline. If you have GNU sed you can try adding the-uor--unbufferedoption
– steeldriver
Sep 11 at 1:03
Great! From one of the answers in the link above, adding-uflag tosedavoids the buffering.
– adatum
Sep 11 at 1:07
It's the same question, and the same answer, as at unix.stackexchange.com/questions/467039 but withsedinstead ofgrep. The specific C program being invoked makes little difference.
– JdeBP
Sep 11 at 8:57
1
1
Apparently related: Process each line of output from
ping immediately in pipeline. If you have GNU sed you can try adding the -u or --unbuffered option– steeldriver
Sep 11 at 1:03
Apparently related: Process each line of output from
ping immediately in pipeline. If you have GNU sed you can try adding the -u or --unbuffered option– steeldriver
Sep 11 at 1:03
Great! From one of the answers in the link above, adding
-u flag to sed avoids the buffering.– adatum
Sep 11 at 1:07
Great! From one of the answers in the link above, adding
-u flag to sed avoids the buffering.– adatum
Sep 11 at 1:07
It's the same question, and the same answer, as at unix.stackexchange.com/questions/467039 but with
sed instead of grep. The specific C program being invoked makes little difference.– JdeBP
Sep 11 at 8:57
It's the same question, and the same answer, as at unix.stackexchange.com/questions/467039 but with
sed instead of grep. The specific C program being invoked makes little difference.– JdeBP
Sep 11 at 8:57
add a comment |Â
1 Answer
1
active
oldest
votes
up vote
2
down vote
If it's available in your system, use unbuffer. Should be as easy as:
unbuffer ping -D localhost 2>&1 | unbuffer sed 's/^[([0-9]*.[0-9]*)](.*$)/echo "[`date -d @1 +"%Y-%m-%d %H:%M:%S"`] 2"/e' | tee -a ping.log
answered Sep 11 at 1:06
msb
1,15079
sed -ulooks like a simpler and cleaner solution for now, but do you know when it would be preferable to useunbuffer? And which commands would need it?
– adatum
Sep 11 at 1:10
unbufferis generic and works with any command. I've used it extensively with various programs including awk, perl and others. It's needed whenever piping is buffered automatically.
– msb
Sep 11 at 1:13
Good to know. When is piping buffered automatically? It confused me that the piped commands, depending on how many and which commands, sometimes get buffered and sometimes don't. I initially thought it was a problem with the code.
– adatum
Sep 11 at 4:22
1
That's a good question, idk myself. I've read around aboutstdbufwhich is another program analogous tounbufferthat controls stdin/stdout/stderr streams. I guess it's something that either the shell or the kernel controls, but I'd have to do some research on it to know who controls. All I know is that it's external to the program. I had my own scripts with autoflush and everything, and they'd still get buffered, which was really puzzling and how I found unbuffer.
– msb
Sep 11 at 4:29
add a comment |Â
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
2
down vote
If it's available in your system, use unbuffer. Should be as easy as:
unbuffer ping -D localhost 2>&1 | unbuffer sed 's/^[([0-9]*.[0-9]*)](.*$)/echo "[`date -d @1 +"%Y-%m-%d %H:%M:%S"`] 2"/e' | tee -a ping.log
answered Sep 11 at 1:06
msb
1,15079
sed -ulooks like a simpler and cleaner solution for now, but do you know when it would be preferable to useunbuffer? And which commands would need it?
– adatum
Sep 11 at 1:10
unbufferis generic and works with any command. I've used it extensively with various programs including awk, perl and others. It's needed whenever piping is buffered automatically.
– msb
Sep 11 at 1:13
Good to know. When is piping buffered automatically? It confused me that the piped commands, depending on how many and which commands, sometimes get buffered and sometimes don't. I initially thought it was a problem with the code.
– adatum
Sep 11 at 4:22
1
That's a good question, idk myself. I've read around aboutstdbufwhich is another program analogous tounbufferthat controls stdin/stdout/stderr streams. I guess it's something that either the shell or the kernel controls, but I'd have to do some research on it to know who controls. All I know is that it's external to the program. I had my own scripts with autoflush and everything, and they'd still get buffered, which was really puzzling and how I found unbuffer.
– msb
Sep 11 at 4:29
add a comment |Â
up vote
2
down vote
If it's available in your system, use unbuffer. Should be as easy as:
unbuffer ping -D localhost 2>&1 | unbuffer sed 's/^[([0-9]*.[0-9]*)](.*$)/echo "[`date -d @1 +"%Y-%m-%d %H:%M:%S"`] 2"/e' | tee -a ping.log
answered Sep 11 at 1:06
msb
1,15079
sed -ulooks like a simpler and cleaner solution for now, but do you know when it would be preferable to useunbuffer? And which commands would need it?
– adatum
Sep 11 at 1:10
unbufferis generic and works with any command. I've used it extensively with various programs including awk, perl and others. It's needed whenever piping is buffered automatically.
– msb
Sep 11 at 1:13
Good to know. When is piping buffered automatically? It confused me that the piped commands, depending on how many and which commands, sometimes get buffered and sometimes don't. I initially thought it was a problem with the code.
– adatum
Sep 11 at 4:22
1
That's a good question, idk myself. I've read around aboutstdbufwhich is another program analogous tounbufferthat controls stdin/stdout/stderr streams. I guess it's something that either the shell or the kernel controls, but I'd have to do some research on it to know who controls. All I know is that it's external to the program. I had my own scripts with autoflush and everything, and they'd still get buffered, which was really puzzling and how I found unbuffer.
– msb
Sep 11 at 4:29
add a comment |Â
up vote
2
down vote
up vote
2
down vote
If it's available in your system, use unbuffer. Should be as easy as:
unbuffer ping -D localhost 2>&1 | unbuffer sed 's/^[([0-9]*.[0-9]*)](.*$)/echo "[`date -d @1 +"%Y-%m-%d %H:%M:%S"`] 2"/e' | tee -a ping.log
answered Sep 11 at 1:06
msb
1,15079
If it's available in your system, use unbuffer. Should be as easy as:
unbuffer ping -D localhost 2>&1 | unbuffer sed 's/^[([0-9]*.[0-9]*)](.*$)/echo "[`date -d @1 +"%Y-%m-%d %H:%M:%S"`] 2"/e' | tee -a ping.log
answered Sep 11 at 1:06
msb
1,15079
answered Sep 11 at 1:06
msb
1,15079
answered Sep 11 at 1:06
msb
1,15079
answered Sep 11 at 1:06
msb
1,15079
1,15079
sed -ulooks like a simpler and cleaner solution for now, but do you know when it would be preferable to useunbuffer? And which commands would need it?
– adatum
Sep 11 at 1:10
unbufferis generic and works with any command. I've used it extensively with various programs including awk, perl and others. It's needed whenever piping is buffered automatically.
– msb
Sep 11 at 1:13
Good to know. When is piping buffered automatically? It confused me that the piped commands, depending on how many and which commands, sometimes get buffered and sometimes don't. I initially thought it was a problem with the code.
– adatum
Sep 11 at 4:22
1
That's a good question, idk myself. I've read around aboutstdbufwhich is another program analogous tounbufferthat controls stdin/stdout/stderr streams. I guess it's something that either the shell or the kernel controls, but I'd have to do some research on it to know who controls. All I know is that it's external to the program. I had my own scripts with autoflush and everything, and they'd still get buffered, which was really puzzling and how I found unbuffer.
– msb
Sep 11 at 4:29
add a comment |Â
sed -ulooks like a simpler and cleaner solution for now, but do you know when it would be preferable to useunbuffer? And which commands would need it?
– adatum
Sep 11 at 1:10
unbufferis generic and works with any command. I've used it extensively with various programs including awk, perl and others. It's needed whenever piping is buffered automatically.
– msb
Sep 11 at 1:13
Good to know. When is piping buffered automatically? It confused me that the piped commands, depending on how many and which commands, sometimes get buffered and sometimes don't. I initially thought it was a problem with the code.
– adatum
Sep 11 at 4:22
1
That's a good question, idk myself. I've read around aboutstdbufwhich is another program analogous tounbufferthat controls stdin/stdout/stderr streams. I guess it's something that either the shell or the kernel controls, but I'd have to do some research on it to know who controls. All I know is that it's external to the program. I had my own scripts with autoflush and everything, and they'd still get buffered, which was really puzzling and how I found unbuffer.
– msb
Sep 11 at 4:29
sed -u looks like a simpler and cleaner solution for now, but do you know when it would be preferable to use unbuffer? And which commands would need it?– adatum
Sep 11 at 1:10
sed -u looks like a simpler and cleaner solution for now, but do you know when it would be preferable to use unbuffer? And which commands would need it?– adatum
Sep 11 at 1:10
unbuffer is generic and works with any command. I've used it extensively with various programs including awk, perl and others. It's needed whenever piping is buffered automatically.– msb
Sep 11 at 1:13
unbuffer is generic and works with any command. I've used it extensively with various programs including awk, perl and others. It's needed whenever piping is buffered automatically.– msb
Sep 11 at 1:13
Good to know. When is piping buffered automatically? It confused me that the piped commands, depending on how many and which commands, sometimes get buffered and sometimes don't. I initially thought it was a problem with the code.
– adatum
Sep 11 at 4:22
Good to know. When is piping buffered automatically? It confused me that the piped commands, depending on how many and which commands, sometimes get buffered and sometimes don't. I initially thought it was a problem with the code.
– adatum
Sep 11 at 4:22
1
1
That's a good question, idk myself. I've read around about
stdbuf which is another program analogous to unbuffer that controls stdin/stdout/stderr streams. I guess it's something that either the shell or the kernel controls, but I'd have to do some research on it to know who controls. All I know is that it's external to the program. I had my own scripts with autoflush and everything, and they'd still get buffered, which was really puzzling and how I found unbuffer.– msb
Sep 11 at 4:29
That's a good question, idk myself. I've read around about
stdbuf which is another program analogous to unbuffer that controls stdin/stdout/stderr streams. I guess it's something that either the shell or the kernel controls, but I'd have to do some research on it to know who controls. All I know is that it's external to the program. I had my own scripts with autoflush and everything, and they'd still get buffered, which was really puzzling and how I found unbuffer.– msb
Sep 11 at 4:29
add a comment |Â
Â
draft saved
draft discarded
Â
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f468142%2fhow-to-prevent-i-o-buffering-for-piped-commands%23new-answer', 'question_page');
);
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
1
Apparently related: Process each line of output from
pingimmediately in pipeline. If you have GNU sed you can try adding the-uor--unbufferedoption– steeldriver
Sep 11 at 1:03
Great! From one of the answers in the link above, adding
-uflag tosedavoids the buffering.– adatum
Sep 11 at 1:07
It's the same question, and the same answer, as at unix.stackexchange.com/questions/467039 but with
sedinstead ofgrep. The specific C program being invoked makes little difference.– JdeBP
Sep 11 at 8:57