How to prevent I/O buffering for piped commands

The name of the pictureThe name of the pictureThe name of the pictureClash Royale CLAN TAG#URR8PPP











up vote
0
down vote

favorite












The following chain of commands are to



  1. ping with datestamp (UNIX),


  2. convert the UNIX datestamp to more human-readable format, and


  3. output to the terminal and a log file.


ping -D localhost 2>&1 | sed 's/^[([0-9]*.[0-9]*)](.*$)/echo "[`date -d @1 +"%Y-%m-%d %H:%M:%S"`] 2"/e' | tee -a ping.log



Problem is that as written, the output seems to be buffered in chunks of almost a minute or ~50 lines, unlike the usual second-by-second and line-by-line output from ping.



What is causing the buffering and how can it be avoided?






bash







share|improve this question

















  • 1




    Apparently related: Process each line of output from ping immediately in pipeline. If you have GNU sed you can try adding the -u or --unbuffered option
    – steeldriver
    Sep 11 at 1:03











  • Great! From one of the answers in the link above, adding -u flag to sed avoids the buffering.
    – adatum
    Sep 11 at 1:07










  • It's the same question, and the same answer, as at unix.stackexchange.com/questions/467039 but with sed instead of grep. The specific C program being invoked makes little difference.
    – JdeBP
    Sep 11 at 8:57














up vote
0
down vote

favorite












The following chain of commands are to



  1. ping with datestamp (UNIX),


  2. convert the UNIX datestamp to more human-readable format, and


  3. output to the terminal and a log file.


ping -D localhost 2>&1 | sed 's/^[([0-9]*.[0-9]*)](.*$)/echo "[`date -d @1 +"%Y-%m-%d %H:%M:%S"`] 2"/e' | tee -a ping.log



Problem is that as written, the output seems to be buffered in chunks of almost a minute or ~50 lines, unlike the usual second-by-second and line-by-line output from ping.



What is causing the buffering and how can it be avoided?






bash







share|improve this question

















  • 1




    Apparently related: Process each line of output from ping immediately in pipeline. If you have GNU sed you can try adding the -u or --unbuffered option
    – steeldriver
    Sep 11 at 1:03











  • Great! From one of the answers in the link above, adding -u flag to sed avoids the buffering.
    – adatum
    Sep 11 at 1:07










  • It's the same question, and the same answer, as at unix.stackexchange.com/questions/467039 but with sed instead of grep. The specific C program being invoked makes little difference.
    – JdeBP
    Sep 11 at 8:57












up vote
0
down vote

favorite









up vote
0
down vote

favorite











The following chain of commands are to



  1. ping with datestamp (UNIX),


  2. convert the UNIX datestamp to more human-readable format, and


  3. output to the terminal and a log file.


ping -D localhost 2>&1 | sed 's/^[([0-9]*.[0-9]*)](.*$)/echo "[`date -d @1 +"%Y-%m-%d %H:%M:%S"`] 2"/e' | tee -a ping.log



Problem is that as written, the output seems to be buffered in chunks of almost a minute or ~50 lines, unlike the usual second-by-second and line-by-line output from ping.



What is causing the buffering and how can it be avoided?






bash







share|improve this question













The following chain of commands are to



  1. ping with datestamp (UNIX),


  2. convert the UNIX datestamp to more human-readable format, and


  3. output to the terminal and a log file.


ping -D localhost 2>&1 | sed 's/^[([0-9]*.[0-9]*)](.*$)/echo "[`date -d @1 +"%Y-%m-%d %H:%M:%S"`] 2"/e' | tee -a ping.log



Problem is that as written, the output seems to be buffered in chunks of almost a minute or ~50 lines, unlike the usual second-by-second and line-by-line output from ping.



What is causing the buffering and how can it be avoided?






bash




bash






share|improve this question













share|improve this question











share|improve this question




share|improve this question










asked Sep 11 at 0:55









adatum

577




577







  • 1




    Apparently related: Process each line of output from ping immediately in pipeline. If you have GNU sed you can try adding the -u or --unbuffered option
    – steeldriver
    Sep 11 at 1:03











  • Great! From one of the answers in the link above, adding -u flag to sed avoids the buffering.
    – adatum
    Sep 11 at 1:07










  • It's the same question, and the same answer, as at unix.stackexchange.com/questions/467039 but with sed instead of grep. The specific C program being invoked makes little difference.
    – JdeBP
    Sep 11 at 8:57












  • 1




    Apparently related: Process each line of output from ping immediately in pipeline. If you have GNU sed you can try adding the -u or --unbuffered option
    – steeldriver
    Sep 11 at 1:03











  • Great! From one of the answers in the link above, adding -u flag to sed avoids the buffering.
    – adatum
    Sep 11 at 1:07










  • It's the same question, and the same answer, as at unix.stackexchange.com/questions/467039 but with sed instead of grep. The specific C program being invoked makes little difference.
    – JdeBP
    Sep 11 at 8:57







1




1




Apparently related: Process each line of output from ping immediately in pipeline. If you have GNU sed you can try adding the -u or --unbuffered option
– steeldriver
Sep 11 at 1:03





Apparently related: Process each line of output from ping immediately in pipeline. If you have GNU sed you can try adding the -u or --unbuffered option
– steeldriver
Sep 11 at 1:03













Great! From one of the answers in the link above, adding -u flag to sed avoids the buffering.
– adatum
Sep 11 at 1:07




Great! From one of the answers in the link above, adding -u flag to sed avoids the buffering.
– adatum
Sep 11 at 1:07












It's the same question, and the same answer, as at unix.stackexchange.com/questions/467039 but with sed instead of grep. The specific C program being invoked makes little difference.
– JdeBP
Sep 11 at 8:57




It's the same question, and the same answer, as at unix.stackexchange.com/questions/467039 but with sed instead of grep. The specific C program being invoked makes little difference.
– JdeBP
Sep 11 at 8:57










1 Answer
1






active

oldest

votes

















up vote
2
down vote













If it's available in your system, use unbuffer. Should be as easy as:



unbuffer ping -D localhost 2>&1 | unbuffer sed 's/^[([0-9]*.[0-9]*)](.*$)/echo "[`date -d @1 +"%Y-%m-%d %H:%M:%S"`] 2"/e' | tee -a ping.log





share|improve this answer




















  • sed -u looks like a simpler and cleaner solution for now, but do you know when it would be preferable to use unbuffer? And which commands would need it?
    – adatum
    Sep 11 at 1:10










  • unbuffer is generic and works with any command. I've used it extensively with various programs including awk, perl and others. It's needed whenever piping is buffered automatically.
    – msb
    Sep 11 at 1:13











  • Good to know. When is piping buffered automatically? It confused me that the piped commands, depending on how many and which commands, sometimes get buffered and sometimes don't. I initially thought it was a problem with the code.
    – adatum
    Sep 11 at 4:22







  • 1




    That's a good question, idk myself. I've read around about stdbuf which is another program analogous to unbuffer that controls stdin/stdout/stderr streams. I guess it's something that either the shell or the kernel controls, but I'd have to do some research on it to know who controls. All I know is that it's external to the program. I had my own scripts with autoflush and everything, and they'd still get buffered, which was really puzzling and how I found unbuffer.
    – msb
    Sep 11 at 4:29










Your Answer







StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "106"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
convertImagesToLinks: false,
noModals: false,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);













 

draft saved


draft discarded


















StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f468142%2fhow-to-prevent-i-o-buffering-for-piped-commands%23new-answer', 'question_page');

);

Post as a guest

















1 Answer
1






active

oldest

votes








1 Answer
1






active

oldest

votes









active

oldest

votes






active

oldest

votes








up vote
2
down vote













If it's available in your system, use unbuffer. Should be as easy as:



unbuffer ping -D localhost 2>&1 | unbuffer sed 's/^[([0-9]*.[0-9]*)](.*$)/echo "[`date -d @1 +"%Y-%m-%d %H:%M:%S"`] 2"/e' | tee -a ping.log





share|improve this answer




















  • sed -u looks like a simpler and cleaner solution for now, but do you know when it would be preferable to use unbuffer? And which commands would need it?
    – adatum
    Sep 11 at 1:10










  • unbuffer is generic and works with any command. I've used it extensively with various programs including awk, perl and others. It's needed whenever piping is buffered automatically.
    – msb
    Sep 11 at 1:13











  • Good to know. When is piping buffered automatically? It confused me that the piped commands, depending on how many and which commands, sometimes get buffered and sometimes don't. I initially thought it was a problem with the code.
    – adatum
    Sep 11 at 4:22







  • 1




    That's a good question, idk myself. I've read around about stdbuf which is another program analogous to unbuffer that controls stdin/stdout/stderr streams. I guess it's something that either the shell or the kernel controls, but I'd have to do some research on it to know who controls. All I know is that it's external to the program. I had my own scripts with autoflush and everything, and they'd still get buffered, which was really puzzling and how I found unbuffer.
    – msb
    Sep 11 at 4:29














up vote
2
down vote













If it's available in your system, use unbuffer. Should be as easy as:



unbuffer ping -D localhost 2>&1 | unbuffer sed 's/^[([0-9]*.[0-9]*)](.*$)/echo "[`date -d @1 +"%Y-%m-%d %H:%M:%S"`] 2"/e' | tee -a ping.log





share|improve this answer




















  • sed -u looks like a simpler and cleaner solution for now, but do you know when it would be preferable to use unbuffer? And which commands would need it?
    – adatum
    Sep 11 at 1:10










  • unbuffer is generic and works with any command. I've used it extensively with various programs including awk, perl and others. It's needed whenever piping is buffered automatically.
    – msb
    Sep 11 at 1:13











  • Good to know. When is piping buffered automatically? It confused me that the piped commands, depending on how many and which commands, sometimes get buffered and sometimes don't. I initially thought it was a problem with the code.
    – adatum
    Sep 11 at 4:22







  • 1




    That's a good question, idk myself. I've read around about stdbuf which is another program analogous to unbuffer that controls stdin/stdout/stderr streams. I guess it's something that either the shell or the kernel controls, but I'd have to do some research on it to know who controls. All I know is that it's external to the program. I had my own scripts with autoflush and everything, and they'd still get buffered, which was really puzzling and how I found unbuffer.
    – msb
    Sep 11 at 4:29












up vote
2
down vote










up vote
2
down vote









If it's available in your system, use unbuffer. Should be as easy as:



unbuffer ping -D localhost 2>&1 | unbuffer sed 's/^[([0-9]*.[0-9]*)](.*$)/echo "[`date -d @1 +"%Y-%m-%d %H:%M:%S"`] 2"/e' | tee -a ping.log





share|improve this answer












If it's available in your system, use unbuffer. Should be as easy as:



unbuffer ping -D localhost 2>&1 | unbuffer sed 's/^[([0-9]*.[0-9]*)](.*$)/echo "[`date -d @1 +"%Y-%m-%d %H:%M:%S"`] 2"/e' | tee -a ping.log






share|improve this answer












share|improve this answer



share|improve this answer










answered Sep 11 at 1:06









msb

1,15079




1,15079











  • sed -u looks like a simpler and cleaner solution for now, but do you know when it would be preferable to use unbuffer? And which commands would need it?
    – adatum
    Sep 11 at 1:10










  • unbuffer is generic and works with any command. I've used it extensively with various programs including awk, perl and others. It's needed whenever piping is buffered automatically.
    – msb
    Sep 11 at 1:13











  • Good to know. When is piping buffered automatically? It confused me that the piped commands, depending on how many and which commands, sometimes get buffered and sometimes don't. I initially thought it was a problem with the code.
    – adatum
    Sep 11 at 4:22







  • 1




    That's a good question, idk myself. I've read around about stdbuf which is another program analogous to unbuffer that controls stdin/stdout/stderr streams. I guess it's something that either the shell or the kernel controls, but I'd have to do some research on it to know who controls. All I know is that it's external to the program. I had my own scripts with autoflush and everything, and they'd still get buffered, which was really puzzling and how I found unbuffer.
    – msb
    Sep 11 at 4:29
















  • sed -u looks like a simpler and cleaner solution for now, but do you know when it would be preferable to use unbuffer? And which commands would need it?
    – adatum
    Sep 11 at 1:10










  • unbuffer is generic and works with any command. I've used it extensively with various programs including awk, perl and others. It's needed whenever piping is buffered automatically.
    – msb
    Sep 11 at 1:13











  • Good to know. When is piping buffered automatically? It confused me that the piped commands, depending on how many and which commands, sometimes get buffered and sometimes don't. I initially thought it was a problem with the code.
    – adatum
    Sep 11 at 4:22







  • 1




    That's a good question, idk myself. I've read around about stdbuf which is another program analogous to unbuffer that controls stdin/stdout/stderr streams. I guess it's something that either the shell or the kernel controls, but I'd have to do some research on it to know who controls. All I know is that it's external to the program. I had my own scripts with autoflush and everything, and they'd still get buffered, which was really puzzling and how I found unbuffer.
    – msb
    Sep 11 at 4:29















sed -u looks like a simpler and cleaner solution for now, but do you know when it would be preferable to use unbuffer? And which commands would need it?
– adatum
Sep 11 at 1:10




sed -u looks like a simpler and cleaner solution for now, but do you know when it would be preferable to use unbuffer? And which commands would need it?
– adatum
Sep 11 at 1:10












unbuffer is generic and works with any command. I've used it extensively with various programs including awk, perl and others. It's needed whenever piping is buffered automatically.
– msb
Sep 11 at 1:13





unbuffer is generic and works with any command. I've used it extensively with various programs including awk, perl and others. It's needed whenever piping is buffered automatically.
– msb
Sep 11 at 1:13













Good to know. When is piping buffered automatically? It confused me that the piped commands, depending on how many and which commands, sometimes get buffered and sometimes don't. I initially thought it was a problem with the code.
– adatum
Sep 11 at 4:22





Good to know. When is piping buffered automatically? It confused me that the piped commands, depending on how many and which commands, sometimes get buffered and sometimes don't. I initially thought it was a problem with the code.
– adatum
Sep 11 at 4:22





1




1




That's a good question, idk myself. I've read around about stdbuf which is another program analogous to unbuffer that controls stdin/stdout/stderr streams. I guess it's something that either the shell or the kernel controls, but I'd have to do some research on it to know who controls. All I know is that it's external to the program. I had my own scripts with autoflush and everything, and they'd still get buffered, which was really puzzling and how I found unbuffer.
– msb
Sep 11 at 4:29




That's a good question, idk myself. I've read around about stdbuf which is another program analogous to unbuffer that controls stdin/stdout/stderr streams. I guess it's something that either the shell or the kernel controls, but I'd have to do some research on it to know who controls. All I know is that it's external to the program. I had my own scripts with autoflush and everything, and they'd still get buffered, which was really puzzling and how I found unbuffer.
– msb
Sep 11 at 4:29

















 

draft saved


draft discarded















































 


draft saved


draft discarded














StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f468142%2fhow-to-prevent-i-o-buffering-for-piped-commands%23new-answer', 'question_page');

);

Post as a guest
































































-

Popular posts from this blog

How to check contact read email or not when send email to Individual?

Image
Clash Royale CLAN TAG #URR8PPP up vote 1 down vote favorite I'm using WordPress 4.9.8, CiviCRM to 5.5.1, I usually send email to contact by Search> Find contacts View contact details Action> Send email Send email ok, Contact received mail ok like picture But status only Email sent though contact read email or not. So, can CiviCRM can change status to Email read when contact read email? wordpress email share | improve this question asked Sep 26 at 0:12 ToanLuong 49 9 add a comment  |  up vote 1 down vote favorite I'm using WordPress 4.9.8, CiviCRM to 5.5.1, I usually send email to contact by Search> Find contacts View contact details Action> Send email Send email ok, Contact received mail ok like picture But status only Email sent though contact read email or not. So, can CiviCRM can change status to Email read when contact read email? wordpress email share | improve this questi...
Read more

Christian Cage

Image
Christian Cage Christian in March 2015 Birth name William Jason Reso Born ( 1973-11-30 ) November 30, 1973 (age 45) Kitchener, Ontario, Canada Residence Tampa, Florida, United States Spouse(s) Denise Hartmann ( m.   2001 ) Children 1 Professional wrestling career Ring name(s) Christian [1] Christian Cage [2] Conquistador Dos [3] Male Nurse [4] Billed height 6 ft 1 in (1.85 m) [1] Billed weight 212 lb (96 kg) [1] Billed from Tampa, Florida (WM18, TNA) Toronto, Ontario, Canada [1] Trained by Dory Funk Jr. [2] Ron Hutchison [5] Debut 1995 [5] Retired 2014 [6] William Jason Reso (born November 30, 1973) is a Canadian actor, podcaster and retired professional wrestler best known for his time in WWE under the ring name Christian , a shortened version of his original ring name Christian Cage that was also used during his tenure in Total Nonstop Action Wrestling (TNA). Reso was trained by former professional wrestlers Ron Hutchison and Dory Funk Jr. and m...
Read more

Running qemu-guest-agent on windows server 2008

Image
Clash Royale CLAN TAG #URR8PPP up vote 0 down vote favorite I know how I need to configure any linux guestvm in order to make qemu-guest-agent work. This is because there is a lot of documentation on it available, this is not the same with a windows guest. I wanted to know what steps I need to take to make qemu-GA work with a windows guest. I know I need to specify a channel type in the xml file of my windows vm, but further than that?? Does somebody has some more experience with this? Thanks. Running system: Debian 9 (on VMware) 3GB ram & 100GB DISK Running libvirt Daemon Several other guest vms running in background Windows guest: Windows server 2012 512MB RAM & 32GB DISK Virtio drivers working windows virtual-machine kvm qemu libvirtd share | improve this question asked Jan 13 at 0:02 codenerd123 1 1 add a comment  |  up vote 0 down vote favorite I know how I need to configure any linux guestvm in...
Read more