mjhjmtu

Update: This behavior is observed on Windows Subsystem for Linux. It seems there are two issues we are dealing with here:

1. Some bug/race condition internal to the system. This is incorrect, see answers.




2. Default buffer size for head.

For (2), as @kusalanda mentioned, head may have some default buffer size that consumes the input up to a certain point. On ArchLinux, we can see that for i < 10, we consistently see no output from tail. The same is true for Windows Subsystem for Linux (i.e. no inconsistent output for tail).
For (1), it is possible that there is some bug internal to the Windows Subsystem for Linux itself that causes this race condition, as we do not observe such behavior in ArchLinux. This is incorrect, see answers. There is a "point 1", but it is different.

I am trying to run the following commands in bash version 4.4.19:

 for ((i = 0; i < 1000; ++i)); do echo $i; done; | head -n 1; echo ...; tail -n 1; 

Sometimes, I see the expected results:

$ ~ for ((i = 0; i < 1000; ++i)); do echo $i; done; | head -n 1; echo ...; tail -n 1; 
0
...
999
$ ~

However, more often than not, I see the following:

$ ~ for ((i = 0; i < 1000; ++i)); do echo $i; done; | head -n 1; echo ...; tail -n 1; 
0
...
$ ~

I suspect this is a race condition. However, if I add a sleep at the beginning of the second block of commands, the "race condition" still happens:

$ ~ for ((i = 0; i < 1000; ++i)); do echo $i; done; | sleep 10; head -n 1; echo ...; tail -n 1; 
0
...
$ ~

Is this actually a race condition? What should I do to make the second block of code see the whole input? Note that if I use 10000 instead of 1000, then I do not see this issue (it is possible that these all just happen to be lucky cases though):

$ ~ for ((i = 0; i < 10000; ++i)); do echo $i; done; | head -n 1; echo ...; tail -n 1; 
0
...
9999
$ ~ for ((i = 0; i < 10000; ++i)); do echo $i; done; | head -n 1; echo ...; tail -n 1; 
0
...
9999
$ ~ for ((i = 0; i < 10000; ++i)); do echo $i; done; | head -n 1; echo ...; tail -n 1; 
0
...
9999
$ ~ for ((i = 0; i < 10000; ++i)); do echo $i; done; | head -n 1; echo ...; tail -n 1; 
0
...
9999
$ ~ for ((i = 0; i < 10000; ++i)); do echo $i; done; | head -n 1; echo ...; tail -n 1; 
0
...
9999
$ ~ for ((i = 0; i < 10000; ++i)); do echo $i; done; | head -n 1; echo ...; tail -n 1; 
0
...
9999
$ ~

This is no race condition and no bug in WSL or ArchLinux.

As you mention, it's because head is reading more than it "should", and so it may not leave enough or anything at all for tail to work on. But there is nothing in the standard or elsewhere which says that head should only read a certain amount of bytes; it could just as well read the whole file and then discard everything but its first line.

In order to "fix" that in all possible cases, head would have to always read its input byte by byte (ie do a system call for each byte) and that would be horrendously inefficient, and absolutely useless in 99.999% of cases.

If you want to avoid that, you can

1) use a temporary file instead of a pipe; then

 head -n <tmpfile; tail -n <tmpfile; 

will work as expected.

2) reimplement your head/tail combination with something else, eg. in awk:

$ seq 10000 20000 | awk -vH=2 -vT=3 'if(NR<=H)print; else a[i++%T]=$0ENDif((j=i-T)>0)print "..."; else j=0; while(j<i)print a[j++%T]'
10000
10001
...
19998
19999
20000

Note: If any information is incorrect, please comment so I can fix or delete.

As @mosvy and @MichaelHomer mentioned in the comments, this is due to the scheduler scheduling each side of the pipe differently, and at different times. To be clear, we are answering why the following has inconsistent output:

 for ((i = 0; i < 1000; ++i)); do echo $i; done | head -n 1; echo ...; tail -n 1; 

With output like:

0
...

and:

0
...
999

Two key points are at play here. The short answer is that because the input into the right side of the pipe is not always all available at once (point 1), head will "consume" different amounts. If the whole input is available (meaning the left side finished first), then the whole input will be consumed due to the implementation of head as explained by @Kusalananda and @mosvy (point 2).

We will first show point 1. The easiest way to show this is to replace tail with head:

$ ~ for ((i = 0; i < 1000; ++i)); do echo $i; done | head -n 1; echo ...; head -n 1; 
0
...
878
$ ~ for ((i = 0; i < 1000; ++i)); do echo $i; done | head -n 1; echo ...; head -n 1; 
0
...
820
$ ~ for ((i = 0; i < 1000; ++i)); do echo $i; done | head -n 1; echo ...; head -n 1; 
0
...
$ ~ for ((i = 0; i < 1000; ++i)); do echo $i; done | head -n 1; echo ...; head -n 1; 
0
...
796

As we can see, the output of the second head is different each time. This shows that the input from the left side is not always available all at once (point 1).

For each of the case in which there is a number after ..., we will get an output of 999 if we used tail instead. For the case in which nothing came after ..., we will see the same for tail. To prove this, we will show point 2.

Although there is nothing we can really do about point 1, we can make it more stable by writing it to a file:

$ ~ for ((i = 0; i < 1000; ++i)); do echo $i; done >input

With the file, we will read it through a pipe (see below for redirection case):

$ ~ cat input | head -n 1; echo ...; tail -n 1; 
0
...

And indeed, head consumes everything, leaving nothing for tail. As such, we have point 2. So with point 1 and point 2, we can explain the inconsistent behavior:

In my version of head, at least 1000 lines will be consumed at a time if read through a pipe, and at least 1000 lines are available (the whole thing if less). If all of the left side finishes before the right side even starts, head will consume everything, leaving nothing for tail. If, however, the left side does not finish, head will only consume those that are done. This means something is leftover for tail, thus leaving an output.

Redirection

So in the above example, we used a pipe to provide the result. The reasoning is that if we used redirection, we will end up with the following result:

$ ~ head -n 1; echo ...; tail -n 1; <input
0
...
999

Which is different from the explanation above. The reasoning is that when used this way, it seems head only reads 1 line:

$ ~ head -n 1; echo ...; head -n 1; <input
0
...
1

The way to explain this is to reference the answer here. In short:

• pipes are not lseek()'able so commands can't read some data and then rewind back, but when you redirect with > or < usually it's a file which is lseek() able object, so commands can navigate however they please.

In other words, head need not consume everything if it is able to seek the file directly. It only need to read as much as it need. Once it finds a newline, it can put everything back. We can prove this by using a file with 1 byte after a newline:

$ ~ cat input
0123456789
1
$ ~ head -n 1; head -c 1; <input
0123456789
1$ ~

If we were to use a pipe, the whole input is consumed, with nothing left for the second head:

$ ~ cat input | head -n 1; head -c 1; 
0123456789
$ ~

As a side note, if we used process substitution (which results in a non-seekable read as I understand it), we will get the same result:

$ ~ head -n 1; head -c 1; < <(cat input)
0123456789
$ ~