mjhjmtu

If you have a very large file (which is a common reason to take a sample) you will find that:

1. 
   shuf exhausts memory


2. Using $RANDOM won't work correctly if the file exceeds 32767 lines

If you don't need "exactly" n sampled lines you can sample a ratio like this:

cat input.txt | awk 'BEGIN srand() !/^$/ if (rand() <= .01) print $0' > sample.txt

This uses constant memory, samples 1% of the file (if you know the number of lines of the file you can adjust this factor to sample a close to a limited number of lines), and works with any size of file but it will not return a precise number of lines, just a statistical ratio.

Note: The code comes from: https://stackoverflow.com/questions/692312/randomly-pick-lines-from-a-file-without-slurping-it-with-unix

Not aware of any single command which could do what you ask but here is a loop I put together which can do the job:

for i in `seq 1000`; do sed -n `echo $RANDOM % 1000000 | bc`p alargefile.txt; done > sample.txt

sed will pick up a random line on each of the 1000 passes. Possibly there are more efficient solutions.

You can save the follow code in a file (by example randextract.sh) and execute as:

randextract.sh file.txt

---- BEGIN FILE ----

#!/bin/sh -xv

#configuration MAX_LINES is the number of lines to extract
MAX_LINES=10

#number of lines in the file (is a limit)
NUM_LINES=`wc -l $1 | cut -d' ' -f1`

#generate a random number
#in bash the variable $RANDOM returns diferent values on each call
if [ "$RANDOM." != "$RANDOM." ]
then
 #bigger number (0 to 3276732767)
 RAND=$RANDOM$RANDOM
else
 RAND=`date +'%s'`
fi 

#The start line
START_LINE=`expr $RAND % '(' $NUM_LINES - $MAX_LINES ')'`

tail -n +$START_LINE $1 | head -n $MAX_LINES

---- END FILE ----

In case the shuf -n trick on large files runs out of memory and you still need a fixed size sample and an external utility can be installed then try sample:

$ sample -N 1000 < FILE_WITH_MILLIONS_OF_LINES 

The caveat is that the sample (1000 lines in the example) must fit into memory.

Disclaimer: I am the author of the recommended software.

Or like this:

LINES=$(wc -l < file) 
RANDLINE=$[ $RANDOM % $LINES ] 
tail -n $RANDLINE < file|head -1 

From the bash man page:

 RANDOM Each time this parameter is referenced, a random integer
 between 0 and 32767 is generated. The sequence of random
 numbers may be initialized by assigning a value to RAN‐
 DOM. If RANDOM is unset, it loses its special proper‐
 ties, even if it is subsequently reset.

If you file size isn't huge, you can use Sort random. This takes a little longer than shuf, but it randomizes the entire data. So, you could easily just do the following to use head as you requested:

sort -R input | head -1000 > output

This would sort the file randomly and give you the first 1000 lines.

If you know the number of lines in the file (like 1e6 in your case), you can do:

awk -v n=1e6 -v p=1000 '
 BEGIN srand()
 rand() * n-- < p p--; print' < file

If not, you can always do

awk -v n="$(wc -l < file)" -v p=1000 '
 BEGIN srand()
 rand() * n-- < p p--; print' < file

That would do two passes in the file, but still avoid storing the whole file in memory.

Another advantage over GNU shuf is that it preserves the order of the lines in the file.

Note that it assumes n is the number of lines in the file. If you want to print p out of the first n lines of the file (which has potentially more lines), you'd need to stop awk at the n^th line like:

awk -v n=1e6 -v p=1000 '
 BEGIN srand()
 rand() * n-- < p p--; print
 !n exit' < file

I like using awk for this when I want to preserve a header row, and when the sample can be an approximate percentage of the file. Works for very large files:

awk 'BEGIN srand() !/^$/ ' data.txt

Similar to @Txangel's probabilistic solution but approaching 100x faster.

perl -ne 'print if (rand() < .01)' huge_file.csv > sample.csv

If you need high performance, an exact sample size, and are happy to live with a sample gap at end of the file, you can do something like the following (samples 1000 lines from a 1m line file):

perl -ne 'print if (rand() < .0012)' huge_file.csv | head -1000 > sample.csv

.. or indeed chain a second sample method instead of head.