pick a random line from a very big file, from command line
Question: Suppose you have a very big file, and it'd be to expensive to go through all
the lines, or to slow.

How would you pick a line at random (preferably from command line, or python)?


Answer: You can try this from the command line - not sure if totally random, but at
least is a beginning.

    
    
    $ lines=$(wc -l file | awk '{ print $1 }'); sed -n "$((RANDOM%lines+1))p" file  
    

This works like this:

  * First, it sets a variable containing the number of lines in the file. 
    
        lines=$(wc -l file | awk '{ print $1 }')
    

  * Later, it prints a random line within that range:
    
        sed -n "$((RANDOM%lines+1))p" file
    

* * *

As Mark Ransom pointed out, the above solution reads the entire file. I have
found a way to choose a random line without (necessarily) having to read the
entire file, but just part of it. Using (I think) the same algorithm, here are
the links to both Perl and Python solutions:

  * Perl: [How do I pick a random line from a file?](http://www.perlmonks.org/?node_id=1910)
    
        perl -e 'srand;' \
         -e 'rand($.) < 1 && ($it = $_) while <>;' \
         -e 'print $it' FILE
    

  * Python: [Retrieving a Line at Random from a File of Unknown Size](http://my.safaribooksonline.com/book/programming/python/0596001673/files/pythoncook-chp-4-sect-6)
    
        import random
    
    def randomLine(file_object):
        "Retrieve a random line from a file, reading through the file once"
        lineNum = 0
        selected_line = ''
    
        while 1:
            aLine = file_object.readline(  )
            if not aLine: break
            lineNum = lineNum + 1
            # How likely is it that this is the last line of the file?
            if random.uniform(0,lineNum)<1:
                selected_line = aLine
        file_object.close(  )
        return selected_line
    

