How do I efficiently subset a very large line-based file? This has come up repeatedly recently: I have a very large text file (in the order of several GiB) and I need to perform line-based subsetting for around 10,000 lines. There exist solutions for specific scenarios (e.g. samtools view -s for randomly sampling BAM files) but sometimes my use-case doesn’t fit into these categories.

Unfortunately a naïve sed-based solution is extremely slow:

time sed -n -f &lt;(awk -vOFS='' '{print $0, "p"}' line_numbers.txt) input_file &gt; selected_lines.txt


Where line_numbers.txt is a file containing one line number per line.

Forget running this for 10,000 lines; it’s already grinding to a halt for a mere 1000.

How can I speed this up, ideally so that it scales only with the size of the input file, and has more or less constant runtime n the number of lines that I subset?
Turns out, simply keeping track of the next candidate line (after sorting the sample line numbers) fixes the performance issue, and most of the remaining slowness seems to be due to the overhead of actually reading the file so there’s not very much to improve.

Since I don’t know how how to do this in sed, and it’s not trivial in awk either, here’s a Perl script:

#!/usr/bin/env perl

use strict;
use warnings;

my $file = $ARGV[0];
my $lines_file = $ARGV[1];

open my $lines_fh, '&lt;', $lines_file or die "Cannot read file $lines_file";
chomp (my @lines = &lt;$lines_fh&gt;);
close $lines_fh;

@lines = sort {$a &lt;=&gt; $b} @lines;

open my $fh, '&lt;', $file or die "Cannot read file $file";
my $line = 1;
my $next_line = 0;
while (&lt;$fh&gt;) {
    last if $next_line == scalar @lines;
    if ($line++ == $lines[$next_line]) {
        $next_line++;
        print;
    }
}
close $fh;


I’ve implemented a similar function in C++ for an R package, that's only slightly longer than the Perl script. It is ~3 times faster than the Perl script on my test file.