Merging text files based on a prefix key
perl

I'm sure this can be done a few percent faster in some other language, but perl makes this super quick to implement at the optimal big-O level, and likely performs well enough. This code preserves order.

#!/usr/bin/perl -w

use strict;
use warnings;

my $key_size = $ARGV[0] =~ /--key-size=(\d+)/ ?
               (shift @ARGV, $1) : 4;

sub slurp {
  my $fn = shift;
  open my $fh, "&lt;", $fn || die "can't open $fn for reading: $!";
  my (%keys, @order);
  while(&lt;$fh&gt;) {
    chomp;
    my $key = substr $_, 0, $key_size;
    push @order, $key;
    $keys{$key} = substr $_, $key_size;
  }
  close $fh || die "can't close $fn: $!";
  return [ \@order, \%keys ];
}

$_ = slurp shift @ARGV;
my @order = @{ $_-&gt;[0] };
my %keys  = %{ $_-&gt;[1] };
while(@ARGV) {
  my ($order2, $keys2) = @{slurp shift @ARGV};
  $keys{$_} = delete $keys2-&gt;{$_}  for (grep { exists $keys2-&gt;{$_} } @order);
  for (grep { exists $keys2-&gt;{$_} } @{$order2}) {
    $keys{$_} = $keys2-&gt;{$_};
    push @order, $_;
  }
}

print "$_$keys{$_}\n"  for  @order;


Test output:

ski@anito$ perl merger.pl --key-size=4 file1 file2  file3
0010THIS_IS_LINE_1_FROM_FILE2
0011EMPTY_LINE
00A0THIS_IS_LINE_3_FROM_FILE1
0100EMPTY_LINE


And here's a slightly faster version that doesn't preserve order:

#!/usr/bin/perl -w

use strict;
use warnings;

my $key_size = $ARGV[0] =~ /--key-size=(\d+)/ ?
               (shift @ARGV, $1) : 4;

my %keys;
while(@ARGV) {
  my $fn = shift;
  open my $fh, "&lt;", $fn || die "can't open $fn for reading: $!";
  while(&lt;$fh&gt;) {
    chomp;
    $keys{substr $_, 0, $key_size} = substr $_, $key_size;
  }
  close $fh || die "can't close $fn: $!";
}

print "$_$keys{$_}\n"  for  keys %keys;