Parsing large CSV in Perl
I am suspicious of this line here:


my ($time, $report) = ($line =~ /(\d+),(.*)$/); # time, XML file



That line needs to scan and group the entire XML doc, but my instinct is that a 2-limited split (see note on LIMIT) would be more efficient:

my ($time, $report) = split /,/, $line, 2;


Apart from that, really, everything looks sane, and far better than a typical perl hack. I am impressed.

About perl using all your memory... are you sure? perl may just be struggling to allocate more than it is allowed. Do you have a ulimit condition that is constraining memory allocation? Are there problems with heavily fragmented memory? (You're not running 32-bit, are you... just checking?)

As it happens, for a task like this, you may find that everything is better with a language other than perl. Not to dismiss perl as incapable, but with the XPath expressions and other items your code looks surprisingly complicated. 2.5gig files are way beyond the normal I would consider for perl performance.