Reading a dynamic record length binary file I am trying to read a dynamic record length binary file, which has data like this:

field1field2field3field4vector1vector2


Here, field3 defines the occurrence of vectors.

For example, field3 is 2, then vector1&amp;2 would be present and if the value is 3 then vector1,2&amp;3 would be present.

Let's say field1-4 is fixed, the length is 2652 and each vector length is 301. field3 is 3 bytes from position 2396.

I have written the below code which is working fine but giving very bad performance.

my $string;
my $rep_factor;
my $size;

open (FILE, $ARGV[0]) or die $!;
my $re = 2396;
my $rec = 0;
while (&lt;FILE&gt;) {
   seek(FILE,$re,0);
   read(FILE,$rep_factor,2);
   my $rep_fact = undefined2defined(convert2ascii_decimal($rep_factor,0));
   $size = ($rep_fact * 301) + 2652;
   seek(FILE,$rec,0);
   read FILE,$string,$size;

   filewrite ($ARGV[1], recordparse($string));
   $rec = $size + $rec;
   $re = $size + 2396;
}
open (FILE, $ARGV[0]) or die $!;


It won't really make a difference in this script, but the standard is to use a variable rather than a bareword.  

open (my $fh, $ARGV[0]) or die $!;


This gives you the benefits of lexical scoping.  

while (&lt;FILE&gt;) {


What this says is while there are still lines in &lt;FILE&gt;, do the things in the block.  It separates lines by the line ending marker, which is normally a linefeed.  This would normally be used:  

while ( my $line = &lt;$fh&gt; ) {
    chomp $line;


Then you could break the line up into parts with unpack.  

    my ( $field1and2, $vector_count, $field4, $vector_field ) = unpack( 'A2395 A3 A254 A*', $line );


Note that I may have the format wrong.  It's easier if one can test against a valid file.  You may not even need that much:

    my ( undef, $vector_field ) = unpack( 'A2652 A*', $line );


It seems like you only use the vectors portion.  

Instead, you use the while loop just to keep count.  However, that wouldn't normally work unless records were terminated with linefeeds.  

open (my $fh, $ARGV[0]) or die $!;
while ( my $line = &lt;$fh&gt; ) {
    chomp $line;

    filewrite ($ARGV[1], recordparse($line));
}


See how that's simpler?  No more seeks.  This doesn't even require unpack, as it looks like you pass the whole line to recordparse.

One more thing:

filewrite ($ARGV[1], recordparse($string));


I'm not sure what this function does, but it looks like it takes a file name and writes to it.  This suggests that it opens the file each time.  Better would be to open the file once and pass a filehandle to it.  



If you were getting multiple records with the original code but there are no linefeeds in the file, then I suspect that the seek and read commands are resetting the line input operator.  

Alternate version to handle reading the entire file at once (assumes no linefeeds in the data).  

open (my $fh, $ARGV[0]) or die $!;
my $file = &lt;$fh&gt;;
while ( '' ne $file ) {
    my $rep_factor = substr $file, 2396, 3;
    my $vector_count = undefined2defined(convert2ascii_decimal($rep_factor, 0));
    my $size = ($vector_count * 301) + 2652;

    my $record = substr $file, 0, $size;
    $file = substr $file, $size;

    filewrite ($ARGV[1], recordparse($record));
}


This uses substr rather than unpack because it's easier to specify different lengths with substr.  The first version asks for $size bytes starting at the beginning.  The second version says to return everything but the first $size bytes.  

I'm a little worried that there is some maximum length that the &lt;&gt; operator returns, but I don't know what it might be.  Since I don't have an example file, I'm not testing the code at all.