title: Optimize perl script to filter rows based on date in the file
body: <p>I am a beginner with programming not just perl ! 
  Please let me know what needs to change or how else this can be done.</p>
  
  <p>Need to optimize the perl code to run faster. 
  For a test run, with around a 500MB file with 3 million rows in it, runtime is 28 minutes.</p>
  
  <p>I know a tool which processes the 39 million rows in 15 mins, but i want to acheive this running on the command prompt without resorting to the tool.</p>
  
  <p>Earlier I used Date::Manip and Date::Parse and moved on to DateTime, thinking it should be faster.</p>
  
  <p>My approach was If the dates are ISO-8601 (ie, YYYY-MM-DD) and we do not need to validate them, 
  we can compare lexicographically (ie, the lt and gt operators.) </p>
  
  <ul>
  <li>Input File Date Format is 07/18/2013 13:45:49 </li>
  <li>Input File Size 42GB.</li>
  <li>Number of Rows 39 Million.</li>
  <li>Column Delimiter : |~|</li>
  <li>Platform :   GNU/Linux</li>
  </ul>
  
  <p>I have tried ">" and "gt" and did not find any difference in runtime.</p>
  
  <pre><code>Code snippet:
  
  use DateTime::Format::Strptime;
  
  my $idate = "07/17/2013 00:00:00";
  
  my $Strp = DateTime::Format::Strptime-&gt;new(
                     pattern     =&gt; '%m/%d/%Y %H:%M:%S',
                    );
  
  my $inputdt = $Strp-&gt;parse_datetime($idate);
  
  open (FILE,"myinputfile.dat") or die "could not input File\n";
  while (defined(my $line = &lt;FILE&gt;)) {
      my @chunks = split '[|]~[|]', $line;
      my $fdate = $Strp-&gt;parse_datetime($chunks[6]);
      if ( $fdate &gt; $inputdt) {
      open(FILEOUT, "&gt;&gt;myoutputfile.dat") or die "Could not write\n";
      print FILEOUT "$line";
                           }
  }
  close(FILE);
  close (FILEOUT);
  </code></pre>