title: C-like arrays in perl
body: <p>I want to create and manipulate large arrays of (4 byte) integers in-memory. By large, I mean on the order of hundreds of million.  Each cell in the array will act as a counter for a position on a chromosome.  All I need is for it to fit in memory, and have fast (O(1)) access to elements.  The thing I'm counting is not a sparse feature, so I can't use a sparse array. </p>
  
  <p>I can't do this with a regular perl list, because perl (at least on my machine) uses 64 bytes per element, so the genomes of most of the organisms I work with are just too big.  I've tried storing the data on-disk via SQLite and hash tying, and though they work, are very slow, especially on ordinary drives. (It works reasonably ok when I run on 4-drive raid 0's).  </p>
  
  <p>I thought I could use PDL arrays, b/c PDL stores its arrays just as C does, using only 4 bytes per element.  However, I found that update speed to be excruciatingly slow compared to perl lists:</p>
  
  <pre><code>use PDL;
  use Benchmark qw/cmpthese/;
  
  my $N = 1_000_000;
  my @perl = (0 .. $N - 1);
  my $pdl = zeroes $N;
  
  cmpthese(-1,{ 
      perl =&gt; sub{
          $perl[int(rand($N))]++;
      },
      pdl =&gt; sub{
          # note that I'm not even incrementing here just setting to 1
          $pdl-&gt;set(int(rand($N)), 1);
      }
  });
  </code></pre>
  
  <p>Returns:</p>
  
  <pre><code>          Rate  pdl perl
  pdl   481208/s   -- -87%
  perl 3640889/s 657%   --    
  </code></pre>
  
  <p>Does anyone know how to increase pdl set() performance, or know of a different module that can accomplish this?</p>