title: Scientific dataset manipulation in Clojure -- reading ByteBuffers into matrices
body: <p>I'm looking to use Clojure and Incanter for processing of a large scientific dataset; specifically, the 0.5 degree version of <a href="http://iridl.ldeo.columbia.edu/SOURCES/.NOAA/.NCEP/.CPC/.PRECL/.prcp/" rel="noreferrer">this dataset</a> (only available in binary format). </p>
  
  <p>My question is, what recommendations do you have for elegant ways to deal with this problem in Java/Clojure? Is there a simple way to get this dataset into Incanter, or some other java matrix package?</p>
  
  <p>I managed to read the binary data into a <code>java.nio.ByteBuffer</code> using the following code:</p>
  
  <pre><code>(defn to-float-array [^String str]
    (-&gt; (io/to-byte-array (io/to-file str))
        java.nio.ByteBuffer/wrap
        (.order java.nio.ByteOrder/LITTLE_ENDIAN)))
  </code></pre>
  
  <p>Now, I'm really struggling with how I can begin to manipulate this <code>ByteBuffer</code> as an array. I've been using Python's NumPy, which makes it very easy to manipulate these huge datasets. Here's the python code for what I'm looking to do:</p>
  
  <pre><code>// reshape row vector into (time, lat_slices, lon_slices)
  // then cut out every other row
  rain_data = np.fromfile("path/to/file", dtype="f")
  rain_data = rain_data.reshape(24, 360, 720);
  rain_data = rain_data[0:23:2,:,:];
  </code></pre>
  
  <p>After this slicing, I want to return a vector of these twelve arrays. (I need to manipulate them each separately as future function inputs.)</p>
  
  <p>So, any advice on how to get this dataset into Incanter would be much appreciated.</p>