Efficient use of python objects to iterate over a large set of data calling python functions
<p>My application has generated a large (>4gb) of data and I need to iterate over it calling a python function on every data element.</p>

<p>My data will be broken down into "rows" and I'll be defining several python variables (objecs) which will reference this data. Ideally these objects would be in native form, i.e. my data will have doubles, integers and arrays of char etc, i.e.</p>

<pre><code>struct DataRow
{
    char key[ 32 ];
    double value;
    int source;
};
DataRow *rows = new DataRow[ 40000000 ];
</code></pre>

<p>I've read a little on the PyBuffer object which can "wrap" an area of memory and I'd assumed that I'd be able to construct 3 of these to "point" to the first row of data (key, value, source) and then construct a tuple and call PyObject_CallObject() on a code object I'd prepared earlier.</p>

<pre><code>PyObject *keyBuffer = PyBuffer_FromMemory( rows[ 0 ].key, 32 );
PyObject *valueBuffer = PyBuffer_FromMemory( &amp;rows[ 0 ].value, sizeof( double ) );
PyObject *sourceBuffer = PyBuffer_FromMemory( &amp;rows[ 0 ].source, sizeof( int ) );
</code></pre>

<p>However, this approach seems to present several issues..</p>

<ol>
<li>How do I advance the pointer held by the PyBuffer objects to point to the next row.</li>
<li>How do I dererence the double and int buffers to use thier values inside the python script</li>
</ol>

<p>I'm sure that this must be possible. If it's not then it may be that using python script across all the rows may be too slow given the amount of PyObjects I'd have to construct.</p>
