Quickly loading numerical data in C++
<p>I'm working on a feature detection program that uses statistical models of what various sorts of landmarks in images look like. The model uses around 100 different landmarks, and the relevant data for each landmark consists of 16 matrices of doubles, each of size about 160x160.</p>

<p>I'm currently using one text file for each landmark, and storing the values from each matrix as a space-separated line. To read the data, I read a line at a time from each file and pass it to a function that generates a stringstream from the line and then reads the values of the matrix from this stream one at a time. </p>

<p>On my computer this takes about 90 seconds to load the ~40 million doubles the model uses. There must surely be a far quicker way of doing this, but I've not found anything useful from googling, and I've got no experience with this sort of thing. </p>

<p>I'd be very grateful for any suggestions.</p>

<p><strong>Edit:</strong> Loki asked me to post code, so I've shown it below. loadFromFile is called once for each landmark. The first line of each landmark file states how many levels the model uses for this landmark (each level uses four matrices; there are four levels by default). It's a horrible mess, but I'm not sure why this is so spectularly slow.</p>

<pre><code>void loadFromFile(string filename)
{
    ifstream modelData(filename.c_str(), ifstream::in);
    string line;    
    getline(modelData,line);
    int numberOfLevels = atoi(line.c_str());

    for(size_t ii = 0; ii &lt; numberOfLevels; ++ii)
        readProfileStats(modelData);        

    modelData.close();              
}

void readProfileStats(ifstream&amp; fileStream)
{
    string line;
    getline(fileStream, line);
    Vector meanProfile = readMatrixFromString(line);

    getline(fileStream, line);
    Matrix principalComponents = readMatrixFromString(line);

    getline(fileStream, line);  
    Matrix covarianceMatrixInverse = readMatrixFromString(line);

    m_statsLevels.push_back(ProfileStats(meanProfile, principalComponents, covarianceMatrixInverse));
}

Matrix readMatrixFromString(const string&amp; line)
{
    stringstream stream(line);

    size_t numRows;
    size_t numCols; 

    stream &gt;&gt; numRows;  
    stream &gt;&gt; numCols;      

    Matrix matrix(numRows,numCols);

    for(int ii = 0; ii &lt; numRows; ++ii)
    {                                       
        for(int jj = 0; jj &lt; numCols; ++jj)             
            stream &gt;&gt; matrix(ii,jj);                                    
    }                                                       

    return matrix;                      
}
</code></pre>
