<p dir="auto">Hello,</p>
<p dir="auto">I dont know if that is possible, but it would great to find a way to speed up the <code class="notranslate">to_csv</code> method in Pandas.</p>
<p dir="auto">In my admittedly large dataframe with 20 million observations and 50 variables, it takes <strong>literally hours</strong> to export the data to a csv file.</p>
<p dir="auto">Reading the <code class="notranslate">csv</code> in Pandas is much faster though. I wonder what is the bottleneck here and what can be done to improve the data transfer.</p>
<p dir="auto">Csv files are ubiquitous, and a great way to share data (without being too nerdy with <code class="notranslate">hdf5</code>and other subtleties). What do you think?</p>
<p dir="auto">iotop and a simple-mined c program indicates we're nowhere<br>
near IO-bound in df.to_csv, at about ~10-15x.</p>
<p dir="auto">It might be possible to speed things up considerably with a fast path<br>
for special cases (numerical only) that don't need fancy quoting and other<br>
bells and whistles provided by the underlying csv python module.</p>
<div class="highlight highlight-source-c notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="#include &lt;stdio.h&gt;
#include &lt;stdlib.h&gt;

int main(int argc,char **argv)
{
    int i;
    FILE *f;
    char fmt[] = &quot;%f,%f,%f,%f,%f\n&quot;;
    while (1) {
    f = fopen(&quot;out.csv&quot;,&quot;wb&quot;);
    for(i=0;i&lt;1000000;i++) {
        fprintf(f,fmt, 1.0,2.0,3.0,4.0,5.0);
    }
    fclose(f);
    }
}"><pre class="notranslate"><span class="pl-k">#include</span> <span class="pl-s">&lt;stdio.h&gt;</span>
<span class="pl-k">#include</span> <span class="pl-s">&lt;stdlib.h&gt;</span>

<span class="pl-smi">int</span> <span class="pl-en">main</span>(<span class="pl-smi">int</span> <span class="pl-s1">argc</span>,<span class="pl-smi">char</span> <span class="pl-c1">*</span><span class="pl-c1">*</span><span class="pl-s1">argv</span>)
{
    <span class="pl-smi">int</span> <span class="pl-s1">i</span>;
    <span class="pl-smi">FILE</span> <span class="pl-c1">*</span><span class="pl-s1">f</span>;
    <span class="pl-smi">char</span> <span class="pl-s1">fmt</span>[] <span class="pl-c1">=</span> <span class="pl-s">"%f,%f,%f,%f,%f\n"</span>;
    <span class="pl-k">while</span> (<span class="pl-c1">1</span>) {
    <span class="pl-s1">f</span> <span class="pl-c1">=</span> <span class="pl-en">fopen</span>(<span class="pl-s">"out.csv"</span>,<span class="pl-s">"wb"</span>);
    <span class="pl-k">for</span>(<span class="pl-s1">i</span><span class="pl-c1">=</span><span class="pl-c1">0</span>;<span class="pl-s1">i</span><span class="pl-c1">&lt;</span><span class="pl-c1">1000000</span>;<span class="pl-s1">i</span><span class="pl-c1">++</span>) {
        <span class="pl-en">fprintf</span>(<span class="pl-s1">f</span>,<span class="pl-s1">fmt</span>, <span class="pl-c1">1.0</span>,<span class="pl-c1">2.0</span>,<span class="pl-c1">3.0</span>,<span class="pl-c1">4.0</span>,<span class="pl-c1">5.0</span>);
    }
    <span class="pl-en">fclose</span>(<span class="pl-s1">f</span>);
    }
}</pre></div>
<p dir="auto">sustains about 30MB/s on my machine (without even batching writes)<br>
vs ~2-3MB/s for the new (0.11.0) cython df.to_csv().</p>
<p dir="auto">need to check if it's the stringifying, quoting logic, memory layout, or something<br>
else that constitutes the difference.</p>
<p dir="auto">Should also yield insights for any future binary serialization format<br>
implemented.</p>