title: split large xml into small chunks
body: <p>We want to process large xml file around 6 GB.</p>
  
  <p>Here we are reading the large xml into dataframe and then export it to csv file
  we are using lxml with iterparse to read xml line by line and load it into dataframe.but this process is almost taking 6 minutes.</p>
  
  <p>So we decide to split this xml into small chunks like 1GB each and then process it concurrently.</p>
  
  <p>Can you suggest the fastest way of splitting this large xml files ?</p>
  
  <p>I have already tried link <a href="https://gist.github.com/benallard/8042835" rel="nofollow noreferrer">https://gist.github.com/benallard/8042835</a> which is almost taking 8-10 just for splitting the file</p>
  
  <p>The structure of my xml is as below. In real application we have almost 200 tags under  and there are almost 100000 records with  tag</p>
  
  <pre><code>&lt;?xml version="1.0" encoding="UTF-8"?&gt;
  &lt;ACADEMICS&gt;  
    &lt;STUDENTS ASOF_DATE="11/21/2019" CREATE_DATE="11/22/2019" RECORDS="108881"&gt;      
      &lt;STUDENT&gt;      
        &lt;NAME&gt;JOHN&lt;/NAME&gt;      
        &lt;REGNUM&gt;1000&lt;/REGNUM&gt;      
        &lt;COUNTRY&gt;USA&lt;/COUNTRY&gt;      
        &lt;ID&gt;JH1&lt;/ID&gt;
        &lt;SHORT_STD_DESC&gt;JOHN IS A GOOD STUDENT&lt;/SHORT_STD_DESC&gt;
      &lt;/STUDENT&gt;
     &lt;STUDENT&gt;      
      &lt;NAME&gt;ADAM&lt;/NAME&gt;      
      &lt;REGNUM&gt;1001&lt;/REGNUM&gt;      
      &lt;COUNTRY&gt;FRANCE&lt;/COUNTRY&gt;      
      &lt;ID&gt;AD2&lt;/ID&gt;
      &lt;SHORT_STD_DESC&gt;ADAM IS A GOOD STUDENT&lt;/SHORT_STD_DESC&gt;
    &lt;/STUDENT&gt;
    &lt;STUDENT&gt;      
      &lt;NAME&gt;PETER&lt;/NAME&gt;      
      &lt;REGNUM&gt;1003&lt;/REGNUM&gt;      
      &lt;COUNTRY&gt;BELGIUM&lt;/COUNTRY&gt;      
      &lt;ID&gt;PE5&lt;/ID&gt;
      &lt;SHORT_STD_DESC&gt;PETER IS A GOOD STUDENT&lt;/SHORT_STD_DESC&gt;
  &lt;/STUDENT&gt;
  &lt;STUDENT&gt;      
      &lt;NAME&gt;ERIC&lt;/NAME&gt;      
      &lt;REGNUM&gt;1006&lt;/REGNUM&gt;      
      &lt;COUNTRY&gt;AUSTRALIA&lt;/COUNTRY&gt;      
      &lt;ID&gt;ER7&lt;/ID&gt;
      &lt;SHORT_STD_DESC&gt;ERIC IS A GOOD STUDENT&lt;/SHORT_STD_DESC&gt;
  &lt;/STUDENT&gt;
  &lt;STUDENT&gt;      
      &lt;NAME&gt;NICHOLAS&lt;/NAME&gt;      
      &lt;REGNUM&gt;1009&lt;/REGNUM&gt;      
      &lt;COUNTRY&gt;GREECE&lt;/COUNTRY&gt;      
      &lt;ID&gt;NI8&lt;/ID&gt;
      &lt;SHORT_STD_DESC&gt;NICHOLAS IS A GOOD STUDENT&lt;/SHORT_STD_DESC&gt;
  &lt;/STUDENT&gt;
  </code></pre>
  
  <p></p>