title: Large XML File Parsing in Python
body: <p>I have an XML file of size 4 GB. I want to parse it and convert it to a Data Frame to work on it. But because the file size is too large the following code is unable to convert the file to a Pandas Data Frame. The code just keeps loading and does not provide any output. But when I use it for a similar file of smaller size I obtain the correct output.</p>
  <p>Can anyone suggest any solution to this. Maybe a code that speeds up the process of conversion from XML to Data Frame or splitting of the XML file into smaller sub sets.</p>
  <p>Any suggestion whether I should work with such large XML files on my personal system (2 GB RAM) or I should use Google Colab. If Google Colab, then is there any way to upload such large files quicker to drive and thus to Colab?</p>
  <p>Following is the code I had used:</p>
  <pre><code>import xml.etree.ElementTree as ET
  tree = ET.parse(&quot;Badges.xml&quot;)
  root = tree.getroot()
  
  #Column names for DataFrame
  columns = ['row Id',&quot;UserId&quot;,'Name','Date','Class','TagBased']
  
  #Creating DataFrame
  df = pd.DataFrame(columns = columns)
  
  #Converting XML Tree to a Pandas DataFrame
  
  for node in root: 
      
      row_Id = node.attrib.get(&quot;Id&quot;)
      UserId = node.attrib.get(&quot;UserId&quot;)
      Name = node.attrib.get(&quot;Name&quot;)
      Date = node.attrib.get(&quot;Date&quot;)
      Class = node.attrib.get(&quot;Class&quot;)
      TagBased = node.attrib.get(&quot;TagBased&quot;)
      
      df = df.append(pd.Series([row_Id,UserId,Name,Date,Class,TagBased], index = columns), ignore_index = True)
  </code></pre>
  <p>Following is my XML File:</p>
  <pre><code>&lt;badges&gt;
    &lt;row Id=&quot;82946&quot; UserId=&quot;3718&quot; Name=&quot;Teacher&quot; Date=&quot;2008-09-15T08:55:03.923&quot; Class=&quot;3&quot; TagBased=&quot;False&quot; /&gt;
    &lt;row Id=&quot;82947&quot; UserId=&quot;994&quot; Name=&quot;Teacher&quot; Date=&quot;2008-09-15T08:55:03.957&quot; Class=&quot;3&quot; TagBased=&quot;False&quot; /&gt;
    &lt;row Id=&quot;82949&quot; UserId=&quot;3893&quot; Name=&quot;Teacher&quot; Date=&quot;2008-09-15T08:55:03.957&quot; Class=&quot;3&quot; TagBased=&quot;False&quot; /&gt;
    &lt;row Id=&quot;82950&quot; UserId=&quot;4591&quot; Name=&quot;Teacher&quot; Date=&quot;2008-09-15T08:55:03.957&quot; Class=&quot;3&quot; TagBased=&quot;False&quot; /&gt;
    &lt;row Id=&quot;82951&quot; UserId=&quot;5196&quot; Name=&quot;Teacher&quot; Date=&quot;2008-09-15T08:55:03.957&quot; Class=&quot;3&quot; TagBased=&quot;False&quot; /&gt;
    &lt;row Id=&quot;82952&quot; UserId=&quot;2635&quot; Name=&quot;Teacher&quot; Date=&quot;2008-09-15T08:55:03.957&quot; Class=&quot;3&quot; TagBased=&quot;False&quot; /&gt;
    &lt;row Id=&quot;82953&quot; UserId=&quot;1113&quot; Name=&quot;Teacher&quot; Date=&quot;2008-09-15T08:55:03.957&quot; Class=&quot;3&quot; TagBased=&quot;False&quot; /&gt;
  </code></pre>