title: How to prevent spyder from crashing if there is too much data in list?
body: <p>I am running the below code to get XML objects from an API in <code>Anaconda Spyder</code>. The XML objects appended in a list, keep getting larger. For some reason <code>spyder</code> keeps crashing, from what I believe is the list getting to large. </p>
  
  <p>I have to make roughly ~300K api calls for which I need data from which I think is causing my application to crash because when I am appending into the list, there is too much data. </p>
  
  <p>Below is code to call api and save results into list. <code>lst1</code> is a list of ids that I pass into the api. I have to take into account that if the http request times out or I can build a mechanism to clear the appended list of data, that I can restart the requests from the ID I left off at from <code>lst1</code> which is passed into the api url. </p>
  
  <pre><code>import requests
  import pandas as pd
  import xml.etree.ElementTree as ET
  from bs4 import BeautifulSoup 
  import time
  from concurrent import futures
  
  lst1=[1,2,3]
  
  lst =[]
  
  for i in lst1:
      url = 'urlId={}'.format(i)
      while True:
          try:
              xml_data1 = requests.get(url).text
              print(xml_data1)
              break
          except requests.exceptions.RequestException as e:
              print(e)
      lst.append(xml_data1)
  </code></pre>
  
  <p>I am thinking that if I can apply my below function to unpack the xml into a dataframe and perform the needed dataframe manipulation, while clearing the list of appended data <code>lst</code> that it could free memory. IF this is not the case, I am open to any suggestions to allowing code or application to not crash with what I believe is too much xml data in the list: </p>
  
  <pre><code>def create_dataframe(xml):
      soup = BeautifulSoup(xml, "xml")
      # Get Attributes from all nodes
      attrs = []
      for elm in soup():  # soup() is equivalent to soup.find_all()
          attrs.append(elm.attrs)
      # Since you want the data in a dataframe, it makes sense for each field to be a new row consisting of all the other node attributes
      fields_attribute_list = [x for x in attrs if 'Id' in x.keys()]
      other_attribute_list = [x for x in attrs if 'Id' not in x.keys() and x != {}]
      # Make a single dictionary with the attributes of all nodes except for the `Field` nodes.
      attribute_dict = {}
      for d in other_attribute_list:
          for k, v in d.items():
              attribute_dict.setdefault(k, v)
      # Update each field row with attributes from all other nodes.
      full_list = []
      for field in fields_attribute_list:
          field.update(attribute_dict)
          full_list.append(field)
      # Make Dataframe
      df = pd.DataFrame(full_list)
      return df
  
  
  with futures.ThreadPoolExecutor() as executor:  # Or use ProcessPoolExecutor
      df_list = executor.map(create_dataframe, lst)
  
  full_df = pd.concat(df_list)
  print(full_df)
  
  
  #final pivoted dataframe
  final_df = pd.pivot_table(full_df, index='Id', columns='FieldTitle', values='Value', aggfunc='first').reset_index()
  </code></pre>