title: Reading in multiple large .dta files in loops/enumerate
body: <p><strong>Problem:</strong> Efficiently reading in multiple .dta files at once without crashing.</p>
  <p><strong>Progress:</strong> Can currently read in one large .dta file, pickle it, and merge without it exceeding memory capacity. When I try to loop it by setting a dynamic variable n, and calling it from a list or dictionary, I do not output 3 separate DataFrame objects, but instead get a list with one of the values being a DataFrame object.</p>
  <p><strong>Current Code:</strong></p>
  <pre><code>import pandas as pd
  import pickle
  import glob
  import os
  
  n = 0 # Change variable value when reading in new DF
  
  # Name of variables, lists, and path
  path = r&quot;directory\directory\\&quot;
  fname = [&quot;survey_2000_ucode.dta&quot;, &quot;2010_survey_ucode.dta&quot;, &quot;2020_survey.dta&quot;]
  chunks = [&quot;chunks_2000&quot;, &quot;chunks_2010&quot;, &quot;chunks_2020&quot;]
  input_path = [path + fname[0], path + fname[1], path + fname[2]]
  output_path = [path + chunks[0], path + chunks[1], path + chunks[2]]
  
  # Create folders and directory if it does not exist
  for chunk in chunks:
      if not os.path.exists(os.path.join(path, chunk)):
          os.mkdir(os.path.join(path, chunk))
  
  CHUNK_SIZE = 100000 # Input size of chunks ~ around 14MB
  
  # Read in .dta files in chunks and output pickle files
  reader = pd.read_stata(input_path[n], chunksize=CHUNK_SIZE, convert_categoricals=False)
  
  for i, chunk in enumerate(reader):
      output_file = output_path[n] + &quot;/chunk_{}.pkl&quot;.format(i+1)
      with open(output_file[n], &quot;wb&quot;) as f:
          pickle.dump(chunk, f, pickle.HIGHEST_PROTOCOL) 
  
  # Read in pickle files and append one by one into DataFrame
  pickle_files = []
  for name in glob.glob(output_path[n] + &quot;chunk_*.pkl&quot;):
      pickle_files.append(name)
  
  # Create a list/dictionary of dataframes and append data
  dfs = [&quot;2000&quot;, &quot;2010&quot;, &quot;2020&quot;]
  
  dfs[n] = pd.DataFrame([])
  for i in range(len(pickle_files)):
      dfs[n] = dfs[n].append(pd.read_pickles(pickle_files[i]), ignore_index=True)
  </code></pre>
  <p><strong>Current Output:</strong> No df2000, df2010, df2020 DataFrames outputted. Instead, my DataFrame with data is the first object in the dfs list. Basically, the dfs list:</p>
  <ul>
  <li><p>index 0 is a DataFrame with 2,442,717 rows and 34 columns;</p>
  </li>
  <li><p>index 1 is a string value of 2010 and;</p>
  </li>
  <li><p>index 2 is a string value of 2020.</p>
  </li>
  </ul>
  <p><strong>Desired Output:</strong></p>
  <ul>
  <li>Read in multiple large data files efficiently and create separate multiple DataFrames at once.</li>
  <li>Advice/suggestions on interacting (i.e. data cleaning, wrangling, manipulation, etc) with the read-in multiple large DataFrames without crashing or taking long periods when running a line of code.</li>
  </ul>
  <p>All help and input is greatly appreciated. Thank you for your time and consideration. I apologize for not being able to share pictures of my results and datasets as I am accessing it through a secured connection and have no access to internet.</p>