title: Pandas: import multiple csv files into dataframe using a loop and hierarchical indexing
body: <p>I would like to read multiple CSV files (with a different number of columns) from a target directory into a single Python Pandas DataFrame to efficiently search and extract data.</p>
  
  <p>Example file:</p>
  
  <pre><code>Events 
  1,0.32,0.20,0.67
  2,0.94,0.19,0.14,0.21,0.94
  3,0.32,0.20,0.64,0.32
  4,0.87,0.13,0.61,0.54,0.25,0.43 
  5,0.62,0.21,0.77,0.44,0.16
  </code></pre>
  
  <p>Here is what I have so far:</p>
  
  <pre><code># get a list of all csv files in target directory
  my_dir = "C:\\Data\\"
  filelist = []
  os.chdir( my_dir )
  for files in glob.glob( "*.csv" ) :
      filelist.append(files)
  
  # read each csv file into single dataframe and add a filename reference column 
  # (i.e. file1, file2, file 3) for each file read
  df = pd.DataFrame()
  columns = range(1,100)
  for c, f in enumerate(filelist) :
      key = "file%i" % c
      frame = pd.read_csv( (my_dir + f), skiprows = 1, index_col=0, names=columns )
      frame['key'] = key
      df = df.append(frame,ignore_index=True)
  </code></pre>
  
  <p>(the indexing isn't working properly)</p>
  
  <p>Essentially, the script below is exactly what I want (tried and tested) but needs to be looped through 10 or more csv files:</p>
  
  <pre><code>df1 = pd.DataFrame()
  df2 = pd.DataFrame()
  columns = range(1,100)
  df1 = pd.read_csv("C:\\Data\\Currambene_001y09h00m_events.csv", 
                    skiprows = 1, index_col=0, names=columns)
  df2 = pd.read_csv("C:\\Data\\Currambene_001y12h00m_events.csv", 
                    skiprows = 1, index_col=0, names=columns)
  keys = [('file1'), ('file2')]
  df = pd.concat([df1, df2], keys=keys, names=['fileno'])
  </code></pre>
  
  <p>I have found many related links, however I am still not able to get this to work:</p>
  
  <ul>
  <li><a href="https://stackoverflow.com/questions/15843123/reading-multiple-csv-files-into-python-pandas-dataframe">Reading Multiple CSV Files into Python Pandas Dataframe</a></li>
  <li><a href="https://stackoverflow.com/questions/13572550/merge-of-multiple-data-frames-of-different-number-of-columns-into-one-big-data-f">Merge of multiple data frames of different number of columns into one big data frame</a></li>
  <li><a href="https://stackoverflow.com/questions/20906474/import-multiple-csv-files-into-python-pandas-and-concatenate-into-one-dataframe">Import multiple csv files into pandas and concatenate into one DataFrame</a></li>
  </ul>