Combine multiple .csv files with python from different directory paths
Question: I am trying to combine multiple .csv files into one .csv file using the
dataframe in pandas. the tricky part about this is, i need to grab multiple
files from multiple days. Please let me know if this does not make sense. As
it currently stands i cannot figure out how to loop through the directory.
Could you offer some assistance?

    
    
    import csv
    import pandas as pd
    import datetime as dt
    import glob, os
    
    
    startDate = 20160613
    endDate = 20160614
    
    dateRange = endDate - startDate
    dateRange = dateRange + 1
    todaysDateFilePath = startDate
    
    
    for x in xrange(dateRange):
        print startDate
        startDate = startDate + 1
        filePath = os.path.join(r"\\export\path", startDate, "preprocessed")
        os.chdir(filePath)
        interesting_files = glob.glob("trade" + "*.csv")
    
    print interesting_files
    
    df_list = []
    for filename in sorted(interesting_files):
        df_list.append(pd.read_csv(filename))
        full_df = pd.concat(df_list)
    
    
    
    
    saveFilepath = r"U:\Chris\Test_Daily_Fails"
    fileList = []
    
    
    full_df.to_csv(saveFilepath + '\\Files_For_IN' + "_0613_" + ".csv", index = False)
    


Answer: IIUC you can create `list` `all_files` and in loop append output from `glob`
to `all_files`:

    
    
    all_files = []
    for x in xrange(dateRange):
        print startDate
        startDate = startDate + 1
        filePath = os.path.join(r"\\export\path", startDate, "preprocessed")
        os.chdir(filePath)
        all_files = all_files + glob.glob("trade" + "*.csv")
    
    print interesting_files
    

Also you need first append all values to `df_list` and then only once `concat`
(I indented code for `concat`):

    
    
    df_list = []
    for filename in sorted(interesting_files):
        df_list.append(pd.read_csv(filename))
    
    full_df = pd.concat(df_list)
    

