Processing Experimential Measurements in Python
Question: I need to process data from a series of experiments. Each experiment has
several sensor measurements in a 'csv' file, for example:

_experiment1.csv:_

    
    
    time, sensor1, sensor2, sensor3
    0, 1.3, 4.7, 2.9, 6.6
    1, 2.8, 7.1, 4.2, 1.1
    .
    .
    

_experiment2.csv_

    
    
    time, sensor1, sensor3, sensor6
    0, 3.8, 7.1, 2.2, 1.1
    1, 1.6, 4.1, 14.1, 3.7
    .
    .
    

I need to orginize the data so that I can easily compare measurements between
trials. For example I might want to subtract the values measured by sensor 1
in experiment 1 from those measured by sensor 6 in experiment 2 and plot that
difference as a function of time.

Right now I'm using pandas data frames to work with the data

    
    
    exp1 = pd.read_csv('experiment1.csv')
    exp2 = pd.read_csv('experiment2.csv')
    
    diff = exp1['sensor1'] - exp2['sensor3']
    plot(exp1['time'],diff)
    

This is cumbersome since I need to go in and manually create a new data frame
when I want to add another experiment. Typically I might be working with data
from 50 trials simultaneously and the method outlined does not scale well.

Ideally I'd like to have a script I can run in a directory with an arbitrary
number of data files which would create a data frame for each file that I
could then refer to by experiment designator and sensor number.

Is there some way I can modify the above code to create a data frame for each
file in the directory without manually adding 'pd.read_csv' commands for every
case?

Thanks.


Answer: What you really are interested in is the `os` library -- specifically the
`listdir` function.

But while you're at it, why not just combine all your measurements into one
dataframe? You can add them as a column or as a level of a MultiIndex,
whichever you prefer.

Here's the column approach:

    
    
    import os
    import pandas as pd
    directory_contents = os.listdir(your_directory)
    experiment_files = [fn for fn in directory_contents if '.csv' in fn]
    df = pd.DataFrame()
    
    for filename in experiment_files:
        experiment_df = pd.read_csv(filename)
        experiment_df['experiment_name'] = file_name
        df = pd.concat([df,experiment_df])
    

