Inconsistent results when concatenating parsed csv files
Question: I am puzzled with the following problem. I have a set of csv files, which I
parse iterativly. Before collecting the dataframes in a list, I apply some
function (as simple as `tmp_df*2`) to each of the `tmp_df`. It all worked
perfectly fine at first glance, until I've realized I have inconsistencies
with the results from run to run. For example, when I apply `df.std()` I might
receive for a first run:

    
    
    In[2]:  df1.std()
    Out[2]:
      some_int      15281.99
      some_float    5.302293
    

and for a second run after:

    
    
    In[3]:  df2.std()
    Out[3]:
      some_int      15281.99
      some_float    6.691013
    

Strangly, I don't not observe inconsistencies like this one when I don't
manipulate the parsed data (simply comment out `tmp_df = tmp_df*2`). I also
noticed that for the columns where I have datatypes `int` the results are
consistent from run to run, which does not hold for `floats`. I suspect it has
to do with the precision points. I also cannot establish a pattern how they
vary, it might be that I have the same results for two or three consecutive
runs. Maybe someone has an idea if I am missing something here. I am working
on a replication example, I'll edit asap, as I cannot share the underlying
data. Maybe someone can shed some light on this in the meantime. I am using
win8.1, pandas 17.1, python 3.4.3.

Code example:

    
    
     import pandas as pd
     import numpy as np
    
     data_list = list()
     csv_files = ['a.csv', 'b.csv', 'c.csv']
    
     for csv_file in csv_files:
    
        #  load csv_file
        tmp_df = pd.read_csv(csv_file, index_col='ID', dtype=np.float64)
    
        # replace infs by na
        tmp_df.replace([np.inf, -np.inf], np.nan, inplace=True)
    
        # manipulate tmp_df
        tmp_df = tmp_df*2
    
        data_list.append(tmp_df)
    
    df = pd.concat(data_list, ignore_index=True)
    df.reset_index(inplace=True)
    

Update:

Running the same code and data on a UX system works perfectly fine.

Edit: I have managed to re-create the problem, it should run on win and ux.
I've tested on win8.1 facing the same problem when `with_function=True`
(typically after 1-5 runs), on ux the it runs without problems.
`with_function=False` runs without differences for win and ux. I can also
reject the hypothesis that it is related to `int` or `float` issue as also the
simulated `int` are different...

Here is the code:

    
    
    import pandas as pd
    import numpy as np
    from pathlib import Path
    from tempfile import gettempdir
    
    
    def simulate_csv_data(tmp_dir,num_files=5):
        """ simulate a csv files
        :param tmp_dir: Path, csv files are saved to
        :param num_files: int, how many csv files to simulate
        :return:
        """
    
        rows = 20000
        columns = 5
        np.random.seed(1282)
    
        for file_num in range(num_files):
    
            file_path = tmp_dir.joinpath(''.join(['df_', str(file_num), '.csv']))
            simulated_df = pd.DataFrame(np.random.standard_normal((rows, columns)))
            simulated_df['some_int'] = np.random.randint(0,100)
            simulated_df.to_csv(str(file_path))
    
    
    def get_csv_data(tmp_dir,num_files=5, with_function=True):
        """ Collect various csv files and return a concatenated dfs
        :param tmp_dir: Path, csv files are saved to
        :param num_files: int, how many csv files to simulate
        :param with_function: Bool, apply function to tmp_dataframe
        :return:
        """
    
        data_list = list()
    
        for file_num in range(num_files):
            # current file path
            file_path = tmp_dir.joinpath(''.join(['df_', str(file_num), '.csv']))
    
            #  load csv_file
            tmp_df = pd.read_csv(str(file_path), dtype=np.float64)
    
            # replace infs by na
            tmp_df.replace([np.inf, -np.inf], np.nan, inplace=True)
    
            # apply function to tmp_dataframe
            if with_function:
                tmp_df = tmp_df*2
    
            data_list.append(tmp_df)
    
        df = pd.concat(data_list, ignore_index=True)
        df.reset_index(inplace=True)
    
        return df
    
    def main():
    
        # INPUT ----------------------------------------------
        num_files = 5
        with_function = True
        max_comparisons = 50
        # ----------------------------------------------------
    
        tmp_dir = gettempdir()
        # use temporary "non_existing" dir for new file
        tmp_csv_folder = Path(tmp_dir).joinpath('csv_files_sdfs2eqqf')
    
        # if exists already don't simulate data/files again
        if tmp_csv_folder.exists() is False:
            tmp_csv_folder.mkdir()
            print('Simulating temp files...')
            simulate_csv_data(tmp_csv_folder, num_files)
    
        print('Getting benchmark data frame...')
        df1 = get_csv_data(tmp_csv_folder, num_files, with_function)
        df_is_same = True
        count_runs = 0
    
        # Run until different df is found or max runs exceeded
        print('Comparing data frames...')
        while df_is_same:
            # get another data frame
            df2 = get_csv_data(tmp_csv_folder, num_files, with_function)
            count_runs += 1
            # compare data frames
            if df1.equals(df2) is False:
                df_is_same = False
                print('Found unequal df after {} runs'.format(count_runs))
                # print out a standard deviations (arbitrary example)
                print('Std Run1: \n {}'.format(df1.std()))
                print('Std Run2: \n {}'.format(df2.std()))
    
            if count_runs > max_comparisons:
                df_is_same = False
                print('No unequal df found after {} runs'.format(count_runs))
    
        print('Delete the following folder if no longer needed: "{}"'.format(
                str(tmp_csv_folder)))
    
    
    if __name__ == '__main__':
        main()
    


Answer: Your variations are caused by something else, like input data change between
executions, or source code changes.

Float precision does not ever gives different results between executions.

Btw, clean your examples and you will find the bug. At this moment you say
something about and int but display a decimal value instead!!

