Transformed Data Takes up 4x more space for a doubling of variables in pandas for python
Question: I've performed some simple z-transforms on some variables I have in a pandas
DataFrame. There was a total of 216 columns in the dataframe, I transformed
196 of them and then concatenated the 197 onto the original 216 for a total of
412 total columns. I then used the `to_csv` function to write the new
dataframe to a csv. The original data is about 300mb, while the new dataset is
1.2gb. It seems odd that adding less than double the columns leads to around a
4x increase in size for the final csv. The code I used is below. Am I missing
something? Or is there a more efficient way to write DataFrames to .csv files?
Everything looks fine when I take a look at the first row of the data. Also,
the number of rows are the same between all three of the DataFrames created in
the code below.

    
    
    import pandas as pd
    
    
    full_data = pd.read_csv('data.csv')
    
    names = full_data.columns.tolist()
    names = names[16:-2]
    len(names) #197 as expected
    transform = (full_data[names] - full_data[names].mean())/full_data[names].std() #Transform has 197 columns as expected. 
    
    column_names = transform.columns.tolist()
    
    new_names = {}
    for name in column_names:
        new_names[name] = name + '_standardized'
    
    transform = transform.rename(columns=new_names)
    
    
    to_concat = [full_data, transform]
    
    final_data = pd.concat(to_concat, axis=1)
    
    final_data.to_csv('transformed_data.csv', index = False)
    


Answer: The CSV stores string representations of data, so it's not necessarily going
to scale in an obvious way with the number of columns unless all columns have
roughly the same size in string representation. It's quite plausible that your
CSV could increase a lot in size if your original data had only a few decimal
places. If you read in numbers like 0.1, 0.2, 3, 1.7, whatever, and then
z-scale them, you're likely to get results with many decimal places. As a
simple example, I did this:

    
    
    >>> df = pandas.DataFrame([[2, 3, 5]], columns=["A", "B", "C"])
    >>> df
       A  B  C
    0  2  3  5
    >>> df.to_csv('someCSV.csv')
    >>> df**0.5
              A         B         C
    0  1.414214  1.732051  2.236068
    >>> (df**0.5).to_csv('someCSV2.csv')
    

I didn't add any rows or columns to the data at all, just took the square
root, but the second CSV is 4 times the size of the first, because the second
one has lots of decimal places that take more bytes to write out in string
form. You're likely to get similarly long decimals when you divide by the
standard deviation.

