title: Generate multiple 2D arrays from column data of a pandas dataframe and store them in a data structure
body: <p>I have several versions of this type of DataFrame. </p>
  
  <p>My idea was to structure each individual value column in a 2D mesh/array <strong>for each time step</strong>. These 2D arrays should be sequenced by increasing TIME values and stored as a separate dataset (in pandas or numpy??) per variable.
  This way i could call the value and load all the TIME instances of it. If i plot these consecutive time steps it should give me a temporal representation of the 2D space (a moving image of the 2D space in time) for each variable.</p>
  
  <pre><code>INDEX TIME ELEM   var1   var2    var3 ....
      0    0   h1  0.555   0.97   1.555
      1    0   t5      0    0.8     1.2
      2    0   y7      1      7       1
    ...
    300   15   h1    0.6  0.477       0
    301   15   t5    0.9  0.777       1
    302   15   y7  0.555  0.897       5
    ...
    800   23   h1     20      7       2
    801   23   t5      5      7       5
    802   23   y7    0.1      3      55
    ...
   1010   58   h1      9    0.7      11
   1011   58   t5     10    977       6
   1012   58   y7      5     71      52 
    ...
  </code></pre>
  
  <p>Hierarchically what i want to achieve is essentially this data structure, where each variable is stored in a 2D array :</p>
  
  <ul>
  <li>Full dataset (dataframe)
  
  <ul>
  <li>Sub-dataset version (dataframe)
  
  <ul>
  <li>Time instance (dataframe)
  
  <ul>
  <li>var1 (2D array)</li>
  <li>var2 (2D array)</li>
  <li>var3 (2D array)</li>
  </ul></li>
  </ul></li>
  </ul></li>
  </ul>
  
  <p>My first idea was to do <a href="http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.groupby.html" rel="nofollow noreferrer">groupby</a> TIME and ELEM. But i don't think this is the way to go. I have also looked into the <a href="http://pandas.pydata.org/pandas-docs/stable/generated/pandas.melt.html" rel="nofollow noreferrer">melt</a> function but that doesn't seem to cut it either.</p>
  
  <p>Logically i think i first need to slice the data per unique TIME values, then reshape each variable column of that slice based on element code to a 2D matrix/array as discussed <a href="https://stackoverflow.com/questions/38696546/make-a-2d-matrix-of-strings-to-match-mesh-coordinates">here</a>. Lastly each 2D matrix/array should be added in a datastructure as discussed above. How could i make this work?</p>
  
  <p>My understanding is that ideally the data structure should be in pandas for increased efficiency of operations and broadcasting. Is there a better way? I have looked into panels but it's not so clear yet</p>
  
  <p>This is what i got so far:</p>
  
  <pre><code>import pandas as pd
  import numpy as np
  
  # read the csv file
  b = pd.read_csv('D:/myfile.csv', skipinitialspace=False, skiprows=0)
  
  # remove possible empty spaces from the headers
  b.rename(columns=lambda x:x.strip(), inplace=True)
  
  # extract unique times and variable names
  times = b.TIME.unique()
  
  #make an empty list
  listt = []
  
  # for each time instance
  for i in range(len(times)):
      # generate the sub-dataset for each uniqe time
      foo = b.loc[b.TIME==times[i]]
      # re-extract the column names for each iteration
      colnames=b.columns.unique().tolist()
  
      # for each columnname in the dataset
      for k in range(len(colnames)):
          # reshape and assign the reshaped arrays inside the colnames list
          colnames[k] = np.reshape(foo[colnames[k]], (-1,51)) 
      # append each TIME instance with it's respective structured data to the list    
      listt.append(colnames)
  
  # convert the generated list to a 4D panel
  mypanel = pd.Panel4D(listt)
  </code></pre>
  
  <p>This way my indexes are numerical so i am not able to keep the actual values for unique time and the column names which is not optimal. I have the feeling this can be done better and in a more efficient way, just don't know how
  Suggestions are welcome.... :-)</p>