title: converting pandas dataframe with categorical values into binary values
body: <p>I am trying to convert categorical data into binary to be able to classify with an algorithm like <em>logistic regression</em>. I thought of using OneHotEncoder from 'sklearn.preprocessing' module but the problem is the dataframe entries are A, B pairs of arrays with <em>different</em> lengths, each row has pair of same-length arrays not equal to array lengths in other rows.
  <strong>OneHotEncoder</strong> does not accept dataframe like mine</p>
  
  <blockquote>
    <p>In [34]: data.index</p>
    
    <p>Out[34]: Index([train1, train2, train3, ..., train7829, train7830,
    train7831], dtype=object)</p>
  </blockquote>
  
  <pre><code>In [35]:  data.columns
  
  Out[35]:  Index([A, B], dtype=object)
  
  SampleID                      A                                B
  train1      [2092.0, 1143.0, 390.0, ...]          [5651.0, 4449.0, 4012.0...]
  train2      [3158.0, 3158.0, 3684.0, 3684.0....]  [2.0, 4.0, 2.0, 1.0...]
  train3      [1699.0, 1808.0 ,...]                 [0.0, 1.0...]
  </code></pre>
  
  <p>So, I want to highlight again that each A and B pair has the same length but the length is variable across different pairs. Dataframe contains numerical, categorical and binary values.
  I have another csv file with the information about every entry type. I read the file filter out categorical entries in both columns like this:</p>
  
  <pre><code>info=data_io.read_train_info()
  col1=info.columns[0]
  col2=info.columns[1]
  info=info[(info[col1]=='Categorical')&amp;(info[col2]=='Categorical')]
  </code></pre>
  
  <p>Then I use <code>info.index</code> to filter my training dataframe</p>
  
  <pre><code>filtered = data.loc[info.index]
  </code></pre>
  
  <p>Than I wrote an utility function to change dimensions of each array so that I can encode them later</p>
  
  <pre><code>def setDim(df):
      for item in x[x.columns[0]].index:
          df[df.columns[0]][item].shape=(1,df[df.columns[0]][item].shape[0])
          df[df.columns[1]][item].shape=(1,df[df.columns[1]][item].shape[0])
  
  setDim(filtered)
  </code></pre>
  
  <p>Then I thought to combine each pair of arrays into 2-row matrix so that I can pass it to encoder then to separate them again after encoding, like this:</p>
  
  <pre><code>import numpy as np
  from sklearn.preprocessing import OneHotEncoder
  
  def makeSparse(df):
     enc = OneHotEncoder()
     for i in df.index:
       cd=np.append(df['A'][i],df['B'][i],axis=0)
       a=enc.fit_transform(cd)
       df['A'][i] = a[0,:]
       df['B'][i] = a[1,:]
  
  makeSparse(filtered)
  </code></pre>
  
  <p>After all these steps get a sparse dataframe. My questions are:</p>
  
  <ol>
  <li>is this a right way to encode this dataframe?(I highly doubt it)</li>
  <li>if no, then what alternatives do you offer?
  <br/>Thanks a lot for your time helping me. </li>
  </ol>