title: One-hot encoding using scikit-learn
body: <p>I am working on a machine learning project and one feature of my dataset consists of categorical data. This data is first stored in an panda series (<code>&lt;class 'pandas.core.series.Series'&gt;</code>) <code>mesh</code> with dimmensions of<code>(2000,)</code>. The number of rows corresponds to the total number of data instances. Each row contains a string of the categories to which that data instance belongs to where the categories are separated by a comma. There are several hundred different categories. For example,</p>
  
  <pre><code>0       Aged, Angiotensin-Converting Enzyme Inhibitors...
  1       Aged, Angiotensin-Converting Enzyme Inhibitors...
  2       Adult, Aged, Aged, 80 and over, Angiotensin-Co...
  ....
  </code></pre>
  
  <p>In this example <code>Aged</code> is one category and <code>Angiotensin-Converting Enzyme Inhibitors</code> another. As you can see in the example, the same category may occur multiple times but this should encode no differently if the category was only in the string once.</p>
  
  <p>I wish to use one-hot encoding to represent them. To try to do this I use this code:</p>
  
  <pre><code>mlb = MultiLabelBinarizer(sparse_output=True)
  for s in data:
     pre_data = mlb.fit_transform(str(s).split(', '))
  return pre_data, len(mlb.classes_)
  </code></pre>
  
  <p>However, this produces a numpy array of dimensions <code>(19, 37)</code>. Why is this the case?</p>
  
  <p><strong>In response to <a href="https://stackoverflow.com/users/5741205/maxu">MaxU's</a> answer:</strong></p>
  
  <p>When repalcing <code>str(s).split(', ')</code> with <code>s.str.split(',\s*')</code>, this produces this error:</p>
  
  <pre><code>Traceback (most recent call last):
    File ".../guidedLearning.py", line 166, in &lt;module&gt;
      X, y = processTrainingData(directory, filename)
    File ".../guidedLearning.py", line 130, in processTrainingData
      pre_mesh, meshN = oneHot(mesh)
    File ".../guidedLearning.py", line 73, in oneHot
      pre_data = mlb.fit_transform(data.str.split(',\s*'))
    File ".../sklearn/preprocessing/label.py", line 723, in fit_transform
      yt = self._transform(y, class_mapping)
    File ".../sklearn/preprocessing/label.py", line 781, in _transform
      indices.extend(set(class_mapping[label] for label in labels))
  TypeError: 'float' object is not iterable
  </code></pre>