<p dir="auto">Suppose we wish to form a prediction by bagging a number of SVCs.</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import numpy as np
from sklearn.svm import SVC
from sklearn.ensemble import BaggingClassifier

# get data
feat=np.random.normal(size=(100,2))
lbl=np.random.choice(a=3,p=[.45,.45,.1],size=100,replace=True)
wghts=np.ones(shape=(100,))

# fit ensemble
bc=BaggingClassifier(base_estimator=SVC(probability=True),n_estimators=10,max_samples=.1)
bc=bc.fit(feat,lbl,sample_weight=wghts)

# predict
pred=bc.predict(feat) # raises Exception"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">numpy</span> <span class="pl-k">as</span> <span class="pl-s1">np</span>
<span class="pl-k">from</span> <span class="pl-s1">sklearn</span>.<span class="pl-s1">svm</span> <span class="pl-k">import</span> <span class="pl-v">SVC</span>
<span class="pl-k">from</span> <span class="pl-s1">sklearn</span>.<span class="pl-s1">ensemble</span> <span class="pl-k">import</span> <span class="pl-v">BaggingClassifier</span>

<span class="pl-c"># get data</span>
<span class="pl-s1">feat</span><span class="pl-c1">=</span><span class="pl-s1">np</span>.<span class="pl-s1">random</span>.<span class="pl-en">normal</span>(<span class="pl-s1">size</span><span class="pl-c1">=</span>(<span class="pl-c1">100</span>,<span class="pl-c1">2</span>))
<span class="pl-s1">lbl</span><span class="pl-c1">=</span><span class="pl-s1">np</span>.<span class="pl-s1">random</span>.<span class="pl-en">choice</span>(<span class="pl-s1">a</span><span class="pl-c1">=</span><span class="pl-c1">3</span>,<span class="pl-s1">p</span><span class="pl-c1">=</span>[<span class="pl-c1">.45</span>,<span class="pl-c1">.45</span>,<span class="pl-c1">.1</span>],<span class="pl-s1">size</span><span class="pl-c1">=</span><span class="pl-c1">100</span>,<span class="pl-s1">replace</span><span class="pl-c1">=</span><span class="pl-c1">True</span>)
<span class="pl-s1">wghts</span><span class="pl-c1">=</span><span class="pl-s1">np</span>.<span class="pl-en">ones</span>(<span class="pl-s1">shape</span><span class="pl-c1">=</span>(<span class="pl-c1">100</span>,))

<span class="pl-c"># fit ensemble</span>
<span class="pl-s1">bc</span><span class="pl-c1">=</span><span class="pl-v">BaggingClassifier</span>(<span class="pl-s1">base_estimator</span><span class="pl-c1">=</span><span class="pl-v">SVC</span>(<span class="pl-s1">probability</span><span class="pl-c1">=</span><span class="pl-c1">True</span>),<span class="pl-s1">n_estimators</span><span class="pl-c1">=</span><span class="pl-c1">10</span>,<span class="pl-s1">max_samples</span><span class="pl-c1">=</span><span class="pl-c1">.1</span>)
<span class="pl-s1">bc</span><span class="pl-c1">=</span><span class="pl-s1">bc</span>.<span class="pl-en">fit</span>(<span class="pl-s1">feat</span>,<span class="pl-s1">lbl</span>,<span class="pl-s1">sample_weight</span><span class="pl-c1">=</span><span class="pl-s1">wghts</span>)

<span class="pl-c"># predict</span>
<span class="pl-s1">pred</span><span class="pl-c1">=</span><span class="pl-s1">bc</span>.<span class="pl-en">predict</span>(<span class="pl-s1">feat</span>) <span class="pl-c"># raises Exception</span></pre></div>
<p dir="auto">Because the third case has a relatively low probability (10%), some of the bootstrap draws do not contain it. As a result, some SVCs are fitted to 3 classes, and some SVCs are fitted to 2. This is a problem because  bc.predict does a simple sum of probabilities across the 100 SVCs, where the number of classes is not always 3. This bug triggers the following error:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="ValueError: operands could not be broadcast together with shapes (100,3) (100,2) (100,3)"><pre class="notranslate"><span class="pl-v">ValueError</span>: <span class="pl-s1">operands</span> <span class="pl-s1">could</span> <span class="pl-c1">not</span> <span class="pl-s1">be</span> <span class="pl-s1">broadcast</span> <span class="pl-s1">together</span> <span class="pl-k">with</span> <span class="pl-en">shapes</span> (<span class="pl-c1">100</span>,<span class="pl-c1">3</span>) (<span class="pl-c1">100</span>,<span class="pl-c1">2</span>) (<span class="pl-c1">100</span>,<span class="pl-c1">3</span>)</pre></div>
<p dir="auto">numpy version 1.11.3<br>
sklearn version 0.18.1</p>
<p dir="auto">Thank you.</p>
<p dir="auto">Hi,</p>
<p dir="auto">The BaggingClassifier does not check if the number of classes of the random drawn samples for one of its estimators matches the number of classes in the dataset, resulting in an error message when its predict method is used:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="---------------------------------------------------------------------------
Sub-process traceback:
---------------------------------------------------------------------------
ValueError                                         Thu Feb 16 11:17:13 2017
PID: 25287                                    Python 2.7.5: /usr/bin/python
...........................................................................
.local/lib/python2.7/site-packages/sklearn/externals/joblib/parallel.py in __call__(self=&lt;sklearn.externals.joblib.parallel.BatchedCalls object&gt;)
    126     def __init__(self, iterator_slice):
    127         self.items = list(iterator_slice)
    128         self._size = len(self.items)
    129
    130     def __call__(self):
--&gt; 131         return [func(*args, **kwargs) for func, args, kwargs in self.items]
        func = &lt;function _parallel_predict_proba&gt;
        args = ([SVC(C=100, cache_size=200, class_weight=None, co...5695085, shrinking=True,
  tol=0.001, verbose=10), SVC(C=100, cache_size=200, class_weight=None, co...8919740, shrinking=True,
  tol=0.001, verbose=10)], [array([   0,    1,    2, ..., 2045, 2046, 2047]), array([   0,    1,    2, ..., 2045, 2046, 2047])], memmap([[  1.21488877e-01,  -6.84937861e-01,  -5...022708e-05,   1.09372859e-06,  -8.55540498e-06]]), 1000)
        kwargs = {}
        self.items = [(&lt;function _parallel_predict_proba&gt;, ([SVC(C=100, cache_size=200, class_weight=None, co...5695085, shrinking=True,
  tol=0.001, verbose=10), SVC(C=100, cache_size=200, class_weight=None, co...8919740, shrinking=True,
  tol=0.001, verbose=10)], [array([   0,    1,    2, ..., 2045, 2046, 2047]), array([   0,    1,    2, ..., 2045, 2046, 2047])], memmap([[  1.21488877e-01,  -6.84937861e-01,  -5...022708e-05,   1.09372859e-06,  -8.55540498e-06]]), 1000), {})]
    132
    133     def __len__(self):
    134         return self._size
    135

...........................................................................
.local/lib/python2.7/site-packages/sklearn/ensemble/bagging.py in _parallel_predict_proba(estimators=[SVC(C=100, cache_size=200, class_weight=None, co...5695085, shrinking=True,
  tol=0.001, verbose=10), SVC(C=100, cache_size=200, class_weight=None, co...8919740, shrinking=True,
  tol=0.001, verbose=10)], estimators_features=[array([   0,    1,    2, ..., 2045, 2046, 2047]), array([   0,    1,    2, ..., 2045, 2046, 2047])], X=memmap([[  1.21488877e-01,  -6.84937861e-01,  -5...022708e-05,   1.09372859e-06,  -8.55540498e-06]]), n_classes=1000)
    130     for estimator, features in zip(estimators, estimators_features):
    131         if hasattr(estimator, &quot;predict_proba&quot;):
    132             proba_estimator = estimator.predict_proba(X[:, features])
    133
    134             if n_classes == len(estimator.classes_):
--&gt; 135                 proba += proba_estimator
        proba = array([[ 0.00130233,  0.00013968,  0.00144125, ....  0.00016293,
         0.00010567,  0.00053245]])
        proba_estimator = array([[  1.02577963e-03,   3.75469340e-04,   9....362413e-05,   1.45631109e-04,   3.04322015e-04]])
    136
    137             else:
    138                 proba[:, estimator.classes_] += \
    139                     proba_estimator[:, range(len(estimator.classes_))]

ValueError: operands could not be broadcast together with shapes (8009,1000) (8009,999) (8009,1000)
___________________________________________________________________________"><pre class="notranslate"><code class="notranslate">---------------------------------------------------------------------------
Sub-process traceback:
---------------------------------------------------------------------------
ValueError                                         Thu Feb 16 11:17:13 2017
PID: 25287                                    Python 2.7.5: /usr/bin/python
...........................................................................
.local/lib/python2.7/site-packages/sklearn/externals/joblib/parallel.py in __call__(self=&lt;sklearn.externals.joblib.parallel.BatchedCalls object&gt;)
    126     def __init__(self, iterator_slice):
    127         self.items = list(iterator_slice)
    128         self._size = len(self.items)
    129
    130     def __call__(self):
--&gt; 131         return [func(*args, **kwargs) for func, args, kwargs in self.items]
        func = &lt;function _parallel_predict_proba&gt;
        args = ([SVC(C=100, cache_size=200, class_weight=None, co...5695085, shrinking=True,
  tol=0.001, verbose=10), SVC(C=100, cache_size=200, class_weight=None, co...8919740, shrinking=True,
  tol=0.001, verbose=10)], [array([   0,    1,    2, ..., 2045, 2046, 2047]), array([   0,    1,    2, ..., 2045, 2046, 2047])], memmap([[  1.21488877e-01,  -6.84937861e-01,  -5...022708e-05,   1.09372859e-06,  -8.55540498e-06]]), 1000)
        kwargs = {}
        self.items = [(&lt;function _parallel_predict_proba&gt;, ([SVC(C=100, cache_size=200, class_weight=None, co...5695085, shrinking=True,
  tol=0.001, verbose=10), SVC(C=100, cache_size=200, class_weight=None, co...8919740, shrinking=True,
  tol=0.001, verbose=10)], [array([   0,    1,    2, ..., 2045, 2046, 2047]), array([   0,    1,    2, ..., 2045, 2046, 2047])], memmap([[  1.21488877e-01,  -6.84937861e-01,  -5...022708e-05,   1.09372859e-06,  -8.55540498e-06]]), 1000), {})]
    132
    133     def __len__(self):
    134         return self._size
    135

...........................................................................
.local/lib/python2.7/site-packages/sklearn/ensemble/bagging.py in _parallel_predict_proba(estimators=[SVC(C=100, cache_size=200, class_weight=None, co...5695085, shrinking=True,
  tol=0.001, verbose=10), SVC(C=100, cache_size=200, class_weight=None, co...8919740, shrinking=True,
  tol=0.001, verbose=10)], estimators_features=[array([   0,    1,    2, ..., 2045, 2046, 2047]), array([   0,    1,    2, ..., 2045, 2046, 2047])], X=memmap([[  1.21488877e-01,  -6.84937861e-01,  -5...022708e-05,   1.09372859e-06,  -8.55540498e-06]]), n_classes=1000)
    130     for estimator, features in zip(estimators, estimators_features):
    131         if hasattr(estimator, "predict_proba"):
    132             proba_estimator = estimator.predict_proba(X[:, features])
    133
    134             if n_classes == len(estimator.classes_):
--&gt; 135                 proba += proba_estimator
        proba = array([[ 0.00130233,  0.00013968,  0.00144125, ....  0.00016293,
         0.00010567,  0.00053245]])
        proba_estimator = array([[  1.02577963e-03,   3.75469340e-04,   9....362413e-05,   1.45631109e-04,   3.04322015e-04]])
    136
    137             else:
    138                 proba[:, estimator.classes_] += \
    139                     proba_estimator[:, range(len(estimator.classes_))]

ValueError: operands could not be broadcast together with shapes (8009,1000) (8009,999) (8009,1000)
___________________________________________________________________________
</code></pre></div>
<p dir="auto">It would be nice to get a warning message, if the number of classes used to train an estimator in the BaggingClassifier would not match the overall number of classes in the trainingsset.</p>