<p dir="auto">Hello, everyone!</p>
<p dir="auto">I am unable to replicate Weka's behavior on scikit learn using the Gaussian Naive Bayes classifier.<br>
I am using StratifiedKFold (which was double-checked).<br>
Nevertheless, Weka's and sklearn's results are inconsistent.<br>
The procedure (in Weka) is: load CSV file, select NaiveBayes classifier, select 2-fold cross validation, run. In Python, I am running the source code that I pasted inline.<br>
Since this problem also happens with SMO classifiers, I am guessing that there is something different in a pre-processing step, which Weka might be doing implicitly.</p>
<p dir="auto">On Weka, I get this confusion matrix:<br>
a b c   &lt;-- classified as<br>
3 0 0 | a = A<br>
3 0 0 | b = B<br>
0 0 3 | c = C<br>
On Sklearn, I get:<br>
[[3 1 1]<br>
[0 2 0]<br>
[0 0 2]]</p>
<p dir="auto">Thanks for everything!</p>
<p dir="auto">My data points are (in a file named fiction.csv):<br>
P0,P1,P2,LABEL<br>
0.1,0.1,0.5,A<br>
0.2,0.3,0.7,A<br>
1.1,1.1,0.0,B<br>
1.1,1.0,0.2,B<br>
0.8,1.3,0.1,B<br>
0.0,0.0,0.0,A<br>
5,7,5,C<br>
9,7,9,C<br>
10,10,10,C</p>
<p dir="auto">The source code is:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import csv
import numpy as np

from sklearn.preprocessing import StandardScaler, normalize, MinMaxScaler
from sklearn.cross_validation import StratifiedKFold
from sklearn.naive_bayes import GaussianNB
from sklearn.metrics import f1_score, confusion_matrix, classification_report

def elements(x):
    p = {}
    for k in x:
        if k not in p:
            p[k] = 1
        else:
            p[k] += 1
    return p.keys(), p

def open_dataset(dataset_filename):

    with open(dataset_filename, 'rb') as f:
        data = list(csv.reader(f))

    data = np.array(data)
    x_ = data[1:, 0:-1].astype(float)
    y_ = data[1:, -1].astype(str)
    attr_names = data[0, :].astype(str)

    labels, nel = elements(y_)
    labels.sort()

    folds = StratifiedKFold(y_, n_folds=2)
    train_index, test_index = next(iter(folds))

    X_train = x_[train_index,:]
    y_train = y_[train_index]
    X_test = x_[test_index,:]
    y_test = y_[test_index]

    return X_train, y_train, X_test, y_test, labels, nel

X_train, y_train, X_test, y_test, labels, nel = open_dataset(&quot;fiction.csv&quot;)

machine = GaussianNB()
machine.fit(X_train, y_train)
p1 = machine.predict(X_test)
machine.fit(X_test, y_test)
p2 = machine.predict(X_train)
f = f1_score(list(p1) + list(p2), list(y_test) + list(y_train), average='weighted')
print f
a = confusion_matrix(list(p1) + list(p2), list(y_test) + list(y_train))
print a"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">csv</span>
<span class="pl-k">import</span> <span class="pl-s1">numpy</span> <span class="pl-k">as</span> <span class="pl-s1">np</span>

<span class="pl-k">from</span> <span class="pl-s1">sklearn</span>.<span class="pl-s1">preprocessing</span> <span class="pl-k">import</span> <span class="pl-v">StandardScaler</span>, <span class="pl-s1">normalize</span>, <span class="pl-v">MinMaxScaler</span>
<span class="pl-k">from</span> <span class="pl-s1">sklearn</span>.<span class="pl-s1">cross_validation</span> <span class="pl-k">import</span> <span class="pl-v">StratifiedKFold</span>
<span class="pl-k">from</span> <span class="pl-s1">sklearn</span>.<span class="pl-s1">naive_bayes</span> <span class="pl-k">import</span> <span class="pl-v">GaussianNB</span>
<span class="pl-k">from</span> <span class="pl-s1">sklearn</span>.<span class="pl-s1">metrics</span> <span class="pl-k">import</span> <span class="pl-s1">f1_score</span>, <span class="pl-s1">confusion_matrix</span>, <span class="pl-s1">classification_report</span>

<span class="pl-k">def</span> <span class="pl-en">elements</span>(<span class="pl-s1">x</span>):
    <span class="pl-s1">p</span> <span class="pl-c1">=</span> {}
    <span class="pl-k">for</span> <span class="pl-s1">k</span> <span class="pl-c1">in</span> <span class="pl-s1">x</span>:
        <span class="pl-k">if</span> <span class="pl-s1">k</span> <span class="pl-c1">not</span> <span class="pl-c1">in</span> <span class="pl-s1">p</span>:
            <span class="pl-s1">p</span>[<span class="pl-s1">k</span>] <span class="pl-c1">=</span> <span class="pl-c1">1</span>
        <span class="pl-k">else</span>:
            <span class="pl-s1">p</span>[<span class="pl-s1">k</span>] <span class="pl-c1">+=</span> <span class="pl-c1">1</span>
    <span class="pl-k">return</span> <span class="pl-s1">p</span>.<span class="pl-en">keys</span>(), <span class="pl-s1">p</span>

<span class="pl-k">def</span> <span class="pl-en">open_dataset</span>(<span class="pl-s1">dataset_filename</span>):

    <span class="pl-k">with</span> <span class="pl-en">open</span>(<span class="pl-s1">dataset_filename</span>, <span class="pl-s">'rb'</span>) <span class="pl-k">as</span> <span class="pl-s1">f</span>:
        <span class="pl-s1">data</span> <span class="pl-c1">=</span> <span class="pl-en">list</span>(<span class="pl-s1">csv</span>.<span class="pl-en">reader</span>(<span class="pl-s1">f</span>))

    <span class="pl-s1">data</span> <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-en">array</span>(<span class="pl-s1">data</span>)
    <span class="pl-s1">x_</span> <span class="pl-c1">=</span> <span class="pl-s1">data</span>[<span class="pl-c1">1</span>:, <span class="pl-c1">0</span>:<span class="pl-c1">-</span><span class="pl-c1">1</span>].<span class="pl-en">astype</span>(<span class="pl-s1">float</span>)
    <span class="pl-s1">y_</span> <span class="pl-c1">=</span> <span class="pl-s1">data</span>[<span class="pl-c1">1</span>:, <span class="pl-c1">-</span><span class="pl-c1">1</span>].<span class="pl-en">astype</span>(<span class="pl-s1">str</span>)
    <span class="pl-s1">attr_names</span> <span class="pl-c1">=</span> <span class="pl-s1">data</span>[<span class="pl-c1">0</span>, :].<span class="pl-en">astype</span>(<span class="pl-s1">str</span>)

    <span class="pl-s1">labels</span>, <span class="pl-s1">nel</span> <span class="pl-c1">=</span> <span class="pl-en">elements</span>(<span class="pl-s1">y_</span>)
    <span class="pl-s1">labels</span>.<span class="pl-en">sort</span>()

    <span class="pl-s1">folds</span> <span class="pl-c1">=</span> <span class="pl-v">StratifiedKFold</span>(<span class="pl-s1">y_</span>, <span class="pl-s1">n_folds</span><span class="pl-c1">=</span><span class="pl-c1">2</span>)
    <span class="pl-s1">train_index</span>, <span class="pl-s1">test_index</span> <span class="pl-c1">=</span> <span class="pl-en">next</span>(<span class="pl-en">iter</span>(<span class="pl-s1">folds</span>))

    <span class="pl-v">X_train</span> <span class="pl-c1">=</span> <span class="pl-s1">x_</span>[<span class="pl-s1">train_index</span>,:]
    <span class="pl-s1">y_train</span> <span class="pl-c1">=</span> <span class="pl-s1">y_</span>[<span class="pl-s1">train_index</span>]
    <span class="pl-v">X_test</span> <span class="pl-c1">=</span> <span class="pl-s1">x_</span>[<span class="pl-s1">test_index</span>,:]
    <span class="pl-s1">y_test</span> <span class="pl-c1">=</span> <span class="pl-s1">y_</span>[<span class="pl-s1">test_index</span>]

    <span class="pl-k">return</span> <span class="pl-v">X_train</span>, <span class="pl-s1">y_train</span>, <span class="pl-v">X_test</span>, <span class="pl-s1">y_test</span>, <span class="pl-s1">labels</span>, <span class="pl-s1">nel</span>

<span class="pl-v">X_train</span>, <span class="pl-s1">y_train</span>, <span class="pl-v">X_test</span>, <span class="pl-s1">y_test</span>, <span class="pl-s1">labels</span>, <span class="pl-s1">nel</span> <span class="pl-c1">=</span> <span class="pl-en">open_dataset</span>(<span class="pl-s">"fiction.csv"</span>)

<span class="pl-s1">machine</span> <span class="pl-c1">=</span> <span class="pl-v">GaussianNB</span>()
<span class="pl-s1">machine</span>.<span class="pl-en">fit</span>(<span class="pl-v">X_train</span>, <span class="pl-s1">y_train</span>)
<span class="pl-s1">p1</span> <span class="pl-c1">=</span> <span class="pl-s1">machine</span>.<span class="pl-en">predict</span>(<span class="pl-v">X_test</span>)
<span class="pl-s1">machine</span>.<span class="pl-en">fit</span>(<span class="pl-v">X_test</span>, <span class="pl-s1">y_test</span>)
<span class="pl-s1">p2</span> <span class="pl-c1">=</span> <span class="pl-s1">machine</span>.<span class="pl-en">predict</span>(<span class="pl-v">X_train</span>)
<span class="pl-s1">f</span> <span class="pl-c1">=</span> <span class="pl-en">f1_score</span>(<span class="pl-en">list</span>(<span class="pl-s1">p1</span>) <span class="pl-c1">+</span> <span class="pl-en">list</span>(<span class="pl-s1">p2</span>), <span class="pl-en">list</span>(<span class="pl-s1">y_test</span>) <span class="pl-c1">+</span> <span class="pl-en">list</span>(<span class="pl-s1">y_train</span>), <span class="pl-s1">average</span><span class="pl-c1">=</span><span class="pl-s">'weighted'</span>)
<span class="pl-k">print</span> <span class="pl-s1">f</span>
<span class="pl-s1">a</span> <span class="pl-c1">=</span> <span class="pl-en">confusion_matrix</span>(<span class="pl-en">list</span>(<span class="pl-s1">p1</span>) <span class="pl-c1">+</span> <span class="pl-en">list</span>(<span class="pl-s1">p2</span>), <span class="pl-en">list</span>(<span class="pl-s1">y_test</span>) <span class="pl-c1">+</span> <span class="pl-en">list</span>(<span class="pl-s1">y_train</span>))
<span class="pl-k">print</span> <span class="pl-s1">a</span></pre></div>
<p dir="auto">I dig into <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="51316849" data-permission-text="Title is private" data-url="https://github.com/scikit-learn/scikit-learn/issues/3945" data-hovercard-type="pull_request" data-hovercard-url="/scikit-learn/scikit-learn/pull/3945/hovercard" href="https://github.com/scikit-learn/scikit-learn/pull/3945">#3945</a> and <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="23842322" data-permission-text="Title is private" data-url="https://github.com/scikit-learn/scikit-learn/issues/2640" data-hovercard-type="issue" data-hovercard-url="/scikit-learn/scikit-learn/issues/2640/hovercard" href="https://github.com/scikit-learn/scikit-learn/issues/2640">#2640</a>. PR <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="51316849" data-permission-text="Title is private" data-url="https://github.com/scikit-learn/scikit-learn/issues/3945" data-hovercard-type="pull_request" data-hovercard-url="/scikit-learn/scikit-learn/pull/3945/hovercard" href="https://github.com/scikit-learn/scikit-learn/pull/3945">#3945</a> claims</p>
<blockquote>
<p dir="auto">Due to round off errors, the formula used to compute the weighted covariance matrix may lead to matrices with large negative eigenvalues.</p>
</blockquote>
<p dir="auto">Actually, the reason why the sample code given by <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/AlexisMignon/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/AlexisMignon">@AlexisMignon</a> in <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="23842322" data-permission-text="Title is private" data-url="https://github.com/scikit-learn/scikit-learn/issues/2640" data-hovercard-type="issue" data-hovercard-url="/scikit-learn/scikit-learn/issues/2640/hovercard" href="https://github.com/scikit-learn/scikit-learn/issues/2640">#2640</a> raises an exception  is that the simplified formula C = (\sum_i w_i x_i x_i') - \mu \mu' does not apply when \mu is not the mean of X.</p>
<p dir="auto">The gmm created in this line</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="gmm = mixt.GMM(2, params=&quot;wc&quot;, covariance_type=&quot;full&quot;, min_covar=1e-3)"><pre class="notranslate"><span class="pl-s1">gmm</span> <span class="pl-c1">=</span> <span class="pl-s1">mixt</span>.<span class="pl-v">GMM</span>(<span class="pl-c1">2</span>, <span class="pl-s1">params</span><span class="pl-c1">=</span><span class="pl-s">"wc"</span>, <span class="pl-s1">covariance_type</span><span class="pl-c1">=</span><span class="pl-s">"full"</span>, <span class="pl-s1">min_covar</span><span class="pl-c1">=</span><span class="pl-c1">1e-3</span>)</pre></div>
<p dir="auto">does not estimate <code class="notranslate">mean</code> in M-step. The <code class="notranslate">gmm.means_</code> is initialized by k-means, and does not change in EM. When the code in m_step before <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="51316849" data-permission-text="Title is private" data-url="https://github.com/scikit-learn/scikit-learn/issues/3945" data-hovercard-type="pull_request" data-hovercard-url="/scikit-learn/scikit-learn/pull/3945/hovercard" href="https://github.com/scikit-learn/scikit-learn/pull/3945">#3945</a> merged is executed, in one of component the mean is initialized to <code class="notranslate">[[ 3.42118979  3.53824541]]</code> by kmeans.  However, the real mean is <code class="notranslate">[0.02670257  0.02945094]</code> before the exception is raised. Using C = (\sum_i w_i x_i x_i') - \mu \mu',  of course, the estimated covariance is not positive definite. If</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="gmm = mixt.GMM(2, params=&quot;wmc&quot;, covariance_type=&quot;full&quot;, min_covar=1e-3)"><pre class="notranslate"><span class="pl-s1">gmm</span> <span class="pl-c1">=</span> <span class="pl-s1">mixt</span>.<span class="pl-v">GMM</span>(<span class="pl-c1">2</span>, <span class="pl-s1">params</span><span class="pl-c1">=</span><span class="pl-s">"wmc"</span>, <span class="pl-s1">covariance_type</span><span class="pl-c1">=</span><span class="pl-s">"full"</span>, <span class="pl-s1">min_covar</span><span class="pl-c1">=</span><span class="pl-c1">1e-3</span>)</pre></div>
<p dir="auto">We don't have the exception at all.</p>
<p dir="auto">In <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="23842322" data-permission-text="Title is private" data-url="https://github.com/scikit-learn/scikit-learn/issues/2640" data-hovercard-type="issue" data-hovercard-url="/scikit-learn/scikit-learn/issues/2640/hovercard" href="https://github.com/scikit-learn/scikit-learn/issues/2640">#2640</a>, without the snippet resulting the error, I also cannot figure out why the covariance is not positive definite. Anyway, I think the method <code class="notranslate">log_multivariate_normal_density</code> in gmm.py could be called by methods in other modules, so it is better to check the covariance first before Cholesky decomposition.</p>
<p dir="auto">Besides, when using parameter <code class="notranslate">params='wc'</code> to initialize a gmm, we <del>always</del> have non-positive-definite covariance problems, with <code class="notranslate">full</code> and <code class="notranslate">tied</code> type covariance. You could try it with any random generated data. I think it is strange to build GMM without estimating means. It would be better to deprecate <code class="notranslate">params</code> initialization other than <code class="notranslate">wmc</code>.</p>
<p dir="auto">Finally, the original code using equation C = (\sum_i w_i x_i x_i') - \mu \mu' before <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="51316849" data-permission-text="Title is private" data-url="https://github.com/scikit-learn/scikit-learn/issues/3945" data-hovercard-type="pull_request" data-hovercard-url="/scikit-learn/scikit-learn/pull/3945/hovercard" href="https://github.com/scikit-learn/scikit-learn/pull/3945">#3945</a> merged, I think, it is OK as long as we could guarantee \mu is the mean of X. But I found the code in Matlab use the way as <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="51316849" data-permission-text="Title is private" data-url="https://github.com/scikit-learn/scikit-learn/issues/3945" data-hovercard-type="pull_request" data-hovercard-url="/scikit-learn/scikit-learn/pull/3945/hovercard" href="https://github.com/scikit-learn/scikit-learn/pull/3945">#3945</a> does. I cannot see other reasons to use that way except preventing round-off errors, which I really doubt.</p>
<p dir="auto">Ping <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/amueller/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/amueller">@amueller</a></p>