<h3 dir="auto">Describe the issue linked to the documentation</h3>
<p dir="auto">Hi folks,</p>
<p dir="auto">I am somewhat in doubt that the <code class="notranslate">sklearn.feature_selection.chi2</code> function is implemented correctly. At least, checking the source code, it is entirely unclear to me why that kind of scoring would make sense.</p>
<p dir="auto">Best,<br>
Felix</p>
<h3 dir="auto">Suggest a potential alternative/fix</h3>
<p dir="auto">Is there any reference that justifies the computation in the implemented way? Then it should be added to the documentation.</p>
<h3 dir="auto">Describe the bug</h3>
<p dir="auto">When looking for correlation between features (for feature selection), I found that sklearn implementation of Chi2 test of independence produce significantly different results from scipy.stats implementation.</p>
<p dir="auto">My sample data contains 300 records, with 6 anonymized categorical features and the label. My focus is on the feature A. This data is available <a href="https://github.com/altanova/stuff/tree/main/projs/2021-10-chi2">in this folder in github</a> . The file sample300.csv has the file, while the file chi2_showcase.ipynb has the code demonstrating the mismatch.</p>
<p dir="auto">For the feature A, <strong>sklearn's SelectKBest()</strong> returned the lowest ranking, suggesting there is no correlation between A and the target. But <strong>scipy.stats.chi2_contingency()</strong> returned very different result, suggesting the correlation is very high.</p>
<p dir="auto">Because of mismatch between the two, I went a long way performing a number of different tests <a href="https://ondata.blog/articles/dont-trust-data-science-ask-the-people/" rel="nofollow">described in detail in this article</a> The results suggest that the scipy implementation is correct, while sklearn implementation is incorrect.</p>
<h3 dir="auto">Steps/Code to Reproduce</h3>
<p dir="auto">Please see the two links given above, where I provided the full source code and the results.<br>
The piece of code is quite standard:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="fs = SelectKBest(score_func=skfs.chi2, k = 'all')
X, y = df[cat_feature_cols], df[label]
selector = fs.fit(X, y)
kbest = pd.DataFrame({'feature': X.columns, 'score': fs.scores_})
kbest.sort_values(by = 'score', ascending = False).reset_index()"><pre class="notranslate"><span class="pl-s1">fs</span> <span class="pl-c1">=</span> <span class="pl-v">SelectKBest</span>(<span class="pl-s1">score_func</span><span class="pl-c1">=</span><span class="pl-s1">skfs</span>.<span class="pl-s1">chi2</span>, <span class="pl-s1">k</span> <span class="pl-c1">=</span> <span class="pl-s">'all'</span>)
<span class="pl-v">X</span>, <span class="pl-s1">y</span> <span class="pl-c1">=</span> <span class="pl-s1">df</span>[<span class="pl-s1">cat_feature_cols</span>], <span class="pl-s1">df</span>[<span class="pl-s1">label</span>]
<span class="pl-s1">selector</span> <span class="pl-c1">=</span> <span class="pl-s1">fs</span>.<span class="pl-en">fit</span>(<span class="pl-v">X</span>, <span class="pl-s1">y</span>)
<span class="pl-s1">kbest</span> <span class="pl-c1">=</span> <span class="pl-s1">pd</span>.<span class="pl-v">DataFrame</span>({<span class="pl-s">'feature'</span>: <span class="pl-v">X</span>.<span class="pl-s1">columns</span>, <span class="pl-s">'score'</span>: <span class="pl-s1">fs</span>.<span class="pl-s1">scores_</span>})
<span class="pl-s1">kbest</span>.<span class="pl-en">sort_values</span>(<span class="pl-s1">by</span> <span class="pl-c1">=</span> <span class="pl-s">'score'</span>, <span class="pl-s1">ascending</span> <span class="pl-c1">=</span> <span class="pl-c1">False</span>).<span class="pl-en">reset_index</span>()</pre></div>
<h3 dir="auto">Expected Results</h3>
<p dir="auto">I would expect that sklearn.feature_selection.SelectKBest(score_func=skfs.chi2) returns same, or at least similar results (p-value and chi2 statistics) as scipy.stats.chi2_contingency() . In the particular case of feature A from my set, these expected results are:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="chi2 = 127.497517
p-value = 1.445816e-29"><pre class="notranslate"><span class="pl-s1">chi2</span> <span class="pl-c1">=</span> <span class="pl-c1">127.497517</span>
<span class="pl-s1">p</span><span class="pl-c1">-</span><span class="pl-s1">value</span> <span class="pl-c1">=</span> <span class="pl-c1">1.445816e-29</span></pre></div>
<h3 dir="auto">Actual Results</h3>
<p dir="auto">For the feature A in my set, sklearn.feature_selection.chi2()  (encapsulated inside SelectKBest(score_func=skfs.chi2)) returned lowest rank of all features, suggesting no correlation. Feature A has score 1.412797, while other features score between 1647 and 24.</p>
<p dir="auto">In contrast, scipy.stats.chi2_contingency() gave highest rank to feature A, suggesting high correlation. The other tests described in the article suggest that the latter is correct.</p>
<h3 dir="auto">Versions</h3>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="System:
    python: 3.7.7 (default, May  6 2020, 11:45:54) [MSC v.1916 64 bit (AMD64)]
executable: C:\Users\pplaszczak\AppData\Local\Continuum\anaconda3\python.exe
   machine: Windows-10-10.0.18362-SP0

Python dependencies:
          pip: 21.0.1
   setuptools: 52.0.0.post20210125
      sklearn: 0.24.1
        numpy: 1.19.2
        scipy: 1.6.2
       Cython: 0.29.22
       pandas: 1.2.3
   matplotlib: 3.3.4
       joblib: 1.0.1
threadpoolctl: 2.1.0

Built with OpenMP: True"><pre class="notranslate"><code class="notranslate">System:
    python: 3.7.7 (default, May  6 2020, 11:45:54) [MSC v.1916 64 bit (AMD64)]
executable: C:\Users\pplaszczak\AppData\Local\Continuum\anaconda3\python.exe
   machine: Windows-10-10.0.18362-SP0

Python dependencies:
          pip: 21.0.1
   setuptools: 52.0.0.post20210125
      sklearn: 0.24.1
        numpy: 1.19.2
        scipy: 1.6.2
       Cython: 0.29.22
       pandas: 1.2.3
   matplotlib: 3.3.4
       joblib: 1.0.1
threadpoolctl: 2.1.0

Built with OpenMP: True
</code></pre></div>