<p dir="auto">Found when trying to build the wheels for 0.19.2:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="________________________ test_non_meta_estimators[969] _________________________
self = &lt;sklearn.utils.testing._named_check object at 0x1091aaa58&gt;
args = ('GaussianProcess', GaussianProcess(beta0=None, corr='squared_exponential', normalize=True,
        nugget=2.220446049...state=None, regr='constant',
        storage_mode='full', theta0=0.1, thetaL=None, thetaU=None,
        verbose=False))
kwargs = {}
    def __call__(self, *args, **kwargs):
&gt;       return self.check(*args, **kwargs)
args       = ('GaussianProcess', GaussianProcess(beta0=None, corr='squared_exponential', normalize=True,
        nugget=2.220446049...state=None, regr='constant',
        storage_mode='full', theta0=0.1, thetaL=None, thetaU=None,
        verbose=False))
kwargs     = {}
self       = &lt;sklearn.utils.testing._named_check object at 0x1091aaa58&gt;
../venv/lib/python3.4/site-packages/sklearn/utils/testing.py:775: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../venv/lib/python3.4/site-packages/sklearn/utils/testing.py:291: in wrapper
    return fn(*args, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
name = 'GaussianProcess'
estimator_orig = GaussianProcess(beta0=None, corr='squared_exponential', normalize=True,
        nugget=2.2204460492503131e-15, optimiz..._state=None, regr='constant',
        storage_mode='full', theta0=0.1, thetaL=None, thetaU=None,
        verbose=False)
    @ignore_warnings(category=(DeprecationWarning, FutureWarning))
    def check_supervised_y_2d(name, estimator_orig):
        if &quot;MultiTask&quot; in name:
            # These only work on 2d, so this test makes no sense
            return
        rnd = np.random.RandomState(0)
        X = rnd.uniform(size=(10, 3))
        y = np.arange(10) % 3
        estimator = clone(estimator_orig)
        set_random_state(estimator)
        # fit
        estimator.fit(X, y)
        y_pred = estimator.predict(X)
    
        set_random_state(estimator)
        # Check that when a 2D y is given, a DataConversionWarning is
        # raised
        with warnings.catch_warnings(record=True) as w:
            warnings.simplefilter(&quot;always&quot;, DataConversionWarning)
            warnings.simplefilter(&quot;ignore&quot;, RuntimeWarning)
            estimator.fit(X, y[:, np.newaxis])
        y_pred_2d = estimator.predict(X)
        msg = &quot;expected 1 DataConversionWarning, got: %s&quot; % (
            &quot;, &quot;.join([str(w_x) for w_x in w]))
        if name not in MULTI_OUTPUT:
            # check that we warned if we don't support multi-output
            assert_greater(len(w), 0, msg)
            assert_true(&quot;DataConversionWarning('A column-vector y&quot;
                        &quot; was passed when a 1d array was expected&quot; in msg)
&gt;       assert_allclose(y_pred.ravel(), y_pred_2d.ravel())
E       AssertionError: 
E       Not equal to tolerance rtol=1e-07, atol=0
E       
E       (mismatch 40.0%)
E        x: array([ -1.086908e-13,   1.000000e+00,   2.000000e+00,   1.155742e-13,
E                1.000000e+00,   2.000000e+00,   5.062617e-14,   1.000000e+00,
E                2.000000e+00,   9.325873e-15])
E        y: array([ -1.089129e-13,   1.000000e+00,   2.000000e+00,   1.272316e-13,
E                1.000000e+00,   2.000000e+00,   5.051515e-14,   1.000000e+00,
E                2.000000e+00,   9.214851e-15])
X          = array([[ 0.5488135 ,  0.71518937,  0.60276338],
       [ 0.54488318,  0.4236548 ,  0.64589411],
       [ 0.43758721,  ...6147936,  0.78052918],
       [ 0.11827443,  0.63992102,  0.14335329],
       [ 0.94466892,  0.52184832,  0.41466194]])
estimator  = GaussianProcess(beta0=None,
        corr=&lt;function squared_exponential at 0x1063c0730&gt;, normalize=True,
        nugget...constant at 0x1063c02f0&gt;, storage_mode='full',
        theta0=array([[ 0.1]]), thetaL=None, thetaU=None, verbose=False)
estimator_orig = GaussianProcess(beta0=None, corr='squared_exponential', normalize=True,
        nugget=2.2204460492503131e-15, optimiz..._state=None, regr='constant',
        storage_mode='full', theta0=0.1, thetaL=None, thetaU=None,
        verbose=False)
msg        = 'expected 1 DataConversionWarning, got: '
name       = 'GaussianProcess'
rnd        = &lt;mtrand.RandomState object at 0x10ab87e58&gt;
w          = []
y          = array([0, 1, 2, 0, 1, 2, 0, 1, 2, 0])
y_pred     = array([ -1.08690834e-13,   1.00000000e+00,   2.00000000e+00,
         1.15574217e-13,   1.00000000e+00,   2.00000000e+00,
         5.06261699e-14,   1.00000000e+00,   2.00000000e+00,
         9.32587341e-15])
y_pred_2d  = array([[ -1.08912879e-13],
       [  1.00000000e+00],
       [  2.00000000e+00],
       [  1.27231559e-13],
       [  ...0000e+00],
       [  5.05151476e-14],
       [  1.00000000e+00],
       [  2.00000000e+00],
       [  9.21485110e-15]])
../venv/lib/python3.4/site-packages/sklearn/utils/estimator_checks.py:1226: AssertionError"><pre class="notranslate"><code class="notranslate">________________________ test_non_meta_estimators[969] _________________________
self = &lt;sklearn.utils.testing._named_check object at 0x1091aaa58&gt;
args = ('GaussianProcess', GaussianProcess(beta0=None, corr='squared_exponential', normalize=True,
        nugget=2.220446049...state=None, regr='constant',
        storage_mode='full', theta0=0.1, thetaL=None, thetaU=None,
        verbose=False))
kwargs = {}
    def __call__(self, *args, **kwargs):
&gt;       return self.check(*args, **kwargs)
args       = ('GaussianProcess', GaussianProcess(beta0=None, corr='squared_exponential', normalize=True,
        nugget=2.220446049...state=None, regr='constant',
        storage_mode='full', theta0=0.1, thetaL=None, thetaU=None,
        verbose=False))
kwargs     = {}
self       = &lt;sklearn.utils.testing._named_check object at 0x1091aaa58&gt;
../venv/lib/python3.4/site-packages/sklearn/utils/testing.py:775: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../venv/lib/python3.4/site-packages/sklearn/utils/testing.py:291: in wrapper
    return fn(*args, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
name = 'GaussianProcess'
estimator_orig = GaussianProcess(beta0=None, corr='squared_exponential', normalize=True,
        nugget=2.2204460492503131e-15, optimiz..._state=None, regr='constant',
        storage_mode='full', theta0=0.1, thetaL=None, thetaU=None,
        verbose=False)
    @ignore_warnings(category=(DeprecationWarning, FutureWarning))
    def check_supervised_y_2d(name, estimator_orig):
        if "MultiTask" in name:
            # These only work on 2d, so this test makes no sense
            return
        rnd = np.random.RandomState(0)
        X = rnd.uniform(size=(10, 3))
        y = np.arange(10) % 3
        estimator = clone(estimator_orig)
        set_random_state(estimator)
        # fit
        estimator.fit(X, y)
        y_pred = estimator.predict(X)
    
        set_random_state(estimator)
        # Check that when a 2D y is given, a DataConversionWarning is
        # raised
        with warnings.catch_warnings(record=True) as w:
            warnings.simplefilter("always", DataConversionWarning)
            warnings.simplefilter("ignore", RuntimeWarning)
            estimator.fit(X, y[:, np.newaxis])
        y_pred_2d = estimator.predict(X)
        msg = "expected 1 DataConversionWarning, got: %s" % (
            ", ".join([str(w_x) for w_x in w]))
        if name not in MULTI_OUTPUT:
            # check that we warned if we don't support multi-output
            assert_greater(len(w), 0, msg)
            assert_true("DataConversionWarning('A column-vector y"
                        " was passed when a 1d array was expected" in msg)
&gt;       assert_allclose(y_pred.ravel(), y_pred_2d.ravel())
E       AssertionError: 
E       Not equal to tolerance rtol=1e-07, atol=0
E       
E       (mismatch 40.0%)
E        x: array([ -1.086908e-13,   1.000000e+00,   2.000000e+00,   1.155742e-13,
E                1.000000e+00,   2.000000e+00,   5.062617e-14,   1.000000e+00,
E                2.000000e+00,   9.325873e-15])
E        y: array([ -1.089129e-13,   1.000000e+00,   2.000000e+00,   1.272316e-13,
E                1.000000e+00,   2.000000e+00,   5.051515e-14,   1.000000e+00,
E                2.000000e+00,   9.214851e-15])
X          = array([[ 0.5488135 ,  0.71518937,  0.60276338],
       [ 0.54488318,  0.4236548 ,  0.64589411],
       [ 0.43758721,  ...6147936,  0.78052918],
       [ 0.11827443,  0.63992102,  0.14335329],
       [ 0.94466892,  0.52184832,  0.41466194]])
estimator  = GaussianProcess(beta0=None,
        corr=&lt;function squared_exponential at 0x1063c0730&gt;, normalize=True,
        nugget...constant at 0x1063c02f0&gt;, storage_mode='full',
        theta0=array([[ 0.1]]), thetaL=None, thetaU=None, verbose=False)
estimator_orig = GaussianProcess(beta0=None, corr='squared_exponential', normalize=True,
        nugget=2.2204460492503131e-15, optimiz..._state=None, regr='constant',
        storage_mode='full', theta0=0.1, thetaL=None, thetaU=None,
        verbose=False)
msg        = 'expected 1 DataConversionWarning, got: '
name       = 'GaussianProcess'
rnd        = &lt;mtrand.RandomState object at 0x10ab87e58&gt;
w          = []
y          = array([0, 1, 2, 0, 1, 2, 0, 1, 2, 0])
y_pred     = array([ -1.08690834e-13,   1.00000000e+00,   2.00000000e+00,
         1.15574217e-13,   1.00000000e+00,   2.00000000e+00,
         5.06261699e-14,   1.00000000e+00,   2.00000000e+00,
         9.32587341e-15])
y_pred_2d  = array([[ -1.08912879e-13],
       [  1.00000000e+00],
       [  2.00000000e+00],
       [  1.27231559e-13],
       [  ...0000e+00],
       [  5.05151476e-14],
       [  1.00000000e+00],
       [  2.00000000e+00],
       [  9.21485110e-15]])
../venv/lib/python3.4/site-packages/sklearn/utils/estimator_checks.py:1226: AssertionError
</code></pre></div>
<p dir="auto">Details:</p>
<p dir="auto"><a href="https://travis-ci.org/MacPython/scikit-learn-wheels/builds/400338565" rel="nofollow">https://travis-ci.org/MacPython/scikit-learn-wheels/builds/400338565</a></p>
<p dir="auto">we have not changed anything between 0.19.1 and 0.19.2 for this model so it's probably a change on the travis environment that caused this issue to appear now.</p>
<p dir="auto">It would be great if someone with a mac could try to reproduce those failures both on master and 0.19.X branches.</p>
<p dir="auto">In a number of places in tests, <code class="notranslate">numpy.testing.assert_allclose</code> is used with default absolute tolerance parameter which is <code class="notranslate">atol=0</code>.</p>
<p dir="auto">This means in particular that <code class="notranslate">np.testing.assert_allclose(0, 1e-16)</code> will fail. More context for the reasons behind this choice can be found in <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="12645508" data-permission-text="Title is private" data-url="https://github.com/numpy/numpy/issues/3183" data-hovercard-type="issue" data-hovercard-url="/numpy/numpy/issues/3183/hovercard?comment_id=350892896&amp;comment_type=issue_comment" href="https://github.com/numpy/numpy/issues/3183#issuecomment-350892896">numpy/numpy#3183 (comment)</a>  and <a href="https://www.python.org/dev/peps/pep-0485/#absolute-tolerance-default" rel="nofollow">PEP485</a>, which can be summed up with,</p>
<blockquote>
<p dir="auto">If, for a given use case, a user needs to compare to zero, the test will be guaranteed to fail the first time, and the user can select an appropriate value.</p>
</blockquote>
<p dir="auto">The issue is that occasionally, the tests will pass, and but then may fail on some other platform.</p>
<p dir="auto">For instance,  <a href="https://github.com/scikit-learn/scikit-learn/blob/4a9034a97c9a303326e1cc4ae40eb06accea8657/sklearn/utils/estimator_checks.py#L1443">this test in <code class="notranslate">estimator_checks.py</code></a> passes CI on master, but then randomly fail for osx: <a href="https://travis-ci.org/MacPython/scikit-learn-wheels/builds/334709556?utm_source=github_status&amp;utm_medium=notification" rel="nofollow">[1]</a>, <a href="https://travis-ci.org/rth/scikit-learn-wheels/builds/334709530" rel="nofollow">[2]</a> correspond to the same commit, one for the PR, one in the forked repo (this test fails with the bellow message and not necessary for the same Python versions) ,</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="```
________________________ test_non_meta_estimators[1045] ________________________
args = ('GaussianProcess', GaussianProcess(beta0=None, corr='squared_exponential', normalize=True,
  ...orage_mode='full', theta0=0.1, thetaL=None, thetaU=None,
    verbose=False))
[...]
&gt;       assert_allclose(y_pred.ravel(), y_pred_2d.ravel())
E       AssertionError: 
E       Not equal to tolerance rtol=1e-07, atol=0
E       
E       (mismatch 40.0%)
E        x: array([ -1.089129e-13,   1.000000e+00,   2.000000e+00,   1.272316e-13,
E                1.000000e+00,   2.000000e+00,   5.051515e-14,   1.000000e+00,
E                2.000000e+00,   9.214851e-15])
E        y: array([ -1.086908e-13,   1.000000e+00,   2.000000e+00,   1.155742e-13,
E                1.000000e+00,   2.000000e+00,   5.062617e-14,   1.000000e+00,
E                2.000000e+00,   9.325873e-15])
```"><pre class="notranslate"><code class="notranslate">```
________________________ test_non_meta_estimators[1045] ________________________
args = ('GaussianProcess', GaussianProcess(beta0=None, corr='squared_exponential', normalize=True,
  ...orage_mode='full', theta0=0.1, thetaL=None, thetaU=None,
    verbose=False))
[...]
&gt;       assert_allclose(y_pred.ravel(), y_pred_2d.ravel())
E       AssertionError: 
E       Not equal to tolerance rtol=1e-07, atol=0
E       
E       (mismatch 40.0%)
E        x: array([ -1.089129e-13,   1.000000e+00,   2.000000e+00,   1.272316e-13,
E                1.000000e+00,   2.000000e+00,   5.051515e-14,   1.000000e+00,
E                2.000000e+00,   9.214851e-15])
E        y: array([ -1.086908e-13,   1.000000e+00,   2.000000e+00,   1.155742e-13,
E                1.000000e+00,   2.000000e+00,   5.062617e-14,   1.000000e+00,
E                2.000000e+00,   9.325873e-15])
```
</code></pre></div>
<ul dir="auto">
<li>this <a href="https://github.com/scikit-learn/scikit-learn/blob/b6854944fb2b2e13579160e61f3b0b43e12a2e1c/sklearn/ensemble/tests/test_gradient_boosting_loss_functions.py#L152">test in <code class="notranslate">test_gradient_boosting_loss_functions.py</code></a> compares floats to 0 with <code class="notranslate">atol=0</code> but doesn't fail.</li>
<li><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="293119124" data-permission-text="Title is private" data-url="https://github.com/scikit-learn/scikit-learn/issues/10561" data-hovercard-type="issue" data-hovercard-url="/scikit-learn/scikit-learn/issues/10561/hovercard" href="https://github.com/scikit-learn/scikit-learn/issues/10561">#10561</a> is another manifestation of this that fails on ppc64le</li>
</ul>
<p dir="auto">When <code class="notranslate">atol</code> is used, it <a href="https://github.com/scikit-learn/scikit-learn/search?p=2&amp;q=atol&amp;type=&amp;utf8=%E2%9C%93">not very consistent</a>.</p>
<p dir="auto">As to <a href="https://github.com/scikit-learn/scikit-learn/blob/c429a4d72ee49cfdb8f8b9a80559292d69dd14b9/sklearn/utils/testing.py#L388"><code class="notranslate">sklearn.utils.testing.assert_allclose_dense_sparse</code></a> it has by default <code class="notranslate">atol=1e-9</code> and not 0.</p>
<p dir="auto">While the necessary absolute tolerance is test dependent,  it might still be useful to</p>
<ul dir="auto">
<li>have a default value (e.g. 1e-9) when it's needed (e.g. <code class="notranslate">DEFAULT_ATOL</code> in  <code class="notranslate">sklearn.utils.testing</code>) , except for the cases when it has to be increased for specific reasons.</li>
<li>use <code class="notranslate">atol</code> when it's definitely reasonable to do so (e.g. in probability equalities)</li>
<li>make <code class="notranslate">sklearn.utils.testing.assert_allclose_dense_sparse</code> and <code class="notranslate">sklearn.utils.testing.assert_allclose</code> have the same default <code class="notranslate">atol</code>.</li>
<li>Check that we don't have floating point equalities with 0.0 even if CI tests passes (<a href="https://github.com/scikit-learn/scikit-learn/blob/6bd1cf594b05fdf16f827077e719a4ab0c0bd7ab/sklearn/linear_model/tests/test_logistic.py#L928">[1]</a>, <a href="https://github.com/scikit-learn/scikit-learn/blob/6bd1cf594b05fdf16f827077e719a4ab0c0bd7ab/sklearn/cluster/tests/test_mean_shift.py#L41">[2]</a>, <a href="https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/svm/tests/test_svm.py#L902">[3]</a> ..)</li>
</ul>
<p dir="auto">This might help improving the numerical stability of tests and prevent some of the tests failures on less common platforms  cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/yarikoptic/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/yarikoptic">@yarikoptic</a></p>
<p dir="auto">What do you think?</p>