<p dir="auto">Hi,</p>
<p dir="auto">I am giving the function cv2.solveP3P 3 3D ref points (matrix of size 3x3 type float32) and 2 2D points (matrix of size 3x2 and type float32) camera matrix (3x3) distCoeffs(5,) and the flags= cv2.SOLVEPNP_AP3P.</p>
<p dir="auto">But I am getting the error<br>
cv2.error: OpenCV(4.1.0) /io/opencv/modules/calib3d/src/undistort.cpp:387: error: (-215:Assertion failed) CV_IS_MAT(_src) &amp;&amp; CV_IS_MAT(_dst) &amp;&amp; (_src-&gt;rows == 1 || _src-&gt;cols == 1) &amp;&amp; (_dst-&gt;rows == 1 || _dst-&gt;cols == 1) &amp;&amp; _src-&gt;cols + _src-&gt;rows - 1 == _dst-&gt;rows + _dst-&gt;cols - 1 &amp;&amp; (CV_MAT_TYPE(_src-&gt;type) == CV_32FC2 || CV_MAT_TYPE(_src-&gt;type) == CV_64FC2) &amp;&amp; (CV_MAT_TYPE(_dst-&gt;type) == CV_32FC2 || CV_MAT_TYPE(_dst-&gt;type) == CV_64FC2) in function 'cvUndistortPointsInternal'</p>
<p dir="auto">I can't find the error.</p>
<p dir="auto">There may be a bug in <code class="notranslate">solvePnPRansac</code> when less than 6 points are provided.</p>
<p dir="auto">In the sample code below, I'm feeding <code class="notranslate">solvePnPRansac</code> with 6 points in the form of a Nx2 and Nx3 matrices. Everything goes well. This is consistent with the documentation (<a href="https://docs.opencv.org/3.3.1/d9/d0c/group__calib3d.html#ga50620f0e26e02caa2e9adc07b5fbf24e" rel="nofollow">https://docs.opencv.org/3.3.1/d9/d0c/group__calib3d.html#ga50620f0e26e02caa2e9adc07b5fbf24e</a>) and the function asserts (e.g <code class="notranslate"> CV_Assert((opoints.rows == 1 &amp;&amp; opoints.channels() == 3) || opoints.cols*opoints.channels() == 3);</code>)</p>
<p dir="auto">When I remove one or more points, the code fails with the following assert error:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="unknown location(0): fatal error: in &quot;test_template_tracker/test_und&quot;: cv::Exception: OpenCV(3.4.3) opencv-3.4.3/modules/imgproc/src/undistort.cpp:385: error: (-215:Assertion failed) CV_IS_MAT(_src) &amp;&amp; CV_IS_MAT(_dst) &amp;&amp; (_src-&gt;rows == 1 || _src-&gt;cols == 1) &amp;&amp; (_dst-&gt;rows == 1 || _dst-&gt;cols == 1) &amp;&amp; _src-&gt;cols + _src-&gt;rows - 1 == _dst-&gt;rows + _dst-&gt;cols - 1 &amp;&amp; (CV_MAT_TYPE(_src-&gt;type) == CV_32FC2 || CV_MAT_TYPE(_src-&gt;type) == CV_64FC2) &amp;&amp; (CV_MAT_TYPE(_dst-&gt;type) == CV_32FC2 || CV_MAT_TYPE(_dst-&gt;type) == CV_64FC2) in function 'cvUndistortPointsInternal'"><pre class="notranslate"><code class="notranslate">unknown location(0): fatal error: in "test_template_tracker/test_und": cv::Exception: OpenCV(3.4.3) opencv-3.4.3/modules/imgproc/src/undistort.cpp:385: error: (-215:Assertion failed) CV_IS_MAT(_src) &amp;&amp; CV_IS_MAT(_dst) &amp;&amp; (_src-&gt;rows == 1 || _src-&gt;cols == 1) &amp;&amp; (_dst-&gt;rows == 1 || _dst-&gt;cols == 1) &amp;&amp; _src-&gt;cols + _src-&gt;rows - 1 == _dst-&gt;rows + _dst-&gt;cols - 1 &amp;&amp; (CV_MAT_TYPE(_src-&gt;type) == CV_32FC2 || CV_MAT_TYPE(_src-&gt;type) == CV_64FC2) &amp;&amp; (CV_MAT_TYPE(_dst-&gt;type) == CV_32FC2 || CV_MAT_TYPE(_dst-&gt;type) == CV_64FC2) in function 'cvUndistortPointsInternal'
</code></pre></div>
<p dir="auto">The problem is that when there are less than 6 point this branch of <code class="notranslate">solvePnPRansac</code> is executed <a href="https://github.com/opencv/opencv/blob/master/modules/calib3d/src/solvepnp.cpp#L312">https://github.com/opencv/opencv/blob/master/modules/calib3d/src/solvepnp.cpp#L312</a><br>
in which <code class="notranslate">solvePnP</code> is called using directly the input parameters of <code class="notranslate">solvePnPRansac</code>, in this case Nx3 and Nx2 single channel matrices.</p>
<p dir="auto">In turn,  <code class="notranslate">solvePnP</code> calls <code class="notranslate">undistortPoints()</code> again with the same parameters received in input. This leads to  <code class="notranslate">cvUndistortPointsInternal()</code> which only takes 2-channel matrices as input in the format 1xN or Nx1. Hence the assert error.</p>
<p dir="auto">If we look at the code of  <code class="notranslate">solvePnPRansac</code> for the case of &gt;= 6 points we see that, before calling <code class="notranslate">solvePnP</code>, <a href="https://github.com/opencv/opencv/blob/master/modules/calib3d/src/solvepnp.cpp#L373">https://github.com/opencv/opencv/blob/master/modules/calib3d/src/solvepnp.cpp#L373</a> the data is reshaped into multichannel matrices with the calls to <code class="notranslate">reshape(3)</code> and <code class="notranslate">reshape(2)</code>.</p>
<p dir="auto">So, bottom line, I think there are 2 things to fix:</p>
<ol dir="auto">
<li>For the case of the 4 or 5 points in <code class="notranslate">solvePnPRansac</code>, the data has to be converted to multichannel matrices as done for the other case. Something like this (<a href="https://github.com/opencv/opencv/blob/master/modules/calib3d/src/solvepnp.cpp#L312">https://github.com/opencv/opencv/blob/master/modules/calib3d/src/solvepnp.cpp#L312</a>):</li>
</ol>
<div class="highlight highlight-source-c++ notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="...
if( model_points == npoints )
    {
        opoints = opoints.reshape(3);
        ipoints = ipoints.reshape(2);
        bool result = solvePnP(opoints, ipoints, cameraMatrix, distCoeffs, _rvec, _tvec, useExtrinsicGuess, ransac_kernel_method);

        if(!result)
        {
            if( _inliers.needed() )
                _inliers.release();
"><pre class="notranslate">...
<span class="pl-k">if</span>( model_points == npoints )
    {
        opoints = opoints.<span class="pl-c1">reshape</span>(<span class="pl-c1">3</span>);
        ipoints = ipoints.<span class="pl-c1">reshape</span>(<span class="pl-c1">2</span>);
        <span class="pl-k">bool</span> result = <span class="pl-c1">solvePnP</span>(opoints, ipoints, cameraMatrix, distCoeffs, _rvec, _tvec, useExtrinsicGuess, ransac_kernel_method);

        <span class="pl-k">if</span>(!result)
        {
            <span class="pl-k">if</span>( _inliers.<span class="pl-c1">needed</span>() )
                _inliers.<span class="pl-c1">release</span>();
</pre></div>
<ol start="2" dir="auto">
<li><code class="notranslate">undistortPoints()</code> should not be able to take single channel Nx2 matrices as input: the documentation is right about that (<a href="https://docs.opencv.org/3.4/da/d54/group__imgproc__transform.html#ga55c716492470bfe86b0ee9bf3a1f0f7e" rel="nofollow">https://docs.opencv.org/3.4/da/d54/group__imgproc__transform.html#ga55c716492470bfe86b0ee9bf3a1f0f7e</a>) but in the code (<a href="https://github.com/opencv/opencv/blob/master/modules/calib3d/src/undistort.cpp#L568">https://github.com/opencv/opencv/blob/master/modules/calib3d/src/undistort.cpp#L568</a>) we have the following check</li>
</ol>
<div class="highlight highlight-source-c++ notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="CV_Assert( src.isContinuous() &amp;&amp; (src.depth() == CV_32F || src.depth() == CV_64F) &amp;&amp;
              ((src.rows == 1 &amp;&amp; src.channels() == 2) || src.cols*src.channels() == 2));"><pre class="notranslate"><span class="pl-en">CV_Assert</span>( src.isContinuous() &amp;&amp; (src.depth() == CV_32F || src.depth() == CV_64F) &amp;&amp;
              ((src.rows == <span class="pl-c1">1</span> &amp;&amp; src.channels() == 2) || src.cols*src.channels() == 2));</pre></div>
<p dir="auto">and <code class="notranslate">src.cols*src.channels() == 2</code> is what let Nx2 single channel matrices pass. So it should be removed and replaced with <code class="notranslate">src.cols == 1 &amp;&amp; src.channels() == 2</code> to accunt for Nx1 2-channel matrices.</p>
<h5 dir="auto">System information (version)</h5>
<ul dir="auto">
<li>OpenCV =&gt; 3.1</li>
<li>Operating System / Platform =&gt; all</li>
<li>Compiler =&gt; all</li>
</ul>
<h5 dir="auto">Steps to reproduce</h5>
<p dir="auto">Note: it's just a sample code, the values of points and calibration are irrelevant for the matter.</p>
<div class="highlight highlight-source-c++ notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="std::vector&lt;int&gt; inliers{};
    cv::Mat keypoints13D = (cv::Mat_&lt;float&gt;(5, 3) &lt;&lt; 12.00604, -2.8654366, 18.472504,
                                                    7.6863389, 4.9355154, 11.146358,
                                                    14.260933, 2.8320458, 12.582781,
                                                    3.4562225, 8.2668982, 11.300434,
                                                    15.316854, 3.7486348, 12.491116);

    cv::Mat keypoints22D = (cv::Mat_&lt;float&gt;(5, 2) &lt;&lt; 918.1734, 196.77412,
                                                    1341.7848, 946.64838,
                                                    1309.8823, 926.85284,
                                                    1153.3813, 782.78381,
                                                    1399.0817, 488.19058);

/*   
// uncomment the previous 2 lines to see that solvePnPRansac is correctly working with 6 points 
cv::Mat keypoints13D = (cv::Mat_&lt;float&gt;(6, 3) &lt;&lt; 12.00604, -2.8654366, 18.472504,
            7.6863389, 4.9355154, 11.146358,
            7.6863389, 4.9355154, 11.146358,
            7.6863389, 4.9355154, 11.146358,
            14.260933, 2.8320458, 12.582781,
            15.316854, 3.7486348, 12.491116);

    cv::Mat keypoints22D = (cv::Mat_&lt;float&gt;(6, 2) &lt;&lt; 918.1734, 196.77412,
            1341.7848, 946.64838,
            1341.7848, 946.64838,
            1341.7848, 946.64838,
            1309.8823, 926.85284,
            1399.0817, 488.19058);*/

    cv::Mat matK{cv::Mat::eye(3,3,CV_64F)};
    cv::Mat distCoeff{cv::Mat::zeros(1,5,CV_64F)};
    cv::Mat rvec{cv::Mat::zeros(1,3,CV_64F)};
    cv::Mat Tvec{cv::Mat::zeros(1,3,CV_64F)};
    const auto&amp; pnpParams = params.pnpParams;

    cv::solvePnPRansac( keypoints13D, keypoints22D, matK, distCoeff, rvec, Tvec);
/*
    // uncomment these two lines to see that undistortPoints actually takes Nx3 matrices making cvUndistortPointsInternal() fail 
    cv::Mat output;
    cv::undistortPoints(keypoints22D, output, matK, distCoeff);
*/"><pre class="notranslate">std::vector&lt;<span class="pl-k">int</span>&gt; inliers{};
    cv::Mat keypoints13D = (cv::Mat_&lt;<span class="pl-k">float</span>&gt;(<span class="pl-c1">5</span>, <span class="pl-c1">3</span>) &lt;&lt; <span class="pl-c1">12.00604</span>, -<span class="pl-c1">2.8654366</span>, <span class="pl-c1">18.472504</span>,
                                                    <span class="pl-c1">7.6863389</span>, <span class="pl-c1">4.9355154</span>, <span class="pl-c1">11.146358</span>,
                                                    <span class="pl-c1">14.260933</span>, <span class="pl-c1">2.8320458</span>, <span class="pl-c1">12.582781</span>,
                                                    <span class="pl-c1">3.4562225</span>, <span class="pl-c1">8.2668982</span>, <span class="pl-c1">11.300434</span>,
                                                    <span class="pl-c1">15.316854</span>, <span class="pl-c1">3.7486348</span>, <span class="pl-c1">12.491116</span>);

    cv::Mat keypoints22D = (cv::Mat_&lt;<span class="pl-k">float</span>&gt;(<span class="pl-c1">5</span>, <span class="pl-c1">2</span>) &lt;&lt; <span class="pl-c1">918.1734</span>, <span class="pl-c1">196.77412</span>,
                                                    <span class="pl-c1">1341.7848</span>, <span class="pl-c1">946.64838</span>,
                                                    <span class="pl-c1">1309.8823</span>, <span class="pl-c1">926.85284</span>,
                                                    <span class="pl-c1">1153.3813</span>, <span class="pl-c1">782.78381</span>,
                                                    <span class="pl-c1">1399.0817</span>, <span class="pl-c1">488.19058</span>);

<span class="pl-c"><span class="pl-c">/*</span>   </span>
<span class="pl-c">// uncomment the previous 2 lines to see that solvePnPRansac is correctly working with 6 points </span>
<span class="pl-c">cv::Mat keypoints13D = (cv::Mat_&lt;float&gt;(6, 3) &lt;&lt; 12.00604, -2.8654366, 18.472504,</span>
<span class="pl-c">            7.6863389, 4.9355154, 11.146358,</span>
<span class="pl-c">            7.6863389, 4.9355154, 11.146358,</span>
<span class="pl-c">            7.6863389, 4.9355154, 11.146358,</span>
<span class="pl-c">            14.260933, 2.8320458, 12.582781,</span>
<span class="pl-c">            15.316854, 3.7486348, 12.491116);</span>
<span class="pl-c"></span>
<span class="pl-c">    cv::Mat keypoints22D = (cv::Mat_&lt;float&gt;(6, 2) &lt;&lt; 918.1734, 196.77412,</span>
<span class="pl-c">            1341.7848, 946.64838,</span>
<span class="pl-c">            1341.7848, 946.64838,</span>
<span class="pl-c">            1341.7848, 946.64838,</span>
<span class="pl-c">            1309.8823, 926.85284,</span>
<span class="pl-c">            1399.0817, 488.19058);<span class="pl-c">*/</span></span>

    cv::Mat matK{<span class="pl-c1">cv::Mat::eye</span>(<span class="pl-c1">3</span>,<span class="pl-c1">3</span>,CV_64F)};
    cv::Mat distCoeff{<span class="pl-c1">cv::Mat::zeros</span>(<span class="pl-c1">1</span>,<span class="pl-c1">5</span>,CV_64F)};
    cv::Mat rvec{<span class="pl-c1">cv::Mat::zeros</span>(<span class="pl-c1">1</span>,<span class="pl-c1">3</span>,CV_64F)};
    cv::Mat Tvec{<span class="pl-c1">cv::Mat::zeros</span>(<span class="pl-c1">1</span>,<span class="pl-c1">3</span>,CV_64F)};
    <span class="pl-k">const</span> <span class="pl-k">auto</span>&amp; pnpParams = params.pnpParams;

    <span class="pl-en">cv::solvePnPRansac</span>( keypoints13D, keypoints22D, matK, distCoeff, rvec, Tvec);
<span class="pl-c"><span class="pl-c">/*</span></span>
<span class="pl-c">    // uncomment these two lines to see that undistortPoints actually takes Nx3 matrices making cvUndistortPointsInternal() fail </span>
<span class="pl-c">    cv::Mat output;</span>
<span class="pl-c">    cv::undistortPoints(keypoints22D, output, matK, distCoeff);</span>
<span class="pl-c"><span class="pl-c">*/</span></span></pre></div>