<h2 dir="auto"><g-emoji class="g-emoji" alias="bug" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f41b.png">🐛</g-emoji> Bug</h2>
<p dir="auto">Recently, I plan to implement Kalman filter with pytorch. I tried to obtain the batch processing capability through torch. However, when I call "torch.cholesky (covariance)" and run it on the GPU, I randomly encounter an incomprehensible error. " CUDA error: an illegal memory access was encountered ", but it does not happen when it is processed through the loop. At the same time, what is even more surprising is that this will not have any problems on window10.</p>
<h2 dir="auto">To Reproduce</h2>
<p dir="auto">If i combine the covariances like this, the error will happen randomly.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="    means = []
    covariances = []
    for row, track_idx in enumerate(track_indices):
        track = tracks[track_idx]
        means.append(track.mean)
        covariances.append(track.covariance)
    means = torch.cat(means, dim=0)
    covariances = torch.cat(covariances, dim=0)
    gating_distance = kf.gating_distance(means, covariances, measurements, only_position)"><pre class="notranslate"><code class="notranslate">    means = []
    covariances = []
    for row, track_idx in enumerate(track_indices):
        track = tracks[track_idx]
        means.append(track.mean)
        covariances.append(track.covariance)
    means = torch.cat(means, dim=0)
    covariances = torch.cat(covariances, dim=0)
    gating_distance = kf.gating_distance(means, covariances, measurements, only_position)
</code></pre></div>
<p dir="auto">What <code class="notranslate"> kf.gating_distance</code> does is call <code class="notranslate">torch.cholesky(covariance)</code>:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="    def gating_distance(self, mean, covariance, measurements, only_position=False):
        mean, covariance = self.project(mean, covariance)
        # (n, 4, 4)
        cholesky_factor = torch.cholesky(covariance)
        d = - mean + measurements  # (n, m, 4)
        z = torch.triangular_solve(d.permute(0, 2, 1), cholesky_factor, upper=False)[0]
        squared_maha = torch.sum(z ** 2, dim=1)  # (n, m)
        return squared_maha

     def project(self, mean, covariance):
        &quot;&quot;&quot;Project state distribution to measurement space.
        std = torch.tensor([[
            self._std_weight_position,
            self._std_weight_position,
            1e-1,
            self._std_weight_position]], device=mean.device)

        std = mean[:, 3:4] * std  # (*, 4)
        std[:, 2] = 1e-1  # (*, 4)

        # (*, 4, 4)
        innovation_cov = torch.diag_embed(torch.pow(std, 2))

        update_mat_t = self._update_mat.t()

        # (4, 8) dot (*, 8)
        mean = torch.mm(mean, update_mat_t)  # (*, 4)

        # (*, 4, 4)
        covariance = torch.matmul(torch.matmul(covariance.permute(0, 2, 1), update_mat_t).permute(0, 2, 1),
                                  update_mat_t)
        return mean, covariance + innovation_cov"><pre class="notranslate"><code class="notranslate">    def gating_distance(self, mean, covariance, measurements, only_position=False):
        mean, covariance = self.project(mean, covariance)
        # (n, 4, 4)
        cholesky_factor = torch.cholesky(covariance)
        d = - mean + measurements  # (n, m, 4)
        z = torch.triangular_solve(d.permute(0, 2, 1), cholesky_factor, upper=False)[0]
        squared_maha = torch.sum(z ** 2, dim=1)  # (n, m)
        return squared_maha

     def project(self, mean, covariance):
        """Project state distribution to measurement space.
        std = torch.tensor([[
            self._std_weight_position,
            self._std_weight_position,
            1e-1,
            self._std_weight_position]], device=mean.device)

        std = mean[:, 3:4] * std  # (*, 4)
        std[:, 2] = 1e-1  # (*, 4)

        # (*, 4, 4)
        innovation_cov = torch.diag_embed(torch.pow(std, 2))

        update_mat_t = self._update_mat.t()

        # (4, 8) dot (*, 8)
        mean = torch.mm(mean, update_mat_t)  # (*, 4)

        # (*, 4, 4)
        covariance = torch.matmul(torch.matmul(covariance.permute(0, 2, 1), update_mat_t).permute(0, 2, 1),
                                  update_mat_t)
        return mean, covariance + innovation_cov
</code></pre></div>
<p dir="auto">The complete code is here <a href="https://github.com/GlassyWing/sort_torch/blob/master/deep_sort/sort/kalman_filter.py">kalman_filter.py</a></p>
<h2 dir="auto">Expected behavior</h2>
<p dir="auto">There should be no problem。</p>
<h2 dir="auto">Environment</h2>
<p dir="auto">The runtime is from docker image: pytorch/pytorch:1.5-cuda10.1-cudnn7-runtime</p>
<p dir="auto">Collecting environment information...<br>
PyTorch version: 1.5.0<br>
Is debug build: No<br>
CUDA used to build PyTorch: 10.1</p>
<p dir="auto">OS: Ubuntu 18.04.4 LTS<br>
GCC version: Could not collect<br>
CMake version: Could not collect</p>
<p dir="auto">Python version: 3.7<br>
Is CUDA available: Yes<br>
CUDA runtime version: Could not collect<br>
GPU models and configuration: GPU 0: TITAN Xp<br>
Nvidia driver version: 418.56<br>
cuDNN version: Could not collect</p>
<p dir="auto">Versions of relevant libraries:<br>
[pip] numpy==1.18.1<br>
[pip] torch==1.5.0<br>
[pip] torchvision==0.6.0a0+82fd1c8<br>
[conda] blas                      1.0                         mkl<br>
[conda] cudatoolkit               10.1.243             h6bb024c_0<br>
[conda] mkl                       2020.0                      166<br>
[conda] mkl-service               2.3.0            py37he904b0f_0<br>
[conda] mkl_fft                   1.0.15           py37ha843d7b_0<br>
[conda] mkl_random                1.1.0            py37hd6b4f25_0<br>
[conda] numpy                     1.18.1           py37h4f9e942_0<br>
[conda] numpy-base                1.18.1           py37hde5b4d6_1<br>
[conda] pytorch                   1.5.0           py3.7_cuda10.1.243_cudnn7.6.3_0    pytorch<br>
[conda] torchvision               0.6.0                py37_cu101    pytorch</p>
<p dir="auto">cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ezyang/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ezyang">@ezyang</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/gchanan/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/gchanan">@gchanan</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/zou3519/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/zou3519">@zou3519</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ngimel/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ngimel">@ngimel</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/vincentqb/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/vincentqb">@vincentqb</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/vishwakftw/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/vishwakftw">@vishwakftw</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ssnl/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ssnl">@ssnl</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/jianyuh/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/jianyuh">@jianyuh</a></p>
<h2 dir="auto"><g-emoji class="g-emoji" alias="bug" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f41b.png">🐛</g-emoji> Bug</h2>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="&gt;&gt;&gt; import torch
&gt;&gt;&gt; t = torch.Tensor([[0.5]])
&gt;&gt;&gt; model = torch.nn.Linear(1, 1).cuda()
&gt;&gt;&gt; model(t)
Traceback (most recent call last):
  File &quot;&lt;stdin&gt;&quot;, line 1, in &lt;module&gt;
  File &quot;/private/home/tbirch/.conda/envs/torch160/lib/python3.7/site-packages/torch/tensor.py&quot;, line 153, in __repr__
    return torch._tensor_str._str(self)
  File &quot;/private/home/tbirch/.conda/envs/torch160/lib/python3.7/site-packages/torch/_tensor_str.py&quot;, line 371, in _str
    return _str_intern(self)
  File &quot;/private/home/tbirch/.conda/envs/torch160/lib/python3.7/site-packages/torch/_tensor_str.py&quot;, line 351, in _str_intern
    tensor_str = _tensor_str(self, indent)
  File &quot;/private/home/tbirch/.conda/envs/torch160/lib/python3.7/site-packages/torch/_tensor_str.py&quot;, line 241, in _tensor_str
    formatter = _Formatter(get_summarized_data(self) if summarize else self)
  File &quot;/private/home/tbirch/.conda/envs/torch160/lib/python3.7/site-packages/torch/_tensor_str.py&quot;, line 89, in __init__
    nonzero_finite_vals = torch.masked_select(tensor_view, torch.isfinite(tensor_view) &amp; tensor_view.ne(0))
RuntimeError: copy_if failed to synchronize: cudaErrorIllegalAddress: an illegal memory access was encountered"><pre class="notranslate"><code class="notranslate">&gt;&gt;&gt; import torch
&gt;&gt;&gt; t = torch.Tensor([[0.5]])
&gt;&gt;&gt; model = torch.nn.Linear(1, 1).cuda()
&gt;&gt;&gt; model(t)
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "/private/home/tbirch/.conda/envs/torch160/lib/python3.7/site-packages/torch/tensor.py", line 153, in __repr__
    return torch._tensor_str._str(self)
  File "/private/home/tbirch/.conda/envs/torch160/lib/python3.7/site-packages/torch/_tensor_str.py", line 371, in _str
    return _str_intern(self)
  File "/private/home/tbirch/.conda/envs/torch160/lib/python3.7/site-packages/torch/_tensor_str.py", line 351, in _str_intern
    tensor_str = _tensor_str(self, indent)
  File "/private/home/tbirch/.conda/envs/torch160/lib/python3.7/site-packages/torch/_tensor_str.py", line 241, in _tensor_str
    formatter = _Formatter(get_summarized_data(self) if summarize else self)
  File "/private/home/tbirch/.conda/envs/torch160/lib/python3.7/site-packages/torch/_tensor_str.py", line 89, in __init__
    nonzero_finite_vals = torch.masked_select(tensor_view, torch.isfinite(tensor_view) &amp; tensor_view.ne(0))
RuntimeError: copy_if failed to synchronize: cudaErrorIllegalAddress: an illegal memory access was encountered
</code></pre></div>
<h2 dir="auto">Expected behavior</h2>
<p dir="auto">Print out the value of the tensor and not crash</p>
<h2 dir="auto">Environment</h2>
<p dir="auto">Collecting environment information...<br>
PyTorch version: 1.6.0<br>
Is debug build: No<br>
CUDA used to build PyTorch: 10.1</p>
<p dir="auto">OS: Ubuntu 18.04.3 LTS<br>
GCC version: (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0<br>
CMake version: version 3.10.2</p>
<p dir="auto">Python version: 3.7<br>
Is CUDA available: Yes<br>
CUDA runtime version: 10.1.105<br>
GPU models and configuration:<br>
GPU 0: Quadro GP100<br>
GPU 1: Quadro GP100</p>
<p dir="auto">Nvidia driver version: 418.116.00<br>
cuDNN version: Could not collect</p>
<p dir="auto">Versions of relevant libraries:<br>
[pip3] numpy==1.18.5<br>
[pip3] torch==1.6.0<br>
[pip3] torchvision==0.7.0<br>
[conda] blas                      1.0                         mkl<br>
[conda] cudatoolkit               10.1.243             h6bb024c_0<br>
[conda] mkl                       2020.1                      217<br>
[conda] mkl-service               2.3.0            py37he904b0f_0<br>
[conda] mkl_fft                   1.1.0            py37h23d657b_0<br>
[conda] mkl_random                1.1.1            py37h0da4684_0    conda-forge<br>
[conda] numpy                     1.18.5           py37ha1c710e_0<br>
[conda] numpy-base                1.18.5           py37hde5b4d6_0<br>
[conda] pytorch                   1.6.0           py3.7_cuda10.1.243_cudnn7.6.3_0    pytorch<br>
[conda] torchvision               0.7.0                py37_cu101    pytorch</p>
<p dir="auto">cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ezyang/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ezyang">@ezyang</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/gchanan/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/gchanan">@gchanan</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/zou3519/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/zou3519">@zou3519</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ngimel/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ngimel">@ngimel</a></p>