We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
warp_frame_depth fails with different aspect ratios between src and dst
## 🐛 Bug

As in the title: I am trying to get `kornia.geometry.warp_frame_depth` to work for my project and a seemingly intermittent bug appears to boil down to that - `warp_frame_depth` works when the aspect ratios of `src` and `dst` match but fails otherwise.
<!-- A clear and concise description of what the bug is. -->

## To Reproduce
I made a small repro repository with sample images, depths and camera parameters [here](https://github.com/jatentaki/warp-bug). Just clone it and execute `warp.py` to see the behavior (it uses kornia, numpy, matplotlib, imageio and h5py modules). You can comment out lines 6-7 in favor of 10-11 to see a working example with images of equal aspect ratio.

## Expected behavior
The warping behavior should not depend on aspect ratios. It may seem that a workaround is to pad the images until they are of equal shape, but this doesn't seem to be the case.

## Environment
kornia 0.4.0

PyTorch version: 1.6.0
Is debug build: No
CUDA used to build PyTorch: None

OS: Ubuntu 18.04.5 LTS
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: 6.0.0-1ubuntu2 (tags/RELEASE_600/final)
CMake version: version 3.18.2

Python version: 3.8 (64-bit runtime)
Is CUDA available: No
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA

Versions of relevant libraries:
[pip3] numpy==1.19.1
[pip3] torch==1.6.0
[pip3] torch-cluster==1.5.4
[pip3] torch-dimcheck==0.0.1
[pip3] torch-geometric==1.4.3
[pip3] torch-localize==0.1.0
[pip3] torch-scatter==2.0.4
[pip3] torch-sparse==0.6.2
[pip3] torchvision==0.7.0
[conda] blas                      1.0                         mkl  
[conda] cpuonly                   1.0                           0    pytorch
[conda] mkl                       2020.1                      217  
[conda] mkl-service               2.3.0            py38he904b0f_0  
[conda] mkl_fft                   1.1.0            py38h23d657b_0  
[conda] mkl_random                1.1.1            py38h0573a6f_0  
[conda] numpy                     1.19.1           py38hbc911f0_0  
[conda] numpy-base                1.19.1           py38hfa32c7d_0  
[conda] pytorch                   1.6.0               py3.8_cpu_0  [cpuonly]  pytorch
[conda] torch-cluster             1.5.4                    pypi_0    pypi
[conda] torch-dimcheck            0.0.1                    pypi_0    pypi
[conda] torch-geometric           1.4.3                    pypi_0    pypi
[conda] torch-localize            0.1.0                    pypi_0    pypi
[conda] torch-scatter             2.0.4                    pypi_0    pypi
[conda] torch-sparse              0.6.2                    pypi_0    pypi
[conda] torchvision               0.7.0                  py38_cpu  [cpuonly]  pytorch


--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `kornia/geometry/warp/depth_warper.py`
Content:
```
1 from typing import Union
2 import torch
3 import torch.nn as nn
4 import torch.nn.functional as F
5 
6 from kornia.geometry.linalg import transform_points
7 from kornia.geometry.linalg import relative_transformation
8 from kornia.geometry.conversions import convert_points_to_homogeneous
9 from kornia.geometry.conversions import normalize_pixel_coordinates
10 from kornia.geometry.camera import PinholeCamera
11 from kornia.geometry.camera import cam2pixel, pixel2cam
12 from kornia.utils import create_meshgrid
13 
14 
15 __all__ = [
16     "depth_warp",
17     "DepthWarper",
18 ]
19 
20 
21 class DepthWarper(nn.Module):
22     r"""Warps a patch by depth.
23 
24     .. math::
25         P_{src}^{\{dst\}} = K_{dst} * T_{src}^{\{dst\}}
26 
27         I_{src} = \\omega(I_{dst}, P_{src}^{\{dst\}}, D_{src})
28 
29     Args:
30         pinholes_dst (PinholeCamera): the pinhole models for the destination
31           frame.
32         height (int): the height of the image to warp.
33         width (int): the width of the image to warp.
34         mode (str): interpolation mode to calculate output values
35           'bilinear' | 'nearest'. Default: 'bilinear'.
36         padding_mode (str): padding mode for outside grid values
37            'zeros' | 'border' | 'reflection'. Default: 'zeros'.
38         align_corners(bool): interpolation flag. Default: True. See
39         https://pytorch.org/docs/stable/nn.functional.html#torch.nn.functional.interpolate for detail
40     """
41 
42     def __init__(self,
43                  pinhole_dst: PinholeCamera,
44                  height: int, width: int,
45                  mode: str = 'bilinear',
46                  padding_mode: str = 'zeros',
47                  align_corners: bool = True):
48         super(DepthWarper, self).__init__()
49         # constructor members
50         self.width: int = width
51         self.height: int = height
52         self.mode: str = mode
53         self.padding_mode: str = padding_mode
54         self.eps = 1e-6
55         self.align_corners: bool = align_corners
56 
57         # state members
58         self._pinhole_dst: PinholeCamera = pinhole_dst
59         self._pinhole_src: Union[None, PinholeCamera] = None
60         self._dst_proj_src: Union[None, torch.Tensor] = None
61 
62         self.grid: torch.Tensor = self._create_meshgrid(height, width)
63 
64     @staticmethod
65     def _create_meshgrid(height: int, width: int) -> torch.Tensor:
66         grid: torch.Tensor = create_meshgrid(
67             height, width, normalized_coordinates=False)  # 1xHxWx2
68         return convert_points_to_homogeneous(grid)  # append ones to last dim
69 
70     def compute_projection_matrix(
71             self, pinhole_src: PinholeCamera) -> 'DepthWarper':
72         r"""Computes the projection matrix from the source to destinaion frame.
73         """
74         if not isinstance(self._pinhole_dst, PinholeCamera):
75             raise TypeError("Member self._pinhole_dst expected to be of class "
76                             "PinholeCamera. Got {}"
77                             .format(type(self._pinhole_dst)))
78         if not isinstance(pinhole_src, PinholeCamera):
79             raise TypeError("Argument pinhole_src expected to be of class "
80                             "PinholeCamera. Got {}".format(type(pinhole_src)))
81         # compute the relative pose between the non reference and the reference
82         # camera frames.
83         dst_trans_src: torch.Tensor = relative_transformation(
84             pinhole_src.extrinsics, self._pinhole_dst.extrinsics)
85 
86         # compute the projection matrix between the non reference cameras and
87         # the reference.
88         dst_proj_src: torch.Tensor = torch.matmul(
89             self._pinhole_dst.intrinsics, dst_trans_src)
90 
91         # update class members
92         self._pinhole_src = pinhole_src
93         self._dst_proj_src = dst_proj_src
94         return self
95 
96     def _compute_projection(self, x, y, invd):
97         point = torch.FloatTensor([[[x], [y], [1.0], [invd]]])
98         flow = torch.matmul(
99             self._dst_proj_src, point.to(self._dst_proj_src.device))
100         z = 1. / flow[:, 2]
101         x = (flow[:, 0] * z)
102         y = (flow[:, 1] * z)
103         return torch.cat([x, y], 1)
104 
105     def compute_subpixel_step(self) -> torch.Tensor:
106         """This computes the required inverse depth step to achieve sub pixel
107         accurate sampling of the depth cost volume, per camera.
108 
109         Szeliski, Richard, and Daniel Scharstein.
110         "Symmetric sub-pixel stereo matching." European Conference on Computer
111         Vision. Springer Berlin Heidelberg, 2002.
112         """
113         delta_d = 0.01
114         xy_m1 = self._compute_projection(self.width / 2, self.height / 2,
115                                          1.0 - delta_d)
116         xy_p1 = self._compute_projection(self.width / 2, self.height / 2,
117                                          1.0 + delta_d)
118         dx = torch.norm((xy_p1 - xy_m1), 2, dim=-1) / 2.0
119         dxdd = dx / (delta_d)  # pixel*(1/meter)
120         # half pixel sampling, we're interested in the min for all cameras
121         return torch.min(0.5 / dxdd)
122 
123     def warp_grid(self, depth_src: torch.Tensor) -> torch.Tensor:
124         """Computes a grid for warping a given the depth from the reference
125         pinhole camera.
126 
127         The function `compute_projection_matrix` has to be called beforehand in
128         order to have precomputed the relative projection matrices encoding the
129         relative pose and the intrinsics between the reference and a non
130         reference camera.
131         """
132         # TODO: add type and value checkings
133         if self._dst_proj_src is None or self._pinhole_src is None:
134             raise ValueError("Please, call compute_projection_matrix.")
135 
136         if len(depth_src.shape) != 4:
137             raise ValueError("Input depth_src has to be in the shape of "
138                              "Bx1xHxW. Got {}".format(depth_src.shape))
139 
140         # unpack depth attributes
141         batch_size, _, height, width = depth_src.shape
142         device: torch.device = depth_src.device
143         dtype: torch.dtype = depth_src.dtype
144 
145         # expand the base coordinate grid according to the input batch size
146         pixel_coords: torch.Tensor = self.grid.to(device).to(dtype).expand(
147             batch_size, -1, -1, -1)  # BxHxWx3
148 
149         # reproject the pixel coordinates to the camera frame
150         cam_coords_src: torch.Tensor = pixel2cam(
151             depth_src,
152             self._pinhole_src.intrinsics_inverse().to(dtype),
153             pixel_coords)  # BxHxWx3
154 
155         # reproject the camera coordinates to the pixel
156         pixel_coords_src: torch.Tensor = cam2pixel(
157             cam_coords_src, self._dst_proj_src.to(dtype))  # (B*N)xHxWx2
158 
159         # normalize between -1 and 1 the coordinates
160         pixel_coords_src_norm: torch.Tensor = normalize_pixel_coordinates(
161             pixel_coords_src, self.height, self.width)
162         return pixel_coords_src_norm
163 
164     def forward(  # type: ignore
165             self,
166             depth_src: torch.Tensor,
167             patch_dst: torch.Tensor) -> torch.Tensor:
168         """Warps a tensor from destination frame to reference given the depth
169         in the reference frame.
170 
171         Args:
172             depth_src (torch.Tensor): the depth in the reference frame. The
173               tensor must have a shape :math:`(B, 1, H, W)`.
174             patch_dst (torch.Tensor): the patch in the destination frame. The
175               tensor must have a shape :math:`(B, C, H, W)`.
176 
177         Return:
178             torch.Tensor: the warped patch from destination frame to reference.
179 
180         Shape:
181             - Output: :math:`(N, C, H, W)` where C = number of channels.
182 
183         Example:
184             >>> # pinholes camera models
185             >>> pinhole_dst = kornia.PinholeCamera(...)
186             >>> pinhole_src = kornia.PinholeCamera(...)
187             >>> # create the depth warper, compute the projection matrix
188             >>> warper = kornia.DepthWarper(pinhole_dst, height, width)
189             >>> warper.compute_projection_matrix(pinhole_src)
190             >>> # warp the destionation frame to reference by depth
191             >>> depth_src = torch.ones(1, 1, 32, 32)  # Nx1xHxW
192             >>> image_dst = torch.rand(1, 3, 32, 32)  # NxCxHxW
193             >>> image_src = warper(depth_src, image_dst)  # NxCxHxW
194         """
195         return F.grid_sample(patch_dst, self.warp_grid(depth_src),  # type: ignore
196                              mode=self.mode, padding_mode=self.padding_mode,
197                              align_corners=self.align_corners)
198 
199 
200 # functional api
201 
202 def depth_warp(pinhole_dst: PinholeCamera,
203                pinhole_src: PinholeCamera,
204                depth_src: torch.Tensor,
205                patch_dst: torch.Tensor,
206                height: int, width: int,
207                align_corners: bool = True):
208     r"""Function that warps a tensor from destination frame to reference
209     given the depth in the reference frame.
210 
211     See :class:`~kornia.geometry.warp.DepthWarper` for details.
212 
213     Example:
214         >>> # pinholes camera models
215         >>> pinhole_dst = kornia.PinholeCamera(...)
216         >>> pinhole_src = kornia.PinholeCamera(...)
217         >>> # warp the destionation frame to reference by depth
218         >>> depth_src = torch.ones(1, 1, 32, 32)  # Nx1xHxW
219         >>> image_dst = torch.rand(1, 3, 32, 32)  # NxCxHxW
220         >>> image_src = kornia.depth_warp(pinhole_dst, pinhole_src,
221         >>>     depth_src, image_dst, height, width)  # NxCxHxW
222     """
223     warper = DepthWarper(pinhole_dst, height, width, align_corners=align_corners)
224     warper.compute_projection_matrix(pinhole_src)
225     return warper(depth_src, patch_dst)
226
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/kornia/geometry/warp/depth_warper.py b/kornia/geometry/warp/depth_warper.py
--- a/kornia/geometry/warp/depth_warper.py
+++ b/kornia/geometry/warp/depth_warper.py
@@ -3,8 +3,7 @@
 import torch.nn as nn
 import torch.nn.functional as F
 
-from kornia.geometry.linalg import transform_points
-from kornia.geometry.linalg import relative_transformation
+from kornia.geometry.linalg import transform_points, inverse_transformation, compose_transformations
 from kornia.geometry.conversions import convert_points_to_homogeneous
 from kornia.geometry.conversions import normalize_pixel_coordinates
 from kornia.geometry.camera import PinholeCamera
@@ -80,8 +79,10 @@
                             "PinholeCamera. Got {}".format(type(pinhole_src)))
         # compute the relative pose between the non reference and the reference
         # camera frames.
-        dst_trans_src: torch.Tensor = relative_transformation(
-            pinhole_src.extrinsics, self._pinhole_dst.extrinsics)
+        dst_trans_src: torch.Tensor = compose_transformations(
+            self._pinhole_dst.extrinsics,
+            inverse_transformation(pinhole_src.extrinsics),
+        )
 
         # compute the projection matrix between the non reference cameras and
         # the reference.

{"golden_diff": "diff --git a/kornia/geometry/warp/depth_warper.py b/kornia/geometry/warp/depth_warper.py\n--- a/kornia/geometry/warp/depth_warper.py\n+++ b/kornia/geometry/warp/depth_warper.py\n@@ -3,8 +3,7 @@\n import torch.nn as nn\n import torch.nn.functional as F\n \n-from kornia.geometry.linalg import transform_points\n-from kornia.geometry.linalg import relative_transformation\n+from kornia.geometry.linalg import transform_points, inverse_transformation, compose_transformations\n from kornia.geometry.conversions import convert_points_to_homogeneous\n from kornia.geometry.conversions import normalize_pixel_coordinates\n from kornia.geometry.camera import PinholeCamera\n@@ -80,8 +79,10 @@\n                             \"PinholeCamera. Got {}\".format(type(pinhole_src)))\n         # compute the relative pose between the non reference and the reference\n         # camera frames.\n-        dst_trans_src: torch.Tensor = relative_transformation(\n-            pinhole_src.extrinsics, self._pinhole_dst.extrinsics)\n+        dst_trans_src: torch.Tensor = compose_transformations(\n+            self._pinhole_dst.extrinsics,\n+            inverse_transformation(pinhole_src.extrinsics),\n+        )\n \n         # compute the projection matrix between the non reference cameras and\n         # the reference.\n", "issue": "warp_frame_depth fails with different aspect ratios between src and dst\n## \ud83d\udc1b Bug\r\n\r\nAs in the title: I am trying to get `kornia.geometry.warp_frame_depth` to work for my project and a seemingly intermittent bug appears to boil down to that - `warp_frame_depth` works when the aspect ratios of `src` and `dst` match but fails otherwise.\r\n<!-- A clear and concise description of what the bug is. -->\r\n\r\n## To Reproduce\r\nI made a small repro repository with sample images, depths and camera parameters [here](https://github.com/jatentaki/warp-bug). Just clone it and execute `warp.py` to see the behavior (it uses kornia, numpy, matplotlib, imageio and h5py modules). You can comment out lines 6-7 in favor of 10-11 to see a working example with images of equal aspect ratio.\r\n\r\n## Expected behavior\r\nThe warping behavior should not depend on aspect ratios. It may seem that a workaround is to pad the images until they are of equal shape, but this doesn't seem to be the case.\r\n\r\n## Environment\r\nkornia 0.4.0\r\n\r\nPyTorch version: 1.6.0\r\nIs debug build: No\r\nCUDA used to build PyTorch: None\r\n\r\nOS: Ubuntu 18.04.5 LTS\r\nGCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0\r\nClang version: 6.0.0-1ubuntu2 (tags/RELEASE_600/final)\r\nCMake version: version 3.18.2\r\n\r\nPython version: 3.8 (64-bit runtime)\r\nIs CUDA available: No\r\nCUDA runtime version: No CUDA\r\nGPU models and configuration: No CUDA\r\nNvidia driver version: No CUDA\r\ncuDNN version: No CUDA\r\n\r\nVersions of relevant libraries:\r\n[pip3] numpy==1.19.1\r\n[pip3] torch==1.6.0\r\n[pip3] torch-cluster==1.5.4\r\n[pip3] torch-dimcheck==0.0.1\r\n[pip3] torch-geometric==1.4.3\r\n[pip3] torch-localize==0.1.0\r\n[pip3] torch-scatter==2.0.4\r\n[pip3] torch-sparse==0.6.2\r\n[pip3] torchvision==0.7.0\r\n[conda] blas                      1.0                         mkl  \r\n[conda] cpuonly                   1.0                           0    pytorch\r\n[conda] mkl                       2020.1                      217  \r\n[conda] mkl-service               2.3.0            py38he904b0f_0  \r\n[conda] mkl_fft                   1.1.0            py38h23d657b_0  \r\n[conda] mkl_random                1.1.1            py38h0573a6f_0  \r\n[conda] numpy                     1.19.1           py38hbc911f0_0  \r\n[conda] numpy-base                1.19.1           py38hfa32c7d_0  \r\n[conda] pytorch                   1.6.0               py3.8_cpu_0  [cpuonly]  pytorch\r\n[conda] torch-cluster             1.5.4                    pypi_0    pypi\r\n[conda] torch-dimcheck            0.0.1                    pypi_0    pypi\r\n[conda] torch-geometric           1.4.3                    pypi_0    pypi\r\n[conda] torch-localize            0.1.0                    pypi_0    pypi\r\n[conda] torch-scatter             2.0.4                    pypi_0    pypi\r\n[conda] torch-sparse              0.6.2                    pypi_0    pypi\r\n[conda] torchvision               0.7.0                  py38_cpu  [cpuonly]  pytorch\r\n\n", "before_files": [{"content": "from typing import Union\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\n\nfrom kornia.geometry.linalg import transform_points\nfrom kornia.geometry.linalg import relative_transformation\nfrom kornia.geometry.conversions import convert_points_to_homogeneous\nfrom kornia.geometry.conversions import normalize_pixel_coordinates\nfrom kornia.geometry.camera import PinholeCamera\nfrom kornia.geometry.camera import cam2pixel, pixel2cam\nfrom kornia.utils import create_meshgrid\n\n\n__all__ = [\n    \"depth_warp\",\n    \"DepthWarper\",\n]\n\n\nclass DepthWarper(nn.Module):\n    r\"\"\"Warps a patch by depth.\n\n    .. math::\n        P_{src}^{\\{dst\\}} = K_{dst} * T_{src}^{\\{dst\\}}\n\n        I_{src} = \\\\omega(I_{dst}, P_{src}^{\\{dst\\}}, D_{src})\n\n    Args:\n        pinholes_dst (PinholeCamera): the pinhole models for the destination\n          frame.\n        height (int): the height of the image to warp.\n        width (int): the width of the image to warp.\n        mode (str): interpolation mode to calculate output values\n          'bilinear' | 'nearest'. Default: 'bilinear'.\n        padding_mode (str): padding mode for outside grid values\n           'zeros' | 'border' | 'reflection'. Default: 'zeros'.\n        align_corners(bool): interpolation flag. Default: True. See\n        https://pytorch.org/docs/stable/nn.functional.html#torch.nn.functional.interpolate for detail\n    \"\"\"\n\n    def __init__(self,\n                 pinhole_dst: PinholeCamera,\n                 height: int, width: int,\n                 mode: str = 'bilinear',\n                 padding_mode: str = 'zeros',\n                 align_corners: bool = True):\n        super(DepthWarper, self).__init__()\n        # constructor members\n        self.width: int = width\n        self.height: int = height\n        self.mode: str = mode\n        self.padding_mode: str = padding_mode\n        self.eps = 1e-6\n        self.align_corners: bool = align_corners\n\n        # state members\n        self._pinhole_dst: PinholeCamera = pinhole_dst\n        self._pinhole_src: Union[None, PinholeCamera] = None\n        self._dst_proj_src: Union[None, torch.Tensor] = None\n\n        self.grid: torch.Tensor = self._create_meshgrid(height, width)\n\n    @staticmethod\n    def _create_meshgrid(height: int, width: int) -> torch.Tensor:\n        grid: torch.Tensor = create_meshgrid(\n            height, width, normalized_coordinates=False)  # 1xHxWx2\n        return convert_points_to_homogeneous(grid)  # append ones to last dim\n\n    def compute_projection_matrix(\n            self, pinhole_src: PinholeCamera) -> 'DepthWarper':\n        r\"\"\"Computes the projection matrix from the source to destinaion frame.\n        \"\"\"\n        if not isinstance(self._pinhole_dst, PinholeCamera):\n            raise TypeError(\"Member self._pinhole_dst expected to be of class \"\n                            \"PinholeCamera. Got {}\"\n                            .format(type(self._pinhole_dst)))\n        if not isinstance(pinhole_src, PinholeCamera):\n            raise TypeError(\"Argument pinhole_src expected to be of class \"\n                            \"PinholeCamera. Got {}\".format(type(pinhole_src)))\n        # compute the relative pose between the non reference and the reference\n        # camera frames.\n        dst_trans_src: torch.Tensor = relative_transformation(\n            pinhole_src.extrinsics, self._pinhole_dst.extrinsics)\n\n        # compute the projection matrix between the non reference cameras and\n        # the reference.\n        dst_proj_src: torch.Tensor = torch.matmul(\n            self._pinhole_dst.intrinsics, dst_trans_src)\n\n        # update class members\n        self._pinhole_src = pinhole_src\n        self._dst_proj_src = dst_proj_src\n        return self\n\n    def _compute_projection(self, x, y, invd):\n        point = torch.FloatTensor([[[x], [y], [1.0], [invd]]])\n        flow = torch.matmul(\n            self._dst_proj_src, point.to(self._dst_proj_src.device))\n        z = 1. / flow[:, 2]\n        x = (flow[:, 0] * z)\n        y = (flow[:, 1] * z)\n        return torch.cat([x, y], 1)\n\n    def compute_subpixel_step(self) -> torch.Tensor:\n        \"\"\"This computes the required inverse depth step to achieve sub pixel\n        accurate sampling of the depth cost volume, per camera.\n\n        Szeliski, Richard, and Daniel Scharstein.\n        \"Symmetric sub-pixel stereo matching.\" European Conference on Computer\n        Vision. Springer Berlin Heidelberg, 2002.\n        \"\"\"\n        delta_d = 0.01\n        xy_m1 = self._compute_projection(self.width / 2, self.height / 2,\n                                         1.0 - delta_d)\n        xy_p1 = self._compute_projection(self.width / 2, self.height / 2,\n                                         1.0 + delta_d)\n        dx = torch.norm((xy_p1 - xy_m1), 2, dim=-1) / 2.0\n        dxdd = dx / (delta_d)  # pixel*(1/meter)\n        # half pixel sampling, we're interested in the min for all cameras\n        return torch.min(0.5 / dxdd)\n\n    def warp_grid(self, depth_src: torch.Tensor) -> torch.Tensor:\n        \"\"\"Computes a grid for warping a given the depth from the reference\n        pinhole camera.\n\n        The function `compute_projection_matrix` has to be called beforehand in\n        order to have precomputed the relative projection matrices encoding the\n        relative pose and the intrinsics between the reference and a non\n        reference camera.\n        \"\"\"\n        # TODO: add type and value checkings\n        if self._dst_proj_src is None or self._pinhole_src is None:\n            raise ValueError(\"Please, call compute_projection_matrix.\")\n\n        if len(depth_src.shape) != 4:\n            raise ValueError(\"Input depth_src has to be in the shape of \"\n                             \"Bx1xHxW. Got {}\".format(depth_src.shape))\n\n        # unpack depth attributes\n        batch_size, _, height, width = depth_src.shape\n        device: torch.device = depth_src.device\n        dtype: torch.dtype = depth_src.dtype\n\n        # expand the base coordinate grid according to the input batch size\n        pixel_coords: torch.Tensor = self.grid.to(device).to(dtype).expand(\n            batch_size, -1, -1, -1)  # BxHxWx3\n\n        # reproject the pixel coordinates to the camera frame\n        cam_coords_src: torch.Tensor = pixel2cam(\n            depth_src,\n            self._pinhole_src.intrinsics_inverse().to(dtype),\n            pixel_coords)  # BxHxWx3\n\n        # reproject the camera coordinates to the pixel\n        pixel_coords_src: torch.Tensor = cam2pixel(\n            cam_coords_src, self._dst_proj_src.to(dtype))  # (B*N)xHxWx2\n\n        # normalize between -1 and 1 the coordinates\n        pixel_coords_src_norm: torch.Tensor = normalize_pixel_coordinates(\n            pixel_coords_src, self.height, self.width)\n        return pixel_coords_src_norm\n\n    def forward(  # type: ignore\n            self,\n            depth_src: torch.Tensor,\n            patch_dst: torch.Tensor) -> torch.Tensor:\n        \"\"\"Warps a tensor from destination frame to reference given the depth\n        in the reference frame.\n\n        Args:\n            depth_src (torch.Tensor): the depth in the reference frame. The\n              tensor must have a shape :math:`(B, 1, H, W)`.\n            patch_dst (torch.Tensor): the patch in the destination frame. The\n              tensor must have a shape :math:`(B, C, H, W)`.\n\n        Return:\n            torch.Tensor: the warped patch from destination frame to reference.\n\n        Shape:\n            - Output: :math:`(N, C, H, W)` where C = number of channels.\n\n        Example:\n            >>> # pinholes camera models\n            >>> pinhole_dst = kornia.PinholeCamera(...)\n            >>> pinhole_src = kornia.PinholeCamera(...)\n            >>> # create the depth warper, compute the projection matrix\n            >>> warper = kornia.DepthWarper(pinhole_dst, height, width)\n            >>> warper.compute_projection_matrix(pinhole_src)\n            >>> # warp the destionation frame to reference by depth\n            >>> depth_src = torch.ones(1, 1, 32, 32)  # Nx1xHxW\n            >>> image_dst = torch.rand(1, 3, 32, 32)  # NxCxHxW\n            >>> image_src = warper(depth_src, image_dst)  # NxCxHxW\n        \"\"\"\n        return F.grid_sample(patch_dst, self.warp_grid(depth_src),  # type: ignore\n                             mode=self.mode, padding_mode=self.padding_mode,\n                             align_corners=self.align_corners)\n\n\n# functional api\n\ndef depth_warp(pinhole_dst: PinholeCamera,\n               pinhole_src: PinholeCamera,\n               depth_src: torch.Tensor,\n               patch_dst: torch.Tensor,\n               height: int, width: int,\n               align_corners: bool = True):\n    r\"\"\"Function that warps a tensor from destination frame to reference\n    given the depth in the reference frame.\n\n    See :class:`~kornia.geometry.warp.DepthWarper` for details.\n\n    Example:\n        >>> # pinholes camera models\n        >>> pinhole_dst = kornia.PinholeCamera(...)\n        >>> pinhole_src = kornia.PinholeCamera(...)\n        >>> # warp the destionation frame to reference by depth\n        >>> depth_src = torch.ones(1, 1, 32, 32)  # Nx1xHxW\n        >>> image_dst = torch.rand(1, 3, 32, 32)  # NxCxHxW\n        >>> image_src = kornia.depth_warp(pinhole_dst, pinhole_src,\n        >>>     depth_src, image_dst, height, width)  # NxCxHxW\n    \"\"\"\n    warper = DepthWarper(pinhole_dst, height, width, align_corners=align_corners)\n    warper.compute_projection_matrix(pinhole_src)\n    return warper(depth_src, patch_dst)\n", "path": "kornia/geometry/warp/depth_warper.py"}], "after_files": [{"content": "from typing import Union\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\n\nfrom kornia.geometry.linalg import transform_points, inverse_transformation, compose_transformations\nfrom kornia.geometry.conversions import convert_points_to_homogeneous\nfrom kornia.geometry.conversions import normalize_pixel_coordinates\nfrom kornia.geometry.camera import PinholeCamera\nfrom kornia.geometry.camera import cam2pixel, pixel2cam\nfrom kornia.utils import create_meshgrid\n\n\n__all__ = [\n    \"depth_warp\",\n    \"DepthWarper\",\n]\n\n\nclass DepthWarper(nn.Module):\n    r\"\"\"Warps a patch by depth.\n\n    .. math::\n        P_{src}^{\\{dst\\}} = K_{dst} * T_{src}^{\\{dst\\}}\n\n        I_{src} = \\\\omega(I_{dst}, P_{src}^{\\{dst\\}}, D_{src})\n\n    Args:\n        pinholes_dst (PinholeCamera): the pinhole models for the destination\n          frame.\n        height (int): the height of the image to warp.\n        width (int): the width of the image to warp.\n        mode (str): interpolation mode to calculate output values\n          'bilinear' | 'nearest'. Default: 'bilinear'.\n        padding_mode (str): padding mode for outside grid values\n           'zeros' | 'border' | 'reflection'. Default: 'zeros'.\n        align_corners(bool): interpolation flag. Default: True. See\n        https://pytorch.org/docs/stable/nn.functional.html#torch.nn.functional.interpolate for detail\n    \"\"\"\n\n    def __init__(self,\n                 pinhole_dst: PinholeCamera,\n                 height: int, width: int,\n                 mode: str = 'bilinear',\n                 padding_mode: str = 'zeros',\n                 align_corners: bool = True):\n        super(DepthWarper, self).__init__()\n        # constructor members\n        self.width: int = width\n        self.height: int = height\n        self.mode: str = mode\n        self.padding_mode: str = padding_mode\n        self.eps = 1e-6\n        self.align_corners: bool = align_corners\n\n        # state members\n        self._pinhole_dst: PinholeCamera = pinhole_dst\n        self._pinhole_src: Union[None, PinholeCamera] = None\n        self._dst_proj_src: Union[None, torch.Tensor] = None\n\n        self.grid: torch.Tensor = self._create_meshgrid(height, width)\n\n    @staticmethod\n    def _create_meshgrid(height: int, width: int) -> torch.Tensor:\n        grid: torch.Tensor = create_meshgrid(\n            height, width, normalized_coordinates=False)  # 1xHxWx2\n        return convert_points_to_homogeneous(grid)  # append ones to last dim\n\n    def compute_projection_matrix(\n            self, pinhole_src: PinholeCamera) -> 'DepthWarper':\n        r\"\"\"Computes the projection matrix from the source to destinaion frame.\n        \"\"\"\n        if not isinstance(self._pinhole_dst, PinholeCamera):\n            raise TypeError(\"Member self._pinhole_dst expected to be of class \"\n                            \"PinholeCamera. Got {}\"\n                            .format(type(self._pinhole_dst)))\n        if not isinstance(pinhole_src, PinholeCamera):\n            raise TypeError(\"Argument pinhole_src expected to be of class \"\n                            \"PinholeCamera. Got {}\".format(type(pinhole_src)))\n        # compute the relative pose between the non reference and the reference\n        # camera frames.\n        dst_trans_src: torch.Tensor = compose_transformations(\n            self._pinhole_dst.extrinsics,\n            inverse_transformation(pinhole_src.extrinsics),\n        )\n\n        # compute the projection matrix between the non reference cameras and\n        # the reference.\n        dst_proj_src: torch.Tensor = torch.matmul(\n            self._pinhole_dst.intrinsics, dst_trans_src)\n\n        # update class members\n        self._pinhole_src = pinhole_src\n        self._dst_proj_src = dst_proj_src\n        return self\n\n    def _compute_projection(self, x, y, invd):\n        point = torch.FloatTensor([[[x], [y], [1.0], [invd]]])\n        flow = torch.matmul(\n            self._dst_proj_src, point.to(self._dst_proj_src.device))\n        z = 1. / flow[:, 2]\n        x = (flow[:, 0] * z)\n        y = (flow[:, 1] * z)\n        return torch.cat([x, y], 1)\n\n    def compute_subpixel_step(self) -> torch.Tensor:\n        \"\"\"This computes the required inverse depth step to achieve sub pixel\n        accurate sampling of the depth cost volume, per camera.\n\n        Szeliski, Richard, and Daniel Scharstein.\n        \"Symmetric sub-pixel stereo matching.\" European Conference on Computer\n        Vision. Springer Berlin Heidelberg, 2002.\n        \"\"\"\n        delta_d = 0.01\n        xy_m1 = self._compute_projection(self.width / 2, self.height / 2,\n                                         1.0 - delta_d)\n        xy_p1 = self._compute_projection(self.width / 2, self.height / 2,\n                                         1.0 + delta_d)\n        dx = torch.norm((xy_p1 - xy_m1), 2, dim=-1) / 2.0\n        dxdd = dx / (delta_d)  # pixel*(1/meter)\n        # half pixel sampling, we're interested in the min for all cameras\n        return torch.min(0.5 / dxdd)\n\n    def warp_grid(self, depth_src: torch.Tensor) -> torch.Tensor:\n        \"\"\"Computes a grid for warping a given the depth from the reference\n        pinhole camera.\n\n        The function `compute_projection_matrix` has to be called beforehand in\n        order to have precomputed the relative projection matrices encoding the\n        relative pose and the intrinsics between the reference and a non\n        reference camera.\n        \"\"\"\n        # TODO: add type and value checkings\n        if self._dst_proj_src is None or self._pinhole_src is None:\n            raise ValueError(\"Please, call compute_projection_matrix.\")\n\n        if len(depth_src.shape) != 4:\n            raise ValueError(\"Input depth_src has to be in the shape of \"\n                             \"Bx1xHxW. Got {}\".format(depth_src.shape))\n\n        # unpack depth attributes\n        batch_size, _, height, width = depth_src.shape\n        device: torch.device = depth_src.device\n        dtype: torch.dtype = depth_src.dtype\n\n        # expand the base coordinate grid according to the input batch size\n        pixel_coords: torch.Tensor = self.grid.to(device).to(dtype).expand(\n            batch_size, -1, -1, -1)  # BxHxWx3\n\n        # reproject the pixel coordinates to the camera frame\n        cam_coords_src: torch.Tensor = pixel2cam(\n            depth_src,\n            self._pinhole_src.intrinsics_inverse().to(dtype),\n            pixel_coords)  # BxHxWx3\n\n        # reproject the camera coordinates to the pixel\n        pixel_coords_src: torch.Tensor = cam2pixel(\n            cam_coords_src, self._dst_proj_src.to(dtype))  # (B*N)xHxWx2\n\n        # normalize between -1 and 1 the coordinates\n        pixel_coords_src_norm: torch.Tensor = normalize_pixel_coordinates(\n            pixel_coords_src, self.height, self.width)\n        return pixel_coords_src_norm\n\n    def forward(  # type: ignore\n            self,\n            depth_src: torch.Tensor,\n            patch_dst: torch.Tensor) -> torch.Tensor:\n        \"\"\"Warps a tensor from destination frame to reference given the depth\n        in the reference frame.\n\n        Args:\n            depth_src (torch.Tensor): the depth in the reference frame. The\n              tensor must have a shape :math:`(B, 1, H, W)`.\n            patch_dst (torch.Tensor): the patch in the destination frame. The\n              tensor must have a shape :math:`(B, C, H, W)`.\n\n        Return:\n            torch.Tensor: the warped patch from destination frame to reference.\n\n        Shape:\n            - Output: :math:`(N, C, H, W)` where C = number of channels.\n\n        Example:\n            >>> # pinholes camera models\n            >>> pinhole_dst = kornia.PinholeCamera(...)\n            >>> pinhole_src = kornia.PinholeCamera(...)\n            >>> # create the depth warper, compute the projection matrix\n            >>> warper = kornia.DepthWarper(pinhole_dst, height, width)\n            >>> warper.compute_projection_matrix(pinhole_src)\n            >>> # warp the destionation frame to reference by depth\n            >>> depth_src = torch.ones(1, 1, 32, 32)  # Nx1xHxW\n            >>> image_dst = torch.rand(1, 3, 32, 32)  # NxCxHxW\n            >>> image_src = warper(depth_src, image_dst)  # NxCxHxW\n        \"\"\"\n        return F.grid_sample(patch_dst, self.warp_grid(depth_src),  # type: ignore\n                             mode=self.mode, padding_mode=self.padding_mode,\n                             align_corners=self.align_corners)\n\n\n# functional api\n\ndef depth_warp(pinhole_dst: PinholeCamera,\n               pinhole_src: PinholeCamera,\n               depth_src: torch.Tensor,\n               patch_dst: torch.Tensor,\n               height: int, width: int,\n               align_corners: bool = True):\n    r\"\"\"Function that warps a tensor from destination frame to reference\n    given the depth in the reference frame.\n\n    See :class:`~kornia.geometry.warp.DepthWarper` for details.\n\n    Example:\n        >>> # pinholes camera models\n        >>> pinhole_dst = kornia.PinholeCamera(...)\n        >>> pinhole_src = kornia.PinholeCamera(...)\n        >>> # warp the destionation frame to reference by depth\n        >>> depth_src = torch.ones(1, 1, 32, 32)  # Nx1xHxW\n        >>> image_dst = torch.rand(1, 3, 32, 32)  # NxCxHxW\n        >>> image_src = kornia.depth_warp(pinhole_dst, pinhole_src,\n        >>>     depth_src, image_dst, height, width)  # NxCxHxW\n    \"\"\"\n    warper = DepthWarper(pinhole_dst, height, width, align_corners=align_corners)\n    warper.compute_projection_matrix(pinhole_src)\n    return warper(depth_src, patch_dst)\n", "path": "kornia/geometry/warp/depth_warper.py"}]}