Why we use transposed filter as the deconvolution operation instead of the pseudo inverse of filter? I am trying to visualization CNN by the method in this paper "Visualizing and Understanding Convolutional Networks" 

According to this tutorial A guide to convolution arithmetic for deep
learning (P.19)

We can always rewrite convolution as a matrix multiplication

For example, $Y$ is a feature map &amp;  $X$ is input &amp; $W$ is a filter &amp; $\otimes$ is convolution operation

$$
\begin{bmatrix}
y_{11} &amp; y_{12} \\\\
y_{21} &amp; y_{22}
\end{bmatrix}
=
\begin{bmatrix}
x_{11} &amp; x_{12} &amp; x_{13} &amp; x_{14}\\\\
x_{21} &amp; x_{22} &amp; x_{23} &amp; x_{24}\\\\
x_{31} &amp; x_{32} &amp; x_{33} &amp; x_{34}\\\\
x_{41} &amp; x_{42} &amp; x_{43} &amp; x_{44}
\end{bmatrix}
\otimes
\begin{bmatrix}
w_{11} &amp; w_{12} &amp; w_{13} \\\\
w_{21} &amp; w_{22} &amp; w_{23} \\\\
w_{31} &amp; w_{32} &amp; w_{33}
\end{bmatrix}
$$

In the matrix multiplication form

$$
\begin{bmatrix}
y_{11} \\\\
y_{12} \\\\
y_{21} \\\\
y_{22}
\end{bmatrix}
=
C
\begin{bmatrix}
x_{11}\\\\
x_{12}\\\\
\vdots \\\\
x_{44}
\end{bmatrix}
$$

$$
C=
\left[\begin{smallmatrix}
w_{11} &amp; w_{12} &amp; w_{13} &amp; 0 &amp; w_{21} &amp; w_{22} &amp; w_{23} &amp; 0 &amp; w_{31} &amp; w_{32} &amp; w_{33} &amp; 0 &amp; 0 &amp; 0 &amp; 0 &amp; 0\\\\
0 &amp; w_{11} &amp; w_{12} &amp; w_{13} &amp; 0 &amp; w_{21} &amp; w_{22} &amp; w_{23} &amp; 0 &amp; w_{31} &amp; w_{32} &amp; w_{33} &amp; 0 &amp; 0 &amp; 0 &amp; 0\\\\
0 &amp; 0 &amp; 0 &amp; 0 &amp; w_{11} &amp; w_{12} &amp; w_{13} &amp; 0 &amp; w_{21} &amp; w_{22} &amp; w_{23} &amp; 0 &amp; w_{31} &amp; w_{32} &amp; w_{33} &amp; 0\\\\
0&amp; 0 &amp; 0 &amp; 0 &amp; 0 &amp; w_{11} &amp; w_{12} &amp; w_{13} &amp; 0 &amp; w_{21} &amp; w_{22} &amp; w_{23} &amp; 0 &amp; w_{31} &amp; w_{32} &amp; w_{33}\\\\
\end{smallmatrix}
\right]
$$

So we can visualize filter by multiplying $C^{-1}$.

OK, my question is why we use $C^{T}$ instead of the pseudo inverse $(C^{T}C)^{-1}C^{T}$?
First, transposed convolution isn't the inverse operation of convolution.
It only takes the output shape of origin convolution as its input shape and takes the input shape of origin convolution as its output shape.

Second, transposed convolution can be seen to a learnable upsampling operation. It just likes bilinear or bicubic upsampling but all of the parameters are learned by gradient descent.

https://www.youtube.com/watch?v=nDPWywWRIRo&amp;index=12&amp;list=PL3FW7Lu3i5JvHM8ljYj-zLfQRF3EO8sYv&amp;t=1632s