Invertible Autoencoder for domain adaptation

Yunfei Teng
Anna Choromanska
Mariusz Bojarski

YT 1208@ NYU . EDU
AC 5455@ NYU . EDU
MBOJARSKI @ NVIDIA . COM

arXiv:1802.06869v1 [eess.IV] 10 Feb 2018

Abstract
The unsupervised image-to-image translation
aims at finding a mapping between the source
(A) and target (B) image domains, where in
many applications aligned image pairs are not
available at training. This is an ill-posed learning problem since it requires inferring the joint
probability distribution from marginals. Joint
learning of coupled mappings FAB : A → B
and FBA : B → A is commonly used by the
state-of-the-art methods, like CycleGAN (Zhu
et al., 2017), to learn this translation by introducing cycle consistency requirement to the learning problem, i.e. FAB (FBA (B)) ≈ B and
FBA (FAB (A)) ≈ A. Cycle consistency enforces the preservation of the mutual information between input and translated images. However, it does not explicitly enforce FBA to be an
inverse operation to FAB . We propose a new
deep architecture that we call invertible autoencoder (InvAuto) to explicitly enforce this relation. This is done by forcing an encoder to be
an inverted version of the decoder, where corresponding layers perform opposite mappings and
share parameters. The mappings are constrained
to be orthonormal. The resulting architecture
leads to the reduction of the number of trainable
parameters (up to 2 times). We present image
translation results on benchmark data sets and
demonstrate state-of-the art performance of our
approach. Finally, we test the proposed domain
adaptation method on the task of road video conversion. We demonstrate that the videos converted with InvAuto have high quality and show
that the NVIDIA neural-network-based end-toend learning system for autonomous driving,
known as PilotNet, trained on real road videos
performs well when tested on the converted ones.

1. Introduction
Inter-domain translation problem of converting an instance,
e.g.: image or video, from one domain to another is applicable to a wide variety of learning tasks, including object detection and recognition, image categorization, sentiment analysis, action recognition, speech recognition, and
more. High-quality domain translators ensure that an arbitrary learning model trained on the samples from the
source domain, can perform well when tested on the translated samples1 . The translation problem can be posed in
the supervised learning framework, e.g.: (Isola et al., 2017;
Wang et al., 2017), where the learner has access to corresponding pairs of instances from both domains, or unsupervised learning framework, e.g.: (Zhu et al., 2017; Liu et al.,
2017), where no such paired instances are available. This
paper focuses on the latter case, which is more difficult but
at the same time more realistic as acquiring the data set of
paired images is often impossible in practice.
The unsupervised domain adaptation is typically solved
using generative adversarial networks (GAN) framework (Goodfellow et al., 2014), where the generator performs domain translation and is trained to learn the mapping from the source to the target domain and the discriminator is trained to discriminate between original images
from the target domain and those provided by the generator. In this setting, the generator usually has the structure
of the autoencoder. The two most common state-of-theart domain adaptation approaches, CycleGAN (Zhu et al.,
2017) and UNIT (Liu et al., 2017), are built on this basic approach. CycleGAN addresses the problem of adaptation from domain A to domain B by training two translation networks, where one realizes the mapping FAB and
the other realizes FBA . The cycle consistency loss ensures
the correlation between input image and the corresponding
translation. In particular, to achieve cycle consistency, CycleGAN trains two autoencoders, where each minimizes its
own adversarial loss and they both jointly minimize
kFAB (FBA (B)) − Bk22 and kFBA (FAB (A)) − Ak22 . (1)
1
Similarly, an arbitrary learning model trained on the translated samples should perform well on the samples from the target
domain. Training in this framework is however much more computationally expensive.

Invertible Autoencoder for domain adaptation

InvAuto

Auto

Cycle

VAE

(a) MNIST MLP

(b) MNIST MLP

(c) MNIST MLP

(d) MNIST MLP

(e) CIFAR ResNet

(f) CIFAR ResNet

(g) CIFAR ResNet

(h) CIFAR ResNet

Figure 1. Heatmap of the values of matrix DE for InvAuto (a and e), Auto (b and f), Cycle (c and g), and VAE (d and h) on MLP
and ResNet architectures and MNIST and CIFAR data sets. Matrices E and D are constructed by multiplying the weight matrices of
consecutive layers of encoder and decoder, respectively. In case of InvAuto, DE is the closest to the identity matrix.

Cycle consistency loss is also incorporated into the recent
implementations of UNIT. It is implicitly assumed that the
model will learn the mappings FAB and FBA in such a way
−1
that FAB = FBA
, however it is not explicitly imposed.
Consider a simple example. Assume the first autonecoder
is a 2-layer linear multi-layer perceptron (MLP) where the
weight matrix of the first layer (encoder) is denoted as E1
and the weight matrix of the second layer (decoder) is denoted as D1 . Thus, for an input xA ∈ A it outputs yB (xA ) =
D1 E1 xA . The second autoencoder then is a 2-layer MLP
with encoder weight matrix E2 and decoder weight matrix
D2 that for an input data point xB should produce output
yA (xB ) = D2 E2 xB . To satisfy cycle consistency requirement, the following should hold: yA (yB (xA )) = (xA ) and
yB (yA (xB )) = (xB ). These two conditions are equivalent
to D2 E2 D1 E1 = I and D1 E1 D2 E2 = I. This holds for
example when D1 = E2−1 and D2 = E1−1 .
In contrast to this approach, we implicitly require FAB =
−1
FBA
. Thus, in the context of the given simple example, we
correlate encoders and decoders to satisfy inversion conditions D1 = E2−1 and D2 = E1−1 . We avoid performing
prohibitive inversions of large matrices and instead guarantee these conditions to hold through two steps: (i) introducing shared parametrization of encoder E2 and decoder D1 such that D1 = E2> (E1 and D2 is treated simi-

larly) and (ii) appropriate training to achieve orthonormality E2> = E2−1 and E1> = E1−1 , i.e. we train autoencoder
(E2 , D1 ) to satisfy D1 E2 xB = xB for arbitrary input xB
and autoencoder (E1 , D2 ) to satisfy D2 E1 xA = xA for
arbitrary input xA . Since the encoder and decoder are coupled as given in (i), such training leads to satisfying inversion conditions. Practical networks contain linear and
non-linear transformations. We therefore propose specific
architectures, which are invertible.

Figure 2. Comparison of the mean squared error (MSE)
MSE(DE − I) for InvAuto, Auto, Cycle, and VAE on MLP,
convolutional, and ResNet architectures and MNIST and CIFAR
data sets. Matrices E and D are constructed by multiplying the
weight matrices of consecutive layers of encoder and decoder, respectively.

Invertible Autoencoder for domain adaptation

Figure 1 (see also its extended version, Figure 17, in the
Supplement) and 2 illustrate the basic idea behind InvAuto.
The plots were obtained by training a single autoencoder
(E, D) to reconstruct its input. InvAuto has shared weights
satisfying D = E > and inverted non-linearities and clearly
obtains matrix DE that is the closest to identity compared
to other methods, i.e. vanilla autoencoder (Auto), autoencoder with cycle consistency (Cycle), and variational autoencoder (VAE) (Kingma & Welling, 2014). Note also
that at the same time InvAuto requires half of the number
of trainable parameters.
This paper is organized as follows: Section 2 reviews the
literature, Section 3 explains InvAuto in details, Section 4
explains how to apply InvAuto to domain adaptation, Section 5 demonstrates experimental verification of the proposed approach, and Section 6 provides conclusions.

2. Related Work
Unsupervised image-to-image translation models were developed to tackle domain adaptation problem with unpaired
data sets. A plethora of existing approaches utilize autoencoders trained in the GAN framework, where autoencoder
serves as a generator, for this learning problem. This includes approaches based on conditional GAN (Dong et al.,
2017; Wang et al., 2017) and methods introducing additional components to the loss function forcing partial cycle
consistency (Taigman et al., 2016). Another approach (Liu
& Tuzel, 2016) introduces two coupled GANs, where each
generator is an autoencoder and the coupling is obtained
by sharing a subset of weights between autoencoders as
well as between discriminators. This technique was later
on extended to utilize variational autoencoders as generators (Liu et al., 2017). The resulting approach is commonly
known as UNIT. CycleGAN presents yet another way of
addressing the image-to-image translation by specific training scheme that preserves the mutual information between
input and translated images (Vincent et al., 2008). Both
UNIT and CycleGAN constitute the most popular choices
for performing image-to-image translation.
There also exist other learning tasks that can be viewed as
instances of image-to-image translation problem. Among
them, notable approaches focus on style transfer (Gatys
et al., 2016b; Johnson et al., 2016; Ulyanov et al., 2016;
Gatys et al., 2016a). They aim at preserving the content
of the input image while altering its style to mimic the
style of the images from the target domain. This goal is
achieved by introducing content and style loss functions
that are jointly optimized. Finally, inverse problems, such
as super-resolution, also fall into the category of image-toimage translation problems (McCann et al., 2017).

3. Invertible autoencoder
Here we explain the details of the architecture of
InvAuto.
The architecture needs to be symmetric
to allow invertibility, e.g.: the layers should be ar−1
−1
−1
ranged as (T1 , T2 , . . . , TM , TM
, TM
−1 , . . . , T1 ), where
|
{z
} |
{z
}
encoder E

decoder D

T1 , T2 , . . . , TM denote subsequent transformations of the
signal that is being propagated through the network (M is
−1
the total number of those) and T1−1 , T2−1 , . . . , TM
denote
their inversions. Thus, the architecture is inverted layer by
layer, where any layer of the encoder has its mirror inverted
counterpart in the decoder. The autoencoder is trained to
reconstruct its input. Below we explain how to invert different types of layers of the deep model.
3.1. Fully-connected layer
Consider transformation T E of an input signal performed
by an arbitrary fully-connected layer of an encoder E
parametrized with weight matrix W . Let x denote layer’s
input and y denote its output. Thus
T E : y = W x.

(2)

An inverse operation is then defined as
(T E )−1 : x = W −1 y,

(3)

We parametrize the counterpart layer of the decoder with a
transpose of W , thus the considered encoder and decoder
layers will share parametrization. Therefore, we enforce
the counterpart decoder’s layer to perform transformation:
T D : x = W > y.

(4)

By training the autoencoder to reconstruct its input on its
output we will enforce orthonormality W −1 = W > and
thus equivalence of transformations (T E )−1 and T D , i.e.
(T E )−1 ≡ T D .
3.2. Convolutional layer
Consider transformation T E of an input image performed
by an arbitrary convolutional layer of an encoder E. Let x
denote layer’s vectorized input image and y denote corresponding output. 2D convolution can be implemented using matrix multiplication involving a Toeplitz matrix (Vasudevan et al., 2017). Toeplitz matrix is obtained from the
set of kernels of the 2D convolutional filters . Thus transformation T E and its inverse (T E )−1 can be explained
with the same equations as the ones used before, Equations 2 and 8, however now W is a Toeplitz matrix. We will
again parametrize the counterpart layer of the decoder with
a transpose of a Toeplitz matrix W . The transpose of the
Toeplitz matrix is in practice obtained by copying weights
from the considered convolutional layer to the counterpart
decoder’s layer that is implemented as a transposed convolutional layer (also known as a deconvolutional layer).

Invertible Autoencoder for domain adaptation

Therefore, as before, we enforce the counterpart decoder’s
layer to perform transformation T D : x = W > y and by
appropriate training ensure (T E )−1 ≡ T D .

We will parametrize the counterpart residual block of the
decoder with a transpose of matrix W2 · W1 + I as given in
Figure 3b. Therefore we enforce the counterpart decoder’s
residual block to perform transformation:

3.3. Activation function

T D : x = (W1> W2> + I)y.

Invertible activation function should be a bijection. In this
paper we consider a modified LeakyReLU activation function σ and use only this non-linearity in the model. Consider transformation T E of an input signal performed by
this non-linearity applied in the encoder E. This nonlinearity is defined as
(
1
x, if x ≥ 0
E
T : y = σ(x) = α
(5)
αx, otherwise.
An inverse operation is then defined as
(
αy, if x ≥ 0
E −1
−1
(T ) : x = σ (y) = 1
α y, otherwise.

(6)

3.4. Residual block

(b)

Figure 3. (a) Residual block. (b) Inverted residual block.

Consider transformation T E of an input signal performed
by a residual block (He et al., 2016) of an encoder E.
We modify the residual block to remove the internal nonlinearity as given in Figure 3a. The residual block is
parametrized with weight matrices W1 and W2 . Those are
Toeplitz matrices corresponding to the convolutional and
transposed convolutional layers of the residual block. Let
x denote block’s vectorized input and y denote its corresponding output. Thus transformation T E is defined as
T E : y = σ((W2 · W1 + I) · x)

(7)

An inverse operation is then defined as
(T E )−1 : x = (W2 · W1 + I)−1 σ −1 (y).

Similarly as before, at training will enforce orthonormality
(W2 ·W1 +I)−1 = (W2 ·W1 +I)> and thus (T E )−1 ≡ T D .
3.5. Bias
We consider bias as a separate layer in the network. Then,
handling biases is straightforward. In particular, the layer
in the encoder that perform bias addition has its counterpart
layer in the decoder, where the same bias is subtracted.
3.6. Experimental validation of orthonormality

The corresponding non-linearity in the decoder will
therefore realize the operation of an inverted modified
LeakyReLU given in Equation 6. In the experiments we
set α = 2.

(a)

(9)

(8)

In this section, we validate the concept of InvAuto. The
goal of this section is to show that proposed shared
parametrization and training enforce orthonormality and
that at the same time the orthonormality property is not
organically achieved by standard architectures. We compare InvAuto with previously mentioned vanilla autoencoder, autoencoder with cycle consistency, and variational
autoencoder. We experimented with various data sets
(MNIST and CIFAR-10) and architectures (MLP, convolutional (Conv), and ResNet). All the networks were designed to have 2 down-sampling layers and 2 up-sampling
layers. Encoder’s matrix E and decoder’s matrix D are
constructed by multiplying the weight matrices of consecutive layers of encoder and decoder, respectively.
Data set
and model
MNIST
MLP
MNIST
Conv
CIFAR
Conv
CIFAR
ResNet

InvAuto

Auto

Cycle

VAE

0.001
±0.118
0.001
±0.148
0.001
±0.145
0.000
±0.134

0.008
±0.210
0.001
±0.179
0.002
±0.176
0.000
±0.203

0.007
±0.207
0.001
±0.176
0.004
±0.195
0.000
±0.232

0.001
±0.219
−0.001
±0.190
0.003
±0.268
0.001
±0.298

Table 1. Mean and standard deviation of cosine similarity of rows
of E. InvAuto achieves cosine similarity that is the closest to 0.

We test orthonormality by reporting the histograms of the
cosine similarity of each pair of rows of matrix E for all
methods (Figure 4) along with their mean and standard deviation (Table 1) as we expect the cosine similarity to be
close to 0 for InvAuto. We then show the `2 -norm of the
rows of E as we expect the rows of InvAuto to have closeto-unit norm (Table 2). InvAuto enforces the encoder, and
consequently the decoder, to be orthonormal. Other methods do not explicitly demand that and thus the orthonormality of their encoders is weaker. This observation is further

Invertible Autoencoder for domain adaptation

InvAuto

Auto

Cycle

VAE

(a) MNIST MLP

(b) MNIST MLP

(c) MNIST MLP

(d) MNIST MLP

(e) CIFAR ResNet

(f) CIFAR ResNet

(g) CIFAR ResNet

(h) CIFAR ResNet

Figure 4. The histograms of cosine similarity of the rows of E for InvAuto (a and e), Auto (b and f), Cycle (c and g), and VAE (d and p)
on MLP and ResNet architectures and MNIST and CIFAR data sets.

Data set
and model
MNIST
MLP
MNIST
Conv
CIFAR
Conv
CIFAR
ResNet

InvAuto

Auto

Cycle

VAE

0.976
±0.190
0.905
±0.321
0.908
±0.219
0.868
±0.078

1.326
±0.095
1.699
±0.732
3.027
±0.816
2.890
±0.895

1.268
±0.095
1.780
±0.779
2.463
±0.688
2.650
±0.937

1.832
±0.501
1.971
±0.794
1.176
±0.356
1.728
±0.311

Table 2. Mean and standard deviation of the `2 -norm of the rows
of E. InvAuto achieves the `2 -norm of the rows that is the closest
to the unit norm.

confirmed by Figures 1 and 2 shown before in the Introduction. In the Supplement (Section A), we provide three
more figures that complement Figure 2 (recall that the latter reports the MSE of DE − I). They show the MSE of
the diagonal (Figure 14) and off-diagonal of DE − I (Figure 15) as well as the ratio of the MSE of the off-diagonal
and diagonal of DE (Figure 16) for various methods. The
reconstruction loss obtained for all methods is also shown
in Section A in the Supplement (Table 5).
Next we describe how InvAuto is applied to the problem of
domain adaptation.

sponding to domain B. The decoder D, on the other hand,
performs the conversion from the features corresponding
to domain B into the features corresponding to domain A.
Since E and D form InvAuto, E realizes an inversion of
D (and vice versa) and shares parameters with D. This
introduces strong correlations between two generators and
reduces the number of trainable parameters, which distinguishes our approach from CycleGAN. The proposed architecture is illustrated in Figure 5. The details of the architecture and training are provided in Section C in the Supplement.
Next we describe the cost function that we use to train our
deep model. The first component of the cost function is the
adversarial loss (Goodfellow et al., 2014), i.e.
Ladv (GenA , DisA ) =ExA ∼pd (A) [log DisA (xA )]+
ExB ∼pd (B) [log(1 − DisA (GenA (xB )))]
Ladv (GenB , DisB ) =ExB ∼pd (B) [log DisB (xB )]+
ExA ∼pd (A) [log(1 − DisB (GenB (xA )))],
(10)
where pd (A) and pd (B) denote the distribution of data
from A and B, respectively.
The second component of the loss function is the cycle consistency loss defined as
Lcc (GenA , GenB ) = ExA ∼pd (A) [kxA − GenA (GenB (xA ))k1 ]

4. Invertible autoencoder for domain
adaptation
For the purpose of performing domain adaptation we construct the dedicated architecture that is similar to CycleGAN, but we use InvAuto at the feature level of the generators. This InvAuto contains encoder E and decoder D
that themselves have the form of autoencoders. Each of
these internal autoencoders is used to do the conversion between the features corresponding to two different domains.
And thus, the encoder E performs the conversion from the
features corresponding to domain A into the features corre-

+ ExB ∼pd (B) [kxB − GenB (GenA (xB ))k1 ].
(11)
The objective function that we minimize therefore becomes
L(GenA , GenB , DisA , DisB ) = λLcc (GenA , GenB )
+ Ladv (GenA , DisA ) (12)
+ Ladv (GenB , DisB ),
where λ controls the balance between the adversarial loss
and cycle consistency loss. The cycle consistency loss enforces the orthonormality property of InvAuto.

Invertible Autoencoder for domain adaptation

InvAuto
x

Enc

E

Enc

y

Dis

T/F

Dec

y

Dis

T/F

Feat. y

Feat. x
x

Dec

D
Feat. x

Feat. y

Figure 5. The architecture of the domain translator with InvAuto (E, D). xA ∈ A and xB ∈ B are the inputs of the translator. yB is
a converted image xA into the B domain and yA is a converted image xB into the A domain. Invertible autoencoder (E, D) is built
of encoder E and decoder D, where each of those itself is an autoencoder. EncA , EncB are feature extractors, and DecA , DecB are the
final layers of the generators GenB , i.e. (EncA , E, DecB ), and GenA , i.e. (EncB , D, DecA ), respectively. Discriminators DisA and DisB
discriminate whether their input comes from the generator (True) or original data set (False).

5. Experiments

Original

CycleGAN

UNIT

InvAuto

We next demonstrate the experiments on domain adaptation problems. We compare our model against UNIT(Liu
et al., 2017) and CycleGAN(Zhu et al., 2017). We used
publicly available implementations of both methods available from https://github.com/mingyuliutw/
UNIT/ and https://github.com/junyanz/
pytorch-CycleGAN-and-pix2pix/.The details of
our architecture and the training process are summarized
in Section C in the Supplement.
5.1. Experiments with benchmark data sets
We considered the following domain adaptation tasks:
(i) Day-to-night and night-to-day image conversion: we
used unpaired road pictures recorded during the day
and at night obtained from KAIST data set (Hwang
et al., 2015).
(ii) Day-to-thermal and thermal-to-day image conversion:
we used road pictures recorded during the day with a
regular camera and a thermal camera obtained from
KAIST data set(Hwang et al., 2015).

Figure 6. Day-to-night image conversion. Zoomed image is
shown in Figure 18 in Section B of the Supplement.

Original

CycleGAN

UNIT

InvAuto

(iii) Maps-to-satellite and satellite-to-maps: we used
satellite images and maps obtained from Google
Maps (Isola et al., 2017).
The data sets for the last two tasks, i.e. (ii) and (iii), are
originally paired, however we randomly permuted them
and train the model in an unsupervised fashion. The
training and testing images were furthermore resized to
128 × 128 resolution.
The visual results of image conversion are presented in Figures 6-11 (Section B in the Supplement contains the same
figures in higher resolution). We see that InvAuto visually
performs comparably to other state-of-the-art methods.

Figure 7. Night-to-day image conversion. Zoomed image is
shown in Figure 19 in Section B of the Supplement.

Invertible Autoencoder for domain adaptation

Original CycleGAN UNIT

InvAuto

Reference

Figure 8. Day-to-thermal image conversion. Zoomed image is
shown in Figure 20 in Section B of the Supplement.

Original CycleGAN UNIT

InvAuto

Reference

Figure 9. Thermal-to-day image conversion. Zoomed image is
shown in Figure 21 in Section B of the Supplement.

To evaluate the performance of the methods numerically
we use the following approach:
• For the tasks (ii) and (iii), we directly calculated the
`1 loss between the converted images and the ground
truth.
• For the task (i), we trained two autoencoders ΩA and
ΩB on both domains, i.e. we trained each of them to
reconstruct well the images from its own domain and
reconstruct badly the images from the other domain.
Then we use these two autoencoders to evaluate the
quality of the converted images, where high `1 reconstruction loss of the autoencoder for the images converted to resemble those from its corresponding domain implies low-quality image translation.
Table 3 contains the results of the numerical evaluation and
shows that the performance of InvAuto is similar to the
state-of-the-art techniques that we compare InvAuto with
and is furthermore contained within the performance range
established by the CycleGAN (best performer) and UNIT
(consistently slightly worst from CycleGAN).

Original CycleGAN UNIT

InvAuto

Reference

Figure 10. Maps-to-satellite image conversion. Zoomed image is
shown in Figure 22 in Section B of the Supplement.

Original CycleGAN UNIT

InvAuto

Reference

Figure 11. Satellite-to-maps image conversion. Zoomed image is
shown in Figure 23 in Section B of the Supplement.

Tasks
Night-to-day
Day-to-nigth
Thermal-to-day
Day-to-thermal
Maps-to-satellite
Satellite-to-maps

CycleGAN
0.033
0.041
0.287
0.179
0.261
0.069

Methods
UNIT
0.227
0.114
0.339
0.194
0.331
0.104

InvAuto
0.062
0.067
0.299
0.205
0.272
0.080

Table 3. Numerical evaluation of CycleGAN, UNIT, and InvAuto.

5.2. Experiments with autonomous driving system
To test the quality of the image-to-image translations obtained by InvAuto, we use the NVIDIA evaluation system
for autonomous driving described in details in (Bojarski
et al., 2016). The system evaluates the performance of an
already trained NVIDIA neural-network-based end-to-end
learning platform for autonomous driving (PilotNet) on a
test video using a simulator for autonomous driving. The
system uses the following performance metrics for evaluation: autonomy, position precision, and comfort. We do
not describe these metrics as they are described well in the
mentioned paper. We only emphasize that these metrics are

Invertible Autoencoder for domain adaptation

expressed as a percentage, where 100% corresponds to the
best performance. We collected the high-resolution videos
of the same road during the day and night from the camera
inside the car. Each video had ∼ 45K frames. The pictures
were resized to 512 × 512 resolution for the conversion and
then resized back to the original size of 1920 × 1208. We
used our domain translator as well as CycleGAN to convert
the collected day video to a night video (Figure 12) and
also the collected night video to a day video (Figure 13).
To evaluate our model, we used aforementioned NVIDIA
evaluation system, where the converted videos where used
as testing sets for this system. We report results in Table 4.

Video type

Autonomy

Comfort

99.6%
98.6%
99.0%

Position
precision
73.3%
63.1%
69.6%

Original day
Original night
Day-to-night
InvAuto
Night-to-day
InvAuto
Day-to-night
CycleGAN
Night-to-day
CycleGAN

99.3%

68.0%

84.7%

Original

CycleGAN

InvAuto

89.7%
86.3%
83.2%

99.0%

68.4%

84.7%

98.8%

64.0%

87.3%

Figure 12. Experimental results with autonomous driving system:
day-to-night conversion. Zoomed image is shown in Figure 24 in
Section B of the Supplement.

Original

CycleGAN

InvAuto

Table 4. Experimental results with autonomous driving system:
autonomy, position precision, and comfort.

The PilotNet model used for testing was trained mostly on
day videos, thus it is expected to perform worse on night
videos. Therefore the performance for original night video
is worse than for the same video converted to a day video in
terms of autonomy and position precision. The comfort deteriorates due to the inconsistency of consecutive frames in
the converted video, i.e. the videos are converted frame-byframe and we do not apply any post-processing to ensure
smooth transition between frames. The results for InvAuto
and CycleGAN are comparable.

6. Conclusion
We proposed a novel architecture that we call invertible autoencoder, which, as opposed to the common deep learning architectures, allows the layers of the model performing opposite operations (like encoder and decoder) to share
weights. This is achieved by enforcing orthonormal mappings in the layers of the model. We demonstrate the applicability of the proposed architecture to the problem of
domain adaptation and evaluate it on benchmark data sets
and autonomous driving task. The performance of the proposed approach matches state-of-the-art methods and requires less trainable parameters.

Figure 13. Experimental results with autonomous driving system:
night-to-day conversion. Zoomed image is shown in Figure 25 in
Section B of the Supplement.

References
Bojarski, M., Del Testa, D., Dworakowski, D., Firner,
B., Flepp, B., Goyal, P., Jackel, L. D., Monfort, M.,

Invertible Autoencoder for domain adaptation

Muller, U., Zhang, J., Zhang, X., Zhao, J., and Zieba,
K. End to end learning for self-driving cars. CoRR,
abs/1604.07316, 2016.

Vincent, P., Larochelle, H., Bengio, Y., and Manzagol, P.A. Extracting and composing robust features with denoising autoencoders. In ICML, 2008.

Dong, H., Neekhara, P., Wu, C., and Guo, Y. Unsupervised image-to-image translation with generative adversarial networks. CoRR, abs/1701.02676, 2017.

Wang, T.-C., Liu, M.-Y., Zhu, J.-Y., Tao, A., Kautz, J.,
and Catanzaro, B. High-resolution image synthesis and
semantic manipulation with conditional gans. CoRR,
abs/1711.11585, 2017.

Gatys, L. A., Bethge, M., Hertzmann, A., and Shechtman, E. Preserving color in neural artistic style transfer.
CoRR, abs/1606.05897, 2016a.
Gatys, L. A., Ecker, A. S., and Bethge, M. Image style
transfer using convolutional neural networks. In CVPR,
2016b.
Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B.,
Warde-Farley, D., Ozair, S., Courville, A. C., and Bengio, Y. Generative adversarial nets. In NIPS, 2014.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In CVPR, 2016.
Hwang, S., Park, J., Kim, N., Choi, Y., and Kweon, I. S.
Multispectral pedestrian detection: Benchmark dataset
and baseline. In CVPR, 2015.
Isola, P., Zhu, J.-Y., Zhou, T., and Efros, A. A. Image-toimage translation with conditional adversarial networks.
In CVPR, 2017.
Johnson, J., Alahi, A., and Fei-Fei, Li. Perceptual losses for
real-time style transfer and super-resolution. In ECCV,
2016.
Kingma, D. P. and Ba, J. Adam: A method for stochastic
optimization. CoRR, abs/1412.6980, 2014.
Kingma, D. P. and Welling, M. Auto-encoding variational
bayes. In ICLR, 2014.
Liu, M.-Y. and Tuzel, O. Coupled generative adversarial
networks. In NIPS. 2016.
Liu, M.-Y., Breuel, T., and Kautz, J. Unsupervised imageto-image translation networks. In NIPS, 2017.
McCann, M. T., Jin, K. H., and Unser, M. Convolutional
neural networks for inverse problems in imaging: A review. IEEE Signal Process. Mag., 34(6):85–95, 2017.
Taigman, Y., Polyak, A., and Wolf, L. Unsupervised crossdomain image generation. CoRR, abs/1611.02200, 2016.
Ulyanov, D., Lebedev, V., Vedaldi, A., and Lempitsky, V. S.
Texture networks: Feed-forward synthesis of textures
and stylized images. In ICML, 2016.
Vasudevan, A., Anderson, A., and Gregg, D. Parallel multi
channel convolution using general matrix multiplication.
In ASAP, 2017.

Zhu, J.-Y., Park, T., Isola, P., and Efros, A. A. Unpaired
image-to-image translation using cycle-consistent adversarial networks. In ICCV, 2017.

Invertible Autoencoder for domain adaptation

Invertible Autoencoder for domain
adaptation
(Supplementary material)
A. Additional plots and tables for Section 3.6

Data set
and model
MNIST
MLP
MNIST
Conv
CIFAR
Conv
CIFAR
ResNet

InvAuto

Auto

Cycle

VAE

0.189

0.100

0.112

1.245

0.168

0.051

0.057

1.412

0.236

0.126

0.195

1.457

0.032

0.127

0.217

0.964

Table 5. Test reconstruction loss (MSE) for InvAuto, Auto, Cycle,
and VAE on MLP, convolutional (Conv), and ResNet architectures
and MNIST and CIFAR data sets. VAE has significantly higher
reconstruction loss by construction.

Figure 14. Comparison of the MSE of the diagonal of DE − I for
InvAuto, Auto, Cycle, and VAE on MLP, convolutional (Conv),
and ResNet architectures and MNIST and CIFAR data sets.

Figure 15. Comparison of the MSE of the off-diagonal of DE −
I for InvAuto, Auto, Cycle, and VAE on MLP, convolutional
(Conv), and ResNet architectures and MNIST and CIFAR data
set.

Figure 16. Comparison of the ratio of MSE of the off-diagonal
and diagonal of DE for InvAuto, Auto, Cycle, and VAE on MLP,
convolutional (Conv), and ResNet architectures and MNIST and
CIFAR data sets.

Invertible Autoencoder for domain adaptation

InvAuto

Auto

Cycle

VAE

(a) MNIST MLP

(b) MNIST MLP

(c) MNIST MLP

(d) MNIST MLP

(e) MNIST Conv

(f) MNIST Conv

(g) MNIST Conv

(h) MNIST Conv

(i) CIFAR Conv

(j) CIFAR Conv

(k) CIFAR Conv

(l) CIFAR Conv

(m) CIFAR ResNet

(n) CIFAR ResNet

(o) CIFAR ResNet

(p) CIFAR ResNet

Figure 17. Heatmap of the values of matrix DE for InvAuto, (a,e,i,m) Auto (b,f,j,n), Cycle (c,g,k,o), and VAE (d,h,l,p) on MLP, convolutional (Conv), and ResNet architectures and MNIST and CIFAR data sets. Matrices E and D are constructed by multiplying the
weight matrices of consecutive layers of encoder and decoder, respectively. In case of InvAuto, DE is the closest to the identity matrix.

Invertible Autoencoder for domain adaptation

B. Additional experimental results for Section 5
Original

CycleGAN

UNIT

Figure 18. Day-to-night image conversion.

InvAuto

Invertible Autoencoder for domain adaptation

Original

CycleGAN

UNIT

Figure 19. Night-to-day image conversion.

InvAuto

Invertible Autoencoder for domain adaptation

Original

CycleGAN

UNIT

InvAuto

Figure 20. Day-to-thermal image conversion.

Reference

Invertible Autoencoder for domain adaptation

Original

CycleGAN

UNIT

InvAuto

Figure 21. Thermal-to-day image conversion.

Reference

Invertible Autoencoder for domain adaptation

Original

CycleGAN

UNIT

InvAuto

Figure 22. Maps-to-satellite image conversion.

Reference

Invertible Autoencoder for domain adaptation

Original

CycleGAN

UNIT

InvAuto

Figure 23. Satellite-to-maps image conversion.

Reference

Invertible Autoencoder for domain adaptation

Original

CycleGAN

InvAuto

Figure 24. Experimental results with autonomous driving system: day-to-night conversion.

Invertible Autoencoder for domain adaptation

Original

CycleGAN

InvAuto

Figure 25. Experimental results with autonomous driving system: night-to-day conversion.

Invertible Autoencoder for domain adaptation

C. Invertible autoencoder for domain
adaptation: architecture and training
Generator architecture Our implementation of InvAuto
contains 18 invertible residual blocks for both 128 × 128
and 512 × 512 images, where 9 blocks are used in the encoder and the remaining in the decoder. All layers in the
decoder are the inverted versions of encoder’s layers. We
furthermore add two down-sampling layers and two upsampling layers for the model trained on 128×128 images,
and three down-sampling layers and three up-sampling layers for the model trained on 512 × 512 images. The details of the generator’s architecture are listed in Table 7 and
Table 8. For convenience, we use Conv to denote convolutional layer, ConvNormReLU to denote ConvolutionalInstanceNorm-LeakyReLU layer, InvRes to denote invertible residual block, and Tanh to denote hyperbolic tangent
activation function. The negative slope of LeakyReLU
function is set to 0.2. All filters are square and we have
the following notations: K represents filter size and F represents the number of output feature maps. The paddings
are added correspondingly.
Discriminator architecture We use similar discriminator
architecture as PatchGAN (Isola et al., 2017). It is described in Table 6. We use this architecture for training
both on 128 × 128 and 512 × 512 images.
Criterion and Optimization At training, we set λ = 10
and use l1 loss for the cycle consistency in Equation 12. We
use Adam optimizer(Kingma & Ba, 2014) with learning
rate lr = 0.0002, β1 = 0.5 and β2 = 0.999. We also add l2
penalty with weight 10−6 .

Name

Stride

Filter

ConvNormReLU
ConvNormReLU
ConvNormReLU
ConvNormReLU
Conv

2×2
2×2
2×2
1×1
1×1

K4-F64
K4-F128
K4-F256
K4-F512
K4-F1

Table 6. Discriminator for both 128 × 128 and 512 × 512 images.

Name

Stride

Filter

ConvNormReLU
ConvNormReLU
ConvNormReLU
InvRes
InvRes
InvRes
InvRes
InvRes
InvRes
InvRes
InvRes
InvRes
ConvNormReLU
ConvNormReLU
Conv
Tanh

1×1
2×2
2×2
1×1
1×1
1×1
1×1
1×1
1×1
1×1
1×1
1×1
1/2 × 1/2
1/2 × 1/2
1×1

K7-F64
K3-F128
K3-F256
K3-F256
K3-F256
K3-F256
K3-F256
K3-F256
K3-F256
K3-F256
K3-F256
K3-F256
K3-F128
K3-F64
K7-F3

Table 7. Generator for 128 × 128 images.

Name

Stride

Filter

ConvNormReLU
ConvNormReLU
ConvNormReLU
ConvNormReLU
InvRes
InvRes
InvRes
InvRes
InvRes
InvRes
InvRes
InvRes
InvRes
ConvNormReLU
ConvNormReLU
ConvNormReLU
Conv
Tanh

1×1
2×2
2×2
2×2
1×1
1×1
1×1
1×1
1×1
1×1
1×1
1×1
1×1
1/2 × 1/2
1/2 × 1/2
1/2 × 1/2
1×1

K7-F64
K3-F128
K3-F256
K3-F512
K3-F512
K3-F512
K3-F512
K3-F512
K3-F512
K3-F512
K3-F512
K3-F512
K3-F512
K3-F256
K3-F128
K3-F64
K7-F3

Table 8. Generator for 512 × 512 images.

