Conditional Prior Networks for Optical Flow
Yanchao Yang Stefano Soatto


CPNFlow

Y. Yang and S. Soatto
UCLA Vision Lab
 University of California, Los Angeles, CA 90095

{yanchao.yang, soatto}@cs.ucla.edu
Conditional Prior Networks for Optical Flow
    Yanchao Yang    Stefano Soatto
    
===========================================




Classical computation of optical flow involves generic priors (regularizers) that capture rudimentary statistics of images, but not long-range correlations or semantics. On the other hand, fully supervised methods learn the regularity in the annotated data, without explicit regularization and with the risk of overfitting. We seek to learn richer priors on the set of possible flows that are statistically compatible with an image. Once the prior is learned in a supervised fashion, one can easily learn the full map to infer optical flow directly from two or more images, without any need for (additional) supervision. We introduce a novel architecture, called Conditional Prior Network (CPN), and show how to train it to yield a conditional prior. When used in conjunction with a simple optical flow architecture, the CPN beats all variational methods and all unsupervised learning-based ones using the same data term. It performs comparably to fully supervised ones, that however are fine-tuned to a particular dataset. Our method, on the other hand, performs well even when transferred between datasets. Code is available at: <https://github.com/YanchaoYang/Conditional-Prior-Networks>






§ INTRODUCTION


Consider Fig. <ref>: A given image (left) could give rise to many different optical flows (OF) <cit.> depending on what another image of the same scene looks like: It could show a car moving to the right (top), or the same apparently moving to the left due to camera motion to the right (middle), or it could be an artificial motion because the scene was a picture portraying the car, rather than the actual physical scene. A single image biases, but does not constrain, the set of possible flows the underlying scene can generate. We wish to leverage the information an image contains about possible compatible flows to learn better priors than those implied by generic regularizers. Note that all three flows in Fig. <ref> are equally valid under a generic prior (piecewise smoothness), but not under a natural prior (cars moving in the scene).

A regularizer is a criterion that, when added to a data fitting term, constrains the solution of an inverse problem. These two criteria (data term and regularizer) are usually formalized as an energy function, which is minimized to, ideally, find a unique global optimum.[We use the terms regularizer, prior, model, or assumption, interchangeably and broadly to include any restriction on the solution space, or bias on the solution, imposed without full knowledge of the data. In OF, the full data is (at least) two images.]



 §.§ Our approach in context


In classical (variational) OF, the regularizer captures very rudimentary low-order statistics <cit.>, for instance the high kurtosis of the gradient distribution. This does not help with the scenario in Fig. <ref>. There has been a recent surge of (supervised) learning-based approaches to OF <cit.>, that do not have explicit regularization nor do they use geometric reprojection error as a criterion for data fit. Instead, a map is learned from pairs of images to flows, where regularization is implicit in the function class <cit.>,[In theory, deep neural networks are universal approximants, but there is a considerable amount of engineering in the architectures to capture suitable inductive biases.] in the training procedure <cit.> (e.g. noise of stochastic gradient descent – SGD), and in the datasets used for training (e.g. Sintel <cit.>, Flying Chair <cit.>).

Our method does not attempt to learn geometric optics anew, even though black-box approaches are the top performers in several benchmarks. Instead, we seek to learn richer priors on the set of possible flows that are statistically compatible with an image (Fig. <ref>).

Unsupervised learning-based approaches use the same or similar loss functions as variational methods <cit.>, including priors, but restrict the function class to a parametric model, for instance convolutional neural networks (CNNs) trained with SGD, thus adding implicit regularization <cit.>. Again, the priors only encode first-order statistics, which fail to capture the phenomena in Fig. <ref>.

We advocate learning a conditional prior, or regularizer, from data, but do so once and forall, and then use it in conjunction with any data fitting term, with any model and optimization one wishes.

What we learn is a prior in the sense that it imposes a bias on the possible solutions, but it does not alone constraint them, which happens only in conjunction with a data term. Once the prior is learned, in a supervised fashion, one can also learn the full map to infer optical flow directly from data, without any need for (additional) supervision. In this sense, our method is “semi-unsupervised”: Once we learn the prior, anyone can train an optical flow architecture entirely unsupervised. The key idea here is to learn a prior for the set of optical flows that are statistically compatible with a single image. Once done, we train a relatively simple network in an unsupervised fashion to map pairs of images to optical flows, where the loss function used for training includes explicit regularization in the form of the conditional prior, added to the reprojection error.

Despite a relatively simple architecture and low computational complexity, our method beats all variational ones and all unsupervised learning-based ones. It is on par or slightly below a few fully supervised ones, that however are fine-tuned to a particular dataset, and are extremely onerous to train. More importantly, available fully supervised methods perform best on the dataset on which they are trained. Our method, on the other hand, performs well even when the prior is trained on one dataset and used on a different one. For instance, a fully-supervised method trained on Flying Chair beats our method on Flying Chair, but underperforms it on KITTI and vice-versa (Tab. <ref>). Ours is consistently among the top in all datasets. More importantly, our method is complementary, and can be used in conjunction with more sophisticated networks and data terms.



 §.§ Formalization


Let I_1, I_2 ∈ℝ_+^H × W × 3 be two consecutive images and f : ℝ^2 →ℝ^2 the flow, implicitly defined in the co-visible region by I_1 = I_2 ∘ f + n  where n ∼ P_n is some distribution. The posterior P(f|I_1, I_2) ∝ P_n(I_1 - I_2 ∘ f) can be decomposed as 

    log P(f|I_1, I_2) = log P(I_2|I_1, f) + log P(f|I_1) - log P(I_2|I_1)
    ≈log P(I_2|I_1, f) + log P(f|I_1)

We call the first term (data) prediction error, and the second conditional prior. It is a prior in the sense that, given I_1 alone, many flows can have high likelihood for a suitable I_2. However, it is informed by I_1 in the sense of capturing image-dependent regularities such as flow discontinuities often occurring at object boundaries, which may or may not correspond to generic image discontinuities. A special case of this model assumes a Gaussian likelihood (ℓ^2 prediction error) and an ad-hoc prior of the form 

    E(f, I_1, I_2)=∫ ( I_1(x) - I_2(x+f(x)) )^2dx + ∫α(x, I_1) ‖∇ f(x) ‖^2 dx

where α is a scalar function that incorporates our belief in an irradiance boundary of I_1 corresponding to an object boundary.[When α is constant, we get an even more special case, the original Horn & Schunk model where the prior is also Gaussian and unconditional (independent of I_1).] This type of conditional prior has several limitations: First, in the absence of semantic context, it is not possible to differentiate occluding boundaries (where f can be discontinuous) from material boundaries (irradiance discontinuities), or illumination boundaries (cast shadows) where f is smooth. Second, the image I_1 only informs the flow locally, through its gradient, and does not capture global regularities. Fig. <ref> shows that flow fails to propagate into homogeneous region. This can be mitigated by using a fully connected CRF <cit.> but at a heavy computational cost.


Our goal can be formalized as learning the conditional prior P(f|I_1) in a manner that exploits the semantic context of the scene[The word “semantic” is often used to refer to identities and relations among discrete entities (objects). What matters in our case is the geometric and topological relations that may result in occluding boundaries on the image plane. The name of an object does not matter to that end, so we ignore identities and do not require object labels.] and captures the global statistics of I_1. We will do so by leveraging the power of deep convolutional neural networks trained end-to-end, to enable which we need to design differentiable models, which we do next.



§ METHOD


To learn a conditional prior we need to specify the inference criterion (loss function), which we do in Sect. <ref> and the class of functions (architecture), with respect to which the loss is minimized end-to-end. We introduce our choice of architecture next, and the optimization in Sect. <ref>.



 §.§ Conditional Prior Network (CPN)


We construct the conditional prior from a modified autoencoder trained to reconstruct a flow f that is compatible with the given (single) image I. We call this a Conditional Prior Network (CPN) shown in Fig. <ref>.



In a CPN, ψ encodes only the flow f, then φ takes the image I and the output of ψ to generate a reconstruction of f,  f̂ = φ( I, ψ(f)). Both ψ and φ are realized by pure convolutional layers with subsampling (striding) by two to create a bottleneck. Note that φ is a U-shape net <cit.> with skip connections, at whose center a concatenation with ψ(f) is applied. In the appendix, we articulate the reasons for our choice of architecture, and argue that it is better than an ordinary autoencoder that encodes both f and I in one branch. This is connected to the choice of loss function and how it is trained, which we discuss next. 



 §.§ Loss function



We are given a dataset D sampled from the joint distribution D={ (f_j, I_j) }_j=1^n ∼ P(f, I), with n samples. We propose approximating P(f|I) with a CPN as follows 

    Q_w_φ,w_ψ(f|I) = exp( - φ(I, ψ(f)) - f ^2 ) ∝ P(f|I)

where w_φ,w_ψ are the parameters of φ and ψ respectively. Given I, for every flow f, the above returns a positive value whose log, after training, is equal to the negative squared autoencoding loss. To determine the parameters that yield an approximation of P(f|I), we should solve the following optimization problem

    w_φ^*, w_ψ^* = min_w_φ, w_ψ𝔼_I∼ P(I)𝕂𝕃( P(f|I)  Q_w_φ, w_ψ(f|I) )

where the expectation is with respect to all possible images I, and 𝕂𝕃 is the Kullback-Leibler divergence between P(f|I) and the CPN Q_w_φ, w_ψ(f|I). In the appendix we show that the above is equivalent to:

    w_φ^*, w_ψ^* = max_w_φ, w_ψ∫_I ∫_f P(f, I) log [ Q_w_φ, w_ψ(f|I) ]df dI 
    
    = min_w_φ, w_ψ∫_I ∫_f P(f, I) φ_w_φ(I, ψ_w_ψ(f)) - f ^2 df dI

which is equivalent to minimizing the empirical autoencoding loss since the ground truth flow is quantized, ∑_j=1^n f̂_j - f_j ^2. If the encoder had no bottleneck (sufficient information capacity), it could overfit by returning f̂ = φ_w_φ(I, ψ_w_ψ(f)) = f, rendering the conditional prior Q_w_φ,w_ψ(f|I) uninformative (constant). Consistent with recent developments in the theory of Deep Learning <cit.>, sketched in the appendix, we introduce an information regularizer (bottleneck) on the encoder ψ leading to the CPN training loss

    w_φ^*, w_ψ^* = min_w_φ, w_ψ𝔼_I∼ P(I)𝕂𝕃( P(f|I)  Q_w_φ, w_ψ(f|I) ) + βI(f,ψ_w_ψ(f))

where β > 0 modulates complexity (information capacity) and fidelity (data fit), and I(f,ψ_w_ψ(f)) is the mutual information between the flow f and its representation (code) ψ_w_ψ(f). When β is large, the encoder is lossy, thus preventing Q_w_φ,w_ψ(f|I) from being uninformative.[the decoder φ imposes no architectural bottleneck due to skip connections.]



 §.§ Training a CPN



While the first term in Eq. (<ref>) can simply be the empirical autoencoding loss, the second term can be realized in many ways, e.g., an ℓ^2 or ℓ^1 penalty on the parameters w_ψ. Here we directly increase the bottleneck β by decreasing the coding length ℓ_ψ of ψ. Hence the training procedure of the proposed CPN can be summarized as follows:


  
  * Initialize the coding length of the encoder ℓ_ψ with a large number (β=0).
  
  * Train the encoder-decoder ψ, φ jointly by minimizing e = 1n∑_j=1^n f̂_j - f_j ^2 until convergence. The error at convergence is denoted as e^*.
  
  * If e^* > λ, training done.[in our experiments, λ=0.5.] 

  Otherwise, decrease ℓ_ψ, (increase β), and goto step 2.

It would be time consuming to train for every single coding length ℓ_ψ. We only iteratively train for the integer powers, 2^k, k≤ 10.

Inference: suppose the optimal parameters obtained from the training procedure are w_ψ^*, w_φ^*, then for any given pair (f, I), we can use Q_w_φ^*,w_ψ^*(f|I) as the conditional prior up to a constant. In the next section we add a data discrepancy term to the (log) prior to obtain an energy functional for learning direct mapping from images to optical flows.



 §.§ Semi-unsupervised learning optical flow


Unlike a generative model such as a variational autoencoder <cit.>, where sampling is required in order to evaluate the probability of a given observation, here (f, I) is directly mapped to a scalar using Eq. (<ref>), thus differentiable w.r.t f, and suitable for training a new network to predict optical flow given images I_1,I_2, by minimizing the following compound loss:

    E(f|I_1,I_2) = ∫_Ω∖ Oρ( I_1(x) - I_2(x+f(x)) )dx - αlog [ Q_w_φ^*,w_ψ^*(f|I_1) ]
    
    = ∫_Ω∖ Oρ( I_1(x) - I_2(x+f(x)) )dx + αφ^*(I_1, ψ^*(f)) - f ^2

with α>0, Q_w_φ^*,w_ψ^* our learned conditional prior, and ρ(x) = (x^2+0.001^2)^η the generalized Charbonnier penalty function <cit.>. Note that the integration in the data term is on the co-visible area, i.e. the image domain Ω minus the occluded area O, which can be set to empty for simplicity or modeled using the forward-backward consistency as done in <cit.> with a penalty on O to prevent trivial solutions. In the following section, we describe our implementation and report results and comparisons on several benchmarks.



§ EXPERIMENTS




 §.§ Network details


CPN: we adapt the FlowNetS network structure proposed in <cit.> to be the decoder φ, and the contraction part of FlowNetS to be the encoder ψ in our CPN respectively. Both parts are shrunk versions of the original FlowNetS with a factor of 14; altogether our CPN has 2.8M parameters, which is an order of magnitude less than the 38M parameters in FlowNetS. As we mentioned before, the bottleneck in Eq. (<ref>) is controlled by the coding length ℓ_ψ of the encoder ψ, here we make the definition of ℓ_ψ explicit, which is the number of the convolutional kernels in the last layer of the encoder. In our experiments, ℓ_ψ=128 always satisfies the stopping criterion described in Sect. <ref>, which ends up with a reduction rate 0.015 in the dimension of the flow f.

CPNFlow: we term our flow prediction network CPNFlow. The network used on all benchmarks for comparison is the original FlowNetS with no modifications, letting us focus on the effects of different loss terms. The total number of parameters is 38M. FlowNetS is the most basic network structure for learning optical flow <cit.>, i.e., only convolutional layers with striding for dimension reduction, however, when trained with loss Eq. (<ref>) that contains the learned conditional prior (CPN), it achieves better performance than the more complex network structure FlowNetC <cit.>, or even stack of FlowNetS and FlowNetC. Please refer to Sect. <ref> for details and quantitative comparisons.



 §.§ Datasets for training


Flying Chairs is a synthesized dataset proposed in <cit.>, by superimposing images of chairs on background images from Flickr. Randomly sampled 2-D affine transformations are applied to both chairs and background images. Thus there are independently moving objects together with background motion. The whole dataset contains about 22k 512 × 384 image pairs with ground truth flows.

MPI-Sintel <cit.> is collected from an animation that made to be realistic. It contains scenes with natural illumination, objects moving fast, and articulated motion. Final and clean versions of the dataset are provided. The final version contains motion blur and fog effects. The training set contains only 1,041 pairs of images, much smaller compared to Flying Chairs.

KITTI 2012 <cit.> and 2015 <cit.> are the largest real-world datasets containing ground truth optical flows collected in a driving scenario. The ground truth flows are obtained from simultaneously recorded video and 3-D laser scans, together with some manual corrections. Even though the multi-view extended version contains roughly 15k image pairs, ground truth flows exist for only 394 pairs of image, which makes fully supervised training of optical flow prediction from scratch under this scenario infeasible. However, it provides a base for unsupervised learning of optical flow, and a stage to show the benefit of semi-unsupervised optical flow learning, that utilizes both the conditional prior (CPN) learned from the synthetic dataset, and the virtually unlimited amount of real world videos.



 §.§ Training details


We use Adam <cit.> as the optimizer with its default parameters in all our experiments. We train our conditional prior network (CPN) using Flying Chairs dataset due to its large amount of synthesized ground truth flows. The initial learning rate is 1.0e-4, and is halved every 100k steps until the maximum 600k training steps. The batch size is 8, and the autoencoding loss after training is around 0.6.

There are two versions of our CPNFlow, i.e. CPNFlow-C and CPNFlow-K. Both employ the FlowNetS structure, and they differ in the training set on which Eq. (<ref>) is minimized. CPNFlow-C is trained on Flying Chairs dataset, similarly CPNFlow-K is trained on KITTI dataset with the multi-view extension. The consideration here is: when trained on Flying Chairs dataset, the conditional prior network (CPN) is supposed to only capture the statistics of the affine transformations (a) CPNFlow-C is to test whether our learned prior works properly or not. If it works, (b) CPNFlow-K tests how the learned prior generalizes to real world scenarios. Both CPNFlow-C and CPNFlow-K have the same training schedule with the initial learning rate 1.0e-4, which is halved every 100k steps until the maximum 400k steps.[α=0.1, η=0.25 for CPNFlow-C, and α=0.045, η=0.38 for CPNFlow-K.] Note that in <cit.>, layer-wise loss adjustment is used during training to simulate coarse-to-fine estimation, however, we will not adopt this training technique to avoid repeatedly interrupting the training process. In a similar spirit, we will not do network stacking as in <cit.>, which increases both the training complexity and the network size.

In terms of data augmentation, we apply the same augmentation method as in <cit.> whenever our network is trained on Flying Chairs dataset with a cropping of 384x448. When trained on KITTI, resized to 384x512, only vertical flipping, horizontal flipping and image order switching are applied. The batch size used for training on Flying Chairs is 8 and on KITTI is 4.



 §.§ Benchmark results





Tab. <ref> summarizes our evaluation on all benchmarks mentioned above, together with quantitative comparisons to the state-of-the-art methods from different categories: Fully supervised, variational, and unsupervised learning methods. Since CPNFlow has the same network structure as FlowNetS, and both CPNFlow-C and FlowNetS are trained on Flying Chairs dataset, the comparison between CPNFlow-C and FlowNetS shows that even if CPNFlow-C is trained without knowing the correspondences between pairs of image and the ground truth flows, it can still achieve similar performance compared to the fully supervised ones on the synthetic dataset MPI-Sintel. When both are applied to KITTI, CPNFlow-C achieves 11.2% and 21.6% improvement over FlowNetS and FlowNetC respectively on KITTI 2012 Train, hence CPNFlow generalizes better to out of domain data.

One might notice that FlowNet2 <cit.> consistently achieves the highest score on MPI-Sintel and KITTI Train, however, it has a totally different network structure where several FlownetS <cit.> and FlowNetC <cit.> are stacked together, and it is trained in a sequential manner, and on additional datasets, e.g. FlyingThings3D <cit.> and a new dataset designed for small displacement <cit.>, thus not directly comparable to CPNFlow. However, when we simply apply the learned conditional prior to train our CPNFlow on KITTI using Eq. (<ref>), the final network CPNFlow-K surpasses FlowNet2 by 8% on KITTI 2012 Train, yet the training procedure of CPNFlow is much simpler, and there is no need to switch between datasets nor between different modules of the network.

Since the emergence of unsupervised training of optical flow <cit.>, there has not been a single method that beats the variational methods, as shown in Tab. <ref>, even if both variational methods and unsupervised learning methods are minimizing the same type of loss function. One reason might be that when we implement the variational methods, we could apply some “secret” operations as mentioned in <cit.>, e.g. median filtering, such that implicit regularization is triggered. Extra data term can also be added to bias the optimization, as in <cit.>, sparse matches are used as a data term to deal with large displacements. However, when combined with our learned conditional prior, even the simplest data term would help unsupervisedly train a network that outperforms the state-of-the-art variational optical flow methods. As shown in Tab. <ref> our CPNFlow consistently achieves similar or better performance than LDOF <cit.>, especially on KITTI 2012 Train, the improvement is at least 40%.

Compared to unsupervised optical flow learning, the advantage of our learned conditional prior becomes obvious. Although DenseNetF <cit.> and UnFlowC <cit.> employ more powerful network structures than FlowNetS, their EPEs on MPI-Sintel Test are still 1.5 higher than our CPNFlow. Note that in <cit.>, several versions of result are reported, e.g. UnFlowC: trained with brightness data term and second order smoothness term, UnFlowC-oc: census transform based data term together with occlusion modeling and bidirectional flow consistency penalty, and UnFlowCSS-oc: a stack of one FlowNetC and two FlowNetS's sequentially trained using the same loss as in UnFlowC-oc. Our CPNFlow-K outperforms UnFlowC by 47% on KITTI 2012 Train and 32% on KITTI 2015 Train. When occlusion reasoning is effective in Eq. (<ref>) as done in <cit.>, our CPNFlow-K-o outperforms UnFlowC-oc by 17.7% on KITTI 2012 Train, 11.1% on KITTI 2015 Train, and 12.9% on Sintel Train Final, even without a more robust census transform based data term and flow consistency penalty, which demonstrate the effectiveness of our learned conditional prior across different data terms. Note that our CPNFlow-K-o even outperforms UnFlowCSS-oc, which is far more complex in training and network architecture.







Fig. <ref>, Fig. <ref>, Fig. <ref> show the visual comparisons on MPI-Sintel, KITTI 2012 and KITTI 2015 respectively. Note that our CPNFlow is generally much smoother, and at the same time sharper at object boundaries, e.g. the girl in the 3rd, 4th rows and the dragon in the 5th row in Fig. <ref>. This demonstrates that our conditional prior network (CPN) is capable of learning high level (semantic) regularities imposed by object entities. In Fig. <ref>, we can also observe that discontinuities in the flow fields align well with object boundaries, for example the cars in all pairs. This, again, demonstrates that our learned conditional prior is able to generalize to different scenarios. The error of the estimated flows is also displayed in Fig. <ref>.



§ DISCUSSION AND RELATED WORK


Generic priors capturing rudimentary statistics to regularize optical flow have been used for decades, starting with Horn & Schunk's ℓ^2 norm of the gradient, to ℓ^1, Total Variation, etc. We seek to design or learn image-dependent priors that capture long-range correlation and semantics.

Image-dependent priors of the form Eq. (<ref>) include <cit.>, whereas most recent methods learn optical flow end-to-end, without explicitly isolating the likelihood and prior terms, for instance <cit.> are the top performing on MPI-Sintel. Some methods even cast optical flow as dense or semi-dense feature matching <cit.> in order to deal with large displacements, while the regularity is merely imposed by forward-backward matching consistency (see references therein for a detailed review of related literature).

It would be tempting to use a GAN <cit.> to learn the prior distribution of interest. A GAN can be thought of as a method to learn a map g such that its push-forward g_* maps two distributions, one known μ, and one we can sample from, p, so ĝ = min𝕂𝕃(g_* μ || p). It does so via an adversarial process such that a generative model G will capture the data distribution p_data. If we sample from the generative model G, we will have samples that are equivalently sampled from p_data, in order to evaluate p_data(x) of a sample x, we can not circumvent the sampling step, thus making the method unsuitable for our purpose where we want a differentiable scalar function.

Our work entails constructing an autoencoder of the flow, so it naturally relates to <cit.>. Similarly, evaluating the probability of a test example is intractable, even if we can approximately evaluate the lower bound of the probability of a data point, which again can not be computed in closed form due to the expectation over the noise.

Optical flow learning algorithms typically rely on synthesized datasets, due to the extreme difficulty in obtaining ground truth flows for realistic videos. Recently, unsupervised optical flow learning methods have flourished, making use of vast amount of unlabeled videos. Although unsupervised optical flow learning methods are able to learn from unlimited amount of data, when compared to variational methods, their performance usually falls behind, even when a similar loss is employed. A phenomenon observed is that almost all unsupervised optical flow learning methods use the Horn-Schunck type surrogate losses. And there is debate on which feature to use for the data term, e.g., the raw photometric value or the edge response, or on the prior/regularizer term, e.g. , penalizing the first order gradient of the flow or the second order, or on how to weight the prior term in a pixel-wise manner. Surrogate losses are getting more and more complex. Instead of focusing on the data term, we ask what should be the best form for the prior term. Our answer is that structural consistency between an image and the flow, as well as high-order statistics, such as semantic consistency, are important. We show that when combined with the raw photometric warping error, this kind of prior serves as a better regularizer than all the other hand-designed ones. We show its effectiveness on several contemporary optical flow benchmarks, also thanks to its ability to leverage existing limited supervised (synthetic) datasets and unlimited real world videos.



§ APPENDIX




 §.§ Equivalence between Eq. (4) and Eq. (5)


Using definition of Kullback-Leibler divergence we have:


    w_φ^*, w_ψ^* = min_w_φ, w_ψ𝔼_I∼ P(I) KL( P(f|I)  Q_w_φ, w_ψ(f|I) )
    
    = min_w_φ, w_ψ∫_I P(I) KL( P(f|I)  Q_w_φ, w_ψ(f|I) )
    
    = min_w_φ, w_ψ∫_I P(I) ∫_f ( P(f|I) logP(f|I)Q_w_φ, w_ψ(f|I) )df dI
    
    = min_w_φ, w_ψ∫_I ∫_f P(I)P(f|I)logP(f|I)df dI - ∫_I ∫_f P(I)P(f|I) log Q_w_φ, w_ψ(f|I) df dI
    
    = max_w_φ, w_ψ∫_I ∫_f P(f, I) log Q_w_φ, w_ψ(f|I) df dI
    
    = max_w_φ, w_ψ - ∫_I ∫_f P(f, I) φ_w_φ(I, ψ_w_ψ(f)) - f ^2 df dI 
    
    = min_w_φ, w_ψ∫_I ∫_f P(f, I) φ_w_φ(I, ψ_w_ψ(f)) - f ^2 df dI




 §.§ Reasoning behind the CPN structure


We are going to show our reasoning that leads us to the current CPN structure instead of the ordinary autoencoder which encodes both f and I in one branch as follows:


    Q_w_φ,w_ψ(f|I) = exp( - φ∘ψ(f, I) - f ^2 )


where φ is a decoder and ψ is an encoder of both f and I, parameterized by w_φ, w_ψ respectively. The optimal parameters w_φ^*, w_ψ^* are obtained by minimizing the average KL divergence between the proposed conditional Q and P(f|I):


    w_φ^*, w_ψ^* = min_w_φ, w_ψ E_I∼ P(I) KL( P(f|I)  Q_w_φ, w_ψ(f|I) )


similarly to former subsection, we can show that the above optimization problem is equivalent to:


    w_φ^*, w_ψ^* = max_w_φ, w_ψ∫_I ∫_f P(f, I) log [ Q_w_φ, w_ψ(f|I) ]df dI 
    
    = min_w_φ, w_ψ∫_I ∫_f P(f, I) φ_w_φ∘ψ_w_ψ(f, I) - f ^2 df dI


However, we are not done as ψ is an encoder with limited capacity, thus ψ is not one-to-one, which makes the following subset non-empty:


    I_ψ, f = { I | ψ_w_ψ(f, I) = ψ}.


We can rewrite the optimization problem Eq. (<ref>) as:


    w_φ^*, w_ψ^* = min_w_φ, w_ψ∫_f ∫_ψ∫_I ∈I_ψ, f P(f, I) φ_w_φ∘ψ_w_ψ(f, I) - f ^2 dI dψ df 
    
    = min_w_φ, w_ψ∫_f ∫_ψφ_w_φ(ψ) - f ^2 ( ∫_I ∈I_ψ, f P(f, I)dI ) dψ df 
    
    = min_w_φ, w_ψ∫_f ∫_ψφ_w_φ(ψ) - f ^2 P(f)P_w_ψ(ψ |f) dψ df


P_w_ψ(ψ | f) is a probability measure induced by the encoder ψ. Thus, the original optimization problem is essentially minimizing the following quantity:


    w_φ^*, w_ψ^* = KL( P(f)P_w_ψ(ψ | f)  Q_w_φ(ψ, f) )


During the optimization process, the encoder is trying to push P(f)P_w_ψ(ψ | f) towards Q_w_φ(ψ, f) and the decoder is pushing from the other side. After optimization:


    Q_w_φ,w_ψ(f|I) = exp( - φ∘ψ(f, I) - f ^2 ) ∝ P(f)P_w_ψ(ψ(f,I) | f)


which is not P(f|I) nor P(f,I)! In order to let Q_w_φ,w_ψ(f|I) approximate P(f|I), the condition P_w_ψ(ψ|f) = P(I|f) should be true. And this is satisfied when ψ imposes no compression on I. i.e. ψ: (f, I) → (ψ(f), I), which enforces Eq. (<ref>) to be a singleton, and now we have the proposed CPN structure.



 §.§ Relation to Information Bottleneck(IB)


In <cit.>, the IB Lagrangian is proposed to achieve the sufficiency and the minimality of the learned representation. In our CPN training loss (Eq. (6) in the main paper), the first term is equivalent to the autoencoding loss Eq. (<ref>), minimizing which is equivalent to maximizing the following conditional mutual information:


    I( f, ψ_w_ψ(f) | I )


which measures the conditional sufficiency, together with the second term that measures the minimality of the representation ψ_w_ψ(f), our CPN training loss can be viewed as  minimizing:


    E(w_φ, w_ψ; D) = - I( f, ψ_w_ψ(f) | I ) + 
    βI(f, ψ_w_ψ(f))


which is in line with the principles proposed in <cit.>.

splncs04

