title: Measuring the Biases and Effectiveness of Content-Style Disentanglement
authors: Xiao Liu Spyridon Thermos Gabriele Valvano Agisilaos Chartsias Alison O'Neil Sotirios A. Tsaftaris
text: 
  
  Lense-Thirring Precession of Misaligned Discs I
      [
      December 30, 2023
  ===============================================
  
  
  
  
  A recent spate of state-of-the-art semi- and un-supervised solutions disentangle and encode image “content” into a spatial tensor and image appearance or “style” into a vector, to achieve good performance in spatially equivariant tasks (e.g. image-to-image translation). To achieve this, they employ different model design, learning objective, and data biases.
  While considerable effort has been made to measure disentanglement in vector representations, and assess its impact on task performance, such analysis for (spatial) content - style disentanglement is lacking. In this paper, we conduct an empirical study to investigate the role of different biases in content-style disentanglement settings and unveil the relationship between the degree of disentanglement and task performance.
  In particular, we consider the setting where we: (i) identify key design choices and learning constraints for three popular content-style disentanglement models; (ii) relax or remove such constraints in an ablation fashion; and (iii) use two metrics to measure the degree of disentanglement and assess its effect on each task performance.
  Our experiments reveal that there is a “sweet spot” between disentanglement, task performance and - surprisingly – content interpretability, suggesting that blindly forcing for higher disentanglement can hurt model performance and content factors semanticness.
  Our findings, as well as the used task-independent metrics, can be used to guide the design and selection of new models for tasks where content-style representations are useful. Code is available at <https://github.com/vios-s/CSDisentanglement_Metrics_Library>.
  
  
  
  
  
  § INTRODUCTION
  
  
  
  
  Recent work in representation learning argues that to achieve explainable and compact representations, one should separate out, or disentangle, the underlying explanatory factors into different dimensions of the considered latent space <cit.>. 
  In other words, it is beneficial to obtain representations that can separate latent variables that capture sensitive and useful information for the task at hand from the less informative ones <cit.>. 
  Disentanglement has recently shown to improve task performance, model generalization, and representation interpretability <cit.>. 
  Unfortunately, disentangling without supervision is an ill-posed and impossible task <cit.> and, to obtain it, we must introduce restrictions and inductive priors <cit.>. These priors are different forms of “bias” imposed by model design (design bias), learning objectives (learning bias), and data (data bias).
  
  In this work, we set out to reveal such choices of bias  in state-of-the-art (SoTA) disentanglement methods. Our particular focus is on “content-style” disentanglement, which decomposes input images into spatial “content” and vector “style” representations. 
  In principle, content variables (C) should contain the semantic information required for spatially equivariant tasks (e.g. segmentation and pose estimation), whereas style variables (S) contain information on image appearance (e.g. color intensity and texture). 
  However, contrary to extensive research on quantifying the degree of disentanglement between vectors <cit.>, usually there is no analysis of C-S disentanglement. 
  In fact, to the best of our knowledge, there is no study identifying the training biases enforced in C-S disentanglement settings or exposing the true relationship between the degree of disentanglement and model performance.
  
  Herein, we attempt to bridge these gaps with our contributions:
  ∙
      
    * We identify and analyse the key biases in SoTA models that employ C-S disentanglement. We show how the biases affect disentanglement and task performance (utility) in three popular vision tasks: image translation, segmentation, and pose estimation.
      
    * To make a quantitative analysis possible, we propose two complementary metrics building on existing work, to evaluate C-S disentanglement (Fig. <ref>) in terms of amount of information encoded in each latent variable (informativeness) and (un)correlation between the encoded spatial tensor content and vector style (proxy for independence).
      
    * 
      We find that: a) lower C-S disentanglement benefits task performance if a specific style-related prior is not violated; and b) performance is highly correlated with latent variable informativeness. We also assess content semanticness (interpretability).
  
  
  
  
  
  § RELATED WORK
  
  
  Content-Style Disentanglement.
  Image-to-Image translation has extensively explored the decoupling of image style and content <cit.>. Content-style disentanglement was also used in other applications, such as semantic segmentation <cit.> and pose estimation <cit.>, where the content serves as a robust representation for downstream tasks. In general, most methods derive latent spaces capturing C or S information using auto-encoder variants.
  
  These models achieve C-S disentanglement through different biases, such as architectural choices (e.g. AdaIN <cit.>, content binarization <cit.>), learning objectives (e.g. Kullback-Leibler  divergence, latent regression loss, de-correlation losses in vector representations <cit.>), or supervisory signals (e.g. using content for segmentation <cit.>). However, the precise effect of each bias on disentanglement and model performance is not thoroughly explored.
  
  Evaluating Disentanglement.
  
  Recently, several methods have been proposed for assessing the degree of disentanglement in a vector latent variable.
  A classical approach is latent traversals: a visualization showing how traversing single latent dimensions generates variations in the image reconstruction. Latent traversals do not need ground truth information on the factors, and can be used in mixed tensor spaces <cit.> to offer qualitative evaluations. Alternatively, latent traversals can be combined with pre-trained networks to measure the perceptual distance between the produced embeddings <cit.>.
  
  There exist several ways in quantitatively evaluating representations learned by VAEs and GANs. Unfortunately, these methods rely only on vector representations, and some also peruse ground truth knowledge about the latent factors. 
  In particular, some methods try to associate known factors of variations (e.g. rotation) with specific latent dimensions <cit.> or manifold topology <cit.>.
  Others measure the ability to isolate one factor in a single vector latent variable <cit.>, measuring compactness or modularity <cit.>, linear separability <cit.>, consistency and restrictiveness <cit.>, and explicitness of the representation <cit.>. Lastly, there is work on measuring the factor informativeness in a vector latent variable w.r.t. the input, independence among factors, as well as interpretability <cit.>.
  
  The aforementioned metrics cannot be directly employed to C-S disentanglement settings, where the latent factors have different dimensionality (i.e. the style is a vector and the content a spatial multi-channel tensor). However, in this paper we attempt to transfer these concepts to the C-S disentanglement domain, incorporating both spatial (tensor) and vector representations[Note that the metrics used for our analysis are generic and can be readily applied to vector-based C-S disentanglement methods, such as <cit.>.] to expand our understanding of the relation between C-S disentanglement and: a) biases adopted by each model; b) task performance; c) representation interpretability.
  
  
  
  
  § MEASURING PROPERTIES OF DISENTANGLED CONTENT AND STYLE
  
  
  Given N image samples {I_i}_i=1^N, we assume two representations of content and style: {C_i}_i=1^N and {s_i}_i=1^N, respectively.
  Building on existing work in vector-based disentanglement <cit.>, we present two complementary metrics to evaluate two properties in the context of C-S disentanglement: (un)correlation, and informativeness. We provide evidence that the metrics offer complementary information in Appendix F. Then, we discuss two properties of the disentangled representations, namely their utility and interpretability.
  
  
  Distance Correlation (DC).
  
  Disentangled representations separate content and style into independent latent spaces <cit.>, satisfying p(C,s)=p(C)p(s). However directly measuring independence between spatial C and vector S with existing metrics is not feasible. Since independent representations must be uncorrelated <cit.>, we use the empirical Distance Correlation (DC) <cit.> to measure the correlation between tensors of arbitrary dimensionality. Note that DC is bounded in the [0,1] range, while differently from other correlation-independence metrics, such as the kernel target alignment <cit.> and the Hilbert-Schmidt independence criterion <cit.>, it has the advantage of not requiring any pre-defined kernels.
  
  
  For N samples, consider two N-row matrices T_1 and T_2. In general, T_1 and T_2 row dimension varies as they are formed by concatenating images I_i, content features C_i or style features s_i. For I_i and C_i we first concatenate the channels and then row-scan to form a vector; s_i is already a vector.
  DC is then defined as:
  
      DC(T_1,T_2) = dCov(T_1, T_2)/√(dCov(T_1, T_1)dCov(T_2, T_2)),
       with  
          dCov(X, Y) = √(∑_i=1^N∑_j=1^N A_i, jB_i,j/N^2).
  
  Here, dCov is the distance covariance between any two N-row matrices X and Y, while A and B are their respective distance matrices.
  In particular, each matrix element a_i,j of A is the Euclidean distance between two samples X^i-X^j, after subtracting the mean of row i and column j, as well as the matrix mean. B is similarly calculated  for Y.
  We estimate disentanglement between C and S using distance correlation, DC(C,s), with values closer to 0 indicating higher disentanglement. 
  C and S can be uncorrelated, e.g. DC(C,s)=0, either when they encode unrelated information or when one encodes all information and the other encodes noise. The latter indicates posterior collapse, thus full entanglement. To tackle this, DC(C,s) needs a complementary metric to measure the representations' informativeness.
  
  Information Over Bias (IOB). 
  
  
  To explicitly measure the amount of information encoded in C and S, we introduce the Information Over Bias (IOB) metric, aiming to detect posterior collapse when C and S are disentangled, but one (C or S) is not informative about the input.
  Given latent variables z ∈{C, s} produced from N images at inference, we measure the amount of information encoded in each representation. To do so, we train a decoder G_θ_n, a neural network with parameters θ_n, to reconstruct images I, given the features z.
  
  
  Thus, we define IOB as the expectation over the test images of the ratio:
  
  
      IOB(I, z) = 𝔼_i[ MSE(I_i, G_θ_1(1))/MSE(I_i,G_θ_2(z_i))] = 1/N∑_i=1^N(1/K∑_k=1^K||I_i^k-G_θ_1(1)||^2/1/K∑_k=1^K||I_i^k- Ĩ_̃ĩ^k||^2+ε),
  
  
  where I and Ĩ are an image and its reconstruction obtained through G_θ_n; i=1… N, k=1… K, n=1…+∞ are indices iterating on the test images, the image pixels, and the generator model index (different for each run), respectively; ε is a small value that prevents division by zero. We justify the above definition of IOB by observing that a post-hoc minimization of the MSE between Ĩ and I is equivalent to maximizing the log likelihood (see our analysis in Appendix A).
  Note that the ratio aims at ruling out from IOB both data correlations
  
  (common structure, colors, pose, etc., across the images of the dataset) and architectural biases that one could introduce in the design of G_θ_n. In particular, this is done by computing the ratio between the MSE obtained after training G_θ_n to reconstruct the images from their informative representation z (i.e. MSE(I_i,G_θ_2(z_i)), and after training G_θ_n from an uninformative constant tensor 1 (i.e. MSE(I_i, G_θ_1(1))). In the latter case, G_θ_n will only learn the dataset bias it can model, given θ_n. Hence, high values of IOB can be associated with higher information inside the representation z, while the lower bound IOB=1 means that no information of the images I is encoded in z.[Optimising G_θ with stochastic gradient descent can introduce noise and slightly alter the measure. For example, IOB may, in practice, even be slightly smaller than 1. Thus, we average results across multiple runs and initializations of G_θ, which contributes to the computational load of estimating IOB.]
  
  
  
  Utility and Interpretability.
  
  As discussed, we can use DC and IOB to measure the degree of disentanglement between latent representations. 
  However, one of the primary goals of disentanglement is to improve task performance (utility) and representation interpretability, hence we also investigate the relationship between C-S disentanglement and these two notions.
  In particular, we measure utility by quantifying performance on a downstream task, which for disentangled representations is typically image translation <cit.> to translate image content from one domain to another. We also consider tasks using content e.g. to extract segmentations <cit.> or landmarks <cit.>, and therefore assess how effectively it can be used in downstream tasks. We detail performance metrics for each application in Sec. <ref>.
  
  Assessing interpretability is not trivial. Here, we assume that interpretability implies semantic representations. Previously, vector representations were considered semantic if a portion of the latent space corresponded to specific data variations <cit.>. Style semantics were qualitatively evaluated with latent traversals of individual dimensions <cit.>. Thus, we consider a style interpretable if images produced by linear traversals in the style latent space are realistic and smoothly change intensity.
  In spatial representations, such data variation should be confined to individual objects: thus, semantic content should split distinct objects into separate channels of C. Wherever possible, we evaluate this with qualitative visuals.
  
  
  
  
  § VALIDATING THE EFFECTIVENESS OF DC AND IOB
  
  To verify the effectiveness of DC and IOB, we design an experiment using the synthetic teapot dataset <cit.>, which consists of 200k of 64×64 pixel resolution images of a teapot with varying pose and colour. Each image of this dataset is generated using 5 ground truth (GT) generating factors (scalars), i.e. azimuth, elevation, red, green, and blue colour, independently sampled from 5 different uniform distributions. We consider the 3 color factors as the GT style (GT S) representation, while as GT spatial content representation (GT C) we leverage the segmentation mask of the object, which correlates with the azimuth and elevation factors (for visual examples see Appendix B).
  
  We first evaluate DC and IOB using the GT C and S representations, and the input images. Then, we sample from a uniform distribution 𝑈[0, 1] to generate a new, random style and content representations for each image, and evaluate the metrics using the following scenarios: a) random content, GT style and images; b) GT content, random style and images; c) random content, random style and images. Finally, to approximate the highly entangled C and S scenario, we construct the content-correlated style representations (correlated S) as the azimuth, elevation and red colour factors. For each experiment, we randomly sample 5k images and the GT representations, while all results are the average of 3 different runs. 
  
  
  Results. From Table <ref>, we observe that for any combination of C and S (except for the correlated S one), the DC(C,s) is low, which indicates that the representations are highly uncorrelated. This result meets our expectation as the colour (S) and the azimuth or the elevation factors (C) are independent in the teapot dataset. 
  
  However, we also observe a high DC(C,s) value, i.e. 0.53, between GT C and correlated S, which verifies that DC can indeed detect the entangled representations case. Additionally, the effectiveness of the DC metric is validated by the high DC(I,C) values when using GT C representations, versus the low values when using random C ones. Note that the DC between the GT S and image is higher than the one between GT C and image, which is reasonable as S and image have nearly one-to-one mapping relationship, while the segmentation masks for different images can be similar. The IOB results, reported in Table <ref>, also reflect that the segmentation mask is less informative (IOB(I, C)=1.73) about the input image compared to S (IOB(I, s)=2.47) for the GT C and GT S case. This is a result of the strong dataset bias, where given that the object is always a teapot, it is the colour of the reconstructed image that makes it more similar to the input one in terms of MSE. 
  
  
  
  
  
  
  
  § EXPERIMENTING ON VISION AND MEDICAL APPLICATIONS
  
  
  
  Many applications disentangle C from S <cit.> or other attributes, such as pose, geometry, and motion <cit.>, to improve performance in vision tasks.
  For our analysis, we select and discuss three SoTA approaches (see Fig. <ref>) from diverse applications, namely image translation (MUNIT <cit.>), semantic segmentation (SDNet <cit.>), and pose estimation (PANet <cit.>). All resemble auto-encoders, mapping input images to disentangled features but use several biases, which are detailed below. Our scope is to elucidate how each bias affects disentanglement using these models and their chosen biases as exemplars.
  
  Here we describe how each bias is enforced, whilst the detailed model descriptions and a summary of their design and learning biases can be found in Appendix D.
  In particular, for: a) MUNIT we consider ablations removing Instance Normalization (IN) <cit.>, AdaIN layers, or style Latent Regression (LR) loss (for fairness, we do not remove LR of the content as it is fundamental for the functioning of the model);
  b) SDNet we identify content binarization, Gaussian approximation, LR and the FiLM-based <cit.> decoder as the main biases that affect C-S disentanglement. We investigate their impact on the representations and their effect on semantic segmentation;
  c) PANet we remove the Gaussian prior and replace its specific C-S conditioning with AdaIN. We analyse PANet performance in pose estimation. These models help us cover the following diverse cases: i) no supervision and weak C constraints (MUNIT), ii) no supervision with strong C constraints (PANet), and iii) supervision with strong C constraints (SDNet).
  
  Setup. For each model, we analyze the effect that design choices and learning objectives have on disentanglement and task performance, and we evaluate utility and interpretability of the learned representations. We use the implementations provided by the authors, ablating only the components needed for our analysis[Metrics code will be made publicly available]. In all tables, arrows (↑, ↓) indicate direction of metric improvement; best results are in bold. Numbers are the average of 5 different runs. Data description and learning settings can be found in Appendix D (see D.1-D.4).
  
  
  
  
  
   §.§ Image-to-Image Translation
   
  We consider the original MUNIT and three variants:
  i) we replace the AdaIN modules of the decoder with simple style concatenations, reducing the restrictions on the re-combination of C and S. 
  ii) We remove the LR loss, responsible for the style following a Gaussian.
  iii) We remove IN from the content encoder, to confirm that it helps to cancel out original style and retain the content only <cit.>.
  As <cit.> we evaluate quality and diversity of the translated images using the Fréchet Inception Distance (FID) <cit.> and LPIPS <cit.>.
  
  Results.
  Table <ref> reports the results of the ablations on the SYNTHIA <cit.> and City-scapes <cit.> datasets. Replacing AdaIN (w/o AdaIN) with simple concatenation does not affect the level of C-S disentanglement, but it leads to a 0.14 absolute decrease in IOB(I,s) and DC(I,s), indicating that the style becomes less informative and less correlated with the input. Here, we observe an information shift to the content (lower IOB(I,s), higher IOB(I,C)) leading to better translation quality but worse diversity (LPIPS=0.06). We infer that this variant is worse than the original model, which had more balanced quality/diversity scores.
  By removing the LR learning bias (w/o LR), the style becomes more correlated to the input image. If the style distribution is no longer Gaussian, the style has more degrees of freedom to encode non-relevant information, which contributes to higher IOB(I,s) and higher C-S disentanglement. This ablation leads to a significant translation quality decrease, while contrary to the analysis in <cit.>, the diversity is not negatively affected.
  Finally, by removing IN (w/o IN) we expect a more entangled content that is encoding also some style information. Our expectations are confirmed by the decrease in C-S disentanglement (DC(C,S) = 0.66), and a more informative content (which is also more correlated to the input image). Interestingly, relaxing the content constraints for a task that does not require a strictly semantic content (such as image segmentation), leads to the best quality/diversity balance. Note that we define the best balance as achieving the highest average ranking in FID and LPIPS (e.g. the “w/o IN" model variant is the 1^st in LPIPS and 2^nd in FID).
  
  Summary.
  Our experiments reveal a trade-off between the translation quality/diversity and disentanglement in a translation task.[Note that the effect of C-S disentanglement on task performance also depends on the data bias. An indicative example is the “edges-to-shoes" setting where the translation is between zero-style and normal images.] Our metrics indicate that a partially disentangled C-S space –with a near-Gaussian style latent space– leads to the best quality/diversity performance. For MUNIT this is achieved by removing the IN design bias.
  
  
  
   §.§ Medical Segmentation
   
  In SDNet, content binarization and style Gaussianity
  are the key representation constraints. We evaluate their effect and those of decoder design on segmentation performance measuring the Dice Score <cit.> after: 
  i) removing content thresholding (w/o Binarization),
  ii) removing style Gaussianity (w/o Kullback-Liebler Divergence (KLD) and LR), 
  and iii) considering a new decoder, obtained replacing the FiLM style conditioning with SPADE <cit.>. SPADE is less restrictive, allowing the style to encode more image-related information, such as textures, rather than just intensity (see Appendix F.1).
  
  
  
  Results.
  Table <ref> reports our findings on the ACDC <cit.> dataset. We highlight that when using all the available annotations (fully supervised learning), all SDNet variants achieve a similar accuracy (see Appendix F.2 for more details), suggesting that strong learning biases, such as supervised segmentation costs, make disentanglement less important. Thus, we consider the semi-supervised training case with minimal supervision, using only the 1.5% of available labelled data.
  Overall, the style encodes little information in all SDNet variants, probably because all medical images in ACDC have similar styles (data bias), and reconstructing using an average style is enough to have low IOB(I, S). 
  However, C-S disentanglement is still important to obtain a good content representation. For example, intermediate levels of disentanglement (SPADE) lead to the best segmentation performance. In this variant,  disentanglement decreases compared to the original model, as some style information is probably leaked to the content (higher DC(C,s) and IOB(I,C)). On the other hand, also removing C binarization (w/o Binarization) makes content more informative; since the correlation between C and S decreases, we assume that the extra information encoded in C is not part of the style.
  Lastly, removing the Gaussian prior constraints from the style (w/o KLD and LR) leads to the lowest degree of disentanglement as there is no information bottleneck on S, and a slight decrease of the Dice score.
  
  
  
  Summary.
  We find disentanglement to have minimal effect on task performance when training with strong learning signals (i.e. supervised costs). 
  In the semi-supervised setting, a higher (but not full) degree of disentanglement leads to better performance, while the amount of information in C alone is not enough to achieve adequate segmentation performance.
  
  
  
   §.§ Pose Estimation
   
  We consider the original PANet model and four possible variants, relaxing design biases on both C and style, and learning biases. In detail: 
  i) we experiment with a different conditioning mechanism to re-entangle S and C, that consists of the use of AdaIN, rather than just multiplying each S vector with a separate C channel (introducing a bias on S, similar to MUNIT). 
  ii) We consider the case where, instead of learning a different S for each channel of C, we extract a global S vector, predicted by an MLP (relaxing the tight 1:1 correspondence between C and S channels).
  iii) We also consider the case where each C part is not approximated by a Gaussian prior. Since we cannot use the original decoder to combine C and S, we reintroduce S using AdaIN.
  iv) Finally, we evaluated the effect of the equivariance constraint, by removing it from the cost function.
  
  
  
  Results.
  Table <ref> reports results of the ablations on the DeepFashion <cit.> dataset. We assess model performance using SIM <cit.> to measure the similarity between the predicted and ground truth landmarks visualized as heatmaps.
  Whilst the original model is the best to predict landmarks, it only achieves average disentanglement (see DC(C,s)). Using an AdaIN-based decoder consistently improves disentanglement as it has a strong inductive bias on the re-entangled representation (see DC(C,s) for AdaIN, and AdaIN w/o Gaussian), but it leads to worse landmark detection – the representation adapts tightly to the strongly-biased decoder, and the content loses transferability to other tasks, and interpretability (see Figs. <ref>, <ref>).
  Using an MLP to encode S relaxes the specific conditioning between C and S (a design bias) and reduces disentanglement. In fact, there is an information shift from C to S, as indicated by the higher IOB(I,s), and we observe a high DC(C,s). Here, a moderate decrease of disentanglement shows slightly lower task performance.
  Finally, the equivariance cost is the most important factor for disentanglement; removing it (w/o Equivariance) leads to the most entangled representation (high DC(C,s)), and accuracy decrease in landmark detection.
  
  Summary.
  Overall, lowering disentanglement leads to better landmark detection. Again, balance is the key to improve the auxiliary tasks. In PANet, partial disentanglement is achieved by carefully balancing the design biases used to extract the style and to reintroduce it to the content while decoding. Relaxing such biases with AdaIN or MLP makes landmark detection worse.
  
  
  
   §.§ Discussion
  
  We now discuss the relationship between C-S disentanglement and inductive biases, task performance, interpretability of the latent representations.
  
  Do biases affect C-S disentanglement? 
  Results in Sec. <ref> illustrate that learning and design biases critically affect disentanglement. However, no evaluation can specifically characterize the relative importance of each one, since this depends on the task at hand, as well as the utilized data.
  In MUNIT, disentanglement is mainly encouraged by the content-related design and learning biases. In fact, IN is key to removing style information from the content, and the model cannot be successfully trained without LR of the content.
  Disentanglement in SDNet is susceptible to the biases that affect both latent variables. Using a SPADE decoder or removing content thresholding leads to more entanglement, while making the style Gaussian through learning constraints restricts its informativeness and encourages disentanglement.
  Similarly, PANet disentanglement is affected both by designing the content as Gaussian, and by the equivariance of C and S w.r.t. spatial or intensity transformations, respectively.
  
  
  
  What is the relationship between C-S disentanglement and task performance? Our results showcase a clear sweet spot between C-S disentanglement and downstream task performance. In particular, we observe that lowering disentanglement by relaxing constraints on the content (e.g. removing IN), but preserving the biases that enforce style priors, such as C-S equivariance, leads to better performance.
  
  Does disentanglement affect content interpretability?
  Interpretability is hard to quantify without metrics. Here, we consider the C interpretable if distinct objects appear in different channels. We qualitatively analyze C interpretability in Fig. <ref> (see also Appendix I).
  Interpretability varies a lot with different design biases of the model, while learning biases do not seem to affect it. Without restrictive design bottlenecks on C,
  MUNIT spreads the content across channels. Instead, SDNet and PANet original models encourage C to encode different objects, or parts, into different channels.
  In SDNet, a semantic content is encouraged by applying Softmax across channels and then binarize the output features, while PANet approximates body parts as 2D Gaussians enforcing an information bottleneck on each channel of C.
  Removing the C constraints from SDNet and PANet spreads the spatial information across all channels, decreasing interpretability.
  
  
  
  § CONCLUSION
  
  
  In this paper we evaluated the disentanglement between image C and S through experimenting on 3 SoTA models, and showcased how design and learning biases affect disentanglement and by extension task performance. Our findings suggest that whilst content-style disentanglement enables the implementation of certain equivariant tasks, partially (dis)entangled can lead to better performance than fully disentangled ones.
  Additionally, our analysis suggests that strict design constraints on the content space lead to increased interpretability, which could be exploited in post-hoc tasks. Using our findings and the presented metrics will enable the design of better models that achieve the degree of disentanglement that maximizes performance, rather than blindly pursuing very high (or low) disentanglement.
  
  
  
  
  
  
  
  § MAXIMIZING LIKELIHOOD BY MINIMIZING MEAN SQUARE ERROR
  
  
  Let y denote a generic pixel in an image I, and ỹ the respective pixel in the reconstructed image Ĩ, obtained trough a learned decoding function.
  
  If we assume the reconstruction error, denoted as ε, to be normally distributed (i.e. ε∼𝒩(0, σ^2)), 
  then, the predicted value ỹ is normally distributed around the true value y, thus ỹ∼𝒩(y, σ^2).
  Based on this assumption, the probability density function can be defined as:
  
      f(ỹ|y, σ^2)=1/√(2 πσ^2) e^-(ỹ-y)^2/2 σ^2 .
  
  Given a set of observations, e.g. the pixels of the image, we maximize the likelihood ℒ as the product of the probability densities of the observations:
  
      ℒ 
          = ∏_i=1^n f(y_i | ỹ_i, σ^2)
          =(2 πσ^2)^-n / 2 e^-∑_i=1^n(y_i-ỹ_i)^2/2 σ^2 .
  
  Assuming the variance of the error to be independent from the input variables, optimizing the latter formula is equivalent to optimize:
  
      log( ℒ/(2 πσ^2)^-n / 2) = -∑_i=1^n(y_i-ỹ_i)^2/2 σ^2 .
  
  Thus, maximizing the original likelihood function is equivalent to minimizing ∑_i=1^n(y_i-ỹ_i)^2 that is the scaled Mean Squared Error (MSE). Thus, by training the decoder to minimize MSE, we train it to maximize the Mutual Information (MI) between z and I.
  
  After training the decoder G_θ (see Sec. 3), computing MSE equivalent to directly measuring the MI. There is a relationship between likelihood and MSE (shown below), but the likelihood acts as a lower bound to MI.
  
  Relationship MSE - likelihood:
  Note that if we divide both parts of the equation by n and then we multiply by - 2σ^2, we obtain:
  
  
      ∑_i=1^n(y_i-ỹ_i)^2/n = 
          - 2σ^2/n·logℒ/(2 πσ^2)^-n/2,
  
  that is:
  
      MSE = - 2σ^2/nlog (ℒ) - σ^2log(2 πσ^2).
  
  Since we assume homoscedastic distributions, i.e. fixed σ^2, Equation <ref> can be expressed as:
  
      MSE =  - a/nlog (ℒ) - b,
  
  where a and b are positive constants.
  
  
  
  § EMPIRICAL STUDY WITH THE TEAPOT DATASET
  
  
  Visual examples and qualitative results of the empirical study on the proposed metrics with the teapot dataset are included in Fig. <ref>. It is notable that the artifacts in the reconstructed images introduced by the decoder bias are observed in the results of both decoders and bias decoders. 
  
  
  
  
  
  
  
  
  
  
  
  
  
  § MODEL DESIGN AND TRAINING SCHEME FOR IOB
  
  For the structure of IOB decoders, we vary the number of layers for different applications due to the different dimensions of the representations. The design of the decoders for the teapot dataset, MUNIT, SDNet and PANet can be found in Table <ref>, <ref>, <ref> and <ref>. The notations in the tables are:
  O: the number of output channels;
  K: the kernel size;
  S: the stride size;
  P: the padding size; FC: fully-connected layer;
  IN: instance normalization; Overall, G_θ(s) consists of several linear layers, followed by transpose (upsampling steps) and one plain CONV layer that generates the final image. G_θ(C) follows an autoencoder structure with several encoder and decoder CONV layers. For the teapot dataset, the content representation has size 1×64×64 and the style representation has size 3. For MUNIT, the content representation has size 128×64×64 and the style representation has size 8. For SDNet, the content representation has size 8×224×224 and the style representation has size 8. For PANet, the content representation has size 3×64×64 and the style representation has size 1024. Note that it is not necessary to have exactly same design as in the tables, where the key suggestion is to design the decoders to generate as high-quality as possible reconstructed images. 
  
  All the decoders are trained using the Adam optimiser <cit.> (β_1=0.5, β_2=0.999) with a learning rate of 1e^-4 for 40 epochs using batch size 10.
  
  
  
  
  
  § DETAILED APPLICATION DESCRIPTION
  
  
  Table <ref> summarizes the design and learning biases of the methods presented in Sec. 5. Note that the biases are reported as modules, without indicating the way they are used in our experiments (e.g. AdaIN is reported without specifying that it is removed from the original MUNIT, but is added to PANet as a variant).
  
  
  
   §.§ MUNIT for Image-to-Image Translation
  
  Multimodal Unsupervised Image-to-image Translation (MUNIT) <cit.> does not impose strict constraints on the learned representations, and achieves disentanglement with both design and learning biases. 
  
  The basic assumption is that multi-domain images (a necessary data bias), share common content information, but differ in style. A content encoder maps images to multi-channel feature maps, by removing style with IN layers <cit.> (design bias). A second encoder extracts global style information with fully connected layers and global pooling. Finally, style and content are combined in a decoder with AdaIN modules <cit.> (design bias). 
  
  Disentanglement is additionally promoted with a bidirectional reconstruction loss <cit.> that enables style transfer. In order to learn a smooth representation manifold, two LR losses (learning bias) are applied on content and style extracted from input images: content LR penalizes the distance to the content extracted from reconstructed images, whereas style LR encourages encoded style distributions to match their Gaussian priors. Finally, adversarial learning encourages realistic synthetic images.
  
  
  
   §.§ SDNet for Medical Image Segmentation
  
  SDNet <cit.> is a semi-supervised framework that disentangles medical images in anatomical features (content) and imaging-specific characteristics (style). Similarly to other models, SDNet uses separate content and style encoders, but here a segmentation network is applied on the content features trained with supervised objectives and annotated images (data bias). 
  
  However, in contrast to MUNIT, SDNet does not impose a design bias on the encoder, but rather on the content which is represented as multi-channel binary maps of the same resolution as the input (design bias).
  
  This is obtained with a softmax and a thresholding function with the straight-through operator <cit.>, such that any style is removed from the content.
  To encourage style features to encode residual information (and not content), a loss enforces the style representation to approximate a standard Gaussian, following the VAE formulation <cit.> (learning bias). In this setup, any information encoded in style comes at a cost, and thus encoding redundant information is prevented <cit.>. Furthermore, a LR loss of the style is employed to prevent posterior collapse of the decoder (learning bias).
  
  Finally, style and content are combined to reconstruct the input image by applying a series of convolutional layers with feature-wise linear modulation (FiLM) conditioning. Similarly to AdaIN, FiLM modules are restrictive, allowing the style only to normalize the conditioned feature maps, and thus further discouraging the style from encoding content information (design bias).
  
  
  
   §.§ PANet for Pose Estimation
  
  For the pose estimation task, we consider a dual-stream autoencoder denoted as PANet <cit.>. PANet consists of two branches that decouple pose (content) and appearance (style) but employs heavily entangled encoders-decoders. 
  
  The content is represented as a multi-channel feature map, where each channel corresponds to a specific body part (since the number of parts are fixed, this imposes a strong data bias).
  A Gaussian distribution is applied to each feature map to remove any style information, whilst also preserving the spatial correspondence (design bias).
  
  The corresponding style information is extracted from the encoder features using average pooling (design bias). More critically, style vectors do not correspond to global image style, since they are applied to specific content parts during decoding (design bias).
  
  Finally, disentanglement is encouraged with a transformation equivariance loss (learning bias). This ensures that the spatial transformations, such as translations and rotations, affect only the content, while the intensity ones, such as the color and texture information, affect only the style.
  
  
  
  § SYNTHIA-CITYSCAPES DESCRIPTION AND MUNIT TRAINING SETUP
  
  
  Data.
  We use SYNTHIA <cit.>, which consists of over 20,000 rendered images and corresponding pixel-level semantic annotations, where 13 classes of objects are labeled for aiding segmentation and scene understanding problems.
  
  We also use Cityscapes <cit.>, which contains a set of diverse street scene stereo video sequences and over 5k frames of high-quality semantic annotations, where 30 classes of instances are labeled in the segmentation masks.
  
  Training setup.
  MUNIT achieves unsupervised multi-modal image-to-image translation by minimizing the following loss function:
  
      ℒ_total = ℒ_GAN + λ_1ℒ_rec + λ_2ℒ_c-rec + λ_3ℒ_s-rec,
  
  where ℒ_rec is the image reconstruction loss, ℒ_c-rec and ℒ_s-rec are the content and style reconstruction losses, and λ_1=10, λ_2=1 are the hyperparameters used by the authors in <cit.>.
  
  
  
  
  § ACDC DESCRIPTION AND SDNET TRAINING SETUP
  
  
  Data. 
  We use data from the Automatic Cardiac Diagnosis Challenge (ACDC) <cit.>, which contains cardiac cine-MR images acquired from different MR scanners and resolution on 100 patients. Images were resampled to 1.37 mm^2/pixel resolution and cropped to 224× 224 pixels. Manual segmentations are provided for the left ventricular cavity, the myocardium and right ventricle in the end-systolic and end-diastolic cardiac phases. In total there are 1920 images with manual segmentations and 23,530 images with no segmentations.
  
  Training setup.
  SDNet is trained by minimizing the following loss function: 
  
      ℒ_total = λ_1ℒ_KL + λ_2ℒ_seg + λ_3ℒ_rec + λ_4ℒ_z_rec,
  
  where ℒ_KL is the KL Divergence measured between the sampled and the predicted style vectors, ℒ_rec is the image reconstruction loss, ℒ_seg is the anatomy segmentation loss, and ℒ_z_rec is the LR loss between the sampled and the re-encoded style vector. λ_1=0.01, λ_2=10, λ_3=1, and λ_4=1 are the hyperparameters used by the authors in <cit.>.
  
  
  
  
  
   §.§ SPADE Decoder
  
  
  As described in Sec. 5, SDNet relies on a FiLM-based decoder to combine the content and style information and reconstruct the input image. The key characteristic of FiLM is that it gradually adds style information to the content-based reconstruction process. Additionally, an alternative approach for combining the content and style features is investigated, by using a SPADE decoder <cit.> to further expose the design bias added by the decoder architecture.  
  
  A SPADE block receives the content channels and projects them onto an embedding space using two convolutional layers to produce the modulation parameters (tensors) γ and β. These parameters are then used to scale (γ) and shift (β) the normalized activations of the style representation. We utilize multiple SPADE blocks
  to fuse content and style information at different levels of granularity during decoding. A schematic of the utilized SPADE decoder in the context of SDNet is depicted in Fig. <ref>.
  
  
  
  
  
   §.§ Medical Segmentation (100% Annotations)
   
  
  In Sec. 5.2, we present the results of the SDNet model variants trained with minimal supervision, using only the 1.5% of the provided ACDC <cit.> annotations. Here, we provide the results for the same experiment but using the 100% of the provided annotations. From the results reported in Table <ref>, it can be seen that when using strong inductive biases, such as the supervised losses in this experiment, the degree of disentanglement does not significantly affect the segmentation performance (utility).
  
  
  
  
  
  
  
  § DEEPFASHION DESCRIPTION AND PANET TRAINING SETUP
  
  
  Data.
  We use DeepFashion <cit.>, a large-scale dataset with over 800,000 diverse images of people in different poses and clothing, that also has annotations of body joints. We only used full-body images, specifically 32k images for training and 8k images for testing.
  
  Training setup.
  PANet is trained in an unsupervised way with the following loss function:
  
      ℒ_total = λ_1 ℒ_rec + λ_2 ℒ_equiv,
  
  where ℒ_rec is the mean absolute error between the reconstructed and the input image. ℒ_equiv is an equivariance cost, that ensures that the mean and covariance of the parts coordinates don't change after some style transformation. Based on the implementation details presented in <cit.>, we set λ_1=λ_2=1. 
  
  
  
  
  § METRICS CORRELATION AND DISENTANGLEMENT-PERFORMANCE TRADE-OFF
  
  
  As noted in Sec. 3, we report that the proposed metrics are uncorrelated with each other. Here, we present the Pearson correlation computed between disentanglement and performance metrics for each of the investigated models. Intuitively, contrary to the desired low (or no) correlation between disentanglement metrics across all models (see Fig. <ref>), we would expect that the performance metric(s) of each application would be correlated with at least one DC or IOB variant. In fact, this correlation can be exploited to find the “sweet spot" between disentanglement and performance. Fig. <ref> confirms our intuition for all investigated models, highlighting the strong correlation of FID and LPIPS in the MUNIT scenario, which is the only model that utilizes both C and S directly in the main task, i.e. I2I translation, and not in any parallel one.
  
  
  
  § QUALITATIVE EVALUATION
  
  
  We visualize the content and style representations in order to reason about their interpretability. We consider the content semantic if distinct objects appear in different channels, whereas the style is semantic when images reconstructed while traversing the style manifold between two points have smooth appearance changes, and are realistic. 
  
  As an extension of the samples presented and discussed in Sec. 5.4, here we provide visualizations for all model variants. In particular, Figs. <ref> and <ref> depict several channels of content, as well as style traversals for different MUNIT and SDNet model variants, respectively. However, Fig. <ref> presents solely content representations, as PANet does not assume a prior distribution on the style latent vector, thus style traversals are not possible. When interpolating between two style vectors, the originally proposed MUNIT produces realistic images, and smooth appearance changes. Instead, removing the LR constraint affects the image quality. Similarly, the original SDNet presents high image quality and smooth transitions, while removing the content Binarization leads to low intensity (style) diversity.
  
  
  
  
  
  
  
  