PHI-S: Distribution Balancing for Agglomerative Models
Computer Vision;Deep Learning;Knowledge Distillation;Agglomerative Models
Thank you for your review. We agree that the effect size is small on a per-benchmark basis, but rather argue that the performance across a broad spectrum of tasks does meaningfully separate different normalization techniques. Because we’re not introducing any new parameters or data, and because the student model is extremely compressed, it’s unsurprising that the effect sizes can be small, but that doesn’t make them irrelevant. Where PHI-S truly shines is in its ability to match all teachers simultaneously (figure 3, tables 3, 11, and 15) without introducing additional parameters (based on the argument that the linear projection of PHI-S can be rolled into the final adaptor linear layer, thus being representationally equivalent). We also demonstrate some interesting properties of the errors of the student model in table 5, showing how the errors are more uniformly balanced, which can be a useful property when applying euclidean measures (e.g. L2 distance) on the produced features. We also show in figure 9 a stability argument, where PHI-S clearly has the most tightly controlled error distributions, which can meaningfully impact learning dynamics. 

One thing that seems inarguable from the study is that normalizing the teachers is incredibly important. Once you accept that and introduce the code to support this, then there’s not really much difference in algorithmic complexity between regular standardization and PHI-S, but PHI-S makes the student model better.
Thank you for your review. At a high level, we relied on AM-RADIO to motivate the agglomerative model paradigm in general, which allowed us to focus solely on improving it. We think that all of the issues you raised in the weaknesses section are indeed quite valid questions regarding agglomerative modeling, and we think that your first two identified weaknesses are worthy of much deeper and dedicated studies. There is emerging evidence that these models do generalize well in practice, which we talk about in our specific answer regarding your first identified weakness.

**Weakness 1**: The AM-RADIO paper does most of the motivating for distilling multiple foundation models, and this work largely assumes that the technique is important based on AM-RADIO’s assertion. What they found was that different VFMs have different strengths and weaknesses, owing largely to how they were trained. The CLIP family of models is great for holistic reasoning, and seems to have a strong sense of semantic alignment with language, however they struggle significantly with dense perception tasks (e.g. semantic segmentation, or the 3D vision tasks tested here). DINOv2 is pretty much the gold standard for perception right now, but it struggles a lot with OCR (e.g. TextVQA in Table 1 of AM-RADIO). It indeed seems to be the case that fusing multiple teachers into these smaller students works well across a spectrum of tasks [6, 8, 9], and in all of these instances, the student model was smaller than even just DINOv2, ignoring the other teachers.

**Weakness 2**: We agree that choice of dataset can potentially be significant. We purposefully chose to keep the dataset constant in this paper so that we could solely focus on the topic of distribution normalization. Teacher selection and dataset selection are topics that need larger treatments that couldn’t fit into the scope of this study.

**Weakness 3**: Are you referring to how quickly our estimates of the rotation matrix and alpha scalar converge? We’re currently just computing these estimates over the first 3 million examples seen during training (first 3k mini-batches), but agree that it might be interesting to see how quickly those values converge. We did run into issues with PyTorch’s eigh call not returning stable eigenvectors, which was a problem for all methods relying on that function, except for ZCA, as the inverse rotation was counteracting the instability.

**Weakness 4**: We agree that ultimately increasing the scale of the model would be good, but we were bound by computation constraints, as you identified in the next weakness.

**Weakness 5**: Finding PHI-S is mostly negligible, as we only "learn" it over the first 3k steps (of 300k), and afterward the  application of it is a GEMM at each iteration, which is very small relative to the actual model. During inference, all of the studied normalizations can be rolled into the weights of the teacher adaptors, making them free. That said, explicitly, running the GEMM on an A100 with an input size of `(32, 729, 1280)` (from DFN CLIP) and PHI-S transform `(1280, 1280)` takes about 0.46ms, and so we’re looking at roughly 2ms/step of overhead across the four teachers, compared against the approximately 355ms/step of the general training harness. We also note that while training the model is expensive, and while specifically trying to make the training cheaper wasn’t a focus of the study, we did improve over AM-RADIO’s recipe which is 17,408 GPU-hours.

**References**:

[6] Drozdova, Mariia et al. “Semi-Supervised Fine-Tuning of Vision Foundation Models with Content-Style Decomposition.” ArXiv abs/2410.02069 (2024)

[8] Lu, Yuxiang et al. “Swiss Army Knife: Synergizing Biases in Knowledge from Vision Foundation Models for Multi-Task Learning.” (2024).

[9] Guo, Pinxue et al. “VideoSAM: Open-World Video Segmentation.” (2024).
**Weakness 5**: We agree that DINOv2 is a hugely valuable model. Studying the benefits of the teacher models, or even teacher selection itself, is out of scope for this study. There is some recent evidence [6] that suggests that AM-RADIO enjoys better generalization and transfer learning than DINOv2. Theia [7] does study teacher inclusion, however it’s also a relatively small scale study, and they’re not trying to build a general foundation model, but a limited robotic one as evidenced by their exclusion of the summary tokens.

**Weakness 6,7**: Thank you for the formatting feedback. We will incorporate it into the draft.

**Question 1**: We explored combining AdaLoss with PHI-S (tables 11-14), and found that teacher MSE is reduced further when additionally applying PHI-S. We agree that AdaLoss marks a large improvement over the baseline MSE without balancing, but disagree that it’s generally competitive against naive weighting combined with standardization.

**Question 2**: It’s actually the projection head of DINOv2 that is L2 normalized, but the backbone features themselves are unconstrained, and it’s the latter that we’re trying to match. SAM has supervised features at the output, but we’re feature matching before the neck, where they’re also unconstrained. DFN CLIP and SigLIP don’t apply any supervisory signal to the features. We think it would be easy to construct an example where it does make sense to take the final operator of the teacher into account before matching. This is exactly why the summary loss is cosine across the board, because the teacher models used cosine for the summary tokens, and thus it’s possible (or even likely) that the vector magnitudes are meaningless. It seems entirely plausible that LayerNorm could benefit from a boutique normalization technique that operates better for it. What we found in figure 2 was that all of the teachers had mostly gaussian distributions with various means and variances across the dimensions, which is something PHI-S is well designed to handle.

**Question 3**: This is currently the major black box of agglomerative models. If the adaptor head of the student perfectly matched a teacher, then the student would have the same behaviors as the teacher, but only when still applying that adaptor. Once you introduce a second teacher, even assuming that it also perfectly matched that teacher, it’s unclear what properties the student backbone would exhibit, assuming that it wasn’t wide enough to represent both teachers exactly. In this case of our study, our ViT-B only has a width of 768, and ViT-L a width of 1024. DINOv2-g alone has a width of 1536, meaning that our student (in the case of ViT-B) is only half the width of just one of its teachers. What we do consider to be important, all things equal, is that higher teacher fidelity (measured here as lower MSE) is preferable. Our study doesn’t introduce a change of data, and we constrain the experimental setting to be identical everywhere except for the loss function (and associated weights), which makes it fair to care about fidelity a lot (tables 3, 11, and 15). We have also added section A.9.4 with associated tables 18 and 19 to attempt to unify fidelity across the teachers in a way in which it can be aggregated. We agree that finding teachers that are distinctly good at certain tasks and correlating them with their losses is a good topic of future work, but teacher selection was largely outside the scope of this study.

**References**:

[5] Anthony Trioux, François-Xavier Coudoux, Patrick Corlay, M Gharbi. A comparative preprocessing study for softcast video transmission. 2018 9th International Symposium on Signal, Image, Video and Communications (ISIVC), Nov 2018, Rabat, Morocco. pp.54-59, ff10.1109/ISIVC.2018.8709171ff. ffhal-03335982f

[6] Drozdova, Mariia et al. “Semi-Supervised Fine-Tuning of Vision Foundation Models with Content-Style Decomposition.” ArXiv abs/2410.02069 (2024)

[7] Shang, Jinghuan et al. “Theia: Distilling Diverse Vision Foundation Models for Robot Learning.” ArXiv abs/2407.20179 (2024)
Thank you for your in-depth review. We agree that the correlation between fidelity and downstream benchmarks is not 1-to-1, which can be seen in how the relative differences on particular benchmarks is not straightforward. We opted to rely on summarizing the entire suite of benchmarks, under the assumption that a model that tends to perform better across the suite would tend to be a more generalizable model. Rank works well in this regard because we don’t have to normalize the benchmark units in order to aggregate a different way. The primary reason we have the “Avg No COCO” and “Avg No MSE/COCO” columns in Table 1 is so that we could demonstrate that PHI-S works without relying on outlier scores in the MSE task. However, we didn’t have a column which includes everything _except_ for MSE, which we’ve rectified in the updated draft. The added column “Avg No MSE” in table 1 has a consistent conclusion as before, which is that PHI-S does the best on average, followed by global standardization. We did the same thing for table 4, which still has PHI-S as the best method, and in this case, regular standardization remains in second place. We have a detailed response below.

**Weakness 1**: We’re applying distribution balancing to the spatial features of the teacher models, whereas zero shot accuracy is affected by the summary loss (always cosine) between the student and teacher. The CLIP teachers have a supervised summary feature, and it’s trained using cosine similarity with their text encoders. DINOv2 also uses cosine similarity for its summary tokens. Presumably this is why AM-RADIO chose to use cosine similarity loss for the summary of all teachers (SAM withstanding). There is a tension between the summary losses (cosine) and the feature losses (studied in our paper), where changing the weight of one will impact the others.

**Weakness 2**: There are two different statements that the paper is making. The central table for the purposes of the paper is actually table 1, where the setup is tightly controlled between different feature distillation losses, and thus it’s a fair comparison. Table 2 is instead showing that we were able to apply the techniques studied to produce an agglomerative model that is competitive with AM-RADIO, but at 15% and 49% of the parameters for ViT-B and ViT-L respectively. It’s essentially a hook for “why should I care” as opposed to the central result of the study.

**Weakness 3**: We used the comparative ranks as it’s not clear which benchmarks are most important, and also how their magnitude changes relative to each other is important. Instead, the average rank argument is that across a large swath of benchmarks, a better model will tend to do better than its peers across a wider range of tasks. A major problem with agglomerative modeling (or even foundation models in general) is that the downstream use case is unclear, and so the choice of which metrics to target is subjective. If we substantially changed the weights across all summary and feature losses in favor of say, DFN CLIP’s summary loss, then we’d expect to get very strong zero shot (and probably kNN) accuracy, but it would come at the expense of the dense tasks. Applying AdaLoss to the MSE baseline actually does have an effect similar to the standardization methods, as it’s applying the inverse expected loss. This is why we also studied AdaLoss + PHI-S, and found improved teacher matching metrics (table 11) across the board, suggesting that its inclusion, all things equal, yields students with higher fidelity. All said, however, AdaLoss tended to work worse across the full task suite as compared to the standardization methods (Global, Standardize, PHI-S) aside from classification where it strongly produces the best results (see for example the Probe 3D metrics in table 13).

**Weakness 4**: We disagree on the claims of limited novelty. As for the comparison against AM-RADIO, the entire study focuses around a single design decision in their paper, their equation 3. So we’re leveraging their work, not necessarily comparing against it, aside from in table 2 where we eventually bring the details full circle. While the applications of Hadamard matrices are not new, to the best of our knowledge, combining the eigenvector projection of PCA with the Hadamard rotation as a method to get identical variance across the dimensions of a multivariate distribution is entirely novel. Even Hadamard whitening (as described in [3], but also [5]) is not the same as we’re employing it here. In [3, 5] they’re spreading the error in a similar motivation as ours, but not applying statistical whitening (multiplying by the inverse root eigenvalues). They’re especially not using the Hadamard matrix for statistical standardization, nor are they motivated by the isotropic property of our approach.
Thank you for your detailed review, particularly catching the discrepancy in your second question. At a high level, we feel as though PHI-S is applicable to any situation where you’re learning one or more fixed multivariate distributions. AM-RADIO (or the other agglomerative approaches) fall into this category, and we chose to go with AM-RADIO because it’s been around the longest. Learning a fixed distribution could also be applied in settings such as quantization, and we believe PHI-S has a useful grounding here as it would minimize quantization errors across dimensions since the distribution is balanced across. This is actually where the previous literature on Hadamard matrices in our related work section focuses, although they don’t seem to combine it with PCA, which means they aren’t identically spreading the variance. Getting an apples-to-apples comparison across the agglomerative papers is hard because the space hasn’t standardized on a single setting in the same way that DataComp did for CLIP models. Instead, we wanted our study to be internally consistent, which is Table 1, and then we wanted to demonstrate the relevance in Table 2 by showing that we can ultimately get a very strong ViT-L/16 model. We agree that having an apple-to-apples with at least AM-RADIO would be nice, however, their ViT-H is quite costly to train, and it was prohibitive for the scope of this work. In the following we go through the details of your feedback:

**Contribution Weakness**: While we agree that Theia also studied standardization, PHI-S has applications beyond just what we studied in this paper. It could similarly be applied anywhere you’re trying to learn a fixed distribution, which we’re seeing similar techniques recently with quantization [10,11, 12], albeit they rely on random matrices as an approximation to PCA. Our paper provides an exact solution which is suitable in situations where the distribution is not changing. We also argue that while Theia applies standardization, they don’t deeply analyze the choice, and this study serves to reinforce the application from both a theory and applications standpoint.

**Experimental Weakness**: While we do include table 2 as a means for grounding our results with the general literature, it wasn’t our goal to exactly replicate AM-RADIO, but rather rely on the fundamentals introduced in table 1 as a means for comparing the different techniques in isolation of a bunch of design choices. AM-RADIO’s ViT-H model is extremely expensive to train, and thus attempting to exactly match it was beyond our computational budget at the time, which is why we made choices such as multi-stage training for table 2 simply to reduce the computational burden. The net result is that we get rather strong results, which table 2 showcases, but really rely on table 1 as the crux of the study where fair comparisons are made.

**Question 1**: Because PHI-S is a rotation and uniform scale, one way to think about what it’s doing is implicitly applying loss weights to the different teacher’s features, and a result of this re-weighting is that the summary losses are impacted. Roughly, this would be similar to loss weighting these components by $\alpha^2$ from table 8. Similar to our answer below about SAM metrics, it’s not necessarily surprising that changing the weighting between losses will affect benchmarks that are quite sensitive to particular models, in this case, the summary loss for DFN CLIP/SigLIP. The argument we’re trying to make with this paper is that it’s hard to even address high-level balancing across teachers without first getting them onto a level playing field, and we’re arguing that PHI-S does the best job across those methods studied of balancing their distributions.

**Question 2**: These differences were a result of transcription errors. We’ve corrected both tables, and greatly appreciate your catching of the mistake.

**Question 3**: This has to do with how dominant SAM’s feature distribution is relative to the other models. In effect, when applying simple MSE, most of the loss weight is applied to SAM, which you can see in figure 3. So the baseline MSE model is learning SAM much better, at the expense of the other models. Tables 13 and 16 show that basically any type of balancing (as they’ll all reduce the weight of SAM) has a negative effect on the SAM COCO benchmark. Because this benchmark relies on exactly (or at least acceptably) approximating the real SAM model, it’s expected that having a higher SAM MSE (figure 3) would lead to reduced SAM COCO metrics.

**References**:

[10] Ashkboos, Saleh et al. “QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs.”

[11] Tseng, Albert et al. “QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks.” ArXiv abs/2402.04396

[12] Shah, Jay et al. “FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision.” ArXiv abs/2407.08608 (2024): n. pag.
Yes
No ethics review needed.
1) Table 2 suggests that the performance of the AM-RADIO zero-shot/few-shot image classification reduces when PHI-S normalization is applied. However, I see that the baseline and the proposed approach are using different models. Is the reduction in performance caused by using a different backbone or it is caused by the proposed PHI-S normalization. Does your method still hurt AM-RADIO if you switch to ViT-H? Adding apple-to-apple comparisons between the proposed method, AM-RADIO and previous approaches can help understanding the effectiveness of the proposed normalization.

2) Table 2 reports PHI-S-RADIO-L to have an ImageNet-1K accuracy of 81.01 and 84.68 on zero-shot and kNN respectively. However, Table 18 reports 80.45 and 84.57 for the same model (PHI-S-RADIO-L) and the same dataset to my understanding. What is the difference between these two experiments?

3) According to the reported experiments in the paper, the proposed PHI-S normalization almost always hurts the performance on the SAM COCO instance segmentation, even compared to the simple MSE baseline without normalization. Is there a specific reason for this observation? An analysis can help the reader to better understand the shortcomings of the proposed normalization.
+ The proposed PHI-S normalization is easy to be applied to the existing AM-RADIO framework.
+ PHI-S on average performs better than the other normalization techniques considered in the paper.
+ I would also like to appreciate the inclusion of recent similar techniques (such as UNIT and UNIC) in the related works.
The paper proposes a modification to the AM-RADIO multi-task distillation framework (Ranzinger et al.) to improve its performance on downstream tasks. More specifically, it focuses on the problem of normalizing the output distributions of different tasks/teachers before distilling them into a single student. The paper performs ablations on different normalization techniques such as simple mean/std global normalization, whitening, Hadamard whitening and then proposes a modified normalization technique called PHI-S which is rotation invariant. The experiments suggest that PHI-S, on average, is a better normalization for AM-RADIO compared to the other normalizations considered in the experiments.
**Contribution:** 

The paper proposes a new normalization technique to be applied to teacher distributions in multi-teacher distillation settings. Although the importance of normalization has been previously studied in the literature and sufficiently covered in the related works by the authors, the proposed method targets a specific framework, namely AM-RADIO. To me the main finding of the paper is the importance of such normalization for this framework and introduction of the PHI-S normalization technique. However, normalization for AM-RADIO has been also recently explored (such as Theia as also acknowledged by the authors). This makes the paper an experimental follow up by introducing a new type of normalization for which I expected to see stronger experimental results (see below).

**Experimental results**:

 The main contribution of the paper is the introduction of the PHI-S normalization with the goal of improving the existing AM-RADIO pipeline. Most experiments in the paper focus on ablating and comparing PHI-S with simpler normalization techniques. Particularly, I found it hard to find a fair comparison between AM-Radio with and without PHI-S and with other existing related works. Table 2 and Table 18 try to provide such a comparison, but the settings are not apple to apple (i.e. using different models, different training, image resolutions, etc for different methods). This makes it hard to verify the main claim of the submission. Please see the questions sections for additional questions/comments.

**Paper organization/presentation:**

I suggest the authors consider re-organizing the paper. A significant portion of the main paper (page 3, page 4, and page 5) goes over the details of the previously known methods and normalization techniques . This can be greatly summarized and/or moved to the supplementary, freeing up some space for discussing the main contributions of the paper and including important experimental results.
Yes
No ethics review needed.
- Adding the AdaLoss to MSE significantly improves over MSE results in Tables 13 and 14 on ALL tasks. Have you experimented with adding the AdaLoss to some other methods that perform well on dense tasks such as Hyb SmL1 or Standardize?
- Different teacher models also normalize the features in different ways during the pre-training step. For example, the features in the DINO head are L2 normalized and lie on a unit hypersphere. Some models use LayerNorms on the output features. Could there be a benefit in taking into account the normalizations used during pre-training while selecting appropriate feature standardization?
- What is the correlation between the teacher-matching loss and the downstream performance? For example, if the loss to a specific teacher is minimized, should that lead to the resultant model displaying properties most similar to that teacher? If the goal is to demonstrate the effectiveness of the feature standardization and its effect on downstream tasks, would it make more sense to consider teacher models which are distinctly good at specific tasks but significantly worse at other tasks? This might enable one to more clearly evaluate the downstream impact of different teachers and minimizing their losses.
- The problem of distributional differences identified by the paper with regards to the AM-RADIO work is well motivated and can be useful not just for agglomerative foundation models but also for other problems involving distillation of multiple features. 
- The presentation of the different standardization techniques and the theoretical analysis of their error properties is good.
This paper addresses the problem of learning a unified agglomerative model for vision - by distilling knowledge from multiple vision foundation models, used as teacher models. This work builds upon AM-RADIO (Ranzinger et al. (2024)) and identifies a possible limitation that could arise from the differences in the distributions of the features from different models, With this motivation, the paper explores different statistical normalization techniques to improve the teachers' features. Specifically, the paper introduces PCA-Hadamard Isotropic Standardization (PHI-S) that is claimed to produce the best student model compared to prior baselines and other feature standardization techniques.
- The experiments initially focus on the teacher-matching metrics, for example the MSE loss to different teachers in Table 3. However, it is not clear if the a decreasing the MSE loss to a specific teacher necessarily brings downstream benefits. For example, consider the Cosine method for the DFN-CLIP teacher. Its MSE loss is 10-25 times higher than all the other methods (the Cosine method is also the worst for SigLIP). DFN-CLIP and SigLIP are the only strong performing models in zero-shot Imagenet classification (see Table 1 in [1]). One would expect higher MSE for DFN-CLIP and SigLIP to result in poorer performance in zero-shot Imagenet classification. But the finding is to the contrary - the Cosine method performs best in zero-shot Imagenet classification among non Ada- methods (see Table 13). If the initial MSE loss is mainly resulting from the difference in feature norms, then the experiments that depend primarily on the MSE loss do not make sense.
- The empricial comparison of PHI-S to AM-RADIO in Table 2 is not a fair comparison as the teacher models used are different between the 2 works. So, this comparison is not a fair demonstration of the benefits obtained by using the feature standardization proposed in this work.
- The empirical performance on downstream tasks are presented in the main paper based on average ranks on different groups of tasks. This makes it difficult to get a clear understanding of the performance and does not provide a clear comparison between the different methods. Looking at the performance metrics in Tables 13-17, the performance of PHI-S is only on par with other baseline methods or the improvement is too marginal to be considered as a significant improvement. The presentation in terms of ranks is also somwhat misleading when the performance difference between the ranks are so small/insignificant. For instance, one could argue that adding the AdaLoss to MSE brings consistent and significant improvements across all tasks (Ada-MSE vs MSE) but the performance difference between the other methods are comparatively insignificant. This raises the question of whether the usage of Hadamard matrices for standardization (which is claimed to be one of the contributions of the paper) is actually effective.
- The paper is heavily reliant on AM-RADIO [1]. The proposed standardization solution is not completely new [3, 4] but it can still be interesting to show that it can be useful in new applications. However, the effectiveness of the proposed solution is not sufficiently demonstrated in this paper. 
- DINOv2 also brings additional benefits like transfer learning, domain generalization and robustness properties. The paper lacks analysis of other such beneficial properties of the teacher models. This can be a worthy addition that is not explored in [1] either.

- Minor: Inconsistent notations can be avoided - in lines 127-133, the different teachers are indexed by $(t)$ whereas in section 2.2 they are indexed by $k$.
- Minor: The readability of the paper can be improved by formatting the references appropriately in parentheses when they are not part of the sentence. 

[1] Ranzinger, Mike, et al. "AM-RADIO: Agglomerative Vision Foundation Model Reduce All Domains Into One." CVPR 2024.

[2] Hu, Hanzhang, et al. "Learning anytime predictions in neural networks via adaptive loss balancing." Proceedings of the AAAI Conference on Artificial Intelligence 2019.

[3] Kanj, Hind, et al. "A comparative study of the whitening methods in linear video coding and transmission schemes." 2022 11th International Symposium on Signal, Image, Video and Communications (ISIVC). IEEE, 2022.

[4] Pratt, William K., Julius Kane, and Harry C. Andrews. "Hadamard transform image coding." Proceedings of the IEEE 57.1 (1969).
Yes
No ethics review needed.
Overall, this is a good paper. The reviewer would like to see the authors' rebuttal on the general questions listed in the weaknesses.
+ The proposed method is built on a solid empirical observation by accounting for the diverse distributions of different teachers' activations. 
+ The paper is well written and easy to follow.
+ The proposed empirical study is comprehensive in seeking the ``best" activation alignment space. 
+ The identified PHI standarization method works well in experiments compared to baseline approaches.
This paper presents a comprehensive study for training agglomerative models, that is to distill multiple diverse teacher models into a student model without labels by collectively "matching/aligning" the activation distributions, where the teacher models' activations are often significantly different from each other.  This paper is built on a prior art, the AM-RAIDO method and explores, from  the perspective of statistical normalization,  many different  designs of normalizing the activations of teacher models in the distillation loss functions. Based on the Hadamard matrices,  the identified PHI (PCA-Hadamard Isotropic) standarization method works the best in terms of training the best student model across different tasks. In experiments, the proposed method is tested in distilling a diverse set of foundation models (DFN CLIP, SigLIP, DINOv2 and SAM) into ViT-B/16 and ViT-L/16.
- The motivation of distilling multiple foundation models into a student model could be elaborated. Although it is an interesting problem, what is the long-term vision? Will it be practically possible to train a student model that is smaller than all teacher model, yet works comparably well in a broad sense. For example, DINOv2 can produce meaningful latent features that are useful in many downstream tasks beyond those tested in the paper.  Will the distilled student be able to retain those? 
- The proposed method is trained without labels. It might be useful to discuss what effects the training datasets could have considering that different teacher models have been trained with diverse datasets. 
- It might be useful to investigate the effects of batch sizes in computing the PHI-S in distillation. 
- Although the propose method shows competitive performance using ViT-B/L models in comparisons to the ViT-H trained by AM-RADIO, it might be useful to train a ViT-H using the proposed method for a broader understanding of the competitiveness. 
- The proposed method is computational expensive (e.g. 14080 total GPU hours for the ViT-B/16 model). It might be useful to compute the overhead of the propose PHI-S in comparisons.
Yes
No ethics review needed.
n/a
* In depth analysis of the effect of different normalization schemes on representations
* Building the PHI-S normalization, and great illustration of its effect and its invariance to rotation
* A solid list of benchmarks, from various truly heterogenous teachers
The paper is interested in improving the agglomerative model distillation, a setting in which several teachers with different heterogenous representations are used to distilled a single student.

In this context, the author identify the fact that the representations of each teacher can be widely different, having different distribution and variances. This causes challenges on the optimization side, as the losses coming from a few different teachers might have a disproportionate effect on the overall summed loss, biasing the distillation toward those few teachers (the other ones being ignored).

To alleviate this, the authors propose a new normalization technique to apply on the target representations of the teachers, PHI-S, for PCA Hadamard Isotropic Standardization, that is invariant to data rotations (contrary to standard normalization). When equipped with this normalization, the authors report that the distillation is less biased toward SAM representations (which have the largest variance) and produces more balanced results on downstream tasks than the standard MSE loss.

The authors also compare their PHI-S normalization to the other normalization schemes that could be applied on the target teacher representations.

The benchmarks include:
* Zero shot classification
* Segmentation
* VQA 
* Probe 3D
The main weakness of the paper is the lack of real performance improvement coming from PHI-S when compared to more standard normalization schemes.

In Table 4, on ViT-L/16, where the authors mention that PHI-S is more dominant, we can see that the average rank of PHI-S places it as one of the best normalization technique. However, the ranks hide the fact that most differences are tiny and not very significant.
* When we look at Table 16, we see that rank 1 on classification is only due to 0.01% difference with standard standardization
* When we look at Table 17, we see that all normalization techniques are within less than 0.3% for most benchmarks

What those table seems to show (to me) is that in general normalization is really important, but the type of normalization used itself is not that impactful.

Overall, I think this normalization technique is interesting but the application to heterogenous teacher distillation is not that impactful.
We find a novel way to standardize teacher feature distributions so that we can balance the contribution of multiple teachers in a multi-teacher knowledge distillation setting.
@inproceedings{
anonymous2024phis,
title={{PHI}-S: Distribution Balancing for Agglomerative Models},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=rP7rghI7yt},
note={under review}
}
Various visual foundation models have distinct strengths and weaknesses, both of which can be improved through heterogeneous multi-teacher knowledge distillation without labels, termed "agglomerative models." We build upon this body of work by studying the effect of the teachers' activation statistics, particularly the impact of the loss function on the resulting student model quality. We explore a standard toolkit of statistical normalization techniques to better align the different distributions and assess their effects. Further, we examine the impact on downstream teacher-matching metrics, which motivates the use of Hadamard matrices. With these matrices, we demonstrate useful properties, showing how they can be used for isotropic standardization, where each dimension of a multivariate distribution is standardized using the same scale. We call this technique "PHI Standardization" (PHI-S) and empirically demonstrate that it produces the best student model across the suite of methods studied.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Computer Vision Deep Learning Knowledge Distillation Agglomerative Models
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/f0f09667b02a625fcea17a7e01f46bd6fc71d410.pdf
unsupervised, self-supervised, semi-supervised, and supervised representation learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/1e7c02b40fabf3418983e8d14603c5ee63f20616.zip
PHI-S: Distribution Balancing for Agglomerative Models
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission