The Probability Simplex is Compatible
Deep Learning;Representation Learning;Compatible Learning;Neural Collapse
Yes
No ethics review needed.
Please see above.
-	The paper includes comprehensive theoretical analysis.
-	The performance decay is clear on previous methods when the number of steps increases. The proposed method achieves interesting results, even with large number of training steps.
This paper studies the backward compatible ability between the representations of the base model and the updated ones. The study shows that independently trained neural networks can be made compatible simply using feature representations derived from softmax outputs and logits. Experiments on Cifar100, TnyImageNet200, ImageNet1k and Google Landmark v2 are conducted to demonstrate the effectiveness of the proposed method.
-	The theoretical analysis appears complex and challenging to follow. Could the authors clarify how to define the projection matrix $P_{k,k}$.
-	It would be better to add some pseudo codes on LSP and PSP, which could help readers understand how to implement the proposed methods.
-	In Table 1, it is interesting to see the old method BCT works better than new ones on most of cases. Why newer methods, such as CoRes and ETF-CE, show lower performance than BCT?
-	BCT is mainly evaluated on face recognition datasets. Does the proposed methods LSP and PSP also work on face recognition datasets?
Yes
No ethics review needed.
Overall, this paper requires some clarifications on theory and experiments. Given these clarifications, if provided in an author's response, I would consider increasing the score. 

For the theory, there are a few steps that need further clarification, especially regarding novelty. 

1. One aspect of the PSP method remains unclear. In Eqn.(2),  the author defines the feature representation $\textbf{h}^k = \\| \textbf{P}_{k,k} \sigma(\textbf{z}^k) \\|\_2$. When defining a metric or distance function $d$, the following conditions should be considered:

- The distance between an object and itself is always zero.
- The distance between distinct objects is always positive.
- Distance is symmetric: the distance from x to y is always the same as the distance from y to x.
- Distance satisfies the triangle inequality: if x, y, and z are three objects, then $d(x,z) \le d(x,y) + d(y,z)$

However, the softmax function is not one-to-one, and for any constant $c$, we have $\sigma (\textbf{z}^k) = \sigma (c \textbf{z}^k)$. This implies that the distance between two different raw representations, $\textbf{z}$ and $c \textbf{z}$ would reduce to zero, thus violating the first rule. 

2. The contribution of the LSP method appears potentially controversial. The model backbone output in response to an image is usually referred to as its ``embedding". However, we double-checked the code (**eval\_logits.py** and **run\_softmax.sh**) provided in the supplemental materials and confirmed that the authors use the final output (logits) of pretrained models as the feature representation. In this scenario, two pre-trained models on the same dataset would be naturally compatible. For well-trained models, their prediction probabilities for the same image should align naturally. Thus, discussing this phenomenon may add limited value.


For the experiments, the following should be addressed.

1. The author should explore the applications or advantages of PSP and LSP. For example: 
- Model A is trained on both the training and test sets, while Model B is fairly trained on the training set only. PSP or LSP could then be used to perform hypothesis testing on whether ``Model A and Model B are trained on the same image collection''.
- It would have been better also to show the classification accuracy gain of the new query model.
1. The paper proposes two methods, PSP and LSP, to improve backward-compatible training (BCT), potentially allowing the evaluation of the compatibility of the query and gallery model.

2. The study uses diverse comprehensive metrics, including Average Compatibility (AC), Average Accuracy (AA), and Average Compatibility Accuracy (ACA), to evaluate backward compatibility.

3. Experimental results indicate that PSP and LSP can outperform other methods in certain cases, showing promise in improving backward compatibility.
The paper proposes two methods, PSP and LSP, for improving the backward-compatible training (BCT), a framework for training embedding models that enables deploying new models without re-indexing existing image collections. Their main contribution is to show that any independently trained model can be made compatible with any other by simply using features derived from softmax outputs. They perform an empirical study and utilize Average Compatibility (AC), Average Accuracy (AA), and Average Compatibility Accuracy (ACA) to evaluate backward compatibility. Lastly, there are empirical experiments conclude that in some cases, PSP and LSP outperform other methods.
Motivation is unclear. The author seems to bypass the training process of BCT. However, the BCT method aims to make the new query model gain more accuracy (see [1], Table 3). In this paper, the author uses three compatibility metrics to evaluate compatibility between query and gallery models but does not illustrate how PSP and JSP improve model performance.

---

[1] Shen, Y., Xiong, Y., Xia, W., & Soatto, S. (2020). Towards backward-compatible representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 6368-6377).
Yes
No ethics review needed.
See weaknesses.
- The idea proposed in this paper is interesting. Without even involving the old model in the new model training, which most backward compatible solutions need, the proposed solution projects either the new classes/tasks into the probability simplex, thereby achieving backward compatibility
- The paper also presents two types of projections, namely the softmax/logits projections. Analysis of the behavior and performance of both types of projections are given.
- This paper also highlights an interesting aspect of backward compatibility, which is whether we can do it without needing the old model. That means that we can make any two models backward compatible to each other without them knowing of each other when they were trained and built.
This paper proposes a solution to deal with the backwards compatibility problems -- when better models, new tasks, etc., necessitates updating the model in production, it takes a lot of effort to ensure the new model is compatible with the old model in production. This paper proposes a solution that project softmax and logits into the simplex space, and "reconciling" the new and old model there.
- The paper can be written better. The paper talks about a P_{t,k}, P_{k,k} projection functions that take the softmax and project into the simplex space. However, I am not able to find anywhere how these projections came about, how they are learned/trained. If the authors can point me to where in the paper describes how these projections are learned, I would appreciate it. I did comb through the appendix quickly as well as the supp, but did not seem to find it. Otherwise, I may have been confused by your writing.
- The paper only consider the task incremental setting in the formulation, making it a bit hard for me to understand how this is extended to retrieval task or when model architecture is updated (Table 2). For example, what if we are dealing with a representation self-supervised model and the retrieval task, which most BCT papers are dealing with, and a newer model comes along?
- I find the evaluation metrics a bit obscure, namely the average compatibility, average accuracy and average compatibility accuracy. Is it possible for the authors to provide numbers before and after you add classes or change architecture per method, so that I can tell whether the old performance is maintained. Specifically, before you add classes, the performance of the new model should maintain on the old classes.
- Is it possible to add retrieval task and benchmark with baselines? A lot of BCT papers use retrieval to measure their performance. It is hard for me to tell without any retrieval task (I am assuming Table 1 and 2 are all classification results, iiuc).
- Table 2 needs to be compared to other benchmarks.
- I would also like to see a lot more modern architecture benchmarking besides just a Max ViT_T in Table 2. For example, a transition of architectures from ViT-S, to ViT-B, huge, etc.
- As it stands, it is hard to know whether the paper's method works since Table 1 and 2 are not 100% showing that the method always beat the others.
Yes
No ethics concern by this reviewer.
No ethics review needed.
- In Table 1-C, some values are reported as NaN, OOM, and $\times$. Please provide additional explanation or context for these entries.
- The proposed PSP and LSP features can also be computed for certain baseline approaches that involve learning. Have the authors considered such cases? For instance, a new model could be trained with the BCT loss. In that case, would it still be beneficial to use PSP and LSP as features instead of the actual features?
- In Fig. 5(a), regarding the same-class cosine distance, is  $\alpha = 0$ ? Please clarify. If so, does the figure simply illustrate that if the variance of a vMF is reduced through an update, the average distance between vectors decreases?
- In Fig. 5(b), does it make sense to also plot  $1 - \cos{\alpha}$  for different $d$?
- The paper tackles the model compatibility problem from a new angle: using projected softmax/logits vectors as the feature.
- Theoretical results provided in the paper appear sound under the stated assumptions.
- The introduction, related work, and problem definition in Section 3.1 are comprehensive and easy to follow.
- Experimental results demonstrate the proposed approach’s empirical efficacy, especially in cases involving multiple model updates.
The paper addresses the issue of model compatibility in representation learning, where features generated by an updated model cannot be directly compared with those of an old model. It proposes a new approach to this problem: using the (projected) softmax/logits vector as the feature vector (referred to as PSP/LSP, respectively). This approach resolves the compatibility issue. The paper also presents theoretical results showing that, under certain assumptions, using PSP/LSP as a feature ensures compatibility, as defined in the BCT paper[1]. Finally, it provides empirical evidence supporting the effectiveness of this approach on CIFAR100, ImageNet, and Google LandMarkv2 datasets.

[1] Shen, Yantao, et al. "Towards backward-compatible representation learning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.
- The paper proposes using projected softmax/logits vectors as features, resulting in compatibility out-of-the-box (i.e., without the need for additional training). However, the paper does not discuss the accuracy impact of this design choice: for a model $M$ , what would be the recall@1 when using  $M$ ’s features (for both query and gallery) versus using PSP and LSP? Could you please report and discuss test recall@1 for all models and datasets considered in the paper, including open-set setups and cases with different numbers of training classes? For example, the diagonal entries of the compatibility matrices provided in Figure 7d-f show drop in accuracy when using PSP/LSP instead of main backbone features.

- A key practical application of model compatibility in representation learning is in large-scale retrieval systems, such as face recognition. In these cases, training involves supervised learning on a large number of classes/identities ($C$  classes), while the test-time classes are unknown (i.e., an open-set setup). The proposed approach requires maintaining  $C$ -dimensional vectors, which can be very large compared to typical embedding dimension (128-256 as in [1]) and may make the proposed approach impractical. Can the authors propose/explore some dimensionality reduction mechanism to address this issue? A face recognition experimental setup (also discussed below) would be good to explore this.

- Many practical use cases involve infrequent model updates, such as updating the face recognition embedding model every few years in a large retrieval system. The proposed approach appears to perform worse than the baselines (e.g., BCT[1]) in such scenarios (2-step results in Table 1), which significantly limits its applicability for real-world problems. What would be the performance of the proposed approach for a single model update of a face recognition setup (the same as those considered in [1]) in comparison to baselines?

- Additionally, the approach does not apply to self-supervised learning, where there is no notion of logits/softmax.

- The theorem provided is based on several assumptions (listed below based on this reviewer's understanding). Could you please clearly state in the main paper the assumptions and the form of conclusion? Keeping the actual proof in the appendix is fine.
  1. Normalized projected softmax features form a vMF distribution.
  2. The variance of this vMF distribution decreases with model updates.
  3. The proof holds only in an expectation sense: on average, same-class features are closer and different-class features are farther when using the new model for queries. No probabilistic bound is provided.

- The presentation of Sections 3.2 and 3.3 is difficult to follow and could be simplified and restructured. For example, the projection matrix is used in Equation 2, but its actual definition is presented two pages later in Proposition 2. 

- Additional minor issues are listed below:
  - Line 184: “Trained with stochastic gradient descent (SGD)”—clarify the significance of this statement.
  - Equation 2:  $|| \cdot ||_2$  seems to be used to indicate L2-normalization, though it typically denotes the L2-norm.
  - Line 211: Softmax is referred to as a projection, although, mathematically, the softmax function is not a projection:  $\sigma(\sigma(z)) \neq \sigma(z)$.
  - Line 304: typo.
  - Line 405: ETF abbreviation is used without definition.
  - Line 953: $\cos  \phi$  should be $\cos  \alpha$

[1] Shen, Yantao, et al. "Towards backward-compatible representation learning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.
In this paper we show that usig softmax outputs and logits as features leads to backward compatible representations between independetly trained models without requiring any additional losses or network modifications.
@inproceedings{
anonymous2024the,
title={The Probability Simplex is Compatible},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=pPmQvd1NUp},
note={under review}
}
In retrieval systems, updating the base model involves re-extracting feature vectors for all gallery data due to changes in internal feature representations. This process can be computationally expensive and time-consuming, especially for large-scale gallery sets. To address this issue, backward compatible learning was introduced, allowing direct comparison between the representations of the old model and those obtained by the newly trained model. Existing backward compatible methods introduce additional losses or specific network architecture changes, which require the availability of base models, thereby limiting compatibility with models trained independently. In this paper, we show that any independently trained model can be made compatible with any other by simply using features derived from softmax outputs. 
We leverage the geometric properties of the softmax function, which projects vectors into the Probability Simplex, preserving the alignment of softmax vectors across model updates and verifying the definition of compatibility. A similar property is observed when using logits as a feature representation. They distribute during training in a simplex configuration, but with a wider spread in the feature distribution than softmax outputs, leading to a more robust and transferable representation. Our framework achieves state-of-the-art performance on standard benchmarks, where either the number of training classes extends across multiple steps or the base model is updated with advanced network architectures. This demonstrates that any publicly available pretrained model can be made compatible without requiring any additional training or adaptation. Our code will be made available upon acceptance.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Deep Learning Representation Learning Compatible Learning Neural Collapse
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/d690e7e7a4a01720eae1ae8bc8de37039b7d636f.pdf
unsupervised, self-supervised, semi-supervised, and supervised representation learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/53bc7426058683bee349bbfa2252de3797d7f75b.zip
The Probability Simplex is Compatible
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission