Vision models trained to estimate spatial latents learned similar ventral-stream-aligned representations
Vision;Convolutional Neural Networks;Representation Learning;Neural Data Alignment;Ventral Visual Stream;Computational Neuroscience
Hi reviewer wqcV, a huge thank you for giving such a thoughtful and detailed review and helping us improve this paper. We will address your first question in this response and work on addressing your other questions in a later response. 

**Respond to your questions:**
The ground-truth rotation of objects in our dataset used for training is defined relative to a canonical view of the object. The canonical view is tied to the object identity. Although there might be some simple heuristics to determine the canonical view of an unknown object, we believe that determining the canonical view mainly relies on correctly identifying the object first. So, as you pointed out, we also think that object rotation is entangled with object category, which may explain the spike in category decoding in rotation-trained models. The experiment you mentioned is a very interesting idea for testing this empirically; thanks for letting us know.

We are working on addressing your other comments, so stay tuned. Thank you!
Hi, reviewer acUt, thank you for your reviews and efforts in helping us improve the paper!

**Respond to your questions:**
We know some work, such as Conwell et al. 2022, that includes semantic segmentation models. We would appreciate it if you could point us to some other studies that you think are relevant to the neural alignment of semantic segmentation models.

The models we studied in this paper primarily focus on estimating object-centric spatial latent variables, including distance, translation, and rotation. We chose to examine models estimating these latents for three main reasons: (1) these variables constitute the core elements of the spatial information about objects, which people can intuitively estimate; (2) previous electrophysiological studies suggest that the ventral stream encodes these spatial attributes (Hong et al., 2016); and (3) models that predict these latents can be trained on the same dataset with the same architecture, allowing us to isolate the effect of the training objective while controlling for other factors.

Most segmentation models we know require an additional decoder module to output a segmentation map, making their architecture different from the latent estimation models studied here. Our study aimed to analyze the influence of training objectives on learned representations while keeping the dataset and architecture constant. We didn't study the segmentation models here because it is hard to control their architecture to be the same. We also don't know whether segmentation models have higher or lower neural alignment. The TDW dataset we proposed can also contain ground-truth labels for segmentation maps, which could be a resource for people interested in studying segmentation models in the future.

We are working on addressing your other comments, so stay tuned. Thank you!

Reference: 
1. Conwell, C., Prince, J.S., Kay, K.N., Alvarez, G.A. and Konkle, T., 2022. What can 1.8 billion regressions tell us about the pressures shaping high-level visual representation in brains and machines?. BioRxiv, pp.2022-03.
2. Hong, H., Yamins, D.L., Majaj, N.J. and DiCarlo, J.J., 2016. Explicit information for category-orthogonal object properties increases along the ventral stream. Nature neuroscience, 19(4), pp.613-622.
Thank you very much for your thoughtful reviews and help in improving the paper!

**Respond to your questions:**
Thanks for pointing that out. We believe a couple of reasons could underlie the phenomenon you mentioned.
1. Figure 4b shows the CKA results until layer 4.0, an intermediate layer in the CNN (the 15th layer of the 18-layer ResNet). To decode categories from intermediate layers linearly could be challenging even for category-trained models. So, it is possible that even for category-trained models, the decoding performance will not be much better than the decoding performance from spatial latent trained models shown in Figure 5b at layer 4.0. However, this remains unknown; we can do an experiment to address that.
2. It is still possible that representations can have different category decoding performance while having similar CKA since CKA does not capture every aspect of neural representations.

We are working on addressing your other comments, so stay tuned. Thank you!
Thank you, reviewer v5sv, for your thoughtful reviews and for pointing us to the many interesting related works.

**Respond to your questions:**
We were able to decode categories to some extent from the latent trained models. For example, in Figure 5b, we can decode the categories well above chance in a 117-way classification task. In Figure C.1, C.2, and Table C.1, C.2, we analyzed the behavior alignment scores of these latent trained models. This score measures how well the penultimate layer of the model supports categorization in a human-like way (Rajalingham et al., 2018). In summary, we found that spatial latent trained models are not as good as category-trained models in supporting categorization behavior if we decode from the penultimate layer. This may or may not hold in earlier layers; further analysis is needed.

We are working on addressing your other comments, so stay tuned. Thank you! 

References:
Rajalingham, Rishi, et al. "Large-scale, high-resolution comparison of the core visual object recognition behavior of humans, monkeys, and state-of-the-art deep artificial neural networks." Journal of Neuroscience 38.33 (2018): 7255-7269.
Yes
No ethics review needed.
1. Were you able to decode categories from the models trained for spatial latents?
1. **Very important problem**:  
   The paper broadens our understanding of the ventral stream by exploring if spatial tasks, not just object categorization, can also drive effective visual cortex models. This is impactful for both building better models of the cortex, and also provides a computational feasibility argument for better understanding what the cortex does in the brain.

2. **In-depth, exhaustive analysis**:  The authors provide thorough testing across multiple objectives, using extensive synthetic data and well designed metrics. Largely, the evidence matches the claim, and the findings seem quite reliable.

3. **Interesting findings**:  Discovering that spatial-task-trained models achieve similar neural alignment to category-trained ones suggests surprising flexibility in modeling the visual cortex. It might opens an interesting question---how do self-supervised models of the visual cortex behave in comparison to categories and spatial latents?

4. **Interesting new dataset**:   The synthetic dataset with spatial labels could be of use to future researchers.

5. **Writing and figures are high quality**: Clear writing and well-designed figures make complex ideas accessible and enhance the paper’s readability.
Traditionally, neural network based models of the visual cortex have relied on CNN backbones trained for object recognition. Here, the paper investigates if CNNs trained with alternate objectives (specifically, spatial tasks like viewpoint and pose estimation) can be used instead, and how that impacts the learned representations. Their main finding is that models trained to predict such alternate spatial tasks can also be used to learn models of the visual cortex which perform just as well. Digging into it further, the authors find this happens because that the representations learned with these different objectives all end up learning quite similar representations (as measured by CKA). Also, authors find that variability in non-target latents helps models.
1. Missing comparison to self-supervised models: The paper mentions them briefly, but does not provide experiments with these models. The best performing modern models used in AI are all increasingly relying on self supervised objectives.

2. Missing comparisons with Vision Language models: The majority of upcoming new models are all multi-modal. It would be interesting to see how similar or dissimilar these perform.

3. Missing some references: 
- The ideas of non-latent diversity are closely related to work in machine learning focusing on Data Diversity. It would be nice to connect this work to these papers [1,2,3,4,5]
- This paper presents results on BrainScore which is in-distribution. It would be interesting to see how these evolves out of the training data distribution, especially since invariances become very important in out of distribution settings [6,7,8,9]

References

1. Koh, P.W., Sagawa, S., Marklund, H., Xie, S.M., Zhang, M., Balsubramani, A., Hu, W., Yasunaga, M., Phillips, R.L., Gao, I. and Lee, T., 2021, July. Wilds: A benchmark of in-the-wild distribution shifts. In International conference on machine learning (pp. 5637-5664). PMLR.

2. Madan, S., Henry, T., Dozier, J., Ho, H., Bhandari, N., Sasaki, T., Durand, F., Pfister, H. and Boix, X., 2022. When and how convolutional neural networks generalize to out-of-distribution category–viewpoint combinations. Nature Machine Intelligence, 4(2), pp.146-153.

3. Gulrajani, I. and Lopez-Paz, D., 2020. In search of lost domain generalization. arXiv preprint arXiv:2007.01434.

4. Arjovsky, M., Bottou, L., Gulrajani, I. and Lopez-Paz, D., 2019. Invariant risk minimization. arXiv preprint arXiv:1907.02893.

5. Arjovsky, M., 2020. Out of distribution generalization in machine learning (Doctoral dissertation, New York University).

6. Ren, Y. and Bashivan, P., 2024. How well do models of visual cortex generalize to out of distribution samples?. PLOS Computational Biology, 20(5), p.e1011145.

7. Madan, S., Xiao, W., Cao, M., Pfister, H., Livingstone, M. and Kreiman, G., 2024. Benchmarking Out-of-Distribution Generalization Capabilities of DNN-based Encoding Models for the Ventral Visual Cortex. arXiv preprint arXiv:2406.16935.

8. Pierzchlewicz, P., Willeke, K., Nix, A., Elumalai, P., Restivo, K., Shinn, T., Nealley, C., Rodriguez, G., Patel, S., Franke, K. and Tolias, A., 2024. Energy guided diffusion for generating neurally exciting images. Advances in Neural Information Processing Systems, 36.

9. Ponce, C.R., Xiao, W., Schade, P.F., Hartmann, T.S., Kreiman, G. and Livingstone, M.S., 2019. Evolving images for visual neurons using a deep generative network reveals coding principles and neuronal preferences. Cell, 177(4), pp.999-1009.
Yes
No ethics review needed.
* In Figure 5b, the decoding accuracy for classification is quite low for models trained for spatial latents, suggesting that the model trained for spatial latents is not suitable for classification. However, this contrasts with the CKA analysis in Figure 4b where the representation similarity is quite high between models trained for spatial latents and models for object categorization. How would you explain such difference?
* The paper asks an important question of the function role of the ventral stream besides image categorization. The discovery suggests that representation that is similar to the ventral stream neural response can be derived from training the network under spatial latent estimation objective as well. This contrasts the traditional belief that the primilary goal of the ventral stream is for object categorization only. 
* To demonstrate the proposal, the paper utilizes a 3D graphics engine to generate large dataset to tackle the shortage of spatial latent variables in previous studies. In Figure C.3 in supplementary, the paper demonstrates the importance of having larger scale dataset. 
* The paper shows that CNN trained on purely synthetic dataset can achieve a similar neural alignment score as these trained with natural images. 
* The paper has clear writing and presentations.
The paper asks whether the ventral stream only codes object category information. By showing that CNN models that are trained for spatial understanding can also align with the neural reponse in the ventral stream, the author provide a critical reflection on the assumption that ventral stream in the brain primarily encode categorical information. The authors also introduce synthetic datasets generated by a 3D graphics engine (ThreeDWorld) containing precise spatial labels, overcoming the scarcity of natural datasets with spatial information. This dataset enabled systematic experiments to evaluate CNNs trained on spatial latents.
* It could be more comprehensive if the paper utilizes natural images as training, e.g. one could extract sptial variables from SA-1B segmentation dataset to perform the experiments. 
* The evaluation of the neural alignment score is limited to the Brain-Score. More metrics would be helpful to consolidate the results. 
* The idea that ventral pathway is not responsible for classification alone is not new among cognitive neuroscience community. E.g Connor discussed in his paper that the retinotropic spatial information is transformed into relative positional information across ventral stream [1]. A discussion with related works in neuroscience community would be appropriated. 

[1] Connor, Charles E., and James J. Knierim. "Integration of objects and space in perception and memory." Nature neuroscience 20.11 (2017): 1493-1503.
Yes
No ethics review needed.
Left in weakness section.
The paper addresses an interesting problem that has been overlooked for some time. The presentation is highly effective, and the paper is easy to read. It takes an unconventional approach to the problem by focusing on spatial location, which proves to be very effective even in a low data regime. There are several controls in the main text and supplementary material that support the claims.
The paper examines the effect of training models to estimate spatial latents on neural alignment (the "where"), comparing this approach with object recognition (the "what"). It highlights that to achieve high neural alignment, training with a focus on "where" yields similar benefits compared to the more common focus on "what." Interestingly, with fewer presentations in the "where" case, alignment levels appear comparable, and representations in the models' initial layers are strikingly similar.
I know that in  semantic segmentation and other tasks where location and classification of the object is used for training, has been studied for brain-score but have not really succeeded.  Perhaps worth adding a bit of discussion on this topic, because even when its possible to say that the world is 3D and therefore the tasks seem to align better with vision, is also true that the models used in this work are only receiving 2D images as input. Some specific questions:  

1. How this approach differs from previous attempts using semantic segmentation for brain-score alignment? 
2. Why this  spatial latent estimation tasks succeed where others have not
3. Are there any implications of using  2D inputs to estimate 3D spatial properties, how this  relates to biological vision? 

Since the title starts with  "Vision Models...", I largely missed other models like ViTs  to be studied. But it seems that as it stands the results only cover CNNs?
1. I would suggest  a change of the title if that is the case, or inclusion of other kind of vision models in the results to make sure is align with the title.
Yes
No ethics review needed.
- About the spike in category decoding with rotation latent target (bottom row Fig. 5b): I wonder if rotation decoding is somewhat entangled with category decoding. I.e. to know some object has been rotated by 90 degrees you must know what its shape is/how it appears when presented at zero degrees of rotation (and such information is correlated with category). It might be interesting to look and see how the performance on category decoding breaks down by class (if category performance is high for objects whose shape varies strongly with viewing angle this might support such a hypothesis). 

- Why are two different similarity metrics employed when comparing two artificial representations and when comparing artificial representations to neural measurements? Since the current work only considers the subset of datasets from BrainScore that are public, couldn't you use CKA to compare the artificial and biological representations as well (or alternatively use partial least squares regression to compare pairs of CNNs)? This doesn't seem likely to be an important issue but I am curious as to the reasoning, as it might be nice to have all comparisons on the same "scale", subject to the same method-induced biases, and tolerant to the same transformations. 

- In the paragraph beginning on line 334 the authors suggest that their results from decoding/CKA experiments suggest that each model with similar predicitivity is also explaining the same portion of variance in the neural responses. While this seems sensible (especially perhaps for earlier layers/V1-V2 predictivity), I don't understand why we cannot just address this hypothesis directly. For example we can: directly examine the residuals produced by each model-to-brain prediction, predict the residual of one predictive model with another, or predict the neural data using ensembled models to get a sense for how overlapping the explained variances are. Again, given that the study is limited to the publicly available benchmarks all of these experiments seem feasible, and I think very interesting! 

- Semi-Nitpick: the paper repeatedly references the number of units provided supervision. I don't think this is an interesting axis, or fair to compare between units that receive a discrete category label and a continuous latent variable. Meaning, you could discretize each spatial latent into 117 bins without changing the training signal but equalizing the number of outputs to the categorization task. I think rather than the dimensionality of supervisory signal the number of bits of supervision is the relevant thing to compare! I.e. since you know the distribution of locations you can measure the entropy of the translation variable and compare this to the entropy of the category variable (this may be trickier because of discrete vs. differential entropy etc.), but I think is more meaningful than comparing dimensionalities. 

- Note: I would prefer to leave a score of 7 for this paper, but as that is unavailable I will sit at a soft 6 for now. I am very willing to raise my score if the authors can address/justify/explain some of the weaknesses or questions listed above.
- The subject (the degree to which object classification is a unique normative explanation for features of neural representations in the ventral stream) is very interesting (and to a substantially large community).

- Investigating this hypothesis using a carefully controlled *but appropriately large* synthetic dataset is a sensible and novel approach.

- For the most parts experiments are clearly described, easy to follow, and sufficiently thorough to justify the conclusions. 

- While I find the results of Section 3.3 the least convincing of those presented, doing these experiments is certainly a step toward addressing why similar representations are observed in the early layers across many artificial representations (which is "above and beyond" what is necessary for supporting the core arguments laid out in the paper).
The paper assesses the neural predictivity and representational similarity of CNNs trained on a novel large synthetic dataset to decode different combinations of the following latent variables: object category , object identity, object position in 3-D space, and object orientation. The author's make the following observations:
- Training networks using a sufficiently large/realistic/diverse synthetic dataset can produce representations that rival those trained on natural images in terms of predictivity of ventral stream responses.
- Model's trained only to estimate spatial latents (i.e. without any explicit incentive for class invariance) can explain a similar fraction of variance as models trained to do categorization. 
- The ability to decode spatial latents is correlated with predictivity.
- Model's trained on different tasks learn similar internal representations (as measured by CKA), especially at early layers and to a lesser degree deeper in the networks.
- The amount of variability of some latent variable in a training dataset modulates the extent to which said latent is linearly-decodable from a given representations, even when that latent is not used to train the neural network.
- Layer Mapping Details Missing: I think it would be good to report which layers are mapped to each of the considered brain areas/datasets (unless I missed this detail). For instance, if the same layers are mapped to V1 across tasks this would be interesting especially in light of the fact earlier layers have higher CKA similarity. 

- More fine-grained/controlled predictivity-to-decoding correlation analyses: It would be interesting to see if performance on any particular task is more or less correlated with predictivity in a particular region (i.e. by reproducing Fig. 3 but measuring the correlation with only V1 predictivity etc.). Furthermore, it may be easier to interpret this type of result if the layers used to decode the latent parameters were matched to the layers mapped to a particular region (so you can make statements along the lines of "the ability to decode distance from layer3.0.relu was correlated with the ability to predict V4 from the same activations, suggesting a possible functional link between the two").

- Difficult to draw conclusions from results in Section 3.3: 
  - It would be nice to see the linear decoding performance for each latent on the pixels as well as early in the network as a baseline. 
  - In Fig. 5b while all models have marginal increases  in category decoding (though most differences seem insignificant based on the depicted error bars, which are not described, but I assume to be 95% confidence intervals?), they are all still generally unable to decode categories at a meaningful rate. I think as well as the chance-performance floor it would be good to include a "trained for category decoding only" ceiling to help assess the scale of the improvement. Furthermore it would be good to include the performance curve (decoding accuracy as a function of depth in the network) for randomly initialized networks. If this curve is above the others a more accurate description would be "learning to decode a spatial latent decreases the decodability of a non-target latent less when that latent is more variable in the training set." Similar baselines would be useful in 5c.
  - The all vs. none variability setup in these experiments produced mostly small differences in the decoding performance with the notable exceptions of layer4.0relu in 5b bottom row and avgpool/fc in the bottom row of Fig 5c. Given this I think the result could be made more convincing by showing an orderly trend in decoding performance as the variability of non-target latents is parametrically decreased in steps between the Full variability case and the very low variability case explored here.  Finally in this setup it would be good to see how decoding performance varies in terms of the variability. Consider an experiment of this type:
    - You have (in addition to the models at full translation variability and zero translation variability already trained) a model trained using a new 50% translation variability dataset. You may then want to measure the translation decoding performance of all models both the full-variability and half-variability test-sets to get the clearest picture possible. 
  - In sum, I think the language describing these results could be softened to reflect how difficult it is to draw definitive conclusions about internal representations from this type of experiment. This isn't a big issue at all but maybe changing line 430 to: "In summary, most of our observations are in line with the hypothesis that models develop better intermediate representations of nontarget latents in the presence of their variability, though future work is needed to understand both the size of and mechanism behind this effect." 

- Model-Brain Comparisons Leave out Relevant Datasets: the author's only compare model responses to 1/4 of the neural recording datasets in both areas V4 and IT available on the BrainScore platform. I understand that predictivities across these datasets are correlated with each other, but I see no reason not to include them in the analyses. Especially given that the images presented in the Majaj-Hong benchmarks have a striking similarity in construction to the TDW synthetic dataset (object at random pose superimposed on a natural background), it would be good to confirm that the observations in this work generalize to other types of images!
CNNs trained to estimate spatial latents, such as object position and pose, achieved similar neural alignment with the primate ventral stream as models trained on hundreds of categories.
@inproceedings{
anonymous2024vision,
title={Vision models trained to estimate spatial latents learned similar ventral-stream-aligned representations},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=emMMa4q0qw},
note={under review}
}
Studies of the functional role of the primate ventral visual stream have traditionally focused on object categorization, often ignoring -- despite much prior evidence -- its role in estimating "spatial" latents such as object position and pose. Most leading ventral stream models are derived by optimizing networks for object categorization, which seems to imply that the ventral stream is also derived under such an objective. Here, we explore an alternative hypothesis: Might the ventral stream be optimized for estimating spatial latents? And a closely related question: How different -- if at all -- are representations learned from spatial latent estimation compared to categorization? To ask these questions, we leveraged synthetic image datasets generated by a 3D graphic engine and trained convolutional neural networks (CNNs) to estimate different combinations of spatial and category latents. We found that models trained to estimate just a few spatial latents achieve neural alignment scores comparable to those trained on hundreds of categories, and the spatial latent performance of models strongly correlates with their neural alignment. Spatial latent and category-trained models have very similar -- but not identical -- internal representations, especially in their early and middle layers. We provide evidence that this convergence is partly driven by non-target latent variability in the training data, which facilitates the implicit learning of representations of those non-target latents. Taken together, these results suggest that many training objectives, such as spatial latents, can lead to similar models aligned neurally with the ventral stream. Thus, one should not assume that the ventral stream is optimized for object categorization only. As a field, we need to continue to sharpen our measures of comparing models to brains to better understand the functional roles of the ventral stream.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Vision Convolutional Neural Networks Representation Learning Neural Data Alignment Ventral Visual Stream Computational Neuroscience
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/b36abf37166dc5b8cc94922f9ac74c19ad94785b.pdf
applications to neuroscience & cognitive science
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/53641725f5325ed53b18c1f27349271ccfd7be72.zip
Vision models trained to estimate spatial latents learned similar ventral-stream-aligned representations
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission