Evaluating Perceptual Distances Models by Fitting Binomial Distributions to Two-Alternative Forced Choice Data
visual perception;perceptual distances;two-alternative forced choice
Yes
No ethics review needed.
Please see the weaknesses.
The issue of modeling 2AFC experimental data is an interesting topic.

The proposed method enjoys simplicity.
The paper proposes a probabilistic approach to model the subjective judgment obtained from 2AFC experiments about visual quality using a binomial distribution. The goal is to evaluate perceptual distance models predicting visual quality with respect to the 2AFC experimental data. Three metrics are derived for evaluation and the results on BAPPS are shown.
1. The paper is not easy to follow, mainly due to lack of definition, usage inconsistency, and unusual usage of notations and symbols. For instance, $t$ in $x_0(t)$ is not defined, $T$ is not defined, $d_v$ is not defined, $N$ is not defined, $P$ and $p$ are used without clear distinction, $j$ in Fig. 1 is not defined, the arguments are not consistent in $P$ (sometimes two arguments, sometimes three arguments, sometimes no argument, arguments do not match), {$(d_0, d_1), n=j$} doesn't make much sense, etc. In addition, some sentences are not easily understood, e.g., first sentence of Section 2.1, last sentence in page 6, what does 'preferred smoothness' mean and how can it be determined, etc.

2. Although the proposed method provides (maybe) new ways to evaluate the predicted quality scores by perceptual distance models, it is not experimentally shown whether the proposed method has any advantage over existing approaches including checking whether the image preferred by a majority of human raters is predicted to have a higher quality score by a model, comparing the predicted scores to the MOS estimated using the Thurstone or Bradley-Terry model on the subjective data, etc. In fact, in Table 2, the proposed method shows very similar results to the distance-only case, so the proposed method doesn't seem advantageous. 

3. Section 4 presents experimental results, but they do not draw much meaningful conclusions. Above all, they do not prove that the proposed method is correct. Table 1 only shows that PIM, LPIPS, and DISTS are better than Euclidean, NLPD, and SSIM, which would be obvious without relying on the proposed method. Fig. 3 shows some difference among distance models, but not much more than that; it is said that humans perceive sharper differences when images are far from their reference, which I wouldn't agree. Fig. 4 is an interesting result, but it is not clear whether it can be useful practically. Fig. 5 shows the robustness of the proposed method against the choice of hyperparameters, but it does not prove that the results are correct.

4. It is said that the proposed method is free from assumptions unlike existing methods. However, the proposed method also imposes certain assumptions. For instance, it is assumed that the subjective preference strictly follows the binomial distribution without any noise, the probability in the binomial distribution depends only on the quality distances and nothing else, etc. Justification on these issues would be needed.

5. Even though the assumptions made in the proposed method are correct, the proposed model is fitted through maximum likelihood estimation, where fitting error wouldn't be zero. How accurate is the fitting in the experiments? The fitting error could be from from the inadequacy of the model, the multimodality of the likelihood, the violation of convexity of the optimization problem, or something else. Analysis on this would be needed to justify the adequacy of the proposed model.

6. Sampling for $\hat{n}$ is performed to obtain an 'upper bound' of AJ(?) and 'minimum possible' NLL. However, in Table 1, AJ with $\hat{n}$ doesn't act as an upper bound and NLL with $\hat{n}$ is not a lower bound. The meaning of the results of $\hat{n}$ should be interpreted appropriately. The same applies to Table 3.

(minor) Line 105, based off -> based on
Yes
No ethics review needed.
1. The authors should compare your method with state-of-the-art perceptual distance models.
2. It would be beneficial to have a more detailed and step-by-step explanation of the algorithm.
1. The paper introduces a novel method for evaluating perceptual distance models using binomial distributions to model decision-making in two-alternative forced choice (2AFC) experiments, which is an innovative attempt in the field of image quality assessment.
2. The paper demonstrates the robustness of the model to changes in hyperparameters and offers insights into the interpretability of the results.
The paper presents an approach to evaluate perceptual distance models using two-alternative forced choice (2AFC) experiments, especially with crowd-sourced data where traditional ranking methods are not applicable. The authors use a binomial distribution to model the decision-making process, estimate parameters with maximum likelihood, and propose metrics that surpass simple accuracy measures. This method offers a more robust evaluation of perceptual models compared to classical and neural network approaches.
1. The paper compares the proposed method with classical and neural network methods but does not extensively compare it with the latest state-of-the-art approaches in perceptual distance modeling.
2. The method is evaluated across different datasets, but there is insufficient evidence to demonstrate its generalizability across different types of images and distortions, particularly on traditional perceptual datasets like TID.
3. The experimental section may lack comparative experiments to show the specific improvements of the proposed method over existing ones.
4. The paper may suffer from a lack of clarity in certain sections, Improving the organization of the content can help address this issue.
Yes
No ethics review needed.
Please see the weakness.
1. Rigorous Statistical Framework: By employing binomial distribution and maximum likelihood, the approach brings statistical rigor to perceptual distance model evaluation.
2. Improved Insight into Judgments: The additional metrics offer nuanced insights into how perceptual models align with human judgments, capturing nuances beyond binary agreement.
The paper presents a statistical model to improve the evaluation of perceptual distance models based on crowd-sourced 2AFC (two-alternative forced choice) data. The topic is interesting but there are some serious issues with the applicability and experiment.
**1. Applicability and Generalizability of the Proposed Metrics**

Could the evaluation metrics proposed in this study be extended to no-reference image quality assessment (NR-IQA) settings? In NR-IQA, pairwise comparisons, where participants judge the relative quality of two images, are common. This scenario bears similarities to 2AFC binary decision tasks and could benefit from the uncertainty modeling approach presented here. I recommend that the authors consider and discuss the metrics’ applicability in no-reference scenarios, potentially running experiments on pairwise NR-IQA datasets to further validate the transferability and generalizability of the metrics. This could significantly enhance the utility and impact of the proposed metrics in image quality assessment.

**2. Superiority Over Traditional Evaluation Metrics**

The study does not provide a clear demonstration of how the proposed metrics outperform traditional evaluation methods such as SRCC (Spearman’s Rank Correlation Coefficient), PLCC (Pearson Linear Correlation Coefficient), or KRCC (Kendall Rank Correlation Coefficient). While the authors show that deep learning models perform well using the new metrics, it is unclear whether this suffices to validate the effectiveness of the new metrics. To address this, I recommend a direct experimental comparison with these established metrics, which would highlight whether the new metrics provide additional information or improved accuracy. Moreover, it would be beneficial to discuss the potential of the new metrics to capture aspects like human judgment uncertainty and finer model differences, which traditional metrics may overlook, to enhance the credibility of this research.

**3. Dataset Selection and Applicability of Use Cases**

The experiments primarily rely on two datasets: BAPPS, which is diverse in distortion types, and CLIC, which is compression-specific. This dataset selection might limit the study’s ability to generalize the metrics’ effectiveness across different use cases. For instance, if the goal is to develop a general FR-IQA (full-reference image quality assessment) method, testing on a broader range of distortions—such as noise, blur, and color distortions—would be beneficial, possibly including datasets like TID2013 or LIVE. Conversely, if the focus is specifically on compression distortions, the paper would benefit from more explicitly discussing this application focus and how the proposed metrics could, for example, aid in choosing compression algorithms based on perceptual quality. Clearer examples or case studies illustrating how the metrics would assist in practical applications would improve the paper’s practical relevance.

**4. Relationship Between Uncertainty Modeling and MOS Distribution**

There seems to be a potential relationship between the proposed uncertainty modeling approach and MOS (Mean Opinion Score) distributions. MOS scores typically provide a more fine-grained rating (e.g., scores from 1 to 5), which offers a probabilistic view of perceived quality rather than binary decisions. I encourage the authors to explore how MOS distributions might contribute to refining the uncertainty modeling in this study, leveraging the probability distribution across the MOS scale to capture subtle perceptual differences more effectively. Discussing or experimenting with this connection could improve the interpretability and granularity of the proposed approach.
We fit binomial distributions to two-alternative forced choice data, conditioned on a candidate perceptual distance, in order to evaluate the perceptual distance models..
@inproceedings{
anonymous2024evaluating,
title={Evaluating Perceptual Distances Models by Fitting Binomial Distributions to Two-Alternative Forced Choice Data},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=fRaK0cG9L8},
note={under review}
}
The two-alternative forced choice (2AFC) experimental method is popular in the visual perception literature, where practitioners aim to understand how human observers perceive distances within triplets made of a reference image and two distorted versions. In the past, this had been conducted in controlled environments, with triplets sharing images, so it was possible to rank the perceived quality. This ranking would then be used to evaluate perceptual distance models against the experimental data. Recently, crowd-sourced perceptual datasets have emerged, with no images shared between triplets, making ranking infeasible. Evaluating perceptual distance models using this data reduces the judgements on a triplet to a binary decision, namely, whether the distance model agrees with the human decision - which is suboptimal and prone to misleading conclusions. Instead, we statistically model the underlying decision-making process during 2AFC experiments using a binomial distribution. Having enough empirical data, we estimate a smooth and consistent distribution of the judgements on the reference-distorted distance plane, according to each distance model. By applying maximum likelihood, we estimate the parameter of the local binomial distribution, and a global measurement of the expected log-likelihood of the measured responses. We calculate meaningful and well-founded metrics for the distance model, beyond the mere prediction accuracy as percentage agreement, even with variable numbers of judgements per triplet -- key advantages over both classical and neural network methods.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
visual perception perceptual distances two-alternative forced choice
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/ca64af68d0f3209d4a8029914570c1a5be74246a.pdf
applications to computer vision, audio, language, and other modalities
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/3e9e2d248fa73d27938bf7f7a2d64bc70d5f0ce5.pdf
Evaluating Perceptual Distances Models by Fitting Binomial Distributions to Two-Alternative Forced Choice Data
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission