Assessing Image Quality Issues for Real-World Problems
Tai-Yin Chiu Yinan Zhao Danna Gurari
cs.CV







Assessing Image Quality Issues for Real-World Problems
     Tai-Yin Chiu, Yinan Zhao, Danna Gurari 
  University of Texas at Austin 
    December 30, 2023
=============================================================================






We introduce a new large-scale dataset that links the assessment of image quality issues to two practical vision tasks: image captioning and visual question answering.  First, we identify for 39,181 images taken by people who are blind whether each is sufficient quality to recognize the content as well as what quality flaws are observed from six options.  These labels serve as a critical foundation for us to make the following contributions: (1) a new problem and algorithms for deciding whether an image is insufficient quality to recognize the content and so not captionable, (2) a new problem and algorithms for deciding which of six quality flaws an image contains, (3) a new problem and algorithms for deciding whether a visual question is unanswerable due to unrecognizable content versus the content of interest being missing from the field of view, and (4) a novel application of more efficiently creating a large-scale image captioning dataset by automatically deciding whether an image is insufficient quality and so should not be captioned.  We publicly-share our datasets and code to facilitate future extensions of this work: . 










§ INTRODUCTION

Low-quality images are an inevitable, intermittent reality for many real-world, computer vision applications.  At one extreme, they can be life threatening, such as when they impede the ability of autonomous vehicles <cit.> and traffic controllers <cit.> to safely navigate environments.  In other cases, they can serve as irritants when they convey a negative impression to the viewing audiences, such as on social media or dating websites.  

Despite that low-quality images often emerge in practical settings, there has largely been a disconnect between research aimed at recognizing quality issues and research aimed at performing downstream vision tasks.  For researchers focused on uncovering what quality issues are observed in an image, their progress largely has grown from artificially-constructed settings where they train and evaluate algorithms on publicly-available datasets that were constructed by distorting high quality images to simulate quality issues (e.g., using JPEG compression or Gaussian blur) <cit.>.  Yet, these contrived environments typically lack sufficient sophistication to capture the plethora of factors that contribute to quality issues in natural settings (e.g., camera hardware, lighting, camera shake, scene obstructions).  Moreover, the quality issues are detangled from whether they relate to the ability to complete specific vision tasks.  As for researchers focusing on specific tasks, much of their progress has developed from environments that lack low-quality images.  That is because the creators of popular publicly-available datasets that support the development of such algorithms typically included a step to filter out any candidate images that are deemed insufficient quality for the final dataset <cit.>.  Consequently, such datasets lack data that would enable training algorithms to identify when images are of insufficient quality to complete a given task.



Motivated by the aim to tie the assessment of image quality to practical vision tasks, we introduce a new image quality assessment (IQA) dataset that emerges from a real use case.  Specifically, our dataset is built around 39,181 images that were taken by people who are blind who were authentically trying to learn about images they took using the VizWiz mobile phone application <cit.>.  Of these images, 17% were submitted to collect image captions from remote humans.  The remaining 83% were submitted with a question to collect answers to their visual questions.  As discussed in prior work <cit.>, users submitted these images and visual questions (i.e., images with questions) to overcome real visual challenges that they faced in their daily lives.  They typically waited nearly two minutes to receive a response from the remote humans <cit.>.  For each image, we asked crowdworkers to either supply a caption describing it or clarify that the quality issues are too severe for them to be able to create a caption.  We call this task the unrecognizability classification task.  We also ask crowdworkers to label each image with quality flaws that are more traditionally discussed in the literature <cit.>: blur, overexposure (bright), underexposure (dark), improper framing, obstructions, and rotated views.  We call this task the quality flaws classification task.  Examples of resulting labeled images in our dataset are shown in Figure <ref>.  Altogether, we call this dataset VizWiz-QualityIssues.

We then demonstrate the value of this new dataset for several new purposes.  First, we introduce a novel problem and algorithms for predicting whether an image is sufficient quality to be captioned (Section <ref>).  This can be of immediate use to blind photographers, who otherwise must wait nearly two minutes to learn their image is unsuitable quality for image captioning.  We next conduct experiments to demonstrate an additional benefit of this prediction system for creating large-scale image captioning datasets with less wasted human effort (Section <ref>).  Finally, we introduce a novel problem and algorithms that inform a user who submits a novel visual question whether it can be answered, cannot be answered because the image content is unrecognizable, or cannot be answered because the image content is missing from the image (Section <ref>).  This too can be of immediate benefit to blind photographers by enabling them to both fail fast and gain valuable insight into how to update the visual question to make it become answerable. 

More generally, our work underscores the importance of defining quality within the context of specific tasks.  We expect our work can generalize to related vision tasks such as object recognition, scene classification, and video analysis. 


§ RELATED WORK





  
Image Quality Datasets. 
A number of image quality datasets exist to support the development of image quality assessment (IQA) algorithms, including LIVE <cit.>, LIVE MD <cit.>, TID2008 <cit.>, TID2013 <cit.>, CSIQ <cit.>, Waterloo Exploration <cit.>, and ESPL-LIVE<cit.>. A commonality across most such datasets is that they originate from high quality images that were artificially distorted to introduce image quality issues. For example, LIVE <cit.> consists of 779 distorted images, which are derived by applying five different types of distortions at numerous distortion levels to 29 high-quality images.  Yet, image quality issues that arise in real-world settings exhibit distinct appearances than those that are found by simulating distortions to high-quality images.  Accordingly, our work complements recent efforts to create large-scale datasets that flag quality issues in natural images <cit.>.  However, our dataset is considerably larger, offering approximately a 19-fold increase in the number of naturally distorted images; i.e., 20,244 in our dataset versus 1,162 images for <cit.>.  In addition, while <cit.> assigns a single quality score to each image to capture any of a wide array of image quality issues, our work instead focuses on recognizing the presence of each distinct quality issue and assessing the impact of the quality issues on the real application needs of real users.





  
Image Quality Assessment. 
Our work also relates to the literature that introduces methods for assessing the quality of images.  One body of work assumes that developers have access to a high-quality version of each novel image, whether partially or completely.  For example, distorted images are evaluated against original, intact images for full-reference IQA algorithms <cit.> and distorted images are evaluated against partial information about the original, intact images for reduced-reference IQA algorithms <cit.>.  Since our natural setting inherently limits us from having access to original, intact images, our work instead aligns with the second body of work which is built around the assumption that no original, reference image is available; i.e., no-reference IQA (NR-IQA).  NR-IQA algorithms instead predict a quality score for each novel image <cit.>.  While many algorithms have been introduced for this purpose, our analysis of five popular NR-IQA models (i.e., BRISQUE <cit.>, NIQE <cit.>, CNN-NRIQA <cit.>, DNN-NRIQA <cit.>, and NIMA <cit.>) demonstrates that they are inadequate for our novel task of assessing which images are unrecognizable and so cannot be captioned (discussed in Section <ref>).  Accordingly, we introduce new algorithms for this purpose, and demonstrate their advantage.





  
Efficient Creation of Large-Scale Vision Datasets. 
Progress in the vision community has largely been measured and accelerated by the creation of large-scale vision datasets over the past 20 years.  Typically, researchers have scraped images for such datasets from online image search databases <cit.>.  In doing so, they typically curate a large collection of high-quality images, since such images first passed uploaders' assessment that they are of sufficient quality to be shared publicly.  In contrast, when employing images captured “in the wild," it can be a costly, time-consuming process to identify and remove images with unrecognizable content.  Accordingly, we quantify the cost of this problem, introduce a novel problem and algorithms for deciphering when image content would be unrecognizable to a human and so should be discarded, and demonstrate the benefit of such solutions for more efficiently creating a large-scale image captioning dataset.





  
Assistive Technology for Blind Photographers.
Our work relates to the literature about technology for assisting people who are blind to take high-quality pictures <cit.>.  Already, existing solutions can assist photographers in improving the image focus <cit.>, lighting <cit.>, and composition <cit.>.  Additionally, algorithms can inform photographers whether their questions about their images can be answered <cit.> and why crowds struggle to provide answers <cit.>.  Complementing prior work, we introduce a suite of new AI problems and solutions for offering more fine-grained guidance when alerting blind photographers about what image quality issue(s) are observed.  Specifically, we introduce novel problems of (1) recognizing whether image content can be recognized (and so captioned) and (2) deciphering when a question about an image can be answered, cannot be answered because the image content is unrecognizable, or cannot be answered because the content of interest is missing from the image.  


§ VIZWIZ-QUALITYISSUES

We now describe our creation of a large-scale, human-labeled dataset to support the development of algorithms that can assess the quality of images.  We focus on a real use case that is prone to image quality issues.  Specifically, we build off of 39,181 publicly-available images <cit.> that originate from blind photographers who each submitted an image with, optionally, a question to the VizWiz mobile phone application <cit.> in order to receive descriptions of the image from remote humans.  Since blind photographers are unable to verify the quality of the images they take, the dataset exemplifies the large diversity of quality issues that occur naturally in practice.  We describe below how we create and analyze our new dataset.
 


 §.§ Creation of the Dataset

We scoped our dataset around quality issues that impede people who are blind in their daily lives.  Specifically, a clear, resounding message is that people who are blind need assistance in taking images that are sufficiently high-quality that sighted people are able to either describe them or answer questions about them <cit.>.





  
Quality Issues Taxonomy.
One quality issue label we assess is whether image content is sufficiently recognizable for sighted people to caption the images.  We also label numerous quality flaws to situate our work in relation to other papers that similarly focus on assessing image quality issues <cit.>.  Specifically, we include the following categories: blur (is the image blurry?), bright (is the image too bright?), dark (is the image too dark?), obstruction (is the scene obscured by the photographer's finger over the lens, or another unintended object?), framing (are parts of necessary items missing from the image?), rotation (does the image need to be rotated for proper viewing?), other, and no issues (there are no quality issues in the image). 





  
Image Labeling Task.
To efficiently label all images, we designed our task to run on the crowdsourcing platform Amazon Mechanical Turk.  The task interface showed an image on the left half and the instructions with user-entry fields on the right half.  First, the crowdworker was instructed to either describe the image in one sentence or click a button to flag the image as being insufficient quality to recognize the content (and so not captionable).  When the button was clicked, the image description was automatically populated with the following text: “Quality issues are too severe to recognize the visual content."  Next, the crowdworker was instructed to select all image quality flaws from a pre-defined list that are observed.  Shown were the six reasons identified above, as well as Other (OTH) linked to a free-entry text-box so other flaws could be described and None (NON) so crowd workers could specify the image had no quality flaws.  The interface enabled workers to adjust their view of the image, using the toolbar to zoom in, zoom out, pan around, or rotate the image if needed. To encourage higher quality results, the interface prevented a user from completing the task until a complete sentence was provided and at least one option from the “image quality flaw" options was chosen.  A screen shot of the user interface is shown in the Supplementary Materials.  
 




  
Crowdsourcing Labels.
To support the collection of high quality labels, we only accepted crowdworkers who previously had completed over 500 HITs with at least a 95% acceptance rate.  Also, we collected redundant results.  Specifically, we recruited five crowdworkers to label each image.  We deemed a label as valid only if at least two crowdworkers chose that label.



 §.§ Characterization of the Dataset



  
Prevalence of Quality Issues.
We first examine the frequency at which images taken by people who are blind suffer from the various quality issues to identify the (un)common reasons.  To do so, we tally how often unrecognizable images and each quality-flaw arise.  

Roughly half of the images suffer from image quality flaws (i.e., 1-P(NON)=51.6%).  We observe that the most common reasons are image blur (i.e., 41.0%) and inadequate framing (i.e., 55.6%).  In contrast, only a small portion of the images are labeled as too bright (i,e., 5.3%), too dark (5.6%), having objects obscuring the scene (3.6%), needing to be rotated for successful viewing (17.5%), or other reasons (0.8%).  The statistics reveal the most promising directions for how to improve assistive photography tools to improve blind users' experiences.  Specifically, the main functions should be focused on camera shake detection and object detection to mitigate the possibility of taking images with blur or framing flaws. 

We also observe that the image quality issues are so severe that image content is deemed unrecognizable for 14.8% of the images.  In absolute terms, this means that $3,829 and 379 hours of human annotation were wasted on employing crowdworkers to caption images that contained unrecognizable content.[Crowdworkers were paid $0.132 for each image and spent an average of 47 seconds captioning each image.]  In other words, great savings can be achieved by automatically filtering such uncaptionable images such that they are not sent to crowdworkers.  We explore this idea further in Section <ref>.





  
Likelihood Image Has Unrecognizable Content Given its Quality-Flaw.
We next examine the probability that an image's content is unrecognizable conditioned on each of the reasons for quality flaws.  Results are shown in Figure <ref>. 

Almost all reasons led to percentages that are larger than the overall percentage of unrecognizable images, which is 14.8% of all images.  This demonstrates what we intuitively suspected, which is that images with quality flaws are more likely to have unrecognizable content.  We observe that this trend is the strongest for images that suffer from obstructions (OBS) and inadequate lighting (BRT and DRK), with percentages just over 40%.

Interestingly, two categories have percentages that are smaller than the overall percentage of unrecognizable images, at 14.8% of all images.  First, images that are flagged as needing to be rotated for proper viewing (ROT) have only 8.3% deemed unrecognizable.  In retrospect, this seems understandable, as the content of images with a rotation flaw could still be recognized if viewers tilt their heads (or apply visual display tools to rotate the images).   Second, images labeled with no flaws (NON) have only 3.9% deemed unrecognizable.  This tiny amount aligns with the concept that “unrecognizable" and “no flaws" are two conflicting ideas.  Still, the fact the percentage is not 0% highlights that humans can offer different perspectives.  Put differently, the image quality assessment task can be subjective.












  
Likelihood Image Has Each Quality-Flaw Given its Content is Unrecognizable.
We next examine the probability that an image manifests each quality flaw given that its content is unrecognizable.  Results are shown in Figure <ref>.  Overall, our findings parallel those identified in the “Prevalence of Quality Issues" paragraph.  For example, we again observe the most common reasons are blurry images (71.0%) and improper framing (71.2%).  Similarly, unrecognizable images are found to be associated less frequently with the other quality flaws.





  
Relationship Between Quality Flaws in Images.
Finally, we quantify the relationship between all possible pairs of quality flaws.  In doing so, we were motivated to provide a measure that offers insight into causality and co-occurrence when comparing any pair of quality flaws, while avoiding measuring joint probabilities.  To meet this aim, we introduce a new measure which we call interrelation index I(A,B), which is defined as follows:

    I(A,B)=P(B|A)/P(B) - P(B|A̅)/P(B).

 
 More details about this measure and the motivation for it are provided in the Supplementary Materials.
 Briefly, larger positive I(A,B) values indicate that A and B tend to co-occur with A causing B to happen more often.  Results are shown in Figure <ref>.  
 
We observe that almost all quality flaws tend to occur with one another, as shown with the positive values of I.  At first, we were surprised to observe that there is a relationship between BRT and DRK (i.e., I(BRT,DRK)=73 is greater than zero), since these flaws are seemingly incompatible concepts.  However, from visual inspection of the data, we found some images indeed suffered from both lighting flaws.  We exemplify this and other quality flaw correlations in the Supplementary Materials.  From our findings, we also observe that “no flaws" does not co-occur with other quality flaws; i.e., the values in the grid are all negative for the row and column for NON.  This finding aligns with our intuition that an image labeled with NON is less likely to have a quality flaw at the same time. 


§ CLASSIFYING UNRECOGNIZABLE IMAGES


A widespread assumption when captioning images is that the image quality is good enough to recognize the image content.  Yet, people who are blind cannot verify the quality of the images they take and it is known their images can be very poor in quality <cit.>.  Accordingly, we now examine the benefit of our large-scale quality dataset for training algorithms to detect when images are unrecognizable and so not captionable.  



 §.§ Motivation: Inadequate Existing Methods

Before exploring novel algorithms, it is important to first check whether existing methods are suitable for our purposes.  Accordingly, we check whether related NR-IQA systems can detect when images are unrecognizable.  To do so, we apply five NR-IQA methods on the complete VizWiz-QualityIssues dataset: BRISQUE <cit.>, NIQE <cit.>, CNN-NRIQA <cit.>, DNN-NRIQA <cit.>, and NIMA <cit.>. The first two are popular conventional methods that rely on hand-crafted features.  The last three are based on neural networks and trained on IQA datasets mentioned in Section <ref>. For example, DNN-NRIQA-TID and DNN-NRIQA-LIVE in Figure <ref> are trained on the TID dataset and LIVE dataset, respectively.  Intuitively, if the algorithms are effective for this task, we would expect that the scores for recognizable images are distributed mostly in the high-score region, while the scores for unrecognizable images are distributed mostly in the low-score region.  

Results are shown in Figure <ref>.  A key finding is that the distributions of scores for recognizable and unrecognizable images heavily overlap.  That is, none of the methods can distinguish recognizable images from unrecognizable images in our dataset.  This finding shows that existing methods trained on existing datasets (i.e., LIVE, TID, CSIQ) are unsuitable for our novel task on the VizWiz-QualityIssues dataset.  This is possibly in part because quality issues resulting from artificial distortions, such as compression, Gaussian blur, and additive Gaussian noise, differ from natural distortions triggered by poor camera focus, lighting, framing, etc.  This also may be because there is no 1-1 mapping between scores indicating overall image quality and our proposed task, since an image with a low quality score may still have recognizable content. 



 §.§ Proposed Algorithm
 

Having observed that existing IQA methods are inadequate for our problem, we now introduce models for our novel task of assessing whether an image is recognizable.  





  
Architecture. We use ResNet-152 <cit.> to extract image features, which are then processed by 2-dimensional global-pooling followed by two fully connected layers. The final layer is a single neuron with a sigmoid activation function.[Due to space constraints, we demonstrate the effectiveness of this architecture for assessing the quality flaws in the Supplementary Materials.  The primary difference for that architecture is that we replace ResNet-152 with XceptionNet <cit.>, use three fully connected layers, and a final layer of eight neurons with eight sigmoid functions.]  We train this algorithm using an Adam optimizer with the learning rate set to 0.001 for 8 epochs.  We fix the ResNet weights pre-trained on ImageNet <cit.> and only learn the weights in the two fully connected layers.    





  
Dataset Splits.
For training and evaluation of our algorithm, we apply a 52.5%/37.5%/10% split to our dataset to create the training, validation, and test splits.  




 





  
Baselines.
We compare our algorithm to numerous baselines.  Included is random guessing, which means an image is unrecognizable with probability 0.148.  We also analyze a linear SVM that predicts with scale-invariant feature transform (SIFT) features.  Intuitively, a low-quality image should have few/no key points.  We also evaluate a linear SVM that predicts from histogram of oriented gradients (HOG) features.  





  
Evaluation Metrics.
We evaluate each method using average precision, recall, and f1 scores.  Accuracy is excluded because the distributions of unrecognizability are highly biased to “false" and such unbalanced data suffer from the accuracy paradox.





  
Results.
Results are shown in Table <ref>. We observe that both SIFT and HOG are much stronger baselines than random guessing and get high scores on precision, especially 87.2 for SIFT. However, they both get low scores on recall. This means that SIFT and HOG are good at capturing a subset of unrecognizable images but still miss many others. On the other hand, the ResNet model gets much higher recall scores while maintaining decent average precision scores, implying that it is more effective at learning the characteristics of unrecognizable images.[Again, due to space constraints, results showing prediction performance for quality flaw classification is in the Supplementary Materials.]  This is exciting since such an algorithm can be of immediate use to blind photographers, who otherwise must wait nearly two minutes to learn their image is unsuitable quality for image captioning. 



 §.§ Application: Efficient Dataset Creation
 

We now examine another potential benefit of our algorithm in helping to create a large scale training dataset.  

To support this effort, we divide the dataset into three sets.  One set is used to train our image unrecognizability algorithm.  A second set is used to train our image captioning algorithms, which we call the captioning-training-set.  The third set is used to evaluate our image captioning algorithms, which we call the captioning-evaluation-set.  

We use our method to identify which images in the captioning-training-set to use for training image captioning algorithms.  In particular, the N images flagged as recognizable are included and the remaining images are excluded.  We compare this method to three baselines, specifically training on: all images in the captioning-training-set, a random sample of N images in the captioning-training-set, a perfect sample of N images in the captioning-training-set that are known to be recognizable images.

We evaluate two state-of-art image captioning algorithms, trained independently on each training set, with respect to eight evaluation metrics:  BLEU-1-4 <cit.>, METEOR <cit.>, ROUGE-L <cit.>, CIDEr-D <cit.>, and SPICE <cit.>. 

Results are shown in Table <ref>.  Our method performs comparably to when the algorithms were trained on all images as well as the perfect set.  In contrast, our method yields improved results over the random sample.  Altogether, these findings offer promising evidence that our prediction system is successfully retaining meaningful images while removing images that are not informative for the captioning task (i.e., unrecognizable).  This reveals that a benefit of using the recognizability prediction system is to save time and money when crowdsourcing captions (by first removing unrecognizable images), without diminishing the performance of downstream trained image captioning algorithms.


§ RECOGNIZING UNANSWERABLE VISUAL QUESTIONS


The visual question “answerability" problem is to decide whether a visual question can be answered <cit.>.  Yet, as exemplified in Figure <ref>, visual questions can be unanswerable because the image is unrecognizable or because the answer to the question is missing in a recognizable image.
Towards enabling more fine-grained guidance to photographers regarding how to modify the visual question so it is answerable, we move beyond predicting whether a visual question is unanswerable <cit.> and introduce a novel problem of predicting why a visual question is unanswerable.





 §.§ Motivation

We extend the VizWiz-VQA dataset <cit.>, which labels each image-question pair as answerable or unanswerable.  We inspect how answerability relates to recognizability and each quality flaw. For convenience, we use the following notations: A: answerable, A̅: unanswerable, R: recognizable: R̅: unrecognizable, Q: quality issues, and P(·): probability function.  Results are shown in Figure <ref>. We can observe that for most quality flaws Q, P(A̅|Q) is larger than P(A̅), and P(A̅)=28.7% increases to P(A̅|R̅)=58.7%. Additionally, the probability P(R̅) increases from 14.8% to P(R̅|A̅)=30.2% when questions are known to be unanswerable. Observing that a large reason for unanswerable questions is that images are unrecognizable images, we are motivated to equip VQA systems with a function that is able to clarify why their questions are unanswerable.





 §.§ Proposed Algorithm



  
Algorithm.
Our algorithm extends the Up-Down VQA model <cit.>.  It takes as input encoded image features and a paired question.  Image features could be grid-level features extracted by ResNet-152 <cit.> as well as object-level features extracted by Faster-RCNN <cit.> or Detectron <cit.>. The input question is first encoded by a GRU cell. Then, a top-down attention module computes a weighted image feature from the encoded question representation and the input image features. The image and question features are coupled by element-wise multiplication. This coupled feature is processed by the prediction module to predict answerability and recognizability.  We employ two different activation functions at the end of the model to make the final prediction. The first one is softmax which predicts three exclusive classes: answerable, unrecognizable, and insufficient content information (answers cannot be found in images).  The second activation function is two independent sigmoids, one for answerability and the other for recognizability.  We train the network using an Adam optimizer with a learning rate of 0.001, only for the layers after feature extraction.





  
Dataset Splits.
We split VizWiz dataset into training/validation/test sets according to a 70%/20%/10% ratio.  





  
Evaluation Metrics.
We evaluate performance using average precision, precision, recall, and f1 scores, for which a simple threshold 0.5 is used to binarize probability values. For inter-model comparisons, we also report the precision-recall curve for each variant. 





  
Baselines.
For comparison, we consider a number of baselines.  One approach is the original model for predicting whether a visual question is answerable, and also employs a top-down attention model <cit.>. We also evaluate the random guessing, SIFT, and HOG baselines used to evaluate the recognizability algorithms in the previous section.





  
Results.
Results are shown in Table <ref> and Figure <ref>.  Our models perform comparably to the answerability baseline <cit.>.  This is exciting because it shows that jointly learning to predict answerability with recognizability does not degrade the performance; i.e., the average precision scores from TD+softmax and TD+sigmoid models are better than the one from the baseline <cit.> (72.6, 73.6 > 71.7) as well as the F1 scores (67.0, 68.0 > 64.8).

Our results also highlight the importance of learning to predict jointly the answerability with recognizability task (i.e., rows 5–9) over relying on more basic baselines (i.e., rows 2–4).  As shown in Table <ref>, low recall values imply that SIFT and HOG fail to capture many unrecognizable images, while our models learn image features and excel in recall and f1 scores.



Next, we compare the results from TD+softmax and TD+sigmoid.  We observe they are comparable in unanswerability prediction due to comparable average precision scores and F1 scores. For unrecognizability prediction, TD+softmax is a bit weaker than TD+sigmoid because due to slightly lower average precision and F1 scores. One reason for this may be the manual assignment of unrecognizability to false when answerability is true. Originally, 14.8% of images are unrecognizable, but after assignment, the portion drops to 8.7%. Learning from more highly biased data is a harder task, which could in part explain the weaker performance of TD+softmax model.


§ CONCLUSIONS

We introduce a new image quality assessment dataset that emerges from an authentic use case where people who are blind struggle to capture high-quality images towards learning about their visual surroundings. We demonstrate the potential of this dataset to encourage the development of new algorithms that can support real users trying to obtain image captions and answers to their visual questions.  The dataset and all code are publicly available at .




  
Acknowledgements.
We gratefully acknowledge funding support from the National Science Foundation (IIS-1755593), Microsoft, and Amazon.  We thank Nilavra Bhattacharya and the crowdworkers for their valuable contributions to creating the new dataset.



ieee





§ APPENDIX

This document supplements Sections 3 and 4 of the main paper.  In particular, it includes the following:



  * Details and motivation for quality flaw interrelation index (supplements Section 3.2). 

  * Result of quality flaw prediction (supplements Section 4.2).

  * Figures illustrating the crowdsourcing interface used to curate our labels (supplements Section 3.1), diversity of resulting unrecognizable images (supplements Section 3.2), performance of our prediction system in classifying unrecognizable images (supplements Section 4.2), and performance of the prediction of the reason for unanswerable questions (supplements Section 5.2).

  * Clarification about baselines used for Section 4.3.






§ QUALITY FLAW INTERRELATION INDEX



  
Details and motivation
The most straightforward way to explore the relation of two quality flaws A and B is to look at their the co-occurrence or their joint probability P(A,B). However, P(A,B) cannot really capture the interrelation between quality flaws. For instance, we cannot say that the relation between DRK and FRM is stronger than the one between DRK and OBS simply because of P(DRK, FRM) ≫ P(DRK, OBS). The reason for P(DRK, FRM) ≫ P(DRK, OBS) is actually due to P(FRM)=55.6%≫ P(OBS)=3.6% but has nothing to do with the interrelation of quality flaws. 

Consequently, we introduce a new measure which we call interrelation index I(A,B), which is defined as follows:

    I(A,B)=P(B|A)/P(B) - P(B|A̅)/P(B).

There are several advantages of this measure:

    
  * It measures causality from A to B: we can show that if P(A) and P(B) are both greater than zero, either P(B|A) ≥ P(B) ≥ P(B|A̅) or P(B|A) < P(B) < P(B|A̅) holds. Therefore, if I(A,B) > 0, then the existence of A must trigger B to happen more (i.e., P(B|A) ≥ P(B)) and the inexistence of A must make B happen less (i.e., P(B) ≥ P(B|A̅)), and vice versa. 
    
  * It measures co-occurence of A and B: We can show that if P(B|A) ≥ P(B) ≥ P(B|A̅), then P(A|B) ≥ P(A) ≥ P(A|B̅) (it is also true for < sign). Hence, we have I(A,B) > 0 ⇔ I(B,A) > 0. In other words, if A makes B happen more often, then B must make A happen more as well, and vice versa. 
    
  * It avoids the aforementioned problem of using joint probability. That is, if P(A) ≫ P(B), it is very likely P(A,C) ≫ P(B,C). However, which of the values of I(A,C) and I(B,C) is greater and how greater it is cannot be told from P(A) ≫ P(B).




  
Co-occurrence of DRK and BRT. Since the values I(DRK,BRT) = 74 and I(BRT,DRK) = 73 are both greater than zero, it means that the quality flaws of DRK and BRT tend to co-occur despite their contradictory concepts. Nevertheless, the examples of such images in Figure <ref> explain why this phenomenon happens. The main reason for this phenomenon is when blind people take pictures in places with poor lighting, they are not aware that the flashlights on mobile devices are turned on automatically, and therefore pictures taken are usually of dark surroundings and a bright spot. Note that this phenomenon is not captured by the joint probability of DRK and BRT, since P(DRK,BRT) = 0.53% is an extremely small value which does not manifest too much. 



  
Co-occurrence of quality flaws. We exemplify the co-occurrence of other pairs of quality flaws in Figure <ref>.











§ QUALITY FLAW PREDICTION

Performance of quality flaw classification is shown in Table <ref>. We can tell that the Xception model outperforms the random guessing baseline for each quality flaw, with respect to precision, recall, and f1 score. Furthermore, Xception predicts much better in NON, BLR and FRM flaws for large portions of the dataset. On the other hand, quality flaws that represent small portions of the dataset are prone to few-shot learning, and so learning to predict them is harder. In the extreme case of OTH, with it representing 0.8% of the data, the Xception model yields very poor scores of 9.1 and 15.8 for recall and f1 score, respectively.




§ MISCELLANEOUS


    
  * Figure <ref> illustrates the diversity of unrecognizable images that can arise from different quality flaws.
    
  * Figure <ref> shows a screen shot of the crowdsourcing interface used to collect the labels for the dataset.  
    
  * Figure <ref> shows the examples of unrecognizability prediction by the Xception model.
    
  * Figure <ref> shows the examples of the prediction of the reason for unanswerable questions. The prediction model used is “TD+sigmoid" model.





§ SECTION 4.3: CLARIFICATION ABOUT BASELINES

The two baselines, “random flag" and “perfect flag", use the same number of images from the captioning-training-set as our method for algorithm training.  That count is determined by our predictor, specifically the number of images that remain after removing all images that are deemed to be unrecognizable.  “Random flag" chooses a random sample from the captioning-training-set.  “Perfect flag" chooses images based on a ranking of images based on how many crowdworkers flag the images as unrecognizable, with selection starting from those where all five crowdworkers agreed the image is unrecognizable.






















































