Most publicly available image quality databases have been created under highly controlled conditions by introducing graded simulated distortions onto high-quality photographs. However, images captured using typical real-world mobile camera devices are usually afflicted by complex mixtures of multiple distortions, which are not necessarily well-modeled by the synthetic distortions found in existing databases. The originators of existing legacy databases usually conducted human psychometric studies to obtain statistically meaningful sets of human opinion scores on images in a stringently controlled visual environment, resulting in small data collections relative to other kinds of image analysis databases. Toward overcoming these limitations, we designed and created a new database that we call the LIVE In the Wild Image Quality Challenge Database, which contains widely diverse authentic image distortions on a large number of images captured using a representative variety of modern mobile devices. We also designed and implemented a new online crowdsourcing system, which we have used to conduct a very large-scale, multi-month image quality assessment (IQA) subjective study. Our database consists of over 350 000 opinion scores on 1162 images evaluated by over 8100 unique human observers. Despite the lack of control over the experimental environments of the numerous study participants, we demonstrate excellent internal consistency of the subjective data set. We also evaluate several top-performing blind IQA algorithms on it and present insights on how the mixtures of distortions challenge both end users as well as automatic perceptual quality prediction models. The new database is available for public use at http: live.ece.utexas.edu research ChallengeDB index.html .
@cite_7: 
@cite_3: Visual quality of color images is an important aspect in various applications of digital image processing and multimedia. A large number of visual quality metrics (indices) has been proposed recently. In order to assess their reliability, several databases of color images with various sets of distortions have been exploited. Here we present a new database called TID2013 that contains a larger number of images. Compared to its predecessor TID2008, seven new types and one more level of distortions are included. The need for considering these new types of distortions is briefly described. Besides, preliminary results of experiments with a large number of volunteers for determining the mean opinion score (MOS) are presented. Spearman and Kendall rank order correlation factors between MOS and a set of popular metrics are calculated and presented. Their analysis shows that adequateness of the existing metrics is worth improving. Special attention is to be paid to accounting for color information and observers focus of attention to locally active areas in images.
@cite_23: Measurement of visual quality is of fundamental importance for numerous image and video processing applications, where the goal of quality assessment (QA) algorithms is to automatically assess the quality of images or videos in agreement with human quality judgments. Over the years, many researchers have taken different approaches to the problem and have contributed significant research in this area and claim to have made progress in their respective domains. It is important to evaluate the performance of these algorithms in a comparative setting and analyze the strengths and weaknesses of these methods. In this paper, we present results of an extensive subjective quality assessment study in which a total of 779 distorted images were evaluated by about two dozen human subjects. The "ground truth" image quality data obtained from about 25 000 individual human quality judgments is used to evaluate the performance of several prominent full-reference image quality assessment algorithms. To the best of our knowledge, apart from video quality studies conducted by the Video Quality Experts Group, the study presented in this paper is the largest subjective image quality study in the literature in terms of number of images, distortion types, and number of human judgments per image. Moreover, we have made the data from the study freely available to the research community . This would allow other researchers to easily report comparative results in the future
@cite_49: A number of quality measures are evaluated for gray scale image compression. They are all bivariate, exploiting the differences between corresponding pixels in the original and degraded images. It is shown that although some numerical measures correlate well with the observers' response for a given compression technique, they are not reliable for an evaluation across different techniques. A graphical measure called Hosaka plots, however, can be used to appropriately specify not only the amount, but also the type of degradation in reconstructed images.
@cite_5: 
@cite_51: This paper considers the use of the image quality metric for still image compression systems comparison. Considering the fact that the conventional PSNR cannot sufficiently reflect the result of subjective assessment other quality measures have been considered to design the variable bit-rate coders. Distortion measures are developed for the purpose of predicting the subjective quality of pictures. Subjectively relevant distortion measures that mirror viewers' assessments of picture quality would make the task of designing and optimizing coding schemes considerably easier. One significant problem associated with these measures is that there is little information on how these metrics perform in comparison to each other. The purpose of this paper is to provide a rigorous evaluation of three metrics to assess the quality of compressed images. The compression system used in this evaluation is the classical JPEG coder. Both objective and subjective tests were performed on a 50 natural image data base with a panel of experimented and non-experimented observers. The results show that these metrics are highly correlated with the subjective quality grading but also depends on the complexity of the images under study. They can be improved by integrating all masking effects and typical artifacts of the compression method. Their combination as a multidimensional quality measure in which each dimension is related to a properly identified artifact, should be based on a subjective evaluation.
Most of the top-performing IQA models (full, reduced, and no-reference) have been extensively evaluated on two benchmark databases: the LIVE IQA Database which was designed in @math and the TID2008 Database, designed and released in @math . The LIVE IQA Database, one of the first comprehensive IQA databases, consists of @math images, much larger than the small databases that existed at the time of its introduction @cite_49 - @cite_51 . This legacy database contains @math pristine reference images and models five distortion types - jp2k, jpeg, Gaussian blur, white noise, and fast fading noise @cite_23 . The TID2008 Database is larger, consisting of @math reference and @math distorted images over @math distortion categories. TID2013 @cite_3 is a very recently introduced image quality database with an end goal to include the peculiarities of color distortions in addition to the 17 simulated spatial distortions included in TID2008. It consists of 3000 images and includes seven new types of distortions, thus modeling a total of 24 distortions. We refer the reader to @cite_5 @cite_7 for more details on the categories and severities of image distortions contained in this database.