Despite tremendous progress in computer vision, there has not been an attempt for machine learning on very large-scale medical image databases. We present an interleaved text image deep learning system to extract and mine the semantic interactions of radiology images and reports from a national research hospital's Picture Archiving and Communication System. With natural language processing, we mine a collection of representative 216K two-dimensional key images selected by clinicians for diagnostic reference, and match the images with their descriptions in an automated manner. Our system interleaves between unsupervised learning and supervised learning on document- and sentence-level text collections, to generate semantic labels and to predict them given an image. Given an image of a patient scan, semantic topics in radiology levels are predicted, and associated key-words are generated. Also, a number of frequent disease types are detected as present or absent, to provide more specific interpretation of a patient scan. This shows the potential of large-scale learning and prediction in electronic patient records available in most modern clinical institutions.
@cite_4: We consider the problem of modeling annotated data---data with multiple types where the instance of one type (such as a caption) serves as a description of the other type (such as an image). We describe three hierarchical probabilistic mixture models which aim to describe such data, culminating in correspondence latent Dirichlet allocation, a latent variable model that is effective at modeling the joint distribution of both types and the conditional distribution of the annotation given the primary type. We conduct experiments on the Corel database of images and captions, assessing performance in terms of held-out likelihood, automatic annotation, and text-based image retrieval.
@cite_51: In this paper, we describe three different methods for the classification and annotation of medical radiographs. The methods were applied in the medical image annotation tasks of ImageCLEF in 2005, 2006, and 2007. Image annotation can be used to access and find images in a database using textual queries when no textual image description is available. One of the methods is a non-linear model taking into account local image deformations to compare images which are then classified using the nearest neighbour decision rule. The other two methods use local image descriptors for a bag-of-features approach. The bags of local image features are classified using discriminative classifiers. Our methods performed best in the 2005 and 2006 evaluations and second best in 2007.
@cite_33: We present a new approach for modeling multi-modal data sets, focusing on the specific case of segmented images with associated text. Learning the joint distribution of image regions and words has many applications. We consider in detail predicting words associated with whole images (auto-annotation) and corresponding to particular image regions (region naming). Auto-annotation might help organize and access large collections of images. Region naming is a model of object recognition as a process of translating image regions to words, much as one might translate from one language to another. Learning the relationships between image regions and semantic correlates (words) is an interesting example of multi-modal data mining, particularly because it is typically hard to apply data mining techniques to collections of images. We develop a number of models for the joint distribution of image regions and words, including several which explicitly learn the correspondence between regions and words. We study multi-modal and correspondence extensions to Hofmann's hierarchical clustering aspect model, a translation model adapted from statistical machine translation (), and a multi-modal extension to mixture of latent Dirichlet allocation (MoM-LDA). All models are assessed using a large collection of annotated images of real scenes. We study in depth the difficult problem of measuring performance. For the annotation task, we look at prediction performance on held out data. We present three alternative measures, oriented toward different types of task. Measuring the performance of correspondence methods is harder, because one must determine whether a word has been placed on the right region of an image. We can use annotation performance as a proxy measure, but accurate measurement requires hand labeled data, and thus must occur on a smaller scale. We show results using both an annotation proxy, and manually labeled data.
@cite_3: In this paper we compare a variety of unsupervised probabilistic models used to represent a data set consisting of textual and image information. We show that those based on latent Dirichlet allocation (LDA) out perform traditional mixture models in likelihood comparison. The data set is taken from radiology; a combination of medical images and consultants reports. The task of learning to classify individual tissue, or disease types, requires expert hand labeled data. This is both: expensive to produce and prone to inconsistencies in labeling. Here we present methods that require no hand labeling and also automatically discover sub-types of disease. The learnt models can be used for both prediction and classification of new unseen data.
The ImageCLEF medical image annotation tasks of 2005-2007 by @cite_51 have 9,000 training and 1,000 testing two-dimensional images, converted to @math pixel thumbnails with @math labels. Local image descriptors and intensity histograms are used as a bag-of-features approach in that work for this scene-recognition-like problem. Unsupervised LDA based matching from lung disease words (e.g., fibrosis, emphysema) to two-dimensional image blocks from axial CT chest scans of 24 patients is studied by @cite_3 . The works of @cite_33 @cite_4 using generative models of combining words and images under a very limited word image vocabulary has also motivated this study.