while this information may include meta - data about the image , such as how , when , or where the image was acquired , the majority of image information is encoded in the pixel data . observational or computational descriptions of image features ( including their spatial coordinates ) can be attached to an image , though there is no standard mechanism for doing so in health care . moreover , the majority of the human observed image feature descriptions are captured only as free text . this free text is often not associated with the spatial location of the feature , making it difficult to relate image observations to their corresponding image locations . it is difficult for both humans and machines to index , query , and search free text in order to retrieve images or their features based on these free text descriptions . this limits the value of image data and its interpretation for clinical , research , and teaching purposes . the mission of the national institutes of health s ( nih ) national cancer institute s ( nci ) cancer bioinformatics grid ( cabig)1 is to provide infrastructure for creating , communicating , and sharing bioinformatics tools , data and research results , using shared data standards and shared data models . imaging , critical to cancer research , lies at an almost unique juncture in the translational spectrum between research and clinical practice . image and annotation information obtained , for example , in cancer clinical trial research , is collected in a clinical setting of commercial information systems . the latter adhere to standards , such as dicom2 and hl73 , and technical frameworks such as integrating the healthcare enterprise ( ihe)4 . image annotation , in particular , needs to be available in a standard format that is at the same time syntactically and semantically interoperable with the infrastructure of cabig while supporting the widespread clinical health care standards , such as dicom and hl7 . at the same time , this standard for annotation must support recommendations being developed for the world wide web community , for example , the semantic web , in order to leverage the full spectrum of cancer biomedical image annotation , whether it exists in commercial pacs systems or in cyberspace . selecting a single standard format to store image annotations will streamline software development and enable the work to focus on providing rich annotation features and functionality . designing the tools to be compatible with other standards will enable a high degree of interoperability and allow the incorporation of the annotation standard into commercial , clinical , information systems . such interoperability has the potential to open many existing resources and databases of cancer - related image data and metadata for exploitation not only by cabig , but by the broader research and clinical radiology community . this paper describes the use cases and requirements for annotation and image markup and the resulting aim information model . a free and open source software library that can be used to create and work with instances of aim is available . the software , schemas , and templates necessary to serialize these aim instances as dicom s / r , aim xml and hl7 clinical document architecture ( cda ) can be downloaded from http://cabig.nci.nih.gov/tools/aim . the project began with use case development and requirements elicitation from the in vivo imaging workspace of the cabig project . the workspace consists of a set of subject matter experts , typically scientists working in imaging informatics related to clinical and clinical trials research . these scientists are supplemented by a large community of other research imaging users , developers , and scientists . requirements were derived from the use cases and a formal identification and reconciliation with other efforts and standards for image annotation and markup . in particular , we examined the requirements of the lung image database consortium ( lidc)5 , the response evaluation criteria in solid tumors ( recist ) standard6 , dicom sr7,8 , and hl7 cda . the requirements for teaching file image annotation were captured from radpix ( weadock software , llc , ann arbor , mi , usa ) and the medical image resource center ( mirc)9 from the radiological society of north america ( rsna ) . we also examined the specification for semantic image annotation of the world wide web consortium ( w3c)10 . the aim model was created in unified modeling language ( uml ) using enterprise architect ( ea ; sparx systems , creswick , victoria , australia ) . the uml model was imported into an ontology developed in protg ( stanford university , palo alto , ca , usa ) . the ontology included controlled terminology for anatomic entities , imaging observations and imaging observation characteristics from the radiology lexicon , radlex11 . xml schemata for the aim project were created in a three - step process . when a version of the aim uml model was considered mature , this file was used as input to the cacore software development kit ( sdk ) version 3.2.112 ( cabig project , national cancer institute , bethesda , md , usa ) which translated it into an xml schema . this schema was modified by hand with altova xml spy 2008 ( altova , beverly , ma , usa ) to remove certain incompatibilities generated by the cacore package . many of such issues will be addressed in subsequent cacore software releases so that future translations to xml schema can be simpler . the aim dicom sr template was constructed from the aim model and using the dicom comprehensive sr information object definition1 . we also used the mammography cad dicom sr content tree and the dicom sr cancer clinical trials results6,7 as guiding examples of how to create the sr object for aim . the aim software library was developed in standard c++ using microsoft visual studio 2005 ( microsoft , redmond , wa , usa ) , the standard template library ( stl ) ( silicon graphics , sunnyvale , ca , usa ) , the apache xerces xml library version 2.4.0 ( http://xerces.apache.org ) and the boost c++ library version 1.34.1 ( http://www.boost.org ) . dicom functionality was provided by dcmtk version 3.5.4 ( offis e. v. , oldenburg , germany ) . some software components were developed in java jdk 6 ( sun microsystems , santa clara , ca , usa ) using the eclipse ( http://www.eclipse.org ) integrated development environment . the aim library creates an object model , using a c++ class for each class in the aim schema . each class in the object model provides mutator methods ( set and get methods ) for every attribute in the corresponding aim schema class . thus , the whole aim schema is represented by the object model through containment and inheritance . the set of object model operations supported by the aim library includes serializing the model as xml , dicom sr , or hl7 cda format . the reverse set of operations of reading xml , dicom s / r , and hl7 cda instances into the object model is supported as well . the design of the aim information model was motivated by a key use case for the cancer imaging community : the image - based clinical trial . prerequisites : 
 one series of dicom images from one study of one patient at a first time point.one series of dicom images from one study of same patient at a second time point . one series of dicom images from one study of one patient at a first time point . one series of dicom images from one study of same patient at a second time point . use case : 
 observer 1 annotates observations in images from first time point.observer 1 measures each observation.software application assigns unique identifier to each image annotation.observer 1 assigns human readable name to each image annotation.for each named image annotation from observer 1 , observer 2 ( blinded to observer 1 s result or not ) annotates the same observation in the images from the second time point.observer 2 measures each observation.researcher retrieves annotations from observer1 and observer 2.researcher performs calculations on measurements from both observers.researcher creates annotation on the collection of image annotations to document change . observer 1 assigns human readable name to each image annotation . for each named image annotation from observer 1 , observer 2 ( blinded to observer 1 s result or not ) annotates the same observation in the images from the second time point . degenerate use cases include a clinician documenting observations in routine clinical work or an instructor documenting observations in a teaching case . the requirements , derived from this use case and the collective experience of the workspace , were legion . at a highest level , the aim model encompasses patient , observer , equipment , image , anatomic entities , observations and observation characteristics , two- and three - dimensional coordinates of defined geometric shapes , markup and text annotation , comments , and defined and arbitrary ( defined by an application ) calculations as well as the ability to store arbitrary calculation results . as introduced above , an annotation is explanatory or descriptive information , generated by humans or machines , directly related to the content of a referenced image or images . annotations become a collection of image semantic content that can be used for data mining purposes . markups can be used to depict textual information and regions - of - interest visually along side of an image . computation results may be created from image markup or other comparison methods in order to provide more meaningful information about images . the model is organized into logical groups of classes relating to specific information model components . the latter uses the referencedannotation class , which has a uid of a referenced imageannotation or annotationofannotation . these are the two root objects that inherit all properties of the abstract class annotation . each root object has a type attribute that is used to identify what kind of imageannotation or an annotationofannotation is being instantiated . the type of annotation is used to retrieve type - specific constraints from the ontology13 . annotation class also captures the name and general description of the aim annotation , the version of aim schema , creation date and time , and the annotation unique identifier ( uid ) . fig  1aim uml class diagram.fig  2general information group.table  1types of annotationsimageannotationrecist baseline target lesionrecist baseline non - target lesionrecist follow - up target lesionrecist follow - up non - target lesionlidc chest ct nodulebrain tumor baseline target lesionbrain tumor follow - up target lesionteachingquality controlclinical findingotherannotationofannotationrecist baseline sum of longest diameterrecist follow - up sum of longest diameterinterval changesummary statisticother aim uml class diagram . anatomicentity , imagingobservation , and imagingobservationcharacteristic classes , figure 3 , are the key classes used to gather the respective features of the particular annotation . the possible values that can be populated in these classes come from controlled terminologies and are generated by the aim library through its api to the aim ontology in protg . the term must be from a recognized controlled vocabulary ( radlex11 , snomed - ct14 , umls15 , etc ) . for example ,  mass ,   opacity ,   foreign body ,   artifact  are all examples of imagingobservation . so , for example ,  spiculated  ontologically is_a kind of  margin  and it might , therefore , be an imagingoberservationcharacteristic for an imagingobservation of  mass .  a new class , inference , pleural effusion ,  for example , is a disease which in imaging is an inference from a collection of imagingobservations , imagingobservationcharacteristics , and anatomy . it allows the user to define an observation and then assign to each pixel in the region of interest , a probability that that pixel represents the observation . for example , consider a rectangular region of interest of tissue and a software application that computes the probability that each pixel is part of the tumor . another example is a tissue where each pixel is inhomogeneous and an estimate of the contribution of each component is made ; a pixel , for example , that is 70% soft tissue and 30% fat . the probabilitymap class contains references to its own instance uid and referenced instance uid of the image to which the probability map is applied to . an instance of probabilitymap must associate with imagingobservation , which describes what the probabilitymap is specifying . the user class is composed of the user s full name , a user login name , and an optional role and identifier within a clinical trial ( for example ,  reader 37  ) . the user class could alternatively record information about an algorithm or other automated method that created the annotation . the equipment class provides information about the system that is used to create the aim annotations . the patient class contains basic patient demographic information : patient name , local medical record number , birth date , and sex . these may be de - identified pseudonyms at the discretion of the software application creating the aim . the image reference classes specify the image or collection of images being annotated , which can be seen in figure 4 . we mean stand - alone images in common lay formats that may or may not be local to the application . web images , whether local or remote , are referenced by uniform resource identifier ( uri ) . 
 fig  4image reference group it has one study object that has one or more series objects which in turn have one or more image objects . image has sop class uid , sop instance uid , image laterality , patient orientation , pixel spacing , and acquisition date time . imageview is a sequence that describes the projection of the anatomic region of interest on the image receptor . the markup group , figure 5 , captures textual information and the graphical representation of the markup . the coordinates of a textannotation are captured as one of the markup graphic types , namely multipoint . textannotation s multipoint is expected to have no more than two coordinates which represent beginning and ending points of a line connecting textannotation to a point on an image . the other available graphic types are point , polyline , circle , and ellipse , based on existing graphic types in dicom sr7 . this is supported in the model by having an attribute of each markup specify its inclusion in the annotation . each graphic type has appropriate spatialcoordinate abstract classes , which can be in two- or three - dimensional space . twodspatialcoordinate has x and y coordinates , the sop instance uid of the image that contains the pixel and the frame number within the referenced sop instance to which the markup applies . threedspatialcoordinate has x , y , and z coordinates as well as the dicom frame of reference for a series . the calculation group , shown in figure 6 , represents how calculation results are stored in an annotation . calculation results may or may not be directly associated with graphical symbols or markups . in a simple example , given an image with a single ellipse markup , calculation results might include area in square millimeters , and maximum and minimum pixel values . arbitrary calculations are defined by the application and free text description of the calculation can be included . a mathml16 representation of the algorithm the referencedcalculation class is used as a mechanism for referencing other calculations that the current calculation may be based upon . the calculationresult class is used to store the results of calculations . in order to store a wide variety of results , it contains the total number of dimensions to the result , a string representation of the universal common units of measurement ( ucum)17 units for each of the dimensions , and the type of result such as scalar , vector , histogram , or array . the dimension class describes each of the dimensions of the result , the index of the current dimension and the size of the dimension . for example , a calculationresult of an area would have a scalar result type with one dimension , one  mm  unit and one data value . the primary use case is the capture of measurements made by observers in clinical trials to facilitate the use of imaging as a biomarker . the same model can , however , support a broad variety of image annotations for clinical , teaching , and other research purposes . administrative aim annotations , such as for documenting imaging artifacts and other quality control issues , are being considered . the national cancer institute makes large collections of research images available to the imaging community through its national cancer imaging archive ( ncia ) web site ( http://ncia.nci.nih.gov/ ) . we anticipate the development of large collections of aim annotations to accompany these large image collections . in leveraging the methodologies of the cabig project this means that all appropriate data collection fields and attributes of data objects in aim use controlled terminologies reviewed and validated by the cabig vocabularies and common data elements workspace ( vcde workspace ) . common data elements ( cdes ) built from controlled terminologies and according to practices validated by the vcde workspace are used throughout the aim model . these cdes are registered as iso / iec 11179 standard metadata components in the cabig context of the cancer data standards repository ( cadsr)18 . the aim model expressed in uml as class diagrams and as xmi files , since the model was developed using this cabig methodology , other cabig tools can be used to develop software that makes use of aim annotations . other cabig projects , for example , querying of aim annotations , will leverage this work . in the not too distant future , it will be possible to query a set of distributed cagrid services for ,  find all the ct images of the lung containing a spiculated mass with a longest diameter between 5 and 7  mm and having decreased in size by at least 10% from prior measurement . lastly , the aim software not only creates aim instances , but it can serialize them for storage and transmission as xml , dicom s / r , and hl7 cda . most cabig tools will use the native aim xml as part of grid service payloads for a variety of purposes . aim xml can also be easily incorporated into other xml standards such as mirc documents . research imaging software such as the cabig in vivo imaging workspace s extensible imaging platform ( xip)19 also makes direct use of the xml schemata . most clinical and clinical research users do their image annotation and measurement work on commercial clinical and research software . several vendors have expressed interest in the aim model and software . in their case , the vendors have the choice of implementing aim annotations as xml or as dicom s / r . s / r will likely become another dicom s / r object similar to the mammography or chest cad1 objects , dicom part 17 . vendors of commercial picture archiving and communication systems ( pacs ) workstations will then have the choice of implementing aim as either a dicom s / r or an xml object depending on their integration strategies . regardless of their choice , our aim toolkit will enable conversion between these formats and interoperability among imaging systems . it is important for purchasers of clinical imaging equipment to recognize the value in standardizing image annotation formats . as aim becomes widely adopted for image annotation , the community will benefit if purchasers contractually require support for aim annotation in new image display and manipulation equipment . the aim project defines an information model for image annotation and markup in health care . although useful when annotating images for clinical and teaching purposes , standardized image annotation and markup is most critical in clinical trials . this is especially true in multicenter clinical trials where data collection and analysis spans multiple investigators and institutions . as a silver compliant product of the cabig methodology the aim software provides developers a toolkit that will enable them to adopt aim in their applications . aim provides a foundation for standardized image annotation practice in the clinical , research , and translational communities .
image annotation and markup are at the core of medical interpretation in both the clinical and the research setting . 
 digital medical images are managed with the dicom standard format . while dicom contains a large amount of meta - data about whom 
 , where , and how the image was acquired , dicom says little about the content or meaning of the pixel data . 
 an image annotation is the explanatory or descriptive information about the pixel data of an image that is generated by a human or machine observer . 
 an image markup is the graphical symbols placed over the image to depict an annotation . while dicom is the standard for medical image acquisition , manipulation , transmission , storage , and display , there are no standards for image annotation and markup . 
 many systems expect annotation to be reported verbally , while markups are stored in graphical overlays or proprietary formats . 
 this makes it difficult to extract and compute with both of them . 
 the goal of the annotation and image markup ( aim ) project is to develop a mechanism , for modeling , capturing , and serializing image annotation and markup data that can be adopted as a standard by the medical imaging community . 
 the aim project produces both human- and machine - readable artifacts . 
 this paper describes the aim information model , schemas , software libraries , and tools so as to prepare researchers and developers for their use of aim .