our institution has a database of whole - slide images ( wsis ) used at interdepartmental clinical conferences ( i.e. , tumor boards ) and educational seminars . these wsi have great potential for education and quality assurance ; however , they must be further annotated and classified to be of greatest benefit . one potential use of these wsi is to improve rapid pattern recognition ( rpr ) . rpr or so - called  fast thinking  is a key attribute of experienced pathologists . the most established method to improve rpr among trainees is to encounter a large number of cases during residency . however , there are additional methods for improving rpr that can be used in parallel to clinical duties during residency . these techniques take advantage of preattentive processing , the phenomenon of forming a quick ( < 200 ms ) impression of an image . herein , the term rpr is used to refer to this phenomenon , although many terms are used across multiple disciplines in the literature . we envisioned a system that would take advantage of our large wsi database to improve novice trainees rpr skills . we developed a web application called novel diagnostic electronic resource ( nder ) that uses still images extracted from annotated wsi , and engages users with an adaptive learning algorithm to improve trainees rpr or our goals for this project : 
 create an efficient workflow for annotating 2000  2000 pixel regions of interest ( roi ) from a large database of wsipreserve annotation labels and create a pipeline to route images for use by ndercreate nder , a user - friendly and aesthetically pleasing web application that provides high - throughput training for rpr . create an efficient workflow for annotating 2000  2000 pixel regions of interest ( roi ) from a large database of wsi preserve annotation labels and create a pipeline to route images for use by nder create nder , a user - friendly and aesthetically pleasing web application that provides high - throughput training for rpr . we chose intraductal proliferative breast lesions ( ipls ) for our first nder module because they represent a challenging biological spectrum with important ramifications for patient care . the wsi were scanned at 20 magnification using an aperio scanscope cs ( leica biosystems imaging , inc . , vista , ca , usa ) digital whole - slide scanner . a 2-year resident , who had completed a subspecialty breast pathology rotation , annotated ( defined herein as outlining and labeling roi ) the wsi using aperio imagescope software ( leica biosystems imaging , inc . , vista , ca , usa ) . each roi was labeled with one of five categories : no intraductal proliferation ( nip ) , usual ductal hyperplasia ( udh ) , atypical ductal hyperplasia ( adh ) , and a binary classification of carcinoma in - situ defined as low to intermediate grade ductal carcinoma in - situ ( lgdcis ) and intermediate to high grade ductal carcinoma in - situ ( hgdcis ) . two methods for extracting still images and creating a large database of labeled images were compared : 
 extraction method 1 : manual extraction of still images from wsi and subsequent validation of each still image by four breast pathologistsextraction method 2 : validation of annotated regions on the wsi by a single experienced breast pathologist and automated extraction of still images tagged by diagnosis . extraction method 1 : manual extraction of still images from wsi and subsequent validation of each still image by four breast pathologists extraction method 2 : validation of annotated regions on the wsi by a single experienced breast pathologist and automated extraction of still images tagged by diagnosis . we determined that extraction method 1 would require validation by multiple pathologists because the still images are removed from the context of the slide . we used the literature as a guide to the level of pathologist agreement needed to qualify a still image as correctly classified . a study of still images of ulcerative - colitis associated dysplasia , an entity of similar diagnostic challenge to ipls , showed interobserver agreement ranging from poor to good ( kappa coefficient 0.180.54 ) . assuming a kappa at the lower end ( 0.2 ) , we estimated that four pathologists would agree on an  incorrect  classification only 2% of the time , an acceptable level of error . thus for extraction method 2 , we built an automated pipeline to extract still images from annotated wsi [ figure 1 ] . custom software was written in the codeigniter php framework ( utilizing aperio imageserver apis ) to authenticate , read annotations stored in the wsi database , and extract roi as jpeg images from the wsi . this file labeling scheme allowed us to sort the jpeg image files into a well - organized database to be used in our web application . slide annotation , whole - slide image are annotated with the following disease labels : no intraductal proliferation , usual ductal hyperplasia , atypical ductal hyperplasia , low to intermediate grade ductal carcinoma in - situ , or intermediate to high grade ductal carcinoma in - situ . an automated pipeline extracted still images from the annotated whole - slide image into a well - organized database to be used in our web application nder is a web - based application written in php and javascript with an sql database and utilizes components of the bootstrap html ( http://getbootstrap.com ) , css , and js framework for responsive display on mobile devices . the welcome page [ figure 2 ] shows the active module , an about section , and contact information . a toolbar at the top of the page allows the user to choose the module they wish to complete . after choosing the module , the user selects  begin test  and selects his / her level of training ( student , resident , fellow , etc . ) . the module begins with an untimed  pretest  consisting of twenty images with multiple - choice options corresponding to the disease categories , in this example ipls of the breast . welcome page , the welcome page shows information regarding the active module , an about section , and contact information . the toolbar at the top of the page allows the user to choose the module he / she wishes to complete after completing the the training module consists of images that are displayed for a specified amount of time , and then disappear [ figure 3 ] . the user must choose from the same multiple - choice options as in the pretest ( nip , udh , adh , lgdcis , hgdcis ) . after choosing the disease category , the image re - appears and the user are given instant feedback [ figure 4 ] . the amount of time the image displays is variable ( range : 1.510 s ) depending on the user 's performance . an adaptive algorithm decreases the image display time of each subsequent image with increasing user accuracy to maintain user engagement . training module , example screenshots from the novel diagnostic electronic resource intraductal proliferative breast lesion module . an image is displayed on the screen for a short time ( range 1.510 s ) . the image disappears after time has expired and the user must choose from the available multiple - choice options . the user is given instant feedback on their selection and can move onto the next image . a progress bar is displayed at the top of the page feedback page , after completing the pretest , training module , and posttest , a feedback page is displayed . there is a thumbnail of each image , the correct answer , and the response given . the yellow arrow indicates the thumbnail image that has been expanded the user takes an untimed twenty image  posttest  after finishing the training module , and a feedback page is displayed . the feedback page contains a thumbnail of each image , the correct answer , and the response given . rpr training lends itself well to short training sessions and is thus a good fit for mobile phones . we enabled users to take advantage of nder on their mobile phones using a responsive image css class in the bootstrap html , css , and js framework . this feature allows for seamless integration with mobile phones and tablet devices of all sizes , allowing users to train with nder anytime in any place with an internet connection . we chose intraductal proliferative breast lesions ( ipls ) for our first nder module because they represent a challenging biological spectrum with important ramifications for patient care . the wsi were scanned at 20 magnification using an aperio scanscope cs ( leica biosystems imaging , inc . , vista , ca , usa ) digital whole - slide scanner . a 2-year resident , who had completed a subspecialty breast pathology rotation , annotated ( defined herein as outlining and labeling roi ) the wsi using aperio imagescope software ( leica biosystems imaging , inc . , vista , ca , usa ) . each roi was labeled with one of five categories : no intraductal proliferation ( nip ) , usual ductal hyperplasia ( udh ) , atypical ductal hyperplasia ( adh ) , and a binary classification of carcinoma in - situ defined as low to intermediate grade ductal carcinoma in - situ ( lgdcis ) and intermediate to high grade ductal carcinoma in - situ ( hgdcis ) . two methods for extracting still images and creating a large database of labeled images were compared : 
 extraction method 1 : manual extraction of still images from wsi and subsequent validation of each still image by four breast pathologistsextraction method 2 : validation of annotated regions on the wsi by a single experienced breast pathologist and automated extraction of still images tagged by diagnosis . extraction method 1 : manual extraction of still images from wsi and subsequent validation of each still image by four breast pathologists extraction method 2 : validation of annotated regions on the wsi by a single experienced breast pathologist and automated extraction of still images tagged by diagnosis . we determined that extraction method 1 would require validation by multiple pathologists because the still images are removed from the context of the slide . we used the literature as a guide to the level of pathologist agreement needed to qualify a still image as correctly classified . a study of still images of ulcerative - colitis associated dysplasia , an entity of similar diagnostic challenge to ipls , showed interobserver agreement ranging from poor to good ( kappa coefficient 0.180.54 ) . assuming a kappa at the lower end ( 0.2 ) , we estimated that four pathologists would agree on an  incorrect  classification only 2% of the time , an acceptable level of error . thus for extraction method 2 , we built an automated pipeline to extract still images from annotated wsi [ figure 1 ] . custom software was written in the codeigniter php framework ( utilizing aperio imageserver apis ) to authenticate , read annotations stored in the wsi database , and extract roi as jpeg images from the wsi . this file labeling scheme allowed us to sort the jpeg image files into a well - organized database to be used in our web application . slide annotation , whole - slide image are annotated with the following disease labels : no intraductal proliferation , usual ductal hyperplasia , atypical ductal hyperplasia , low to intermediate grade ductal carcinoma in - situ , or intermediate to high grade ductal carcinoma in - situ . an automated pipeline extracted still images from the annotated whole - slide image into a well - organized database to be used in our web application nder is a web - based application written in php and javascript with an sql database and utilizes components of the bootstrap html ( http://getbootstrap.com ) , css , and js framework for responsive display on mobile devices . the welcome page [ figure 2 ] shows the active module , an about section , and contact information . a toolbar at the top of the page allows the user to choose the module they wish to complete . after choosing the module , the user selects  begin test  and selects his / her level of training ( student , resident , fellow , etc . ) . the module begins with an untimed  pretest  consisting of twenty images with multiple - choice options corresponding to the disease categories , in this example ipls of the breast . welcome page , the welcome page shows information regarding the active module , an about section , and contact information . the toolbar at the top of the page allows the user to choose the module he / she wishes to complete after completing the pretest ,  the user proceeds to the  training module .  the training module consists of images that are displayed for a specified amount of time , and then disappear [ figure 3 ] . the user must choose from the same multiple - choice options as in the pretest ( nip , udh , adh , lgdcis , hgdcis ) . after choosing the disease category , the image re - appears and the user are given instant feedback [ figure 4 ] . the amount of time the image displays is variable ( range : 1.510 s ) depending on the user 's performance . an adaptive algorithm decreases the image display time of each subsequent image with increasing user accuracy to maintain user engagement . training module , example screenshots from the novel diagnostic electronic resource intraductal proliferative breast lesion module . an image is displayed on the screen for a short time ( range 1.510 s ) . the image disappears after time has expired and the user must choose from the available multiple - choice options . the user is given instant feedback on their selection and can move onto the next image . a progress bar is displayed at the top of the page feedback page , after completing the pretest , training module , and posttest , a feedback page is displayed . there is a thumbnail of each image , the correct answer , and the response given . the yellow arrow indicates the thumbnail image that has been expanded the user takes an untimed twenty image  posttest  after finishing the training module , and a feedback page is displayed . the feedback page contains a thumbnail of each image , the correct answer , and the response given . rpr training lends itself well to short training sessions and is thus a good fit for mobile phones . we enabled users to take advantage of nder on their mobile phones using a responsive image css class in the bootstrap html , css , and js framework . this feature allows for seamless integration with mobile phones and tablet devices of all sizes , allowing users to train with nder anytime in any place with an internet connection . extraction method 1 : the resident performing the annotation required 2 min on average to annotate and extract each image ( 200 images in total ) . four experienced breast pathologists categorized each of the 200 images , taking on average 30 min to categorize all 200 images . ( fleiss kappa = 0.63 ) , only 101 images remained after removal of discordant images . extraction method 2 : the resident required 5 min to annotate one wsi with an average of 6.4 roi . table 1 compares the average time to annotate , extract , and validate 200 images using each method . extraction method 2 was almost an order of magnitude more efficient than extraction method 1 for creating our database of labeled jpeg images . absolute time saving was greater for the resident portion of the workflow using extraction method 2 . the resulting nder breast module takes users 2530 min to complete and displays 240 images . qualitative feedback from initial users regarding website design , usability , and interactivity has been very positive . comparison of efficiency of extraction methods table 2 displays the result of a pilot study involving ten users including four novice trainees ( medical students and junior residents ) , four senior trainees ( senior residents and fellows ) , and two attending pathologists . users of all experience levels showed increased accuracy in the posttest compared to the pretest , with novice trainees showing the largest gains . the overall accuracy improved from 53% pretest to 83% posttest , a statistically significant result ( p < 0.0001 , paired t - test ) . the effect size ( cohen 's d = 2.11 ) was very large , where 0.2 is a small effect , 0.5 moderate , and 0.8 large . a 2530 min high - yield nder training module can be created in roughly 3 h ( resident : two hours , attending : one hour ) . the training module can be accessed via desktop computer , tablet device , or mobile phone web browsers and responds to device screen size . a pilot study of ten users demonstrated large increases in accuracy from pre- to post - test . qualitative feedback from initial users has been positive , focusing on intuitive design , usability , and responsive display features . discussion at the double - headed microscope , didactic lectures , and textbooks are common methods for teaching pathology . thinking . the only current strategy for teaching  fast  thinking or rpr is seeing a large volume of cases during residency . nder is a novel tool for rpr to be used in parallel to traditional residency training . previous studies using electrocardiograms have shown that a combination of pattern - based and feature - based instruction performs better than either method alone . we believe nder , in combination with traditional methods , will enhance residency education through improvement of residents histopathologic intuition . there are many high - quality digital tools developed for pathology training including examples in medical education , histology , and cytopathology . the closest analog to nder is the excellent introductory histopathology training module developed by krasne et al . this tool shows images of basic pathologic processes , forces users to choose an answer within 24 s , and gives immediate , succinct feedback . in contrast , nder forces the user to choose an answer in as little as 1.5 s or at most 10 s , depending on performance . the brief time period forces users to focus solely on improving the accuracy of their initial impression of the image . improved accuracy of this initial impression , or intuition , has never been targeted for pathology training by a digital tool ( to our knowledge ) . an additional advantage of the brief period is an increased volume of images viewed by the user . for example , we are currently creating a module for breast prognostic / predictive immunohistochemical markers in hopes of decreasing both inter- and intra - observer variability in the interpretation of 2013 cap / asco human epidermal growth receptor 2 ( her2 ) immunohistochemistry ( ihc ) guidelines . current evidence suggests an increase in the number of  equivocal 2 +  interpretations since the release of the revised guidelines in 2013 , some of which may be related to overinterpretation . we hope nder will help train pathologists to accurately separate ihc her2 protein expression into the appropriate category ( 0 , 1 + , 2 + , 3 + ) as defined by the guidelines . such a module would decrease costs for pathology departments by decreasing the number of unnecessary reflex her2 fluorescence in situ hybridization tests in cases better interpreted as either negative or positive . additional quality assurance and improvement applications are being explored , of which the possibilities are essentially endless . one current limitation of nder includes the use of still images rather than wsi for the pre- and post - test phases of the module . we have begun work on integrating wsi into the pre- and post - test phases for a more accurate assessment of actual pathology practice . incorporation of wsi will investigate how context affects interpretation when users can evaluate an entire slide rather than an isolated static image . the addition of wsi will be of particular interest in borderline lesions such as the distinction of adh and lgdcis , in which size and span may be the limiting factor in differentiating these two entities . in the future , we aim to create an open - access version of nder available to anyone with an internet connection . nder relies on a well - curated and annotated database of wsi , which is available to users at only a select group of institutions . we envision a system where collaborators from across the world can contribute annotated wsi for the creation of a broad range and depth of modules . nder is a novel web app that uses annotated wsi to rapidly improve pattern recognition and evaluate for diagnostic proficiency . it has wide applicability to education , training , and quality assurance . in the near future , we envision an open - access platform where nder modules can be created from annotated wsi from collaborators across the world , including experts in their fields .
context : whole - slide images ( wsis ) present a rich source of information for education , training , and quality assurance . 
 however , they are often used in a fashion similar to glass slides rather than in novel ways that leverage the advantages of wsi . 
 we have created a pipeline to transform annotated wsi into pattern recognition training , and quality assurance web application called novel diagnostic electronic resource ( nder).aims : create an efficient workflow for extracting annotated wsi for use by nder , an attractive web application that provides high - throughput training.materials and methods : wsi were annotated by a resident and classified into five categories . 
 two methods of extracting images and creating image databases were compared . 
 extraction method 1 : manual extraction of still images and validation of each image by four breast pathologists . 
 extraction method 2 : validation of annotated regions on the wsi by a single experienced breast pathologist and automated extraction of still images tagged by diagnosis . 
 the extracted still images were used by nder . 
 nder briefly displays an image , requires users to classify the image after time has expired , then gives users immediate feedback.results:the nder workflow is efficient : annotation of a wsi requires 5 min and validation by an expert pathologist requires an additional one to 2 min . 
 the pipeline is highly automated , with only annotation and validation requiring human input . 
 nder effectively displays hundreds of high - quality , high - resolution images and provides immediate feedback to users during a 30 min session.conclusions:nder efficiently uses annotated wsi to rapidly increase pattern recognition and evaluate for diagnostic proficiency .