Synthesising Wider Field Images from Narrow-Field Retinal Video Acquired Using a Low-Cost Direct Ophthalmoscope (Arclight) Attached to a Smartphone
Keylor Daniel Chaves Viquez Ognjen Arandjelovic Andrew Blaikie In Ae Hwang































V>c<height 4./imgs/BICICCV-015
empty

Synthesising Wider Field Images from Narrow-Field Retinal Video Acquired Using a Low-Cost Direct Ophthalmoscope (Arclight) Attached to a Smartphone
    
  Keylor Daniel Chaves Viquez^a     Ognjen Arandjelović^† a     Andrew Blaikie^b    In Ae Hwang^b

^aSchool of Computer Science    ^bSchool of Medicine


University of St Andrews

Scotland, United Kingdom
^†
    December 30, 2023
====================================================================================================================================================================================================================



Access to low cost retinal imaging devices in low and middle income countries is limited, compromising progress in preventing needless blindness. The Arclight is a recently developed low-cost solar powered direct ophthalmoscope which can be attached to the camera of a smartphone to acquire retinal images and video. However, the acquired data is inherently limited by the optics of direct ophthalmoscopy, resulting in a narrow field of view with associated corneal reflections, limiting its usefulness. In this work we describe the first fully automatic method utilizing videos acquired using the Arclight attached to a mobile phone camera to create wider view, higher quality still images comparable with images obtained using much more expensive and bulky dedicated traditional retinal cameras.


§ INTRODUCTION

It is estimated that some 285 million people in the world visually impaired, with the majority of cases potentially preventable or treatable if diagnosed promptly <cit.>. Unfortunately the burden of disease is greatest in low and middle income countries (East Asia, Latin America, and Africa) where access to diagnostic devices presents a major challenge <cit.>. Observation of the retina by an eye care professional using an ophthalmoscope or through acquiring a video or photograph allows screening for key blinding diseases such as diabetic retinopathy, glaucoma and retinopathy of prematurity <cit.>.



However, one of the key barriers to fundal screening in low and middle income countries (LMICs) emerges from the lack of access to affordable retinal imaging equipment <cit.>. The Arclight direct ophthalmoscope has been recently developed as a means of addressing this challenge <cit.>. The device, despite its simplified design and low cost, has been shown to be as effective as more expensive traditional devices <cit.>. In addition, the Arclight can be attached to the camera of a mobile phone and acquire a digital recording of the fundus for the purposes of telemedicine. However, the captured images are intrinsically limited to a narrow field of view by the optics of direct ophthalmoscopy. Other limiting factors relating to the optics, sensor, and compression software of mobile phone cameras can further reduce the quality of the acquired data in comparison to more expensive dedicated traditional retinal cameras <cit.>.



§ CONTEXT AND PRACTICAL CHALLENGES

Traditionally, retinal images (see Figure <ref>) are acquired using expensive and bulky tabletop units operated by trained technicians in a hospital setting  <cit.>. Consequently such devices are impractical for use in low income areas where resources to buy and maintain traditional cameras are limited and robustness and portability is key for allowing the sharing and movement between inhospitable sites <cit.>.

However, the availability of low cost yet powerful smartphones in LMICs has led to the development of `m-health' <cit.> whereby mobile phones can play several roles: contacting and reminding patients to take medicines and attend appointments, transferring money to pay for health care, storing guidelines and diagnostic algorithms, acquiring and archiving audio and visual information from patients, analysing data, utilizisng GPS tracking, and transferring information wirelessly.  These approaches <cit.> have quickly been adopted in low resource health care settings where the portability and connectivity of smartphones <cit.> has made them particularly valuable in telemedicine <cit.>, making it possible to reach patients and communities that currently receive suboptimal care due to geographical and financial barriers <cit.>.



 §.§ The Arclight device

The Arclight is a highly portable (dimensions: 110mm ×26mm ×9mm, mass: 18g) low-cost ( 5 if purchased in high volume[See <https://iapb.standardlist.org/get-arclight/>]) ophthalmoscope that has been developed in response to the need for affordable diagnostic eye care tools in LMICs <cit.>. Uniquely the device is illuminated by an LED allowing it to be powered by a slim lithium battery that is charged by an integrated solar panel reducing the need for expensive and hard to find consumables. Employing a small LED allows simplification of the design of the device reducing manufacturing costs and its overall size, see Figure <ref><cit.>. The novel features of the Arclight make it ideal for use in mobile clinics and remote low resource settings with limited access to power. Consequently over 10000 devices have been distributed, enabling thousands of health care workers worldwide to perform fundal examinations for the first time <cit.>.



Studies have shown that the Arclight is just as accurate for screening for signs of diabetic retinopathy and glaucoma as more expensive traditional devices when used manually by an eye care professional <cit.>. However, one of the key inherent limitations of any direct ophthalmoscopy device is the narrow retinal field of view. Single still images are typically of limited utility but a sweeping video of a greater area of the fundus may offer greater clinical information, benefiting a telemedicine approach. Still, transferring video files is challenging in remote areas of a low income country where fast and reliable broadband internet may not be available. Synthesising a single wider field still image from multiple frames of a video fused together is a promising solution for overcoming many of the limitations of the optics of direct ophthalmoscopy as well as the data-heavy nature of video.



§ PROPOSED METHOD

In this section we describe a novel method for processing narrow field retinal video acquired using the Arclight ophthalmoscope while attached to an Apple iPhone 6s. By fusing the information contained in different frames of a video we synthesise a wider field image of higher quality, which is easier to interpret clinically and more compact to transmit and share. We start with an overview of the method and then follow up with detailed descriptions of each of the processing steps.



 §.§ Overview

As we noted earlier, the quality and useful information content of different frames in a video varies greatly. Hence, in order to ensure robustness, our algorithm starts by automatically identifying the highest quality frame. Tracking, and thereafter incremental image stitching, commences both in forward and backward directions in time. Each new frame is processed in order to remove confounding appearance content, processed to extract salient information, registered with the growing synthetic image (initially the single frame which which tracking begun), and stitched with it.



 §.§ Detection of the region of interest

A crucial step in the pipeline outlined in the previous section concerns the detection of the region of interest (ROI) i.e. the part of the image which corresponds to the fundus and contains salient appearance content: the blood vessels and the optic disc. The parts of the frame which are outside of it are of no use for us and correspond to the sclera (the white, outer part of the eyeball), eyelids, etc.

As the frames in Figure <ref> illustrate, the region of the interest is elliptical in shape, and bright. However, colour in the image can not be used reliably as a discriminative feature for segmentation as significant variation can be observed across different individuals (e.g. there are differences between individuals of Caucasian and African descent). Variation in location, ellipse shape and orientation, as well as scale should also be noted as significant challenges.

Though the region of interest is elliptical in shape, observe that there is no sharp delineation, prohibiting the use of edge detection based approaches. Instead, we propose a two step method. In the first step, simple binary segmentation is performed using the well-known method first introduced by Otsu <cit.>. Here the frame is treated as a greyscale image, see Figure <ref>. Then the contour of the largest region used to find the best elliptical fit. We achieve this using a genetic algorithm based approach. Our methodology is inspired by the work of Conn and Arandjelović who used ellipse fitting as the first stage in the task of localizing and segmenting ancient coins in real-world images <cit.>.



An ellipse embedded within a 2-dimensional Euclidean space is parameterized by five parameters. This can be readily seen from the implicit equation:

    x^2 + k_xy x y + k_yy y^2 + k_x x + k_y y + k = 0.

The choice of the parameterization is crucial in making the most of our  genetic search strategy. In particular, the parameterization should be such that when encoded as a `chromosome' (in the context of a genetic algorithm), operations such as crossover, mutations, and others, are likely to effect an improvement in the fitness of a hypothesis. With this goal in mind, we parameterize an ellipse (hypothesis) using five points on its circumference and, following Conn and Arandjelović <cit.>, use a short, non-binary chromosome comprising the coordinates of these points. As argued by Conn and Arandjelović, by enforcing the indivisibility of coordinate values we ensure that the constraint imposed by two circumference points is retained during evolutionary operations, thereby achieving a higher chance of greater generational fitness improvement.

The fitness of a specific hypothesis is evaluated in a straightforward fashion. From chromosomes formed by concatenating the coordinates of five points on an ellipse's circumference –(x_a,y_a), (x_b,y_b), (x_c,y_c), (x_d,y_d), and (x_e,y_e)– the parameters in (<ref>) can be obtained by solving a simple linear equation:

    [
      [   x_a^2 x_a y_a   y_a^2     x_a     y_a       1;   x_b^2 x_b y_b   y_b^2     x_b     y_b       1;   x_c^2 x_c y_c   y_c^2     x_c     y_c       1;   x_d^2 x_d y_d   y_d^2     x_d     y_d       1;   x_e^2 x_e y_e   y_e^2     x_e     y_e       1;         ]]
      [[    1; k_xy; k_yy;  k_x;  k_y;    k;      ]]
      =[[ 0; 0; 0; 0; 0;   ]].


From this parameterization, the fitness of a hypothesis can be readily quantified by (i) uniformly sampling the circumference of the corresponding ellipse, and (ii) computing the number of samples which are located on edge pixels of the edge image. Equispaced samples along the circumference of the corresponding ellipse are readily generated using the canonical form, i.e. using the major and minor radii (a and b) of the ellipse, its centre (x_0, y_0), and the rotation angle θ relative to the coordinate system:

    a = -√(2 ψψ_1)/k_xy^2 - 4k_yy   and       b = -√(2 ψψ_2)/k_xy^2 - 4k_yy,
       x_0=2k_yy k_x  - k_xy k_y/k_xy^2 - 4k_yy   and       y_0=2 k_y  - k_xy k_x/k_xy^2 - 4k_yy,

where:

    ψ=    k_y^2+ k_yyk_x^2 - k_xyk_x k_y + (k_xy^2 - 4k_yy)k,
    ψ_1=    1+k_yy + √((1-k_yy)^2+k_xy^2),
    ψ_2=    1+k_yy - √((1-k_yy)^2+k_xy^2).

Then, if there are n_s samples, and the samples are { (x_1,y_1),(x_2,y_2),…,(x_n_s,y_n_s)}, the fitness becomes:

    ϕ = ∑_i=1^n_s E(x_i,y_i) / n_s,

where E(x,y) is the value of the pixel (x,y) in the edge image 𝐄.



 §.§ Glare detection and removal

As mentioned earlier, one of the challenges with the input videos is the presence of significant glare. It is of critical importance that the areas of the fundus occluded by glare are detected so as not to confound the tracking process, as well as so that the relevant regions can be filled-in by exploiting information from other frames in which they are not occluded.

Glare areas are for practical purposes of unpredictable shape, often but not universally present, and their location varies. All of these characteristics make their detection difficult. Another challenge which emerges in our specific application concerns the appearance of the optic disk which is usually brighter than the rest of the fundus, and could be mistaken for glare. Yet, the appearance of the optic disk is a crucial structure of interest to ophthalmologists, used to diagnose glaucoma and a number of other conditions <cit.>.

Our approach to detecting glare is inspired by the work of Lange <cit.> on images of the uterine cervix. Broadly speaking, we detect glare areas as those regions of the image which exhibit high contrast or saturation, and are approximately white. Formally, the criterion is based on the following local measure g(x,y):

    g(x,y)=min(R_x,y,G_x,y,B_x,y) - min(R_x,y,G_x,y,B_x,y)/max(R_x,y,G_x,y,B_x,y)

where R_x,y, G_x,y, and B_x,y are respectively the red, green, and blue colour components of the pixel at the locus (x,y). After computing g(x,y) for all image loci the corresponding feature image is smoothed using the Alternating Sequential Filter (a sequence of morphological closing and opening operations), and the binary mask 𝐌 computed with M_x,y = 1 iff G_x,y exceeds a predefined threshold, and M_x,y = 0 otherwise. Examples are shown in Figure <ref>.



 §.§ Salient information extraction

Having removed confounding image content (glare, background, etc.) the processed frames are ready for the main part of our algorithms with concerns the tracking of salient structures present in the fundus, and the subsequent stitching. However, notwithstanding the significant processing described thus far, these tasks are still far from straightforward. For example, note that neither simple appearance nor any of the commonly used local features can be used for tracking (or, equivalently, registration). Raw appearance changes greatly due to global illumination changes, as well as local changes caused by the inherent optics of the eye. Local features – such as appearance based SIFT <cit.> and SURF <cit.>, or edge based ones <cit.>– can also not be reliably detected as the appearance of the fundus does not contain corner-like loci.

Our approach is specifically targeted at the type of appearance content of interest in fundal images, namely blood vessels. These are line like features which form complex and characteristic structures, and by virtue of this allow different frames to be mutually registered.

We extract characteristic vessel based features using the method introduced by Frangi et al. <cit.>. Their so-called vesselness filter, first proposed for the use on 3D MRI data but later successfully employed in a number of different applications <cit.>, extracts tubular structures from an image. For a 2D image consider the two eigenvalues λ_1 and λ_2 of the Hessian matrix computed at a certain image locus and at a particular scale. Without loss of generality let us also assume that |λ_1| ≤ |λ_2|. The two key values used to quantify how tubular the local structure at this scale is are ℛ_𝒜 = |λ_1|/|λ_2| and 𝒮 = √(λ_1^2 + λ_1^2). The former of these measures the degree of local 'blobiness'. If the local appearance is blob-like, the Hessian is approximately isotropic and |λ_1|≈|λ_2| making ℛ_𝒜 close to 1. For a tubular structure ℛ_𝒜 should be small. On the other hand, 𝒮 ensures that there is sufficient local information content at all: in nearly uniform regions, both eigenvalues of the corresponding Hessian will have small values. For a particular scale of image analysis s, the two measures, ℛ_𝒜 and 𝒮, are then unified into a single vesselness measure:

    𝒱(s) =
        
          0     if λ_2 > 0
    
          (1-e^-ℛ_ℬ/2β^2) × (1-e^-𝒮/2c^2)     otherwise,

where β and c are the parameters that control the sensitivity of the filter to ℛ_𝒜 and 𝒮. Finally, if an image is analyzed across scales from s_min to s_max, the vesselness of a particular image locus can be computed as the maximal vesselness across the range:

    𝒱_0 = max_s_min≤ s ≤ s_max𝒱(s)

We empirically set the parameter values to β= 0.75 and c = 15, and process frames at scales s=3,4,5. Examples of typical results are shown in Figure <ref>.



  §.§.§ Algorithm initialization

Recall that our algorithm starts the tracking (and thus the stitching process) from an automatically frame detected as the most reliable one i.e. one that contains sufficient salient information content to facilitate robust registration with the subsequent frames, and free of motion or out of focus blur. The vessel extraction method we just described is used to this end. In particular, following the computation of the vesselness image, the richness of salient information content is quantified by computing the entropy of this image. Input frames which do not contain many blood vessels, or which are corrupted by blur, will not produce many vessel detections, and thus result in low entropy vesselness images. Therefore, the frame with the highest entropy of the corresponding vesselness image is selected as the starting frame from which tracking commences, and as the initial synthetic image which is subsequently extended and improved though merging with newly processed frames.



 §.§ Image registration

Recall that our algorithm incrementally expands the synthetic image of the fundus by expanding it by (stitching with) newly processed frames. The first step in the stitching process concerns the registration of a frame with the synthetic image. This is achieved by using the extracted vessel feature images. Specifically, we adopt the use of normalized cross-correlation as the hypothesis matching criterion <cit.>, and exhaustively (though in a coarse to fine fashion, to speed up the process) explore the space of different scaling and translation parameters. The combination of the parameters which results in the highest normalized cross-correlation is accepted as the correct hypothesis but only if matching goodness exceeds that of the second best hypothesis (the second highest local normalized cross-correlation maximum) by a significant enough margin (herein we used a factor of 1.5). If this condition is not satisfied (which can happen, for example, when the newly acquired frame does not contain enough information to allow confident registration) the frame is discarded as unreliable.



 §.§ Incremental stitching

Having registered a new frame with the growing synthetic image, the last step of our algorithm concerns the merging of the two images and with it the expansion of the synthetic result. There are several factors which make the process difficult. Firstly and as we noted before, there may be a significant illumination discrepancy between the two images. Moreover, a gradual darkening of the region of interest in the proximity of its boundary can be readily observed; note that this is a feature of the imaging process, rather than an artefact introduced by any of the steps of our algorithm. Lastly, the regions corresponding to glare areas, void of useful information, should be taken into account.

Ideally, the result of the merging process should not only produce a seamless output image, spatially expanded (in general), but also leverage information content from both input images to increase the quality of information in the overlapping areas too. To achieve this while at the same addressing the challenges summarized above, we adopt an adaptive weighting strategy whereby the relative contributions of the two input images are adjusted depending on the particular locus under the consideration. To motivate out approach intuitively, the idea is to use preferentially data from an image in which a particular location is more central (relative to the region of interest) i.e. where the optical setup of mini ophthalmoscopies and the eye itself, produce better quality data (higher signal to noise ratio). We formalize this through the use of the distance transform <cit.>. Specifically, after a linear illumination transform of brightness which normalizes global changes, for each pixel in an image we compute its approximate distance from the boundaries of the region of interest (the outer boundary and, if present, the boundary of the glare region). Then, when combining two corresponding pixels their relative contributions are computed by considering the inverse value of their distances from the boundaries in the original images, i.e. the output value of a RGB channel c_x,y becomes:

    c_x,y= w_f c^f_x,y + w_s c^s_x,y/w_f w_s

where c^f_x,y is the value of the pixel in a new frame and c^s_x,y in the synthetic image, and:

    w_f = 1/d^f_x,y       w_s = 1/d^s_x,y

where d^f_x,y and d^s_x,y are the corresponding distance transform values (see Figure <ref>). The process can be therefore described as a form of adaptive linear fusion <cit.>.



§ EVALUATION

The data corpus we used in this work was collected from healthy volunteers. The acquisition process was entirely unaffected and unguided by us, guaranteeing realistic video input as would be collected in the field. Thus, we found that the sequences were highly variable in length, lasting from approximately 20 up to 90 seconds. The majority of the sequences start with a distant view of the eye (and hence little useful fundal information), and progressively close in onto the areas of actual interest such as blood vessels and the optic disc. Major motion and out of focus blur can be observed in this early phase. Once a section of the retina is clear the acquisition becomes more controlled, exhibiting purposeful motion in an effort to capture as much of the fundus as possible. Only a small section can be seen at any given moment due to the narrow field of view of direct ophthalmoscopy devices. Throughout this process blur continues to pose problems, with the retina easily going out of focus, drastic changes in the direction of motion occurring, and the imaging stopping unexpectedly. Considering the nature of our data set, it was not possible to obtain objective ground truth high quality images using a traditional ophthalmoscope. Hence, at this stage we limited ourselves to a qualitative assessment of the results. Representative examples of synthetic images and the effects of specific challenges are shown in Figures <ref> and <ref>.



§ SUMMARY AND CONCLUSIONS

The Arclight is a recently developed low-cost direct ophthalmoscope that offers great potential for the prevention of blindness amongst those living in low and middle income countries. However, currently this potential is limited by the inherent optical limitations of direct ophthalmoscopy, most notably the narrow field of view. In this paper we described the first method that uses this low quality data to create a synthetic, higher quality, wider view image comparable to one acquired using expensive and bulky traditional retinal cameras.

ieee