Abstract
Follow the light: projector-based augmented reality intracorporeal system for laparoscopic surgery," J. Med. Imag. 5(2), 021216 (2018), doi: 10.1117/1.JMI.5.2.021216. Abstract. A projector-based augmented reality intracorporeal system (PARIS) is presented that includes a miniature tracked projector, tracked marker, and laparoscopic ultrasound (LUS) transducer. PARIS was developed to improve the efficacy and safety of laparoscopic partial nephrectomy (LPN). In particular, it has been demonstrated to effectively assist in the identification of tumor boundaries during surgery and to improve the surgeon's understanding of the underlying anatomy. PARIS achieves this by displaying the orthographic projection of the cancerous tumor on the kidney's surface. The performance of PARIS was evaluated in a user study with two surgeons who performed 32 simulated robot-assisted partial nephrectomies. They performed 16 simulated partial nephrectomies with PARIS for guidance and 16 simulated partial nephrectomies with only an LUS transducer for guidance. With PARIS, there was a significant reduction [30% (p < 0.05)] in the amount of healthy tissue excised and a trend toward a more accurate dissection around the tumor and more negative margins. The combined point tracking and reprojection root-mean-square error of PARIS was 0.8 mm. PARIS' proven ability to improve key metrics of LPN surgery and qualitative feedback from surgeons about PARIS supports the hypothesis that it is an effective surgical navigation tool.

Follow the light: projector-based augmented reality intracorporeal system for laparoscopic surgery Philip Edgcumbe, a,
Introduction
In 2015, there were 61,560 new cases of kidney cancer diagnosed in the United States. 1 According to the American and European Urological Association guidelines, 2 a partial nephrectomy should be offered to all patients who have T1 tumors (i.e., tumor is <7 cm across and is only in the kidney), whereas patients with tumors that are over 7 cm across should be offered a radical nephrectomy. A partial nephrectomy is the removal of the entire cancer tumor while attempting to preserve the maximum amount of healthy kidney tissue. 3 The preservation of the healthy kidney tissue results in improved health outcomes. 3 Matched patients who underwent partial nephrectomy rather than radical nephrectomy (i.e., removal of the whole kidney) had equivalent long-term cancer outcomes and a lower risk of developing chronic renal insufficiency and proteinuria. 4 For a partial nephrectomy, the standard of care is a 5-mm negative margin. 5 Laparoscopic partial nephrectomies (LPNs) are considered to be technically challenging because the surgeon has a limited field of view and effectively no tactile feedback. Furthermore, it is challenging for the surgeon to excise the entire tumor while also minimizing the excision of healthy tissue. Tumor size, tumor position, and surgeon experience are important factors in determining whether a patient is offered an LPN. Smaller and more peripheral tumors make for a simpler partial nephrectomy surgery and are more likely to be removed by LPN. Surgeons that have higher kidney cancer surgical volumes and practice in larger cities conduct a higher proportion of nephrectomies as LPNs. 6 Between 2002 and 2013, the percentage of nephrectomies that were performed in the USA as LPNs increased from 2% to 17%. 6 With more effective surgical navigation technology available that reveals to the surgeon the tumor outline and tumor depth, it is likely that more surgeons will opt for LPNs and that their patients will benefit accordingly.
Introduction
Planning and executing a successful LPN surgical approach is particularly challenging for endophytic (inward growing) kidney tumor resections. LPNs for endophytic tumors have a 47% complication rate, five times higher than that for exophytic (outward growing) tumors. 7 As defined by Herrell et al., 8 the ideal approach for endophytic tumors is to start as close as feasible to the tumor and excise straight down from the organ surface along the orthographic projection of the tumor. This ideal approach minimizes the excised tissue, while maintaining a negative margin. A negative margin indicates that the entirety of the tumor was removed and that the tumor was not visible in the excised specimen. The ideal excision specimen for a spherical tumor is a cylinder normal to the surface of the kidney. In reality, tumors are only approximately ellipsoidal.
Introduction
When surgeons are interested in visualizing the underlying anatomy of the kidney during an LPN, they commonly use laparoscopic ultrasound (LUS). However, using an LUS effectively during surgery is challenging and complicated and mostly performed by expert surgeons. When using an LUS transducer during an LPN, surgeons put the LUS transducer on the kidney, interpret the LUS image on the ultrasound image display screen, create a mental image of the tumor, mentally register the LUS with the laparoscopic view, and mark the kidney tissue surface using electrocautery. When surgeons remove the LUS transducer, they have to recall the ultrasound images and their poses relative to the kidney throughout the surgery.
Introduction
To assist in surgical navigation, there has been significant research and development in augmented reality (AR). Herrell et al. 8 used the information from a preoperative computer tomography (CT) scan of a phantom tumor model to provide intraoperative guidance during robotic surgery. Bernhardt et al. 9 recently published a comprehensive review about AR in laparoscopic surgery. There have been important advances in surface reconstruction using structured light, 10 surface tracking, and surface registration [11] [12] [13] as well as advances in AR visualization techniques. The five AR visualization techniques for depth perception of blood vessels that were studied by Wang et al.
Introduction
14 are transparent overlay, virtual window, random-dot mask, transparent mask, and ghosting. Jayarathne et al. 15 developed a real-time three-dimensional (3-D) ultrasound reconstruction and visualization strategy for laparoscopy, in which the user views a 3-D ultrasound volume through a transparent window in the surface image. With regard to AR for partial nephrectomy, Hughes-Hallett et al. 16 published a review on the topic. Examples of video-based AR for partial nephrectomy include the following: display of semitransparent segmented preoperative CT using real-time 3D-CT to stereoscopic video registration, 17 display of intraoperative cone-beam CT images of the kidney using radio-opaque needles for tracking and registration, 18 display of an AR overlay of the ultrasound image during the intraoperative ultrasound scan of the kidney using image-based tracking of an ultrasound transducer, 19 display of a 3-D tumor model during the planning and execution stage of the surgery using image-based tracking of an optical marker on the kidney, 20 and showing the surgeon how to access the kidney tumor with an incision tool. 21 Another type of AR for surgical guidance is projector-based AR guidance with the projector outside 22 or inside the patient 23 for open and laparoscopic surgery, respectively. Projector-based AR guidance, rather than video-based AR guidance, is of interest because it does not introduce latency into the video-based display being used by the surgeon. Furthermore, it could provide structured light to improve surface reconstruction and project the display information directly onto the surgical scene, resulting in a natural blending of the augmented and real worlds. Prior work with the projector inside the patient explored the display of blood vessels during laparoscopic surgery. 23 This work incorporates intraoperative ultrasound imaging for real-time tracking and a display of the kidney tumor by the projector inside the patient.
Introduction
The development of the projector-based augmented reality intracorporeal system (PARIS) for image-guided laparoscopic surgery made possible the projector-based AR described already (Fig. 1) . The purpose of PARIS is to improve the efficacy and safety of LPN surgery and render LPN more accessible to surgeons so as to make LPN, not radical nephrectomy, the most common surgery for T1 tumors. In surgical experiments on phantom kidney models, PARIS was compared to standalone LUS. While the use of PARIS is proposed for all types of nonrobotic and robotic minimally invasive surgery, in this work, it is specifically evaluated in the context of robot-assisted LPNs with the da Vinci Surgical System (Intuitive Surgical Inc., Sunnyvale). PARIS includes a projector that displays an outline of the orthographic tumor outline on the surface of the kidney.
Introduction
The sections in this paper include Secs. 2, 3, and 4. Section 2 includes a description of intraoperative tumor model generation, the augmented visualization provided by PARIS, verification and validation tests to determine the accuracy of PARIS, and the user study conducted to test PARIS for LPN on kidney phantoms.
Materials and Methods

Materials
PARIS consists of multiple devices that could be used during partial nephrectomy. PARIS includes a tracked marker called the dynamic augmented reality tracker (DART), 24 projector called the Pico Lantern, 23 and miniature LUS transducer. The DART is placed onto the kidney and anchored in place with its own barbed legs. The Pico Lantern is used for both surface reconstruction and augmentation of the surgical scene. The LUS transducer is used to generate a 3-D model of the tumor. The DART is used as a means for tracking the position of the kidney and accurately creating an AR display of the 3-D model of the tumor. Both the DART and LUS transducer are optically tracked in real time in 6 degrees-of-freedom using only image analysis of the laparoscopic video feed-no external tracking hardware is used. A diagram of PARIS is shown in Fig. 1 . The Pico Lantern is a PicoPro projector (Celluon Inc., Seoul, Republic of Korea) with an attached KeyDot ® , a marker with an asymmetric grid of circles (Key Surgical, Eden Prairie). Using OpenCV, the KeyDot ® is tracked optically with 6 degrees-offreedom relative to a laparoscope. 19 No external tracking hardware is required. Images are projected directly onto the surgical scene via laser raster scanning, and the Pico Lantern has a focus range from 3 cm to infinity. There is some latency for the projected images. However, direct projection implies that unlike video-based AR, no modifications are made to the video pipeline displaying the surgical scene to the surgeon. Compared to our previous work, 23 which included a Pico Lantern projector with VGA resolution (640 × 480) and 15 lumens of brightness, this paper presents a projector prototype [ Fig. 2 (e)] with increased resolution (1920 × 1080), 30 lumens brightness, and with wireless capability. It does not require a dedicated port as it can be placed through the skin incision with a cable beside the trocar or it can be controlled wirelessly. The prototype used in this paper has not been miniaturized as was done previously. 23 However, it shares a similar underlying architecture and component sizing; therefore, the miniaturization techniques and engineering applied previously in Ref. 23 may be applied here, too. In another previous work, we developed the DART, a navigation aid with barbed legs made in either plastic for one-time use or stainless steel for repeated use, with a KeyDot ® optical marker on it. 24 With dimensions of 8.5 × 8.5 × 13 mm, the DART can be inserted into a 12-mm trocar and picked up in a repeatable manner by the da Vinci Pro-Grasp™ (Intuitive Surgical Inc., Sunnyvale). The LUS transducer is a 10-MHz, 28-mm linear array designed for robot-assisted minimally invasive surgery; additionally, it has a KeyDot ® marker fixed to it for tracking. 25 Cylindrical Super Soft Plastic (M-F Manufacturing, Fort Worth) kidney phantoms, with color, elasticity, and size similar to those of human kidneys, were used. Each of the kidney phantoms had a height of 6 cm and diameter of 15 cm and a Young's modulus of 15 kPa, consistent with human kidneys. 26 Spherical inclusions with a 10-to 20-mm diameter are randomly placed at a depth of ∼10 to 30 mm in the phantom. PARIS is evaluated with the da Vinci Si ® surgical system (Intuitive Surgical Inc., Sunnyvale).
Tumor Model Generation
Using PARIS requires two major steps: first, an accurate model of the tumor must be generated; this is followed by the AR visualization of the tumor model. In the first step, the model is generated by tracking the LUS transducer and DART simultaneously as done by Singla et al. 20 A 3-D tumor model is generated in the DART coordinate system by registering the twodimensional LUS ultrasound images to the DART, reconstructing the 3-D volume, and then performing manual segmentation. The 3-D volume reconstruction method tests the proximity of each registered ultrasound plane to a coarse discretization of the scanned volume. Where intersections take place, each voxel value within the respective regions is assigned in accordance with the ultrasound image texture at the closest perpendicularly projected point within a specified distance threshold. Using ITK-Snap's manual segmentation and interpolation features, it takes <5 min to perform the segmentation and 3-D reconstruction of the tumor model. 27 The segmentation is integrated into the surgical workflow by conducting the segmentation on the surgical navigation computer in the operating room as soon as the data are collected. Automatic segmentation could be incorporated in the future. Deformable models and machine learning 28, 29 are likely to be used to develop effective automatic segmentation.
Augmented Reality Visualization
In the second step, the surgeon uses the Pico Lantern and AR visualization to visualize the tumor and determine the ideal excision approach. The goal is complete tumor removal while minimizing the amount of healthy kidney tissue excised. The Pico Lantern first displays a checkerboard pattern on the kidney to provide extra corresponding features, which improve the quality of the kidney surface reconstruction. Surface reconstruction is achieved via semiglobal block matching (SGBM) 30 using OpenCV. It is similar to the projector-enhanced stereo surface reconstruction described previously. 23 Next, PARIS displays the orthographic projection of the tumor model as a dense point set on the surface of the kidney. To perform the orthographic projections, the rays are projected in parallel from the tumor toward the Pico Lantern. The projected image is determined by the intersection of the rays with the surface. In this manner, the reconstructed surface is used to predistort the image and account for the phantom kidney surface when projecting an image onto the nonplanar surface. 31 The excision approach is the vector that the surgeon follows to excise the tumor. The ideal approach, as described by Herrel et al., 8 is normal to the surface at the point closest to the tumor. As the Pico Lantern has a flexible cable, the surgeon can use it as an "eye in the hand" and move it to a variety of positions to visualize the surface projection of the tumor from different perspectives. Ultimately, the surgeon places the tracked Pico Lantern in a position that is unused by the laparoscope and aligns the Pico Lantern such that it is parallel with and centered on the ideal approach vector. This is achieved by manually positioning the Pico Lantern so it is normal to the surface and the center of the projection image intersects the tumor centroid. The surgeon then uses the projected tumor outline and the line between the projector and center of the projected image to identify the ideal approach and plan the LPN. Throughout, the orthographic projection of the tumor in the direction of the Pico Lantern is displayed on the kidney surface.
Verification and Validation
The laparoscope was calibrated using OpenCV. The projector calibration was performed as done previously. 32 The ultrasound calibration was achieved geometrically via a careful alignment of the axes of the KeyDot ® to the axes of the LUS transducer. 19 The pixel-to-millimeter ratio was determined by the known width and resolution of the LUS linear array and the axial depth and resolution of the ultrasound image. After ultrasound calibration, the LUS transducer has a relative root-mean-square (RMS) point reconstruction accuracy of 0.9 mm estimated over 10 ultrasound images covering a working volume of 16 × 10 × 19 mm.
24
It was previously demonstrated by a biomechanical simulation using ANSYS (ANSYS, Pittsburgh, Pennsylvania) that the local deformation between the DART and the centroid of the tumor owing to the pressure of the LUS transducer on the kidney surface is submillimeter. 20 To evaluate the benefit of using the Pico Lantern to improve surface reconstruction, the surface density was measured. The surface density is defined as the percentage of the kidney's surface that is successfully reconstructed by SGBM within the region of interest in the laparoscope view. The surface density of an ex vivo kidney, with and without the projection of extra features, is compared among 12 unique laparoscope and projector poses. The mean surface density change is reported.
24
To evaluate the accuracy of the Pico Lantern's augmentations, the point reprojection error was measured. The point reprojection error is the distance between a point on the DART and its transformed equivalent as projected onto the scene. This captures the error in the tracking of the two KeyDots ® and the error in the laparoscope and projector calibration models. To measure it, the Pico Lantern was moved to five poses, and for each pose, the DART was placed at 10 poses, ∼80 mm from the laparoscope. The RMS error is reported.
24
For the phantoms used in the user study, the segmented tumor volumes are compared to the ground truth values measured during phantom construction. Additionally, a phantom was sectioned in half, and the segmented ultrasound volume of the exposed part of the tumor is projected onto the cut surface. The Hausdorff distance and mean RMS distance between the contours of the actual tumor and the projected tumor were measured for five laparoscope and projector poses. The mean RMS distance is defined as the tumor localization error.
User Study
A novice urologist (second year resident) and an expert urologist (with over 10 years of surgical experience) completed partial nephrectomies on kidney phantoms using either PARIS or only LUS imaging. The RENAL nephrometry score 33 was 12 for all the tumors, indicating a challenging resection task. Prior to the start of the user study, each surgeon was permitted a practice surgery with each of PARIS and only LUS technique. In an equal amount of time for the user study, the novice completed 12 simulated surgeries, and the expert completed 20 simulated surgeries. After each simulated surgery, the surgeons answered a five-point Likert scale questionnaire and provided open-ended qualitative feedback.
User Study
For each simulated surgery, the margin status was recorded as negative, microscopic, or gross. A negative margin indicates that the entirety of the tumor was removed and that the tumor was not visible in the excised specimen. A microscopic margin indicates that the tumor was removed with exposure, while a gross margin indicates that a visible part of the tumor remained in the kidney phantom.
User Study
Additionally, the mean and standard deviation of excision time, tumor volume, total excised specimen volume, adjusted excised specimen volume, and depth of cut are reported for all the user study surgeries that had negative margins. The excision time is defined as the time it takes the surgeon from the moment the surgeon cuts into the kidney phantom to the time when the entire tumor is removed. The total excised specimen volume is the volume of the tissue that is excised by the surgeon minus the tumor volume. The tumor volume is subtracted so that the variable size of the tumor does not affect the results. The adjusted excised specimen volume is the total excised specimen volume minus the volume of the top layer of parenchyma that sits above the tumor. The volume of the top layer of parenchyma and the tumor is subtracted so that the variable depth and size of the tumors do not affect the results. The volumes are determined from the specimen weights and known density. The depth of cut is the thickness of the parenchyma tissue below the tumor in the excised specimen. This is determined by US imaging of the excised specimens.
User Study
Finally, the metrics used in this work to quantify the deviation from the ideal excision include cross-sectional analysis of the specimens. Each excised specimen was sectioned at increments of 5 mm, and the cross section with the largest tumor diameter is analyzed. The tumor and the full cross section of the specimens were segmented, and the RMS distance between their centroids and the Hausdorff distance between their segmented contours is reported. For all the metrics, a two-tailed paired t-test was performed to assess statistical significance with a p-value of 0.05. The Holm-Bonferroni correction is used to account for multiple comparisons.
Results

Verification and Validation
The projected patterns improved surface density by an absolute mean of 15.4% AE 8.3%. An example of how the projected pattern improved surface reconstruction is shown in Fig. 3 .
Verification and Validation
The Pico Lantern's point reprojection error was 0.8-mm RMS. During the data collection, the Pico Lantern was moved over a range of 32 × 9 × 11 mm. The mean ground truth and tumor model volumes were 2.6 AE 0.7 cm 3 and 4.2 AE 1.4 cm 3 , respectively. This corresponds to an overestimation of the radius of the tumor by 1.4 AE 1.4 mm. Given that there is an error in the tumor model estimation, it is better that the tumor model volume be larger, rather than smaller, than the actual tumor volume. As shown in Fig. 4 , for the projection of the tumor onto the actual tumor, the mean Hausdorff distance and RMS distance between the tumor contour and projection contour were 3.9 and 1.7 mm, respectively.
User Study
The quantitative results from the 32 simulated LPNs are summarized in Table 1 . The mean and standard deviation (avg AE stdev) of each metric are listed. The results of the trials of LUS only and PARIS are presented. The asterisk indicates the statistical significance (p < 0.05) of PARIS compared to LUS. Positive margin cases were excluded from quantitative analysis. In summary, the novice surgeon had 5∕6 negative margins with both PARIS and LUS, and the expert surgeon had 10∕10 and 8∕10 negative margins with PARIS and LUS, respectively. Furthermore, when using PARIS, both the surgeons excised a statistically significant less amount of healthy tissue (p < 0.05), as measured via the adjusted excised specimen volume. As examples, the cross sections of the expert surgeon's first four excised specimens with each of PARIS and only LUS are shown in Fig. 5 . As reported in Table 1 , the Hausdorff distance between the tumor and complete cross section of the excised specimen was reduced for both the surgeons when they used PARIS.
User Study
The surgeons reported that they would prefer to use PARIS over the standalone ultrasound imaging method partly because the AR provides persistent guidance by projecting the tumor's outline. Conventionally, the surgeon observes a limited cross section of the tumor during the ultrasound scan. The surgeons considered the drawbacks of PARIS to be that the Pico Lantern is an extra piece of hardware, there is an absence of guidance during the surgical excision of the tumor, and the surgical light must be set to a moderate light intensity to prevent the washing out of the projected image. After each resection, the surgeons answered a five-level Likert scale questionnaire wherein the response of 1 indicates "strongly disagree" and 5 indicates "strongly agree." A comparison of LUS and PARIS results indicates that the surgeons felt more confident (3.3 AE 1.1 versus 5.0 AE 0.0) and had a better spatial understanding (3.5 AE 0.8 versus 4.6 AE 0.5) in the case of PARIS. All these results favor PARIS over LUS visualization.
User Study
Singla et al. 20 conducted a nearly identical user study, in which the same expert surgeon used an AR guidance system that provided guidance during the execution stage of the surgery with computer graphic overlays. Both this user study and the Singla et al.'s user study were shaped by and benefited from early feedback on study design from the same clinical collaborator. When the expert surgeon used the AR guidance system in Singla et al.'s prior work 20 instead of PARIS, the excision time, total excised specimen volume, and adjusted excised special volume were effectively similar. However, there was a statistically significant reduction in the depth of cut under the tumor, from 10 AE 4 mm to 3 AE 2 mm, when the expert surgeon used the AR guidance system instead of PARIS. This is important because it reduces the risk of the surgeon entering the collecting system of the kidney. The AR guidance system provides AR navigation during the execution stage of the surgery. Therefore, it is reasonable that there was a reduction in the depth of cut under the tumor. There is scope for combining the PARIS described in this paper with the AR guidance system. 20 That is likely to provide the optimum guidance for the planning and execution stage of the partial nephrectomy.
Discussion
This work presents, from an engineering and clinical user study perspective, a completely integrated intracorporeal AR system for intraoperative guidance in soft tissue surgery. Given the 5-mm margin of safety for partial nephrectomy, the overall tumor localization error (1.7-mm RMS) is small enough to consider PARIS as acceptable for guidance.
Discussion
From the engineering perspective, it is demonstrated that the integration of the two components, vision-based tracking and PARIS projector augmentation, is feasible. The integration of PARIS with the da Vinci system requires read-only access to the video feed, which will ease dissemination. PARIS includes an improved Pico Lantern prototype as compared to previous work. 23 This prototype provides higher resolution, is brighter, has wireless capability, and is automatically tracked in real time using optical tracking computer vision techniques. These additional features enable ease of use; however, the fundamental Expert surgeon (n ¼ 20) concept of a tracked projector for AR display remains unchanged. Several projection strategies were considered. They included the orthographic or perspective projection of the tumor in the projector and/or laparoscopic coordinate systems. The perspective projections were unintuitive because the perspective projection of the tumor on the organ surface covers an area that is smaller than the actual tumor. Thus, an orthographic surface projection is the only option if the surgeon wishes to achieve a "cylindrical-approach" to the dissection of the tumor. Furthermore, a display in the laparoscopic coordinate system and field of view proved to be confusing to the surgeon because the laparoscope is generally oriented at a shallow angle to the tissue surface.
Discussion
The user study was intended to test the overall concept of PARIS in the clinical setting. The main outcome from the user study presented herein is that there was a statistically significant reduction of 30% in the amount of healthy tissue excised when the surgeon used PARIS. The implications for a reduction in healthy kidney tissue excised are critical because the preservation of the kidney tissue has been established to improve health outcomes. 4, 6 Furthermore, strategies to save healthy kidney tissue may include the provision of continuous guidance throughout the planning and execution of the surgery. Second, the surgeons that used PARIS felt more confident and had a better self-reported spatial understanding of the underlying anatomy. While initial results were positive, further ex vivo and in vivo studies with additional challenges, such as bleeding, smoke, and specular reflection, are required. Maier-Hein et al. 34 conducted a comparative study on the effect of smoke and blood on the surface reconstruction accuracy of eight state-of-the-art passive and active reconstruction methods. They determined that structured light surface reconstruction exhibited optimum performance in the presence of smoke and that smoke generally increased the median RMS stereo surface reconstruction error by 1 to 6 mm. Furthermore, the assumption of submillimeter local deformation between DART and tumor must be validated in vivo.
Discussion
If guidance tools, such as PARIS, continue to improve key surgical metrics when tested in vivo and is approved for clinical use, it could potentially improve LPN outcomes and increase the percentage of kidney cancer surgeries that are conducted as partial nephrectomies. PARIS aids the surgeon to localize the tumor and plan the surgery, thus with less chance of missing a part of the tumor, the surgeon can consider bigger and more complex tumors for LPN surgery. It is noteworthy that based on the key surgical metrics of total excised specimen volume (cm 3 ) and adjusted excised specimen volume (cm 3 ), the novice surgeon with PARIS performed as well as the expert surgeon with only LUS. This data supports the suggestion that surgeons, and in particular novice surgeons, who have access to guidance tools, such as PARIS, would be more likely to perform LPNs. This would imply faster adoption of LPNs and that more of the 61,560 patients per year who develop kidney cancer in the USA would benefit from partial nephrectomies. Furthermore, a new avenue for AR in laparoscopic surgery, with numerous interesting applications, will have been developed.
Discussion
Here are a few important observations from the user study. Each surgeon required a practice trial before she/he was able to use PARIS effectively for guidance. PARIS is intended as an adjunct and not as a replacement for standard practice. With the mobile Pico Lantern, the surgeon can dynamically explore tumor visualization from a wider range of angles, benefit from valuable dynamic visual cues, and align the Pico Lantern with the ideal surgical approach. It also became apparent that there are potential trade-offs associated with PARIS. The extra DART, which is meant to be inserted into the kidney and removed with the kidney tumor specimen, and the Pico Lantern hardware increase the complexity of the surgery. However, the advantage of the DART is that it provides a means of tracking the tumor throughout the surgery so that the Pico Lantern can provide persistent guidance from the projector-a feature that the surgeons found beneficial. A second advantage was that the surgeons observed that PARIS generated a clear image that blended effectively with the phantom surface and that the depiction of the resection line was natural and intuitive. This compares favorably with video-based AR overlays, which generally appear to float above the surgical scene. Finally, PARIS does not interfere with the video-based surgical display.
Discussion
In conclusion, PARIS can be integrated into the existing laparoscopic surgical workflow. It improves surface reconstruction and provides meaningful surgical guidance information resulting in statistically significant improvements in key surgical metrics and outcomes in kidney phantom LPNs.
Disclosures
The authors have no conflicts of interest to report.

{'paper_id': '3699126', 'title': 'Follow the light: projector-based augmented reality intracorporeal system for laparoscopic surgery', 's2_url': 'https://api.semanticscholar.org/CorpusID:3699126', 'source': 'S2ORC'}