2.1.

Imaging, Illumination, and Holographic Display Overview

Our intraoperative imaging system is comprised of the following components: a bioinspired single-chip multispectral imaging sensor,¹⁹ holographic goggles (Microsoft HoloLens, 1st generation), laser light illumination (BWF2-780 and BWF2-660, B&W TEK, Plainsboro, New Jersey, United States), personal computer (Lenovo M Series Tiny PC, i7 Intel Pentium, 16 GB RAM), wireless router (R6220, Netgear, San Jose, California, United States), and custom printed circuit board with NIR LEDs for optical calibration and co-registration. For generating 780 nm illumination, we utilized a laser shaping filter (FF01-769/41 Brightline bandpass filter, Semrock) along with a diffusing lens (ACL2520U-DG15, Thorlabs). Similarly, for producing 665 nm illumination, we employed a laser shaping filter (665.85-1 OD6 Ultra Narrow Bandpass Filter, Alluxa) in addition to a diffusing lens (ACL2520U-DG15, Thorlabs).

Our bioinspired imaging sensor incorporates a unique design featuring vertically stacked photodiodes combined with pixelated spectral filters [Fig. 2(a)].¹⁹ The pixelated spectral filter array consists of two distinct pixels arranged in a checkboard pattern across the imaging area. One pixelated spectral filter allows the transmission of photons in the range of 400 to $\sim 680\mathrm{nm}$, whereas the second pixelated spectral filter transmits photons within the range of 700 to 1000 nm. Under each pixelated spectral filter, there are three vertically stacked photodiodes, resulting in a total of six spectral measurements provided by the sensor. Three spectral observations are obtained in the visible spectrum, and the remaining three observations occur in the NIR spectrum. The merged NIR and color image was presented on a standard screen, allowing the surgical staff to view it easily. This facilitated a broader distribution of fluorescent information within the operating room. The multiple spectral measurements in the NIR spectrum allow the sensor to effectively differentiate between multiple fluorescent dyes present in that range. The main focus of this paper is to capture fluorescence specifically from a single tumor targeted probe and deliver this information to the end user through a 3D hologram. Our bioinspired sensor is capable of imaging multiple NIR dyes excited at different wavelengths. This feature was not integrated with the HoloLens and is the subject of future studies.

Fig. 2

Schematic illustration of the fluorescence-guided surgery system with the bioinspired image sensor and HoloLens. (a) The pixel array of the image sensor with vertically stacked photodiodes, including the top, middle, and bottom layers of photosensitive cells. (b) The experimental setup of the holographic goggles system used for in vivo imaging of tumor-bearing mice.

The bioinspired sensor is housed on a custom printed circuit board with low-noise differential amplifiers, a 14-bit analog-to-digital converter, a low-noise digital-to-analog unit, low-noise voltage regulators, and other auxiliary electronics necessary for the correct operation of the imager. An FPGA board (OpalKelly XEM 7310, Seattle, Washington, United States) is used to read out data from the imager, acquire data from the sensor, and transfer data to the PC. A custom-built camera enclosure houses optical notch filters with optical density $>6$ to suppress excitation light sources at 665 nm (NF03-658E, Semrock) and 785 nm (NF03-785E, Semrock). All printed circuit boards and the camera housing are designed using Altium or Autocad and fabricated by PCBway (Shenzhen, China). The FPGA firmware is developed using Xilinx Verilog, and the data acquisition and display software are written in the Python and C++ programming languages.

The data from the bioinspired image sensor are streamed to a computer, where additional signal processing is performed; this includes image calibration, co-registration with natural eyesight, color compression, and wireless streaming of data to the holographic goggles to provide augmented information to the surgeon’s natural eyesight about the location of tumors [Fig. 2(b)]. On the holographic display, co-registration with the natural eyesight is performed, and 24-bit color image hologram is generated and displayed. However, high-color resolution images require a long transmission time of more than 250 ms and introduce image lag. Therefore, we explored reducing the bit depth of the color images while still displaying the necessary contrast information to the surgeon under 80 ms from end to end.

A threshold was applied to the acquired NIR image from the bioinspired sensor, and only pixels with digital values above the threshold were presented in false color. Pixels below the threshold were represented as black and hence were rendered transparent on the HoloLens. Hence, the operator views the surgical site with their natural eyesight with only fluorescent information above a certain threshold being augmented to the natural eyesight. The threshold level of the NIR regions can be altered during the surgery to accommodate the operators’ preferences. The compressed NIR image was transmitted wirelessly to the holographic goggles, where an embedded processor generated a false-color hologram using a lookup table and projected it in 3D space. The goggles also have an embedded inertial and magnetic unit to track movements of the user to allow for stabilizing the placement of the hologram position even as the user gaze is away from the surgical site.

2.2.

Hologram Co-Registration with Natural Vision

Our intraoperative holographic system requires calibration and co-registration of the hologram projected onto the see-through display screen with the natural vision. Initially, the user was directed to launch the calibration application of HoloLens. This step is crucial as it allows for the adjustment of the interpupillary distance to account for the disparity that is unique for each user of the goggle. The application accomplishes this by guiding the user through a series of finger alignment tasks. Before initiating the co-registration procedure, it is necessary to position the camera above the surgical bed in the operating room for imaging. Subsequently, the co-registration process, which consists of two steps, can be performed. First, a custom printed circuit board with four NIR LEDs placed in assigned positions was imaged using our bioinspired sensor. To ensure that all LEDs were captured within the field of view of the camera, it was necessary to select an appropriate working distance (WD). However, once the WD was determined, the calibration routine proceeded regardless of the specific WD selected. A threshold was then applied to the NIR image, and the locations of the four LEDs were determined by analyzing the contours and its first moments. A homography matrix was then computed using the first moments and pixel locations, which were applied to all frames acquired from the image sensor. The assembled hologram was wirelessly transferred to the holographic goggles and displayed to the user’s natural vision.

The second step involves the operator placing the hologram on the calibrated board such that the visual information from the LEDs is correctly superimposed with the hologram. This was achieved using a series of voice and gesture commands that are recognizable by the holographic goggles. This step is a critical process that ensures that the augmented reality provided by our system accurately aligns with the natural eyesight of the operator, enabling them to locate and remove cancerous tissues. Typically, the calibration and co-registration procedure is required to be performed once for continual ongoing usage and takes $\sim 10$ to 15 min to complete. However, if the surgeon removes the wearable goggle, the co-registration process may need to be repeated to account for any changes in the position of the goggles relative to the surgeon’s eyes.

2.3.

Co-Registration Error Between Holograms and Natural Vision

We conducted an experiment to assess co-registration accuracy between holograms and the natural vision at varying depths. Although the holographic lenses provide a 3D mesh of the real world, it lacks the necessary spatial resolution for intraoperative procedures. We addressed this issue by utilizing planar holograms that were positioned at a fixed distance from the user, resulting in co-registration errors that varied according to changes in distance. We used the calibration printed circuit board and placed the hologram at different offset positions from the calibration board plane. The co-registration error was evaluated as the hologram offset from the calibration board plane, which was incrementally adjusted in 5 mm increments, while the operator’s viewing angle remained fixed at $\sim 60\mathrm{deg}$.

2.5.

FGS Simulation Study

To evaluate the impact of 2D display monitors and 3D holograms on depth perception while displaying fluorescent information to an end user, we conducted a study using six 3D-printed phantoms with varying levels of hills and valleys. The phantoms were painted black to minimize visual cues to identify the 3D topology. Each valley was filled with $100\mu \mathrm{L}$ of ICG, which fluoresced under 780 nm laser excitation ($20\mathrm{mW}/{\mathrm{cm}}^2$) and was detected by the bioinspired image sensor.

We recruited four participants with a minimum of 5 years of experience in animal handling, including administering tumor-targeted dyes and performing tumor resections. The participants were asked to locate and delineate the borders of the fluorescence signals on each of the phantoms using a pointed sharpie, either by viewing information displayed on a 2D monitor or using our 3D holographic system. A curtain was installed to ensure that the participants only relied on the 2D monitors having monocular depth to make decisions. The distance between the participant’s estimation of the target location and the actual depth was recorded for each phantom.

3.1.

Co-registration of Hologram with Natural Vision

Hologram co-registration with the operator’s natural vision is an important system requirement to ensure accurate localization of fluorescent targets. In the previously published study, we utilized affine transformation to ensure that the Euclidean distances between the LEDs on the calibration board matched the distances between the corresponding regions on the hologram.³¹ However, affine transformation, which preserves line parallelism and ratios of distances along parallel lines, does not account for the effects of perspective distortion. In this work, we were able to address this concern by implementing projective transformation or the homography technique.

Figures 3(a)–3(e) display the various stages of the co-registration routine. First, the hologram was positioned and aligned inside the perimeter of the calibration board [Fig. 3(a)], which consisted of four NIR LEDs located at predetermined relative positions. Next, the NIR signals from the LEDs were captured by the image sensor [Fig. 3(b)]. The contours of each of the detected regions, which were used to compute the homography, are displayed in Fig. 3(c). After calibration, the hologram was co-registered with the locations of the LEDs on the board. Figures 3(d) and 3(e) display the composite image of the calibration board and hologram before and after the co-registration routine, respectively. Figure 3(f) shows the outcome of the co-registration routine using a vial of excited quantum dots. The fluorescent signal displayed in false green was co-registered with the location of the quantum dots within the vial.

Fig. 3

Procedure for co-registering hologram with natural vision, which consists of several stages. (a) The hologram is displayed within the borders of the calibration board. (b) Four NIR LEDs mounted on the board are detected by the bioinspired camera. (c) The detected NIR regions are outlined. (d) Co-registration before calibration and (e) after calibration is illustrated. (f) Co-registration is demonstrated using a vial of quantum dots. (g) The co-registration error model is shown, displaying the dependence of error ($e_i$) on the offset from the calibration plane ($h_i$) and viewing angle ($\theta$). (h) The error computed from the error model is compared with results obtained from the chessboard pattern at different offsets from the calibration board plane.

We then assessed the co-registration error at different depths relative to the calibration board. Figure 3(g) illustrates an error model describing how co-registration of natural vision with the hologram deteriorates as a function of hologram offset from the calibration board plane and the view angle relative to the same plane. We conducted an experiment to evaluate the co-registration accuracy by recording the distance between the center of a specific LED of the calibration board and the corresponding point in the hologram that was offset from the calibration board plane in increments of 5 mm. The view angle was maintained at $\sim 60\mathrm{deg}$. Figure 3(h) displays the theoretical error along with the measured errors. The equation below describes the mathematical model that quantifies the co-registration error ($e_i$) between the real-world point $P_i$ and its projection onto the hologram $P_{i^{\prime }}$, as a function of the hologram’s offset from the calibration board plane ($h_i$) and the operator’s viewing angle ($\theta$) relative to the plane:

Based on this equation, for a given offset, the co-registration error is also influenced by the viewing angle. A viewing angle closer to 90 deg would result in a smaller error, given a specific offset. A good correlation was observed between the model and the observations from the study. The co-registration error linearly increased as a function of the offset of the hologram from the calibration board. When the planar hologram was offset by 5 mm from the calibration board plane, a co-registration error of 2 mm was observed between the center of the LED and its corresponding location in the hologram. In a surgical setting, the co-registration error will vary based on the offset between the region with strong fluorescence and its corresponding area within the hologram. Hence, it is crucial to make precise manual adjustments to the hologram’s position before starting the procedure to minimize this offset and effectively reduce the co-registration error.

3.2.

Image Compression for Real-Time Display on the HoloLens

To reduce the delay between image capture from the bioinspired sensor and image display on the HoloLens, the 24-bit NIR image (excluding the alpha channel) was compressed. We assessed the quality of the image displayed on the HoloLens by visually evaluating it after compressing it to 12- and 6-bit depths. The color gradient image, used for the test, was projected onto the HoloLens before and after the image compression, as shown in Fig. 4. The original 24-bit image displayed a wide color range with seamless transitions between colors [Fig. 4(a)]. The hologram captured from the HoloLens, using the mixed reality capture tool, demonstrated that the system could preserve the image quality when the original 24-bit image was compressed to 12-bit as the color transitions appeared continuous [Fig. 4(b)]. However, when the image was further compressed to 6-bit [Fig. 4(c)], significant degradation in the image quality was observed with discrete transitions between colors. In a surgical setting, it is essential to display a wide color gamut to differentiate regions with high fluorescence from those with low fluorescence. Qualitatively, we demonstrated that we were able to achieve this requirement satisfactorily with 12-bit compression, resulting in an average delay of 50 ms between image capture and display on the HoloLens. Consequently, we adopted the 24-bit to 12-bit compression scheme for all of our studies.

Fig. 4

HoloLens displays color gradient images with different bit depths. (a) The original image with 24-bit depth, (b) the image compressed to 12-bit depth, and (c) the image compressed to 6-bit depth. The 12-bit image provides a good balance between reducing data transmission and maintaining a rich and continuous color gamut for the rainbow target.

3.3.

Evaluating Target Delineation Accuracy with Holographic versus 2D Monitor Displays

A longstanding question in fluorescence-guided surgery is whether 3D display goggles provide better localization accuracy of the tumor/target location compared with 2D display monitors and how to measure these differences. Our simulation study involved a set of 3D-printed phantoms that consisted of various hills and valleys to simulate a complex navigational environment. Figure 5(a) illustrates the imaging setup employed for the study with the participant utilizing the holographic display. Figure 5(b) depicts one of the phantoms employed in the study. To improve the visualization of the phantom’s underlying geometry, texture was incorporated into the rendering. The lower images in Fig. 5(b) showcase the actual phantoms utilized, arranged in ascending order of depth. The width and depth of each phantom were randomly chosen. To minimize visual cues about the 3D shape of the object, the phantoms were painted with solid black and the valleys were filled with $100\mu \mathrm{L}$ of ICG solution. The phantom was then placed in a simulated operating room installed with the imaging sensor and light source. The task for the participant was to mark the boundary of the fluorescent target using a marker while avoiding markings outside the target area.

Fig. 5

(a) Imaging setup for the simulation study. The left image illustrates the positioning of the camera, illumination optics, and the estimated placement of the phantom (indicated by the red outline labeled as “sample”). The right image showcases a study involving a participant using the holographic display. (b) Phantoms with varying valley depths used in the fluorescence-guided surgery simulation study. The bottom images show the phantoms arranged in increasing order of valley depth. (c) Results comparing estimation error of fluorescent dye location using the 2D monitor and the holographic goggles. The plot shows the mean and standard deviation of the error for each phantom as a function of valley depth, indicating that holographic goggles with stereoscopic vision offers better depth perception compared with 2D monitors.

We enrolled four participants who possessed a minimum of 5 years of experience in animal handling, which involved tasks such as administering tumor-targeted dyes and conducting tumor resections. Each operator was assigned a phantom randomly. Initially, participants were instructed to perform the task using the information displayed on a 2D screen, where NIR fluorescence images were superimposed on color images. Subsequently, the operators repeated the task using the holographic display, where the NIR fluorescence images were projected as holograms after undergoing the calibration process.

To eliminate any visual biases toward the target that may assist in its delineation, a small curtain was placed between the operator and the phantom when fluorescent information was displayed on the 2D monitor. This experimental setup allowed us to simulate a scenario in which the surgeon solely relied on real-time color and fluorescent information captured by intraoperative sensors when making decisions during surgery. For instance, during lung cancer surgeries involving the resection of small residual tumors marked by tumor-targeted fluorescent probes, the surgeon typically depends exclusively on information provided by intraoperative instruments. However, it is important to note that our experimental setup does not fully replicate scenarios in which the surgeon relies on a variety of cues, including natural vision and intraoperative imaging tools, during SLN mapping or large tumor resection.

Figure 5(c) displays the mean and standard deviation of the estimation error versus the depth of phantom, which were obtained after measuring the distance between the actual location of the ICG solution in the phantom and the participant’s estimation of its location. The plot shows the estimation error for each phantom as a function of the valley depth. The results demonstrate that the participants were able to locate and delineate the fluorescence signal accurately when the valley depth was shallow, resulting in minimal estimation errors. However, an increase in the valley depth led to a proportional increase in the error, indicating impairment of the participant’s depth perception. In contrast, the holographic goggles featuring stereoscopic vision notably reduced the error by providing binocular depth cues that resemble the human visual system in a natural environment.

3.4.

Imaging Small Animal Models of Breast Cancer with Holographic System

We employed the imaging system to assess in vivo imaging and holographic display in small animal models with breast cancer. In a previous study, we conducted an evaluation of the sensitivity and specificity of the single chip, bioinspired imaging sensor for the in vivo detection of prostate tumors using single or multiple tumor-targeted probes (IRDye800CW 2-DG and IRDye680RD EGF).¹⁹ This assessment was performed on seven murine models of prostate cancer. When either one of the tumor-targeted probes was administered to the tumor-bearing mice, the area under the curve was $\sim 75\%$ due to intratumor biomarker variations. However, when both probes were used together to account for tumor heterogeneity, the area under the curve improved to $\sim 92\%$. These experiments confirmed the imaging sensor’s capability to image multiple fluorescent probes with a single imaging chip and improve cancer detection statistics.

Subsequently, we demonstrated the detection and holographic display of fluorescence, co-registered with the tumor locations, using our system in vivo. This was accomplished by utilizing tumor-bearing mice labeled with ICG and three distinct types of IRDye 800 (native, 2DG-conjugated, and RGD-conjugated). The mice were imaged using our holographic system 6 h post-administration of the NIR fluorescent dyes. Figure 6(a) demonstrates the co-registration between the holographic fluorescence image and the tumor location, both with and without the laser excitation sources activated. Following the resection and removal of the tumor, the virtual fluorescence signal, which had been calibrated and superimposed on the tumor, continued to accurately track its location, as shown in Video 1. The fluorescence signal is highlighted in false green to enhance its visibility for demonstration. However, the residual fluorescence within the surgical cavity was not further evaluated in this study as our objective was focused solely on demonstrating co-registration of holograms to tumors during intraoperative settings.

Fig. 6

(a) Holographic display of fluorescence image of a tumor-bearing mouse with excitation source turned off and on. (b) Ex vivo fluorescence image of resected tumor, liver, kidneys, and muscle tissue with corresponding H&E-stained histological images of a tumor and muscle tissue. Scale bar: $100\mu \mathrm{m}$. (c) In vivo fluorescence images of mouse models with breast cancer captured by the HoloLens 6 h post-administration of various NIR fluorophores, including IR800CW (10 nM), IR800CW-2DG ($200\mu \mathrm{M}$), IR800CW-RGD ($20\mu \mathrm{M}$), and ICG ($640\mu \mathrm{M}$), demonstrating the system’s adaptability to various clinical applications (Video 1, MP4, 15.8 MB [URL: https://doi.org/10.1117/1.JBO.28.9.096003.s1]; Video 2, MP4, 6.83 MB [URL: https://doi.org/10.1117/1.JBO.28.9.096003.s2]; Video 3, MP4, 8.97 MB [URL: https://doi.org/10.1117/1.JBO.28.9.096003.s3]; and Video 4, MP4, 11.3 MB [URL: https://doi.org/10.1117/1.JBO.28.9.096003.s4]).

Videos 2–4, featuring two mice, also exhibit the co-registration of the holographic fluorescence signal projected onto the display. Videos 2 and 3 showcase both translation and rotation along the vertical axis (yaw), and Video 4 mainly demonstrates rotation along the horizontal axis (pitch), resulting in variation in the WD. Despite rotational or translational movements, the position of the perceived hologram by the user remained unchanged and correctly superimposed on the tumor tissue. The co-registered hologram consistently aligned with the natural vision, regardless of changes in perspective. It is crucial to note that the observed slow translation and rotation motion can be attributed to the relatively low frame rate of the image sensor, which operates at $\sim 4$ frames per second. This is partly caused by factors such as the low accumulation of fluorescent probes in the tumor, weak laser excitation, and the sensitivity of the camera. Fluorescence emitted by the NIR probes was successfully detected in all eight mice, and this information was accurately superimposed at the appropriate anatomical location in every case. As a result, we achieved a 100% positive predictive value for the detection and display of fluorescent information in the murine model of breast cancer.

The ex vivo biodistribution study revealed varying levels of fluorescence in the harvested organs, demonstrating the preferential accumulation of the dyes at the tumor site and their clearance via the liver and kidneys [Fig. 6(b)]. Histological images of H&E-stained specimens of a tumor and healthy muscle tissue indicated the distinct cellular and tissue structures that confirmed malignancy in the specimen identified as tumor. Finally, the in vivo fluorescence images in Fig. 6(c) highlighted the system’s ability to detect and display the location of tumors with different types and doses of NIR fluorescent dyes, indicating its potential for various clinical applications. When imaging NIR fluorescence through the skin, as illustrated in Figs. 6(a) and 6(c), the fluorescence signal tends to be weak due to signal attenuation within the skin. As a result, the hologram is depicted in green to represent the lower fluorescent signal. However, in the case of ex vivo samples, this issue is not present, and they exhibit a higher fluorescent signal, as indicated by the false color range.

3.5.

Intraoperative Imaging of Canine Patients

To demonstrate the applicability of holographic display in the operating room, we utilized our holographic goggles to guide surgeons operating on canine patients with spontaneously occurring head and neck cancer. In standard head and neck cancer surgery, an SLN biopsy is performed to determine metastatic progression to the lymph node. The conventional method for the SLN biopsy in head and neck cancer involves the use of fluorophores such as ICG or MB that are administered into the lymphatic system near the tumor site, and surgeons rely solely on their unaided eyes to map and remove the visible fluorophore stained SLNs.¹² However, it should be noted that both ICG and MB exhibit NIR fluorescence when excited with an appropriate NIR light source. NIR fluorescence has the capability to radiate from SLNs situated just a few millimeters below the surface, rendering this approach more reliable for SLN mapping compared with visible detection.

In our study, we employed ICG as the fluorescent dye, which was excited using the NIR light source $\sim 15\min$ after its administration. Figure 7(a) demonstrates the co-registration procedure carried out by the clinical staff prior to the surgery. The image sensor was mounted directly above the canine patient as shown in Figs. 7(a) and 7(b). The resulting fluorescence signals were captured and displayed in real time on the holographic goggles to identify SLNs at the surgical site [Fig. 7(c)]. A hologram indicating the location of the SLN was projected and overlaid onto the operator’s visual field. Due to the wireless transfer of images, multiple users, including surgeons and two assistants, were able to access the holographic fluorescence images simultaneously. This facilitated active collaboration and enabled learning during the surgery [Figs. 7(a) and 7(d)]. Back-table imaging of the resected SLNs showed high-intensity fluorescence emitted from the center of the specimen. Both in vivo and ex vivo composite images demonstrated good co-registration of the NIR fluorescence images with the physical specimen and helped the surgeons to identify and resect the SLNs. After the resection was completed, the holographic lenses were removed from the surgeon by the operating room nurse and the surgical cavity was examined for any additional SLNs. Among the 7 canine patients included in our study, a total of 17 SLNs were detected, out of which 15 demonstrated NIR fluorescence. As a result, we achieved an 88% positive predictive value and 100% negative predictive value for the detection of SLNs using ICG fluorescence and displaying this information on the HoloLens.

Fig. 7

Intraoperative imaging of a canine patient with head and neck cancer using holographic display. (a) Co-registration routine carried out by the clinical staff prior to the surgery. (b) Imaging setup in the operating room with the canine patient at the beginning of the surgery. (c) Veterinarian surgeon wearing HoloLens during the surgical procedure. (d) In vivo and ex vivo back-table images showing real-time fluorescence holograms (green dotted box) superimposed over SLNs during and after resection, respectively. (e) HoloLens projecting real-time fluorescence and pre-loaded CT images simultaneously as a hologram during the surgical procedure (Video 5, MP4, 28.2 MB [URL: https://doi.org/10.1117/1.JBO.28.9.096003.s5]).

Figure 7(e) illustrates projection of pre-operatively acquired CT images onto the holographic display from the perspective of the assistant wearing the holographic display. The image also shows the illumination sources (NIR and white light) used for the surgery. The hologram was able to display both real-time fluorescence images and pre-loaded patient records simultaneously in the surgeon’s field of view during the surgery (Video 5). This showcases the adaptability of our imaging system to meet various surgical requirements in the operating room. The advantages of projecting supplementary pre-operative images include reducing disruptions in the surgical workflow and providing multimodal information that caters to various surgical requirements. Nonetheless, through the use of distinct voice commands, this information can be concealed to prevent any obstruction of the view of the surgical site.

One of the modifications in this setup involves the contrasting viewing perspectives of the surgeon and the multispectral camera, which is positioned above the surgical bedside. This difference in perspective may result in a shadowing effect, potentially leading to inaccurate detection and placement of fluorescent information. Although positioning the multispectral camera near the surgeon’s eyes or making it wearable on their head could help mitigate this issue, it is not feasible due to the camera’s size and the absence of the camera’s IMU information.