A.

Image synthesis and uncertainty estimation

A 3D conditional GAN was developed for CBCT-to-CT image synthesis. For training (Section II.C), a high-fidelity, physics-based forward projection framework (including an accurate beam model, absorption / scatter characteristics, and model of the imaging chain) was used to generate simulated CBCT images from corresponding CT images. Two alternative inputs to the synthesis network were investigated: (i) an uncorrected FBP , and (ii) a precorrected FBP for which a simple (constant) scatter correction was applied, hypothesizing that the precorrection to improve synthesis performance.

As illustrated in Fig. 1, a 3D GAN was implemented with a U-Net with a residual block at each level of the encoding / decoding path as the generator, and a convolutional pixel-wise classifier⁶ as the discriminator. The objective function combined GAN and L1 loss as follows:

Figure 1.

Network architecture of the generator for 3D image synthesis using a conditional GAN. Model uncertainty (𝜎) in the synthesis is computed via Monte-Carlo dropout layers.

where

G and D denote the generator and discriminator, and μ^CT and μ^CBCT represent paired CT and CBCT images. The L1 loss helps avoid over smoothing, and the balance between the GAN and L1 loss is controlled by λ.

As described in Gal and Ghahramani,⁷ dropout applied during network training is equivalent to a Bayesian approximation of the Gaussian process, and uncertainty in the model output can be estimated by computing the voxel-wise variance of multiple forward passes. Following such an approach, we added dropout layers (dropout rate = 0.2) prior to the skip connection in each encoder and decoder block and to the final output. Both training and inference were performed with dropout. The predictive mean computed from a collection of 8 network outputs yields the synthesized image (DL-Synthesis, μ^Syn), and the predictive variance (σ²) serves as a proxy for model uncertainty.

B.

The DL-Recon framework

The proposed method (termed DL-Recon) integrates 3D image synthesis with physics-based reconstruction via uncertainty associated with the synthesis model. The method involves three steps: (i) generation of a 3D synthetic CT image (μ^Syn) from a CBCT volume with estimation of model uncertainty (σ) as described above; (ii) physics-based 3D image reconstruction of projection data, including artifact corrections – for example, the pipeline described in [1] – to yield an artifact-corrected CBCT image ; and (iii) voxel-wise combination of μ^Syn and weighted by the estimated uncertainty to yield the DL-Recon image (denoted μ^DL–Recon). The resulting image is:

where uncertainty is contained within a spatially varying map (β, with values in the range [0, 1]) related by a sigmoid function:

where c₁ and c₂ specify the range and level, respectively, of the sigmoid, and β controls the contribution of μ^Syn and in a voxel-wise manner. When predictive uncertainty is high, the β map draws more from the physics-based reconstruction.

The underlying premise in this approach is that the synthesis image (μ^Syn) carries particular benefits (e.g., uniformity and noise reduction) but may be subject to systematic error – for example, in structures unseen in the training data. The uncertainty map [σ(x, y, z), alternatively β(x, y, z)] were shown previously in simulation studies [5] to correlate with deviations from ground truth. The “uncertainty map” therefore offers insight on where the synthesis image may be subject to error and where it is advantageous to draw more from the physics-based 3D image reconstruction .

Note that the physics-based method incorporated in DL-Recon could be FBP or any particular form of MBIR, recognizing that the latter may invite disadvantages of computational load associated with conventional iterative optimization. Alternatively, the synthesis image could be incorporated as a prior within a penalized optimization, as in [5]. In any of these scenarios, the voxel-wise weighting of synthesis and physics-based image reconstructions is intended to leverage the strengths of each, mediated by the model uncertainty. In the work reported below, DL-Recon incorporates (artifact-corrected) FBP reconstruction as a practical implementation that may be compatible with the rapid runtime requirements of image-guided surgery, focusing here on intracranial neurosurgery.

C.

Training data generation

To obtain a large training dataset of matched CT and CBCT images, CBCT projection data were simulated from 35 real, helical CT volumes of 35 healthy subjects using a high-fidelity forward projector [5]. CBCT system geometry and image acquisition were simulated to match data (~745 views over 360°) acquired from the O-arm (the O-arm^TM “O2” imaging system, Medtronic) using nominal head scan protocols (100–120 kV and 75–240 mAs). Volumes were reconstructed with isotropic 0.7 mm voxels via FBP without artifact correction.

Signal normalization linearly transformed the CBCT intensity histogram within the brain parenchyma to [-1, 1]. Volumetric patches (64×64×64 voxels) were stochastically sampled from the brain volume and fed to the network, and a total of 875 patches were used for training. The Adam optimizer (learning rate = 5 × 10^-5, β₁ = 0.5, β₂ = 0.999, L1 regularization λ=100, and batch size = 2) was used and early stopping at 800 epochs was applied.

D.

Experimental studies

D.1.

Image synthesis of simulated and real brain CBCT images

The proposed image synthesis method was validated on both simulated and real CBCT data. Simulated CBCT projections of 5 test CT volumes were generated and reconstructed in the same manner as the training set. Intensity differences between synthesized images and ground truth were measured within the brain region for each volume. Experiments were conducted using the O-arm™ system illustrated in Fig. 2. Real projection data for 3 cadaveric heads (denoted below as cadaver #1-3) were collected at 120 kV and 150 mAs. Volumetric images were reconstructed on a grid of 320×320×280 voxels with isotropic 0.7 mm voxels. The runtime of DL-Synthesis was ~1 min per prediction (NVIDIA TITAN Xp). DL-Synthesis images were evaluated with uncorrected CBCT as input and with a basic (constant-scatter) precorrection . Method performance was quantified in terms of image non-uniformity (NU), the difference in mean voxel value between region of interests (ROIs) in the parenchyma near the dural surface / sphenoid bone and about the lateral ventricles.

Figure 2.

Experimental setup for cadaver studies using the O-arm.

A.

Performance of image synthesis

Fig. 3 shows results of image synthesis on simulated data (high-fidelity CBCT projections generated from CT). DL-Synthesis demonstrated good overall correspondence with the ground truth CT, yielding high image uniformity and reduced noise compared to the uncorrected FBP image. In 5 test volume images, DL-Synthesis exhibited a difference in overall mean intensity (in the brain) of less than 1 HU (compared to > 12 HU for FBP) to the ground truth, with residual differences owing mainly to image noise. The estimated uncertainty highlights regions with anatomical variations such as the lateral ventricles and sulci in the cerebral cortex, which is susceptible to error (e.g., contrast loss) in the synthesis mage.

Figure 3.

Synthesis performance in simulated CBCT data. (a) Sagittal slice of a test CT image volume. (b) Corresponding CBCT reconstruction (network input). (c) Resulting synthesized image. (d) Violin plot quantifying the respective difference in voxel values of uncorrected FBP and DL-Synthesis to the ground truth measured for 5 test data. (e) Sagittal and (f) axial slice of the estimated uncertainty.

Fig. 4 illustrates the performance of image synthesis on real data, in which the input to the synthesis network was either uncorrected or precorrected image data. DL-Synthesis acting on uncorrected FBP input exhibits performance degradation in regions affected by severe artifacts, yielding a higher degree of non-uniformity near the sphenoid bone (yellow arrow). A simple (constant) scatter correction was shown to partially account for biases that were not modeled by the forward projector (e.g., variation in bone density) and improve the overall image uniformity (2–4 HU). As a result, precorrected FBP yielded more accurate synthesis, reducing image NU by ~50% compared to synthesis acting on uncorrected FBP. However, DL-Synthesis exhibited a loss in contrast in structures such as the lateral ventricles (cadaver #1, magenta arrows), demonstrating potential pitfalls in the generalizability of image synthesis to real and highly variable image data.

Figure 4.

Synthesis performance for (a) uncorrected and (b) precorrected FBP of real CBCT images. (c) Boxplot quantifying image non-uniformity in synthesized images of 3 cadavers.

B.

Uncertainty estimation in cadaver studies

Fig. 5 demonstrates the performance of uncertainty estimation on real data with unseen features (calcium deposit in cadaver #2 and brain shift in cadaver #3). For both cases, the uncertainty map highlights the location of the unseen structure as well as at the lateral ventricles, suggesting a lack of reliability in the synthesis result and the need for input from physics-based reconstruction.

Figure 5.

Uncertainty estimation in cadaver CBCT head images. Precorrected FBP and the estimated uncertainty (β map) within the brain parenchyma for (a-b) cadaver #1 with calcium deposit and (c-d) cadaver #3 with brain shift.

C.

Performance of DL-Recon

Fig. 6 shows reconstructed images from conventional methods (FBP and DL-Synthesis) and the proposed DL-Recon framework. As shown in Fig. 6(b), the comprehensive artifact correction pipeline reduced NU by 59%, but led to 38% increase in image noise. DL-Synthesis yielded the lowest NU value and noise but suffered from loss in soft-tissue contrast. In comparison, DL-Recon was able to reduce both NU and noise while preserving image contrast of the ventricles, providing ~15% increase in soft-tissue CNR compared to fully corrected FBP.

Figure 6.

Example axial and sagittal slices of FBP, DL synthesis, and DL-Recon in cadaver CBCT data. Measurements of image non-uniformity (NU), noise, and contrast-to-noise ratio (CNR) of the lateral ventricles are listed below each image. Difference images show the contributions of the physics-based [(d)-(a)] and image synthesis [(d)-(c)] methods to the DL-Recon image (approximate Hounsfield Units).

The intensity profile of a curve across the brain [yellow dashed curve shown in Fig. 6(b)] was plotted in Fig. 7 for fully corrected FBP, DL-Synthesis, and DL-Recon. Fully corrected FBP exhibited residual nonuniformity, especially just inside the cranium due to residual beam-hardening effects, as indicated by the nonuniform intensity profile between the ventricle and cranium. DL-Synthesis improved uniformity in these regions but reduced the contrast in the ventricle, similar to the effects shown above in relation to model uncertainty. By comparison, DL-Recon maintained the benefits of image uniformity from DL-Synthesis while achieving contrast in the ventricles similar to the fully corrected FBP.

Figure 7.

Intensity profiles of FBP, DL-Synthesis, and DL-Recon. The DL-Recon image leverages the improved uniformity of DL-Synthesis (region just inside the cranium) and the improved (accurate) contrast of fully corrected FBP (in the ventricles).