1.1.

Clinical Background

In total hip arthroplasty (THA), also referred to as total hip replacement, the damaged bone and cartilage are replaced with prosthetic components. The procedure relieves pain and disability with a high success rate. In 2010, there were $\sim \mathrm{330,000}$ THAs performed in the US. This is projected to increase to 570,000 THAs by 2030¹ as younger patients and patients in developing countries are considered for THA. Together with a prolonged life expectancy, the consideration of younger, more active patients for THA suggests that implant longevity is of increasing importance as it is associated with the time to revision arthroplasty.¹ The time to repeat surgery is affected by the wear of the implants that is correlated with their physical properties as well as acetabular component positioning. Poor placement leads to increased impingement and dislocation that promotes accelerated wear. Conversely, proper implant placement that restores the hip anatomy and biomechanics decreases the risk for dislocation, impingement, loosening, and limb length discrepancy and thus implant wear and revision rate.^2–6 Steps to ensure accuracy and repeatability of acetabular component positioning are therefore essential. Due to the large volume of THA procedures, small but favorable changes to the risk-benefit profile of this procedure enabled by improved implant positioning will have a significant impact on a large scale.

Unfortunately, optimal placement of the acetabular component is challenging due to two main reasons. First, the ideal position of the implant with respect to the anatomy is unknown, yet a general guideline exists⁷ and is widely accepted in clinical practice. This guideline suggests abduction and anteversion angles of the hip joint measured with respect to bony landmarks defining the so-called safe zone, which is indicative of an acceptable outcome. Recent studies suggest that an even narrower safe zone may be necessary to minimize the risk of hip dislocation.^8,9 Defining the ideal implant position is not as straightforward as the definition of a range of abduction and anteversion angles when considering a large population.¹⁰ A static definition of the safe zone seems even more prone to error when considering that the position of the pelvis varies dramatically from supine to sitting to standing posture among individuals.^11,12

Second, even if a clinically acceptable safe zone is known, it is questionable whether surgeons are, in fact, able to accomplish acetabular component placement within the suggested margin.⁹ In light of previous studies that report malpositioning of up to 30% to 75%^13–15 when free-hand techniques are used, addressing this challenge seems to be imperative.

Most computer-assisted methods consider the direct anterior approach (DAA) to the hip for THA as it allows for convenient integration of intraoperative fluoroscopy to guide the placement of the acetabular component.¹⁶ The guidance methods reviewed below proved effective in reducing outliers and variability in component placement, which equates to more accurate implant positioning.^17–20

1.2.

Related Work

External navigation systems commonly use certain points on the anatomy of interest, as decided by the surgeon, and conform to a “map” of the known morphology of the anatomy of interest. Despite the fact that THA commonly uses x-ray images for navigation and preoperative patient CT may not be available, several computer-assisted THA solutions suggest planning the desired pose of the acetabular component preoperatively on a CT scan of the patient.^21,22 Preoperative CT imaging allows planning of the implants in 3-D, automatically estimating the orientation of the natural acetabular opening, and predicting the appropriate size of the cup implant.²³

Navigation-based THA with external trackers are performed based on preoperative patient CT or image-less computer assisted approaches. The planning outcome in a CT-based navigation approach is used intraoperatively with external optical navigation systems to estimate the relative pose of the implant with respect to the patient anatomy during the procedure. Tracking of the patient is commonly performed using fiducials that are drilled into the patient’s bones. Registration of the preoperative CT data to the patient on the surgical bed is performed by manually touching anatomical landmarks on the surface of the patient using a tracked tool.²¹ In addition to the paired-point transformation estimated by matching the few anatomical landmarks, several points are sampled on the surface of the pelvis and matched to the segmentation of the pelvis in the CT data.²⁴ CT-based navigation showed statistically significant improvement in orienting the acetabular component and eliminating malpositioning, while resulting in increased blood loss, cost, and time for surgery.^25,26 Combined simultaneous navigation of the acetabulum and the femur was used in 10 clinical tests, where the surgical outcome based on postoperative imaging showed 2.98 mm and 4.25 deg error in cup position and orientation, respectively.²⁷

Image-less navigation systems do not require any preoperatively acquired radiology data. In this method, the pelvic plane is located in 3-D by only identifying anatomical landmarks on the surface of the patient using a tracked pointer reference tool and optically visible markers attached to the patient.²⁸ This approach showed improvement in terms of cup positioning.²⁹ However, a few number of samples points for registration as well as pelvis tilts resulted in unreliable registration.³⁰

Robotic systems are developed to provide additional confidence to the surgical team in placing implants during THA.^31,32 In a robotic system, pins are implanted into the patient’s femur prior to acquiring a preoperative CT scan. After the surgeon has performed the planning on the CT data, the robot is introduced into the operating room. To close the registration loop between patient, robot, and CT volume, each preoperatively implanted pin is touched by the robot with manual support. To eliminate the need for fiducial implantation, registration is either achieved by selecting several points on the surface of the bone using a digitizer and using an iterative closest point algorithm to perform registration to patient CT data,³³ or by using intraoperative C-arm fluoroscopy and performing 2-D/3-D registration between the x-ray image and CT volume.³⁴ After registering the preoperative CT data to the patient, the robot assists the surgeon in placing the femoral stem and the acetabular component according to the planning. The outcome of 97 robot-assisted THA procedures indicates performance similar to the conventional technique;³⁵ however, in some cases, additional complications, such as nerve damage, postoperative knee effusion, incorrect orientation of the acetabular component, and deep reaming resulting in leg length discrepancy, were reported when the robotic system was used. To assist the surgeon in placing implants for joint replacement procedures, haptic technology was integrated into robotic solutions to maintain the orientation of the cup according to preoperative planning and to control the operator’s movement.³⁶

If preoperative CT is available, intensity-based 2-D/3-D registration can be used to evaluate and verify the positioning of the acetabular component postoperatively.²² This is done by recovering the spatial relation between a postoperative radiograph (2-D) and the preoperative patient CT data (3-D), followed by a registration of the 3-D CAD model of the component to the 2-D representation of the cup in the postoperative radiograph. To overcome the large variability in individual patient pelvic orientations and to eliminate the need for preoperative 3-D imaging, the use of deformable 2-D/3-D registration with statistical shape models was suggested.³⁷

Aforementioned solutions perform well but require preoperative CT, which increases the time and cost for surgery and requires intraoperative registration to the patient.^25,32 Zheng et al.³⁸ proposed a CT-free approach for navigation in THA. The method relies on tracking of the C-arm, surgical instruments used for placing femoral and acetabular components, and the patient’s femur and pelvis using an external navigation system. Multiple stereo C-arm fluoroscopy images are acquired intraoperatively. Anatomical landmarks are then identified both in these x-ray images as well as percutaneously using a point-based digitizer. Due to the tracking of the C-arm, the relative pose between the x-ray images is known; therefore, the anatomical landmarks are triangulated from the images and reconstructed in 3-D. These points are used later to define the anterior pelvic plane and the center of rotation for the acetabulum. After estimating the pelvis coordinate frame, the impactor is moved by the surgeon until the cup is at the correct alignment with respect to a desired orientation in the anterior pelvic plane coordinate frame. This work reported subdegree and submillimeter accuracy in antetorsion, varus/valgus, and leg length discrepancy. Later, this system was tested in 236 hip replacement procedures, where a maximum of 5-deg inclination error and 6-deg anteversion error was observed.³⁹

The state-of-the-art approaches that provide guidance using image-less or image-based methods have certain drawbacks. Image-less methods require complex navigation and may provide unreliable registration.³⁰ Image-based solutions rely on preoperative CT scans or intraoperative fluoroscopy and often use external navigations systems for tracking.^40,41 Systems based on external navigation are expensive and increase the operative time due to the added complexity. Use of preoperative CT scans increases the radiation exposure and cost to the patient. Moreover, many of the methods used for registering CT to patient seek to solve ill-posed problems that require manual interaction either for initialization or landmark identification and, thus, disrupt the surgical workflow. Manual annotations can take between 3 to 5 min during the intervention for each image registration.²¹ Although proven beneficial for the surgical outcome, neither of these costly and labor-intensive navigation techniques were widely adopted in clinical practice.

Partly due to above drawbacks, surgeons who use the DAA often rely solely on fluoroscopic image guidance.^16,42 These images, however, are a 2-D representation of 3-D reality and have inherent flaws that complicate the assessment. The challenges include finding the true anterior pelvic plane as well as eyeballing acetabular component position by eye on the image. Therefore, a technique that provides a quantitative and reliable representation of the pelvis and acetabular component intraoperatively without increasing either radiation dose or cost, while largely preserving the procedural workflow, is highly desirable.

1.3.

Proposed Solution

This work proposes an augmented reality (AR) solution for intraoperative guidance of THA using DAA, where the C-arm is kept in place until the correct alignment of the acetabular cup is confirmed.^43,44 With the proposed solution, the surgeon first plans the position of the acetabular cup intraoperatively based on two fluoroscopy images that are acquired after the dislocation of the femoral head and the reaming of the acetabulum are completed. The orientation of the cup in the x-ray images could be either automatically preset based on desired angles relative to the APP plane (or other known pelvic coordinate frames) or adjusted by the surgeon. Once the desired pose of the acetabular cup is estimated relative to the C-arm, we use optical information from the cocalibrated RGBD camera that is mounted on the C-arm to provide an AR overlay^45–47 that enables placement of the cup according to the planning. As the cup is not visible in RGBD, we exploit the fact that the acetabular cup is placed using an impactor that is rigidly attached to the cup and is well perceived by the RGBD camera. For accurate cup placement, the surgeon aligns the optical information of the impactor (a cloud of points provided by the RGBD camera) with the planned virtual impactor-cup, which is visualized simultaneously in our AR environment. A schematic of the proposed clinical workflow is shown in Fig. 1.

Fig. 1

After the femoral head is dislocated, the size of the acetabular implant is identified based on the size of the reamer. Next, two C-arm x-ray images are acquired from two different perspectives. While the C-arm is repositioned to acquire a new image, the relative poses of the C-arm are estimated using the RGBD camera on the C-arm and a visual marker on the surgical bed. The surgeon then plans the cup position intraoperatively based on these two stereo x-ray images simultaneously. Next, the pose of the planned cup and impactor is estimated relative to the RGBD camera. This pose is used to place the cup in a correct geometric relation with respect to the RGBD camera and visualize it in an AR environment. Finally, the surgeon observes real-time optical information from the impactor and aligns it with the planned impactor using the AR visualization. The green boxes in this figure highlight the contributions of this work.

2.1.

Cocalibration of the RGBD-C-Arm Imaging Devices

The cocalibration of the RGBD camera and the x-ray imaging devices is performed using a multimodal checkerboard pattern. In this hybrid checkerboard pattern, each black square is backed with a radiopaque thin metal square of the same size.⁴⁸ Calibration data are acquired by simultaneously recording RGB and x-ray image pairs of the checkerboard at various poses. Next, we estimate the intrinsic parameters for both the RGB channel of the RGBD sensor as well as for the x-ray imaging device. Using these intrinsic parameters, we estimate the 3-D locations of each checkerboard corner, ${\mathbf{O}}_{{\mathrm{CH}}_{\mathrm{RGB}}}$ and ${\mathbf{O}}_{{\mathrm{CH}}_{\mathrm{X}}}$, in the RGB and x-ray coordinate frames, respectively. The stereo relation between the x-ray and RGB imaging devices ${\mathrm{X}}_{{\mathbf{T}}_{\mathrm{RGB}}}$ is then estimated via least squares minimization:

The stereo relation between the RGB and IR (or depth) channel of the RGBD sensor is provided by the manufacturer. To simplify the notation, we use ${{}^{\mathrm{X}}\mathbf{T}}_{\mathrm{RGB}}$, which embeds the relation between the RGB, depth, and x-ray imaging devices. We assume that both extrinsic parameters between x-ray and RGBD and the intrinsic parameters of the x-ray remain constant; however, both quantities are subject to minor change while the C-arm rotates to different angles, an observation that is further discussed in Sec. 4. Figure 2(a) illustrates the spatial relation between the RGBD camera and the x-ray source.

Fig. 2

In the transformation chain of the RGBD-C-arm system for THA (a), the RGBD, x-ray, visual marker, and acetabular cup coordinate frames are denoted as RGBD, X, M, and C, respectively. In an offline calibration step, the extrinsic relation between the RGBD and x-ray (${{}^{\mathrm{X}}\mathbf{T}}_{\mathrm{RGBD}}$) is estimated. Once this constant relation is known, the pose of the x-ray source can be estimated for every C-arm repositioning (b) by identifying displacements in the RGBD camera coordinate frame.

2.2.

Vision-Based Estimation of C-Arm Extrinsic Parameters

The stereo relation between C-arm x-ray images acquired at different poses is estimated by first tracking visual markers in the RGBD camera coordinate frame and then transforming the tracking outcome to the x-ray coordinate frame:

2.3.

Intraoperative Planning of the Acetabular Cup on Two x-ray Images

Planning of the acetabular component is performed in a user interface, where the cup could be rotated and translated by the surgeon in 3-D with six degrees of freedom (DoF) rigid parameters and is forward projected ($p_{c_v}$ and $p_{c_v}^{\prime }$) onto the planes of the two x-ray images acquired from different perspectives:

In addition, if the desired orientation of the cup is known relative to an anatomical coordinate frame (e.g., APP plane) and an x-ray image is acquired from a known perspective in relation to that anatomical frame (e.g., AP view), then the orientation of the cup could be automatically adjusted for the user (equivalent to presetting the orientation in ${}^{\mathrm{C}}{\mathbf{T}}_{\mathrm{X}}$). It is worth emphasizing that in several image-guided orthopedic procedures, x-ray images are frequently acquired from the AP view.

The transparency of the cup is adjusted by the surgeon in the user interface such that the ambiguity between the front and the back of the cup is optimally resolved. Finally, the contours around the edge of the cup are estimated and visualized by thresholding the dot product of the unit surface normal ${\mathbf{n}}_v$ and the intersecting ray ${\mathbf{r}}_v$:

The planning of an acetabular cup based on two x-ray images is shown in Fig. 3.

Fig. 3

(a) and (b) The acetabular component is forward projected from an initial 3-D pose onto the respective x-ray image plane. (c) and (d) The surgeon moves the cup until satisfied with the alignment in both views. The x-ray images shown here are acquired from a dry pelvis phantom encased in gelatin. (e) A cubic visual marker is placed near the phantom but outside the x-ray field of view to track the C-arm.

2.4.

Augmented-Reality Visualization

Once the desired cup position is known, guidance of the cup placement using an impactor with an AR visualization is needed to ensure a positioning in agreement with the planning. To construct the AR environment, we first estimate the pose of the RGBD sensor relative to the planned cup as follows:

Within the AR environment, we then render a 3-D mesh of the cup and impactor superimposed with the real-time cloud of points observed by the camera, all in the RGBD coordinate frame. In the interventional scenario, the acetabular cup is hidden under the skin and only the impactor is visible. Therefore, the surgeon will only align the cloud of points from the impactor, a cylindrical object, with the 3-D virtual representation of the planned impactor.

Ambiguities in the AR environment, among others occlusions or the rendering of a 3-D scene in a 2-D display, are eliminated by showing different perspectives of the scene simultaneously. Thus, it is ensured that the surgeon’s execution fully matches the planning once alignment of the current cloud of points of the impactor and the planned model is achieved in all perspectives. We provide an intuitive illustration of these relations in Fig. 4.

Fig. 4

(a) and (b) Multiple virtual perspectives of the surgical site are shown to the surgeon (c) before the cup is aligned. (d)–(f) The impactor is then moved by the user until it completely overlaps with the virtual planned impactor.

To solely visualize the moving objects (e.g., the surgeon’s hands and tools), background subtraction of point clouds is performed with respect to the first static frame observed by the RGBD camera. It is important to note that in an image-guided DAA procedure, most tools other than the impactor, such as retractors, are removed prior to placing the acetabular component; therefore, important details on the fluoroscopy image are not occluded.

3.1.

System Setup

In DAA for THA, the detector is commonly positioned above the surgical bed. This orientation of the C-arm machine is considered to reduce the scattering to the surgical crew. Therefore, we modified the C-arm machine by mounting the RGBD camera near the detector plane of the C-arm, which then allows the detector to remain above the bed. The mount for the RGBD camera extends out from the C-arm detector for nearly 5.00 cm in the XY direction ($Z$ being the principal axis of the x-ray camera) and is screwed to a straight metal plate, which is rigidly tied to the image intensifier. Considering that the RGB camera is used for pose estimation of the C-arm scanner while the depth camera is used for point cloud observation, the RGBD camera needs to be angled such that a maximum of the surgical site is visible in both RGB and depth camera views. The RGBD sensor is, therefore, angled such that it has a direct view onto the surgical site such that the principal axis of the camera is close to the isocenter of the C-arm.

The impactor used for testing is a straight cylindric acetabular trialing from Smith and Nephew. For intraoperative x-ray imaging, we use an Arcadis Orbic 3-D C-arm (Siemens Healthcare GmbH, Forchheim, Germany) with an isocenteric design and an image intensifier. The RGBD camera is a short-range Intel RealSense SR300 (Intel Corporation, Santa Clara, California), which combines depth sensing with HD color imaging. Data transfers from C-arm and the RGBD camera to the development PC are done via Ethernet and powered USB 3.0 connections, respectively.

The AR visualization is implemented as a plug-in application in ImFusion Suite using the ImFusion software development kit (ImFusion GmbH, Germany). We use ARToolkit for visual marker tracking.⁴⁹

3.2.

Experimental Results

3.2.2.

Accuracy in tracking x-ray poses

Tracking accuracy is computed by acquiring x-ray images from a phantom with several radiopaque landmarks and measuring the stereo error between the corresponding landmark points in different images.

The phantom is constructed by attaching nine radiopaque mammography skin markers (bbs) with diameters of 1.5 mm, inside and near the acetabulum on a pelvis model, as shown in Fig. 5. Next, we acquired 11 x-ray images from $-50\mathrm{deg}$ to $+50\mathrm{deg}$ along C-arm oblique rotation and 9 x-ray images from $-40\mathrm{deg}$ to $+40\mathrm{deg}$ on the cranial/caudal direction, with intervals of 10 deg. In the planning software, we placed a virtual sphere with the same diameter as the bbs on each of the bb landmarks and measured the distance of the bb in the second image to the epipolar line from the center of the corresponding virtual sphere in the first image. The error distance is measured as $7.58\pm 3.02\text{pixels}$ (values reported as $\text{mean}\pm \text{standard deviation}$) in an x-ray image with pixel size of $1024\times 1024$ and pixel spacing of $0.22\frac{\mathrm{mm}}{\text{pixel}}$. In addition, we acquired a cone beam CT (CBCT) scan of the phantom and measured a root mean square error of 1.37 mm between the bbs in the CT and those reconstructed using two x-ray images.

Fig. 5

(a) and (b) The geometric error is measured using radiopaque bbs viewed in the stereo x-ray images. (c) The blue line highlights a pair of corresponding bbs in the two images. The phantom is shown in panel (c).

3.2.3.

Planning accuracy in placing the acetabular component using two views

To measure 3-D errors and ensure precise placement of the cup in two x-ray images during planning, we construct a dry phantom, where an implant cup is screwed into the acetabulum. Therefore, the desired implant cup placement is well visible in the x-ray images and serves as a reference. We perform experiments, where a virtual cup with the same size of the implant, shown in Fig. 6, must be aligned precisely with the cup implanted a priori that is visible in the x-ray images. To evaluate the 3-D error, we acquire a CBCT scan of the phantom and measure the error between the planning outcome and the ground-truth pose. This yields a mean translation error of 1.71 mm and anteversion and abduction errors of 0.21 deg and 0.88 deg, respectively.

Fig. 6

An implant cup is placed inside the acetabulum, and two x-ray images and a CBCT scan are acquired using the C-arm. (a)–(c) The x-ray and CBCT images before planning. (d)–(f) The overlay of the real and virtual cup after proper alignment is shown.

3.2.4.

Preclinical feasibility study of acetabular component planning using stereo x-ray imaging

In image-guided DAA hip arthroplasty, the proper alignment of the acetabular component is frequently inferred from AP x-ray images.⁵⁰ Thus, the accuracy in estimating the 3-D pose-based on a single 2-D image heavily depends on the surgeon’s experience. In this experiment, we seek to demonstrate the clinical feasibility of our solution that is based on stereo x-ray imaging and compare the outcome with image-guided DAA solutions that only use AP x-ray images for guidance. We refer to the latter as “classic DAA.” Although the use of a single AP radiograph and the anterior pelvic plane coordinate system has certain drawbacks, it is the frame of reference that is most commonly used in computer-assisted THA solutions.⁵¹ While there may be alternatives (e.g., coronal plane), the use of anterior pelvic plane as the frame of reference will enable direct comparison with the current literature.

We conduct a preclinical user study, where medical experts use the planning software to place acetabular cups on simulated stereo x-ray images. These results are then compared with the conventional AP-based method considering orientational error in abduction and anteversion.

For the purpose of the user study, simulated x-ray images or so-called digitally reconstructed radiographs (DRR) are produced from a cadaver CT data. We generate 21 DRRs from the hip area, starting at $-45\mathrm{deg}$ and ending at $+45\mathrm{deg}$ with increments of $+45\mathrm{deg}$ on the orbital oblique axis of the C-arm, where 0 deg refers to an AP image. Each time the users are given a randomly selected DRR together with the DRR corresponding to the AP plane and are expected to place the acetabular cup such that it is properly aligned in both views.

As the spatial configuration of the DRRs is known relative to the APP plane, we are able to compute the correct rotation of the acetabular component and preset this orientation for the cup in the planning software. This can occur when an AP image is acquired during the intervention and the desired orientation of the component is known relative to the anterior pelvic plane, which allows locking the DoF for rotational parameters. When the orientation is preset, the user only has to adjust a translational component, substantially reducing the task load. Presetting the orientation of the cup is evidently only possible if the x-ray pose is known relative to the APP or the AP image.

Four orthopedic surgery residents from the Johns Hopkins Hospital participated in the user study. The translation error in placing the cup is shown in Fig. 7. The abduction and anteversion errors are measured as zero as a result of presetting the desired angles. The abduction and anteversion adjusted by the user solely using AP image (classic DAA) are $6.52\mathrm{deg}\pm 5.97\mathrm{deg}$ and $1.82\mathrm{deg}\pm 1.89\mathrm{deg}$, respectively. Ground-truth for these statistics includes the five DoF pose of the cup in CT data (as the cup is a symmetric hemisphere, 1 DoF, i.e., rotation around the symmetry axis, is redundant), where abduction and anteversion angles are 40 deg and 25 deg, respectively.

Fig. 7

(a) DRRs were generated from $-45\mathrm{deg}$ to $+45\mathrm{deg}$ around the AP view. Participants were each time given two images, where one was always AP and the other one was generated from a different view. (b) The translational errors are shown for all four participants. Note that 0 deg in the horizontal axis refers to where the user performed planning on only the AP x-ray image.

3.2.5.

Error evaluation in the augmented reality environment

To evaluate the agreement between surgeons’ actions in the AR environment with their intraoperative planning, we measure the orientational error of the impactor after placement with respect to its planning.

The axis-angle error between the principal axis of the true and planned impactor in the AR environment is measured, as shown in Fig. 8. We repeat this experiment for 10 different poses, and each time we use four virtual perspectives of the surgical site. The orientational error is $0.74\mathrm{deg}\pm 0.41\mathrm{deg}$.

Fig. 8

The angle between the principal axis of the virtual impactor and the cloud of points represents the orientation error in the AR environment.

After the cup is placed in the acetabulum using AR guidance, we acquire a CBCT scan of the cup and measure the translation, abduction, and anteversion errors compared with a ground-truth CBCT as 1.98 mm, 1.10 deg, and 0.53 deg, respectively.