1.1.

Study Objectives and Hypotheses

The objective of this study was to characterize whether physicians with different levels of radiologic expertise perform differently in two specific nonradiologic search tasks that involve target identification.

Our hypothesis was that both visual search skill (measured by eye-tracking metrics) and diagnostic accuracy (measured by target identification) would not be different among the groups.

2.1.

Study Design

This prospective study was granted institutional review board approval with exemption from full ethical review granted for this study on the basis that it did not involve collection or display of patient data and used a set of nonclinical open-access images. This paper was prepared in accordance with the Standards for Reporting Diagnostic Accuracy Studies checklist.¹²

This study forms the second half of a two part experiment. In the first half, participants were asked to locate pneumothoraces in chest x-ray (CXR) while having their gaze tracked by a noninvasive eye-tracking device.⁹ Immediately after this experiment, participants began a second experiment in which they were shown three maps [an example is shown in Fig. 1(a)] and were asked to identify all the hospitals, represented by a (+) symbol, [Fig. 1(b)] in that map. The target (+) symbol that designates a hospital was shown to the participants before they began the experiment. They were then presented with three images made up of geometric shapes (an example is shown in Fig. 2) and were asked to move on when they identified a specific shape.

Fig. 1

(a) and (b) A map of Boston, one of three maps used in the experiments: (a) the image as it was encountered by the participants and (b) the three hospital targets enlarged.

Fig. 2

One of three sets of shapes used in the experiments.

Performance in the initial experiment was estimated by the reference standard in the second experiment (i.e., we are comparing performance in nonradiological search tasks to performance in a specific radiological search task). This was a prospective study.

2.2.

Participants

Four groups of participants were included in the study. All were medical doctors in one major Irish university affiliated teaching hospital or one peripheral clinical center within the same hospital group. The first group consisted of seven medical interns in their initial week of clinical internship, all of whom completed the MB, BCh, BAO medical degree. Seven of these interns formed a convenience sample from the annual medical rotations, and one was excluded because of poor eye-tracker sample collection ($<50\%$). The second group consisted of five senior house officers (SHOs) (the equivalent to second-year medical residents in the North American system), who completed their intern year in nonradiologic specialties and were part of the medical on-call rotation. The third group consisted of four radiology registrars (radiology residents) in their fourth year of training. The fourth group consisted of five consultant radiologists, with mean clinical experience of 11 years after radiology board certification. Three of the five radiologists specialized in chest imaging, but all were responsible for reporting the same number of chest radiographs annually at a university teaching hospital.

2.3.

Eye Tracker

The eye movements of participants were tracked with a Tobii TX300 eye tracker (Tobii Technology, Stockholm, Sweden) and an integrated monitor. Display characteristics include an aspect ratio of 16:9 and resolution of $1920\times 1080\text{pixels}$, with a typical screen response time of 5 ms. The eye-tracker sampling rate was 300 Hz. Zooming and panning were not permitted to ensure that eye-tracking data were not influenced by fixations related to these tools rather than the detection task.

2.4.

Experimental Procedure

Experiments were conducted in a viewing area that was adjacent to but separate from the radiology department. Distractions were minimized and conditions, including monitor display, temperature, and ambient lighting, were maintained in accordance with optimum viewing standards.¹³ The search task was explained to the participants, and they were familiar with the interface from the previous study. The eye tracker was calibrated to each individual’s eye movements at the beginning of the experiment, and a full calibration was carried out at the beginning of each of the four days of experiments.

Before the experiments, “regions of interest” (ROI) were applied to each target within the source images. An acceptance radius of 50 pixels was allowed. A tracked visual fixation within the ROI was deemed as positive identification of the target.

The search task began with a set of instructions (search the image until you are satisfied that you have found all the hospitals along with an image of the hospital icon), and they were presented with three images of maps. Two of the maps contained three targets each, and one map had zero targets. Participants were not told how many hospitals were located on the map. After moving on, they were presented with a multiple choice question (MCQ) asking how many hospitals were identified (giving the options 0, 1, 2, 3, 4, 5, and $>5$). Results were recorded electronically. Following these images, they were shown another set of instructions (search the image until you have found the “odd shape out” and then progress to the next image) followed by the sets of shapes. Whether they progressed after fixation on the correct shape was recorded electronically. Each participant’s eye movements were tracked as they reviewed the images. One participant was excluded from the analysis, as there were a large number of missing samples.

2.5.

Analysis

The number of detected targets on the maps was recorded for each participant. Visual search fixations were reviewed to examine if any errors were perceptual (target not fixated) or decision based (target fixated but not recorded). The number of targets correctly identified was deemed to be the measure of diagnostic accuracy. Fixations on the shapes were reviewed to ensure that the correct shape had been identified before proceeding to the next image. Progression to the next image directly after identification of the target shape was deemed to be the measure of diagnostic accuracy.

For the eye-tracking analysis, the following metrics were analyzed: (a) time to first fixation (how many seconds before the participant fixated on the target), (b) fixations before the ROI (how many fixations are made before a fixation was made on the ROI), (c) total fixations, (d) mean fixation time, and (e) maximum visit count (the number of times an ROI was left and revisited). Smaller values for each of these metrics are indicative of better performance. The diagnostic accuracy data (which are the comparison of four means) were analyzed with the Kruskal–Wallis test. Analysis of the eye-tracking metrics was with the Friedman method. $p<0.05$ was considered to indicate a significant difference.

3.1.

Discussion

Image perception has been described in terms of “top down” and “bottom up” theories. Top down theories are those that are global, and bottom up theories are analytical building from small details.¹¹ The reproducible ability of experts in radiology to identify pathology far faster than they can fully search the entire image lends evidence to the top down theories.^14,15 Top down visual search can, in turn, be thought of in two stages,¹⁶ stage one using cognitive parallel processing mechanisms to give an overall impression of the image and stage two being a more exhaustive detail oriented process. In radiology, this could mean that interpretation starts with an overview (which often leads to a diagnosis based on initial Gestalt⁸) and, for more complex cases, proceeds to more detailed analysis of the image as a whole. As the participants experience and “bank” of memorized cases grows, the process becomes “guided” and more efficient and accurate.¹⁷ The bank of experience from which the reader draws also has an effect on perception,¹¹ and those with a different frame of reference will actually perceive images differently.

The influence of expertise on the “reading” of a chest radiograph has long been the subject of research. A pioneering paper¹⁸ explains how the “systematic method” still being taught to medical students and radiology residents today has little bearing on the way an expert examines an image. To examine an image with the highest degree of visual acuity, one needs foveal fixation; however, as stated, most diagnoses are made long before an entire image could be scrutinized with this level of acuity.¹⁴ While a novice will waste timing moving around an image, the expert will sample “high yield” areas and then focus in on crucial elements to make a diagnosis.

The current research was carried out immediately after an experiment that quantified the four groups of clinician’s diagnostic accuracy and search pattern behavior during identification of pneumothoraces on CXR. That experiment found that not only did diagnostic accuracy and search pattern behavior increase with experience but these two metrics also may reach peak performance at different stages. Previous research has shown that expert radiologist do not outperform lay people in complex nonradiologic search tasks.¹ Patterns in developing diagnostic accuracy¹⁹ and visual search⁴ have been identified. The current study aimed to bring these ideas together and try to identify any trends or innate visual search characteristics in groups with different levels of expertise. We found that none of the groups were more successful at identifying targets in complex images or in images that were based on geometric shapes. This is in spite of having significantly different performance in finding a pneumothoraces on chest radiography. The current work also found that when examining participants’ eye-tracking metrics no metrics was statistically different among groups of expertise.

It has been shown that expert radiologists can fixate on an abnormality within 0.25 of a second,⁷ which is much faster than it would take to perform a full foveal search of a given image. The radiologists in our previous work took on average 0.6 s to fixate on a pneumothorax, with interns taking over 2 s.⁷ However, the time to first fixation on any of the targets in the current study was not significantly different (6.74 versus 8.49 s, $p=0.267$).

In a commentary on the role of image perception research in medical imaging, Manning et al.³ summarized the factors that influence the observer in their interpretation as image dependent or image independent. Image-independent factors are mainly cognitive and relate to prior knowledge that the observer has about the image, for example, a consultant knowing a lot more about pathology visible on a CXR than an intern. When readers are asked to view unfamiliar images, this advantage is removed, and image-dependent features, such as how conspicuous a target is from the background, become more important. This “leveling of the playing field” allows us to examine the innate visual search skill of each physician and reveal their baseline ability. The current work demonstrates that this is in fact quite similar. Errors in the current study were all detection rather than decision errors. Previous research has shown that expert errors tend to be decision based rather than detection ones.²⁰ This finding held true in the previous work⁹ in which the radiology registrars and consultant radiologists had the same eye-tracking scores but registrars had lower levels of diagnostic accuracy. When nonradiologic images are used to “level the playing field,” the types of errors change from those that an expert might make to those usually made by novices.

There were several limitations to this study. The first was the sample size, both in terms of images and participants. Similar research¹ has had lower numbers of both images and participants, but we acknowledge that one cannot necessarily generalize conclusions made in one small study to the entire population; intead, this work acts as a proof of concept, and further work is needed. The results can, however, be compared to the CXR experiments with the same number of participants, which did find significant differences in performance among clinician groups of differing expertise. Instead, we employed the expert group (the radiologists) as the gold standard and compared to them as they had already demonstrated their radiologic expertise.

Krupinski²¹ outlines how diagnostic accuracy in observer performance studies is measured using a “figure of merit,” which is usually receiver operator characteristic (ROC) or one of its variants. A free-response ROC (FROC)²² analysis would have been preferred for analysis of the maps and targets as it would have allowed for a more thorough analysis of false positives (FPs). However, as the setup was optimized for the initial experiment and the display software did not facilitate switching from ROC to FROC, a simpler form of analysis (number of targets identified) was used. While we can be certain that there were no FPs in the map that did not contain any targets, it is unlikely but possible that there were FPs in the other two images. The number of fixations on ROIs, however, in every case matched the answer given in the MCQs, suggesting that the method was accurate. While this system of analysis is not optimal, it gleaned interesting and consistent data, which, in the opinion of the authors, warrants further investigation.

Zooming and panning (or changing window width or level, but this is not applicable to these kinds of image) were not permitted. While this is different from normal clinical practice, it enables more legitimate interpretation of the eye-tracking data as we can be sure that eye movements are part of visual search as opposed to image optimization.