1.

Introduction

Optical coherence tomography (OCT) is a relatively new imaging modality that can generate high-resolution and high-contrast cross-sectional images of thin layers of biological tissue.^{1, 2} OCT is ideally suited for ophthalmology since the eye is directly optically accessible.^{2, 3} Quantitative OCT-based measures have become an integral part of macular disease assessment as many retinal disorders have been extensively described and their pathogenesis studied³ by using OCT.

The commercial time-domain Stratus OCT (Carl Zeiss Meditec, Dublin, California) software has a measurement capability limited to retinal thickness and cannot give quantitative information on intraretinal layers. In addition, the quantification provided by this system is often imprecise because of erroneous detection of the inner and outer borders of the retina.^{4, 5} As a result, potentially useful quantitative information is not extracted by the current commercial Stratus OCT. Thus, the addition of a software that could allow for manual correction of the boundaries in case of obvious artifacts and provide local quantitative information of the retinal structure would be very useful. As a matter of fact, a few algorithm developments have been introduced to overcome the Stratus OCT software limitations.^{6, 7, 8, 9} In addition, a manual grading software known as OCTOR offers the ability to perform a detailed quantification of relevant OCT features and correction for Stratus OCT errors.¹⁰ In an effort to provide additional retinal quantifications along with accurate automatic/semiautomatic detection, we developed a software tool for OCT retinal image analysis (OCTRIMA), that is an interactive, user-friendly stand-alone application for analyzing Stratus OCT retinal images.^{11, 12, 13} The OCTRIMA software integrates a denoising and edge-enhancement technique along with a novel segmentation algorithm developed by Cabrera ⁸ Moreover, OCTRIMA is able to minimize segmentation errors, give quantitative information of intraretinal structures, and also facilitates the analysis of other retinal features that may be of diagnostic and prognostic value, such as morphology and reflectivity.^{11, 12, 13}

The OCTRIMA software enables the segmentation of the various cellular layers of the retina. It was initially designed to quantify pathological changes in diabetic eyes with early retinopathy, in which the retinal structure is not yet disrupted by macular edema. However, to evaluate changes in retinal thickness it is first necessary to quantify the reproducibility and repeatability of measurements made by the software. In this study, we aimed to investigate the reliability and reproducibility of OCTRIMA software using Stratus OCT data from normal healthy eyes. Providing an estimate of the repeatability and reproducibility of OCTRIMA-derived retinal thickness measurements in these healthy eyes before applying the methodology to diabetic eyes with early retinopathy will help to determine the degree of change in retinal thickness measurements that may better represent true clinical change rather than measurement variability.

2.

Methods

2.2.

OCT Measurements

For imaging purposes the commercially available Stratus OCT unit (software version 4.0; Carl Zeiss Meditec, Inc., Dublin, California) was used. Subjects underwent three OCT scanning sessions during the first visit on day (D1) by two experienced examiners (E1, E2) with intervals of approximately $5\min$ between scans (sessions 1 and 2, corresponding to S1 and S2, respectively). Thus, two scans (E1D1S1, E1D1S2) were performed by the same examiner (E1) to determine intraobserver repeatability (i.e., E1D1S1 versus E1D1S2). A third scan was performed by a second examiner (E2) and the results were compared with those of the first scan (S1) to determine interobserver reproducibility (i.e., E1D1S1 versus E2D1S1). To assess intervisit reproducibility (i.e., E1D1S1 versus E1D2S1), an additional scan session was performed by one of the examiners (E1) during a second visit (D2) the next day. Figure 1 shows an outline of the experimental setup. Between the examiners, the OCT instrument alignment and controls were randomly changed, so all alignment and focusing had to be restarted. Only scans with a signal strength of 6 or more were accepted.⁴

Fig. 1

Study setup for the reproducibility of OCTRIMA measurements. Ten healthy eyes were scanned during four sessions (E1D1S1, E1D1S2, E2D1S1, and E1D2S1) on two consecutive days (D1, D2) by two OCT examiners (E1, E2) and analyzed by an experienced grader.

The Radial Lines protocol was used for the OCT studies. This protocol acquires six retinal B-scans each of scan length $6\mathrm{mm}$ , each scan oriented $30\mathrm{deg}$ apart from each other, and centered at the fovea. Each B-scan consists of 512 aligned A-scans. Each A-scan consists of $1024\text{pixels}$ with a total scan depth of $2\mathrm{mm}$ in tissue. Thus, each B-scan acquired in this protocol consists of a total $1024\times 512\text{pixels}$ . If the subject moved or blinked during the scan, the image was repeated. In addition, the quality of B-scans was evaluated with OCTRIMA. Generally, the standard deviation of the foveal center point thickness is used as a measure of the scan variance. A high standard deviation ( $>10\%$ of center point thickness) indicates high variability, usually due to patient movement or boundary line error, and therefore incorrect center point thickness. Good quality images have a standard deviation $<10\%$ of center point, good clarity of the layers, and are also well centered. Substantially decentered scans could have a low standard deviation.^{4, 10} Therefore, a scan quality factor (SQF) based on the standard deviation calculation (in percent) of the foveal center point (FCP) for the six radial line scans included in the OCTRIMA software was used to control the variability of measurements associated to image acquisition pitfalls. A good scan has an $\mathrm{SQF}=1$ , indicating that the percentage standard deviation of the foveal center point is $⩽10$ . Data for each measurement were exported to disk using the export feature available in the Stratus OCT version 4.0 analysis software.

2.3.

Computer-Aided OCT Image Analysis Software

OCTRIMA is a powerful computer-aided system designed to facilitate viewing and automatic/semiautomatic OCT retinal image analysis. ^{11, 12, 13} The application essentially provides dual functionality in a single software package by combining image enhancement and speckle denoising of Stratus OCT images along with intraretinal segmentation and error correction using direct visual evaluation of the detected boundaries (see Fig. 2 ). Moreover, the software has the capability to provide quantitative analysis based on measured values of corrected thickness, volume, and reflectance of the various cellular layers of the retina. A total of seven intraretinal layers can be extracted using OCTRIMA, namely, the retinal nerve fiber layer (RNFL), the ganglion cell layer along with the inner plexiform layer $(\mathrm{GCL}+\mathrm{IPL})$ , the inner nuclear layer (INL), the outer plexiform layer (OPL), the outer nuclear layer (ONL), the photoreceptor inner/outer segment (IS/OS) junction, and the outer segment/retinal pigment epithelium (OS/RPE) junction.

Fig. 2

OCTRIMA screenshot: (a) automatic mode showing original raw image (B-scan) with overlaid retinal boundaries labeled and (b) manual mode displaying the manual toolbar when the “Start Manual Corrections Tool” button is clicked.

2.4.

Quantitative Analysis

As a result of repeatedly scanning a total of 10 healthy eyes during four sessions on two consecutive days by two OCT examiners, a total of 240 OCT B-scans were collected and analyzed by an experienced grader. Specifically, the grader segmented all the B-scans from all sessions (E1D1S1, E1D1S2, E2D1S1, and E1D2S1) using OCTRIMA for testing the intraobserver, interobserver, and intervisit variability of repeated measurements performed by the same examiner, by different examiners, and at different visits (see Fig. 1).

After each B-scan was denoised, the inner and outer borders of the retinal structure were identified between the internal limiting membrane (ILM) and the inner boundary of the OS/RPE junction; and a total of seven intraretinal layers were extracted using OCTRIMA⁸ (see Fig. 2). Note that visualizing and quantifying microstructural changes within the photoreceptor and RPE, layers is difficult using Stratus OCT images due to weakened signal energy after penetrating the neuroretina, RPE, and choriocapillaries. Thus, the three outermost hyperreflective layers clearly observed with Fourier domain OCT systems are not certainly visible in Stratus OCT images. Only two hyperreflective layers and one hyporeflective band are observed with the Stratus OCT device. These layers have been identified^{14, 15} as the IS/OS junctional complex, which is the first hyperreflective layer, the hyporeflective band below this junction, which is clearly wider in the fovea and attributed to the photoreceptor OSs, and the second hyperreflective layer corresponding to the outer segments interdigitizing with the microvilli of the RPE (i.e., the OS/RPE junction). In time domain OCT images, the RPE and photoreceptor OSs are too close to be resolved and often appeared as a single hyperreflective band. Thus, the second and third hyperreflective layers have been conventionally assigned to the RPE in previous studies using Stratus OCT images. However, the third hyperreflective layer only visible in spectral domain optical tomography (SDOCT) images and identified as the RPE, is probably due to a signal from the RPE cell bodies, although reflections from choriocapillaris might also be included.^{14, 15} Accordingly, OCTRIMA measurements of the total retinal thickness were made from the innermost point of the retina (ILM) to the inner border of the second hyperreflective band, which has been attributed to the OS/RPE junction in agreement with histological studies^{8, 15} (see Fig. 2). Note that this thickness differs from the thickness measured with Stratus OCT, which calculates the distance between the inner border (ILM) of the retina and the inner border of the highly reflective photoreceptor IS/OS junction (i.e., the first hyperreflective band). Therefore, in contrast to OCTRIMA, thickness calculated with the Stratus OCT algorithm does not take into account the thickness of the junctions of the inner/outer photoreceptor segment and the outer photoreceptor segments (i.e., the hyporeflective band) in the fovea.

All scans in the study had a signal strength of 9 or 10 and were perfectly centered $(\mathrm{SQF}=1)$ . Algorithm performance was visually evaluated by the experienced grader to detect algorithm errors. Criteria for algorithm error included evident disruption of the detected boundary (e.g., small peaks, linear and curve offsets), and/or detected boundary jumping to and from different anatomical structures (i.e., false segmentation, see Fig. 3 ). The average number of manual corrections needed per scan was three. Since the thickness of the inner and outer photoreceptor segments has been found to be relatively constant,^{7, 8} which is consistent with an anatomically uniform thickness, the outer border of the photoreceptor segment junction (IS/OS) can be extracted manually using the semiautomated approach in OCTRIMA. Thus, the outer border of the IS/OS is located $10\text{pixels}$ from the outer border of the ONL, which give⁸ a constant thickness of $20\mu \mathrm{m}$ . Accordingly, the thickness measurements for the IS/OS were not included in this study. Thus, the thickness measurements of the total retina and six intraretinal layers (RNFL, $\mathrm{GCL}+\mathrm{IPL}$ , INL, OPL, ONL, and OS/RPE junction) were actually used in the analysis. Note that the repeatability and reproducibility analysis was performed for the uninterpolated measurements at every A-scan location for all six B-scans (i.e., uninterpolated raw data).

Fig. 3

OCTRIMA segmentation results for a diabetic patient with minimal diabetic retinopathy. Images were outlined with the outer border of the ONL. Segmentation algorithm failure was defined as noticeable deviation of the segmented intraretinal borders from the subjectively recognized borders (a) automatic segmentation results. Note the algorithm failures due to artifacts, noise, and signal distortion due to vessel shadows. (b) Segmentation results after manual error correction.

3.

Results

Retinal thickness measurements at every A-scan location for all six B-scans of all 10 eyes was performed for the total retina and six intraretinal layers. As a result, the average thickness per layer was calculated using the OCTRIMA software for each macular scan group of all 10 eyes for each session. Coefficients of repeatability and reproducibility for the total retina and six intraretinal layers are given in Table 1 . The means and SDs of the differences between measurements obtained under different conditions, ICCs, and Wilcoxon test results are also shown in Table 1. Repeatability coefficients to test intraobserver variability were less than 4% for the total retina and less than 7% for all intraretinal layers except the OS/RPE junction $(\sim 10\%)$ . Reproducibility coefficients to test interobserver variability were less than 5% for the total retina and less than 7% for all intraretinal layers except the OS/RPE junction $(\sim 31\%)$ ; and for intervisit variability it was less than 3% for the total retina and less than 7% for all intraretinal layers except the OS/RPE junction $(\sim 16\%)$ (see Table 1). The ICCs obtained for the intraobserver and intervisit variability tests were greater than 0.75 for the total retina and all intraretinal layers, except INL (intraobserver and interobserver test) and OPL (intraobserver, interobserver, and intervisit test). The lowest ICC values for the total retina were obtained for the interobserver variability test (see Table 1). In addition, the Wilcoxon paired measurements test (5% significance level) showed that there were no statistically significant differences between measurements obtained by different examiners or during different visits.

Table 1

Thickness measurements (mean±SDs) , coefficients of repeatability/reproducibility (CRs), ICCs, and Wilcoxon test results obtained for the total retina and six intraretinal layers.

4.

Discussion

Although a layer-editing tool to manually adjust the retinal layer boundaries for macula and RNFL was recently incorporated in the current Stratus OCT software, its quantitative analysis does not provide thickness measurements of the various intraretinal layers. This limitation in the Stratus OCT system has stimulated interest in developing segmentation algorithms to better detect the local changes in the retinal structure to improve retinal disease detection and its progression.^{6, 7, 8, 9} In this study, we report the reliability and reproducibility of macular segmentation mapping using the OCTRIMA software, which overcomes the limitation of the Stratus OCT software and provides additional quantitative information that can be extracted from OCT data.

The uninterpolated average thickness measurements recorded for all 10 healthy eyes showed that the coefficient of repeatability was less than 4% for the total retina and less than 7% for intraretinal layers except the OS/RPE junction $(\sim 10\%)$ . These values indicate high repeatability of the results of measurements generated by the OCTRIMA software (see Table 1). The high variability in the thickness measurements of the OS/RPE junction is due to the fact that the outer boundary of this layer is not clearly visualized in Stratus OCT images because of the low contrast between the OS/RPE junction (outer border) and the RPE inner boundary, which can be attributed to the limitation of the Stratus OCT system to penetrate deeper structures in the retina. Moreover, the interobserver coefficients of reproducibility calculated for the total retina and intraretinal layers (except for the RNFL) were higher than corresponding values for intervisit reproducibility, which may possibly be explained by the fact that subject fatigue and normal drying of the eye during repeated sessions the same day induced more noise into the overall measurements. In addition, the scans were not aligned between visits because the Stratus OCT did not provide this feature. Thus, this limitation affected the intervisit variability results. Furthermore, there was less than 5% interobserver variability for the total retinal thickness measurements (see Table 1). This is a reassuring finding for an analysis software tool applied to data obtained with a diagnostic instrument, because comparisons of measurements taken for the same subject over a period of time may be compared even when measurements are obtained by different experienced examiners. In summary, intraobserver, interobserver, and intervisit variability combined accounted for less than 5% of total variability for the total retinal thickness measurements and less than 7% for the intraretinal layers except the OS/RPE junction.

5.

Conclusions

We obtained that retinal thickness measurements for the total retina and intraretinal layers (except the OS/RPE junction) performed using OCTRIMA are very repeatable and reproducible. High measurement repeatability and reproducibility is a prerequisite for quantitative application of OCTRIMA in research and clinical work. These findings are particularly useful because they indicate that any retinal thickness change of greater than 5% (or layer average thickness change greater than 7%) in the macular area in healthy undilated subjects are likely to be caused by changes in retinal thickness rather than by inconsistencies in either the OCTRIMA software or in measurements given by the OCT system.

Although the OCTRIMA quantitative analysis of Stratus OCT images described in this paper is potentially useful, new OCT technologies, such as spectral domain, ultrahigh resolution, and adaptive-optics-based OCT technology,¹⁹ are likely to partially afford better solutions to the limitation of existing OCT software by providing images with higher resolution along with a dense map of the retina with precise registration and localization. However, automatic segmentation algorithms for OCT data have a tendency to give erroneous segmentation results especially in pathological cases, which is actually a result of the algorithm performance independently of how well the OCT image could be reproduced with a high level of detail. Consequently, to improve the practicability of OCT technology in ophthalmology, effective data processing requires robust and accurate segmentation algorithms integrated into intelligent software solutions. In addition, computer-aided detection and diagnosis based on automatic/semiautomatic robust algorithms will be essential in clinical studies where large data sets will be impractical for manual grading approaches.

Even though the results presented are based on Stratus OCT images, the main purpose was to establish the feasibility of our quantitative methodology for OCT image analysis independent of the technology used. There is no doubt that if our methodology works well for Stratus OCT images, then it should perform better for SDOCT images which have better resolution. As a matter of fact, OCTRIMA is currently able to analyze B-scans from SDOCT systems. However, a more practical interface to handle the large quantities of measured raw data generated by these systems along with the associated substantial processing is certainly required, and it is currently under development.

In addition, recent studies have shown that retinal thickness measurements between SDOCT devices are significantly different due to different assumptions considered for the detection of the outer retinal boundary, which makes it difficult to compare data obtained by different devices. These differences also make it difficult to adequately evaluate the performance of SDOCT to detect the progression of disease. OCTRIMA could facilitate a well-defined and standardized quantitative analysis for the assessment of retinal diseases and its progression using SDOCT images. Thus, quantitative evaluation of OCT images with OCTRIMA may improve the quality of data and analysis currently being obtained with Stratus OCT and SDOCT devices. From a clinical point of view, it would be possible to understand the mechanism and time-dependence of macular dystrophies and degenerations, and neurodegenerative diseases by understanding the cellular changes of the macula by using the OCTRIMA software. OCTRIMA can be used as an in vivo tool for quantification of the early structural changes in retinal diseases.¹²

Direct comparison of our study with previous reproducibility and repeatability studies is difficult because the age group of healthy subjects along with image segmentation and experimental and statistical methods vary between studies. As with previous findings, change of examiner did not significantly affect the reproducibility of the measurements in healthy eyes.^{20, 21, 22, 23, 24} Future studies will examine the repeatability and reproducibility of macular segmentation mapping with OCTRIMA for each of the nine Early Treatment of Diabetic Retinopathy Study (ETDRS)-like regions in healthy subjects and patients with early diabetic retinopathy and other retinal diseases.