1.1.

Previous Solutions

Various solutions have been proposed for the reduction or elimination of autofluorescence.⁹ These include using narrow bandpass emission filters in an effort to isolate the desired fluorescence signal, and the use of labels that can be excited at wavelengths in the near-infrared (NIR) that are much less likely to induce autofluorescence.¹⁰ However, in many cases, these approaches may be infeasible, and none of them fully eliminates the problem.

1.2.

Multispectral Imaging

Spectral imaging delivers a high-resolution optical spectrum at every pixel of an image. This capability provides useful information beyond that which can be captured using color cameras, or monochrome cameras combined with one or a handful of conventional interference filters. There are a number of ways of acquiring spectral datasets.¹¹ The approach illustrated here involves imaging a sample through a liquid crystal tunable filter (LCTF) that can be set to allow only light of a narrow bandpass (plus or minus $10\text{to}20\mathrm{nm}$ ) to reach the camera; the peak position of this bandpass can be rapidly switched to any other position within milliseconds with about $1\text{-}\mathrm{nm}$ precision.^{12, 13} A series of images (typically 10 to 20) of a particular field can thus be rapidly acquired at different wavelengths to create a spectral data “cube,” in which the three dimensions are $x$ , $y$ , and wavelength (or lambda). In this cube, a spectrum is associated with every pixel. The resulting data can be used to identify, separate, and remove the contribution of autofluorescence in analyzed images, as well as to enable imaging of a multiplicity of signals; the entire process can be completed in a matter of seconds.¹⁴

1.3.

Spectral Unmixing

Since fluorescent light emissions combine linearly, mixtures of signals can be mathematically disentangled (or unmixed), as long as the spectrum of the desired signal(s) and that of the autofluorescence or other components are known or can be deduced from the data, yielding images that reflect the abundance of each component isolated from the others. Since autofluorescence can be considered simply as another spectral signal, it can be unmixed away from exogenous labels. Spectral unmixing algorithms need to be adapted from generic least-squares fitting to reflect the particular nature of the task; basically, negative weightings of component abundances should be avoided, for example,¹⁵ and the abundances are constrained to sum to 100%.

Very similar spectra can be unmixed quantitatively from each other. With reasonably non-noisy spectra, even signals with peak emissions that differ no more than $2\text{to}3\mathrm{nm}$ can be effectively unmixed (data not shown). Moreover, since spectral analysis uses information from the entire measured spectrum, even spectra that have identical peak maxima but different “shoulders” can be separable. A recent publication provides a useful recent discussion on the effect of spectral sensor resolution, signal-to-noise, and spectral similarity between signals on unmixing success.¹⁶

1.4.

Quantitation

Because of the complexities of fluorescence excitation and emission measurements in deep scattering and absorbing tissue, it should be recognized that the intensities projected to the surface (which is what most current imaging systems measure) bear a complex relationship to actual target volume and intensity characteristics. Nevertheless, the simplicity and speed (which translates to achievable throughput) of surface measurements, as opposed to full 3-D imaging approaches, make 2-D projection imaging the technique of choice for most purposes. Various strategies have been applied to extracting meaningful data from surface intensities, including relating the thresholded signal areas or fluorescence intensity^{17, 18} to measured tumor volume. In either case, it is clear that without first removing the autofluorescence, it would be difficult to achieve reliable quantitative metrics.

1.5.

Component Spectra

Unmixing as described here reflects the application of mathematical approaches used for many years in nonimaged-based spectroscopy of chemical mixtures, spawning a field called chemometrics. In one sense, spectral imaging is just chemometric analysis on a spatially arranged 2-D array of spectra.¹⁹ Key to the algorithms that are involved is the ability to measure directly or deduce an accurate spectrum for each component in a mixture.²⁰ While this can be easy for defined chemical mixtures (since often the spectra of these components can be measured in isolation), chemical reactions can spawn novel or transient species whose spectra cannot be measured in a pure state. Various mathematical techniques have thus been developed for detecting such novel or uncharacterized entities and extracting their spectra even in the constant presence of other components.²¹

The same problem exists for multispectral imaging in vivo, for which accurate unmixing requires knowledge of the spectra of the components present in the sample. Autofluorescence is relatively easy to measure, since one can image a control animal containing no exogenous fluorescent labels. There are typically many sources of autofluorescence in animals and tissue samples, so the collected spectra represent an ensemble of constituents. Nevertheless, it is not necessary to decompose such complex signals as long as they are fairly constant in their proportions. However, when additional strong autofluorescent signals, such as the intense chlorophyll-derived red emissions emanating from the gut, are present only locally, often more than one autofluorescence spectrum needs to be included in the analysis.

Another, more difficult problem reflects the need to obtain “pure” in-vivo spectra of exogenous fluorescent signals. In most cases, the spectra are always admixed with autofluorescence signals, so that no pure examples exist in the captured spectral image. Furthermore, the spectra as measured in the living animal, for example, may differ considerably from published spectra due to environmental effects (e.g., pH, ionic milieu) and from the combined influence of light scattering and absorbance due to such constituents as collagen, melanin, hemoglobin, and red blood cells. The approach taken here is to measure the emitted spectra emitted from the surface of the experimental animal, and then, taking into account the presence of one or more admixed spectra (from autofluorescence or other exogenous fluorophores), calculate the presumed “real” spectrum of the species of interest. This calculated spectrum is then used to provide the unmixing algorithms with a correct estimate of all components, allowing for accurate quantitation of each species.

The generation of these accurate spectra (to create spectral libraries or basis sets) can require some skill and a basic knowledge of the spectral properties of the sample. The development of automated methods will be necessary if multispectral imaging is to become a mainstream technique. Such methods are presented next. Once accurate spectra are determined, they can be saved and applied in routine fashion to subsequent datasets, making the routine use of multispectral imaging (MSI) appropriate even for large studies.

2.1.

Semiautomated Spectral Analysis Using Real Component Analysis

A number of approaches have been employed to detect and characterize spectral features in a sample. Principle components analysis (PCA) and its variants are commonly used to detect entities that contribute to the spectral variance in a scene.^{19, 23} However, in their basic form, they are not designed to deal with linear mixtures, but rather they interpret spectrally distinct pixels as separate components rather than as mixtures of known components. Other approaches, such as N-FINDR,²⁴ seek the purest component, or “end-member,” spectra that exist in a sample. However, if no pure examples are present, they do poorly in extrapolating what the pure spectrum should be.

We have developed an algorithm, termed “real component analysis” (RCA), that combines several automated methods to analyze a multispectral image scene and to decompose it into its pure spectral components. Through a combination of unsupervised and supervised classification routines, the RCA algorithm is able to determine accurate spectra of end-members in samples even when these spectra do not exist in pure form.

2.2.

Samples

Quantum dot samples for the phantom mouse were obtained from Quantum Dot Corporation (Hayward, California) and deposited in an aqueous solution onto the back of the phantom.

The in-vivo tumor model¹⁴ was obtained by injecting a nude mouse in the flankwith a C4–2 prostate cell line; tumors were allowed to grow for approximately 3 weeks. A $640-\mathrm{nm}$ quantum dot-labeled antibody targeting prostate-specific membrane antigen (PSMA) was injected i.v. 2 hours before imaging was performed. Spectrally resolved images were taken from $580\text{to}700\mathrm{nm}$ at $10-\mathrm{nm}$ intervals using a prototypre of the CRI Maestro in-vivo imaging system and the resulting spectral data were unmixed using software provided with the system. ²

The five-quantum dot sample was provided by Quantum Dot Corporation. Various spectral species of quantum dots were conjugated with antibodies against cellular components: $525\mathrm{nm}$ , mitochondria; $565\mathrm{nm}$ , microtubules; $605\mathrm{nm}$ , Ki-67, $655\mathrm{nm}$ , another nuclear antigen; and $705\mathrm{nm}$ , actin. This sample was imaged with a $100\times$ oil-immersion objective using a FITC long-pass filter cube, taking images from $510\text{to}720\mathrm{nm}$ in $10\text{-}\mathrm{nm}$ intervals.

3.1.

Phantom Mouse Model Labeled with Two Quantum Dot Fluorophores

Determining the correct basis, or library, spectra with which to unmix, or separate, the dataset is a critical step. However, this can be difficult since many fluorophores change their spectral properties depending on their environment and on the effects that tissue scattering and absorbance can have, particularly for fluorophores located in deep tissue. It is thus preferable to determine the pure spectrum of each fluorophore from an actual sample. The challenge lies in the fact that such signals are usually mixed with an unknown amount of autofluorescence. To deal with this problem, a compute pure spectrum (CPS) methodology was developed that allows the determination of the spectral properties of fluorophores of interest, even when pure examples are not present anywhere in a specimen. The algorithm relies on the user identifying which spectra are “background,” i.e., in this case, autofluorescence, and which represent mixed spectra comprising background plus an additional spectral species. It then estimates the spectral shape of the putative pure component based on spectral differences between the background and the mixed spectra.

Figure 1 shows a phantom sample that exemplifies the problems of obtaining pure spectra of a fluorophore of interest. The left panel shows an RGB image of the fluorescence of the sample (a fuzzy cat toy with squeaking capabilities unrelated to the present discussion). The toy itself has considerable autofluorescence similar in shape and intensity to that of many live mice. The dotted circles show the regions that have been spotted with quantum dots. The left dotted circle was painted with a $570\text{-}\mathrm{nm}$ -emitting quantum dot sample, the right circle with a $620\text{-}\mathrm{nm}$ -emitting quantum dot, and the middle circle with a mixture of the two. The red and green spectra in the right panel show spectra obtained from the autofluorescence-only region of the sample and from a region with both $570\text{-}\mathrm{nm}$ quantum dot and autofluorescence. The blue spectrum is the calculated “pure” spectrum of the quantum dot, and resembles the expected Gaussian shape of such reagents. Conceptually, the algorithm used to obtain this spectrum is a subtraction; however, the actual algorithm as implemented is quite different from this.

Fig. 1

Phantom with quantum dots and autofluorescence. Left panel: RGB image of the fluorescence emission of the sample. Two species of quantum dots ( $570\mathrm{nm}$ , left circle; and $620\mathrm{nm}$ , right circle) were spotted onto a plastic mouse phantom. Center circle: mixture of both quantum dots. Red and green arrows indicate regions from which sample spectra were obtained. Right panel: Spectral data. Red and green spectra correspond to values obtained from the indicated regions. The blue spectrum is the calculated spectrum of the pure quantum dot derived from red and green spectral data using CPS (see text).

Figure 2 shows unmixing results obtained from the phantom using the computed $570\text{-}\mathrm{nm}$ quantum dot spectrum, a similarly computed spectrum of the $620\text{-}\mathrm{nm}$ quantum dot, and the autofluorescence spectrum. In the RGB image of the fluorescence (Fig. 1), the three quantum dot spots can barely be seen. When imaged at the peak wavelength of the $570\text{-}\mathrm{nm}$ quantum dot [Fig. 2(a)], the pure spot (at left) is barely discernable, while the mixed spot (that also contains a $570\text{-}\mathrm{nm}$ quantum dot) is nearly invisible (center). Figures 2(b) and 2(c) show the spectrally unmixed images of the 570- and $620\text{-}\mathrm{nm}$ quantum dots, respectively, showing the improvement in contrast obtained by using multispectral information. Figure 2(d) consists of the composite image obtained by coloring the unmixed quantum dot images [Figs. 2(d) and 2(e)] red and green, respectively, and rendering the autofluorescence in white. The middle mixed quantum dot spot now shows up as yellow-orange (red plus green), reflecting the fact that it is composed of both quantum dots.

Fig. 2

Results obtained from phantom sample in Fig. 1. (a) Image obtained at the peak of one of the quantum dots $(\text{bandpass}=570+∕-10\mathrm{nm})$ . (b) Unmixed image of the $570\text{-}\mathrm{nm}$ quantum dot. (c) Unmixed image of the $620\text{-}\mathrm{nm}$ quantum dot. (d) Combined pseudocolor image of (b) (green), (c), and autofluorescence channel (in white, not shown separately).

3.2.

Mouse Inoculated i.v. with a $620\text{-}\mathrm{nm}$ —Labeled Antibody Targeting C4–2 Prostate Cancer Xenografts

Even though the peak emission of the quantum dots is in the red, the presence of skin autofluorescence (and food autofluorescence) impairs achievable contrast, even when using narrow-band filters centered on the quantum dot peak transmission region. However, a spectral imaging approach can greatly improve the contrast of such specimens.

Figure 3, 4, 5 show the results from a xenografts-bearing nude mouse injected i.v. with labeled antibody approximately 2 hours prior to imaging. Panel A of Fig.3 shows the RGB image of the fluorescence of the sample. The quantum-dot signal is vaguely visible as a reddish coloration over the flank. Not visible in this image is far-red fluorescence arising from chlorophyll-containing food, a common interferent in small-animal imaging. Similar but weaker fluorescence is present ubiquitously (in the skin, for example) and is due to the chlorophyll breakdown products, pheophorbide a and/or pheophytin a.⁸ Feeding the animal a nonfluorescent, alfalfa-free chow for several days prior to imaging can eliminate the majority of the interfering food fluorescence.²⁵

Fig. 3

Spectral analysis of quantum-dot-labeled antibody targeting C4–2 prostate cancer xenografts in a nude mouse. The mouse was injected approximately 3 weeks prior to spectral imaging with a C4–2 prostate cancer cell line. 2 hours prior to imaging, an anti-PSMA antibody coupled to $640\text{-}\mathrm{nm}$ quantum dots was injected i.v. A. RGB fluorescence image. B. Monochrome fluorescence image at $640\mathrm{nm}$ (the peak emission of the quantum dots). C, D, E. Spectrally unmixed quantum-dot signal, food signal, and skin autofluorescence signal, respectively. F. Composite pseudocolor image of C (red), D (green), and E (white).

Fig. 4

Spectra from quantum-dot labeled mouse. Shown are the spectra of the skin (white) as well as a spectrum of mixed skin and quantum dot signals (yellow) manually selected from pixels where the tumors are located. Using CPS, the “pure” spectrum (in red) of the $640\text{-}\mathrm{nm}$ quantum dot was derived, as was food autofluorescence signal (in green).

Fig. 5

Screenshot of RCA plug-in showing results from the quantum-labeled mouse. At left is an RGB image of the fluorescence of the sample. At right are the RCA-computed regions presented for user selection.

At the peak emission wavelength of the quantum dot (Panel B, Fig. 3), the labeled tumors can be appreciated, but autofluorescence in this spectral range still dominates. Panels C, D, and E show the unmixed images obtained from this sample using some of the spectra shown in Fig. 4. The unmixed quantum-dot signal (panel C), in particular, shows a dramatic improvement in signal to noise. Panel D reveals the signals from food and chlorophyll breakdown products, and panel E indicates the contributions to overall signals arising from skin (largely collagen) autofluorescence. The composite image (panel F of Fig. 3) formed from pseudocoloring the images in panels C, D, and E shows the locations of both the food autofluorescence and the quantum dot (tumor) signals superimposed on the skin autofluorescence for orientation.

3.3.

CPS and RCA

Basis spectra for unmixing, like those shown in Fig. 4, can either be measured directly or calculated from the spectral dataset. Obtaining representative spectra of the GFP and of the food as they appear in this sample is not possible, since they are everywhere mixed with skin autofluorescence; the pure spectra have to be calculated. This can be done by manually choosing regions of the sample that contain a mixture of the fluorophore of interest and the autofluorescence, and using that to calculate a “pure” fluorophore spectrum as previously described. Figure 4 shows the spectra of the skin (white) as well as a spectrum of mixed skin and quantum dot signals (yellow) selected from pixels where the tumors are located. Using the Compute Pure Spectrum (CPS) functionality, the “pure” spectrum (in red) of the $640-\mathrm{nm}$ quantum dot was derived. Similarly, the food autofluorescence signal (in green) can be extracted, and matches well with published spectra.⁸ However, in situations where the fluorophore of interest is dim (as here with the food signal particularly), or when several fluorophores are present in overlapping regions, choosing where to select regions to acquire the spectral inputs can be difficult.

To address this problem, an automated spectral searching algorithm was developed that combines an unsupervised (i.e., operating without user intervention or choice) clustering with a supervised step. The initial unsupervised operation presents the user with a series of images that indicate the spatial distribution (location) of different spectral signals. At this point, the user selects images corresponding to biologically significant signals (a subjective decision), identifies which image best represents “background,” and which remaining images represent biological signals of interest. The algorithm then proceeds to calculate pure spectra (as outlined earlier). Using these calculated spectra, the spectral cube is unmixed to indicate the location and abundance of the labels as well as that of the autofluorescence. Generally, cross talk between autofluorescence and the desired signal(s) is eliminated, and the resulting unmixed data are quantitatively accurate.

Figure 5 shows a screen-shot of the Real Component Analysis (RCA) plug-in software in operation, applied to the prostate-tumor/quantum-dot sample. At left is an RGB image of the sample. The user can draw a region of interest (ROI) on the sample to speed up the algorithm and to eliminate extraneous signals (like plastic restraining devices, for example), and/or choose a region of the sample where a pure autofluorescence signal is known to exist. This “starting guess” can help the algorithm’s performance but is not required. RCA then presents the user with a series of images (at right) that show where in the sample it identified various fluorescence spectral features to be found. The user then (optionally) selects which feature represents autofluorescence (labeled “background” and marked white) and which features represent the signals of interest (labeled “signal” and marked with red and green). The algorithm then computes a pure spectrum from each of the regions selected as signals and performs the unmixing. In the present case, the fourth image highlights low-intensity signals that are of no apparent biological significance, and this panel is ignored.

3.4.

Cells Labeled with Five Different Quantum Dots

In addition to being able to remove autofluorescence to increase contrast and signal to noise, spectral imaging offers the possibility of a high degree of multiplexing of fluorophores in a single sample. A test sample created by the Quantum Dot Corporation containing five quantum dots, each labeling a different structure or molecule in the cells (mitochondria, microtubules, proliferation marker Ki-67, nucleus, and actin), was imaged using a long-pass fluorescein cube ( $460\text{-}\mathrm{nm}$ excitation and a $510\text{-}\mathrm{nm}$ long-pass emission filter). Figure 6 shows a portion of the RGB image of the fluorescence of this sample and five spectra extracted approximately from where the different markers would be expected. Each of these five extracted spectra clearly represents a different spectral profile, but they are all mixed signals, derived from emissions from at least two or more quantum dot labels. Separating the spatially overlapping components of the sample either by eye or using an RGB camera is not feasible: spectral imaging is required for visualizing and quantitating each signal independently of the others.

Fig. 6

Raw spectra extracted from a five-quantum dot dataset. The image in the center shows a portion of an RGB fluorescence image of five-quantum-dot-labeled cells $(100\times )$ . Each spectrum shows the raw spectral emissions of the sample at the indicated regions.

Using the CPS algorithm described before, the five spectra in Fig. 7 were obtained. The emission maxima of these five spectra match extremely well with the published maxima for the quantum dots in this sample. These spectra were then used to unmix the dataset to give the images shown in Fig. 8 .

Fig. 7

Calculated quantum dot spectra derived from the five-quantum dot sample. This graph presents the five spectra calculated using CPS, each curve corresponding to one of the five quantum-dot species that were present.

Fig. 8

Unmixing results from five-quantum dot sample. Shows the complete RGB image of the fluorescence of the sample, with the subregion shown in Fig. 6 outlined by the dotted rectangle. Each of the monochrome images corresponds to the unmixed image obtained using the spectra from Fig. 7. From top left to bottom right, they correspond to the 525-, 565-, 605-, 655-, and $705\text{-}\mathrm{nm}$ quantum dots.