4.1.

Point Scanning and Line Scanning

Point scanning and line scanning, collectively called the spatial scanning or imaging spectrograph, acquires the entire spectrum section-by-section and uses mechanical means to scan the entire space. These methods are also known as “whiskbroom”/ “spotlight” and “push-broom,” respectively, which are terminologies originated from satellite imaging. In a point scanning imager, an aperture only allows light from a small section to pass through a monochromator, which will disperse the light onto a sensor array. The specific method of scanning point by point in whiskbroom imagers brings to mind that of confocal microscopy imaging, in which a pinhole is also used to block lights that are out of focus. Hyperspectral confocal microscopy combines both methods with little modifications, producing 3D structures that can be analyzed by spectral values.^82,83 In a line scanning imager, the aperture is a slit that allows a sliver of light to be dispersed onto a two-dimensional (2D) sensor plane (the direction of dispersion is perpendicular to the direction of the slit). Compared with point scanning, line scanning can capture the same field of view in less time, but the higher number of sensors means that there are potentially more elements to calibrate and higher chance of sensors artifact.⁶ Line scanning remains the dominant method of acquisition in biomedical imaging⁷ and remote sensing.⁸¹

As mentioned, usage of spectral imaging for practical purposes began with the MSS in 1972. In this system, the point-scanning imager used a mirror and the motion of the spacecraft to achieve perpendicular and parallel scanning, respectively.⁸⁴ Nowadays, in the field of remote sensing, mirrors combined with motion of the aircrafts are still being used as the mean of acquiring the full spatial component.^71,85 By contrast, in histology, fluorescence, and confocal microscopy imaging, the subject being imaged is moved with the help of programmable platforms.⁶⁰ Several line scanning cameras that used mirrors to image near objects were proposed. However, these devices all required spatial calibration to avoid image distortions.^86,87 In the works by Sigernes et al.⁸⁶ and Gutiérrez-Gutiérrez et al.,⁸⁷ the mirror is on a rotational axis parallel to the line scanning slit. When the mirror rotates, only a line section gets reflected and focused onto the dispersive device.

4.2.

Spectral Scanning

In spectral scanning (also called staring, framing, or band sequential), the datacube is captured spatially all at once but only at selected wavelengths. The process of wavelength selection is done using bandpass spectral filters, which can be either interference filters, variable filters, or interferometers. Because no filter has an infinitely small bandwidth, the resulting image captured at each wavelength should be considered more as a function of the filter’s spectral response, quantified by the following equation:

4.3.

Spatial-Spectral Scanning

Spatial–spectral scanning (also called windowing) captures the datacube in both the spatial and spectral directions within one single exposure and effectively samples a diagonal “slice” out of the datacube. Even though the datacube reconstruction is harder to visualize in spatial–spectral scanning, this method offers advantages in acquisition speed and movements. Methods of spatial–spectral scanning often used dispersion elements that were location-dependent, such as linear variable filters (LVFs).⁹⁷ With the use of a simple grating element in the fore optics, a line scanning spectral camera can be converted into a spatial–spectral scanning camera.⁹⁸ The grating element transmitted lights that pass through the aperture slit based on both spatial and spectral values. Pichette et al.⁷⁹ developed a commercial device called the “Snapscan” imager that used location-dependent interference filters. In their system, a series of Fabry–Perot filters overlaid the imaging sensor. The bandpass values of the filters varied across the translation direction, so by moving the sensor both spatial and spectral components were sampled. Fourier transform spectrometer can also be used as a mechanism to achieve spatial–spectral scanning.^13,99

4.4.

Snapshot Scanning

Snapshot scanning cameras capture the entire spatial structure multiple wavelengths at one exposure.¹⁰⁰ The implementations of snapshot cameras are extremely diverse and can be classified into the following main categories: dispersive-based methods, coded-aperture methods, speckle-based methods, and spectral-resolved detector methods.¹⁰¹ Dispersive-based methods rely on dispersive elements (grating or prisms) to split the incoming images into different wavelengths that are recorded either on a single sensor or on multiple sensors. An example of a system that uses multiple sensors to record the spectral images is the beam splitting camera.¹⁰⁰ These types of cameras are often seen in television and movies production, although they only have three sensors for red, green, and blue images. An example of a system that records the entire datacube on a single sensor is the computed tomographic imaging spectrometry (CTIS). CTIS is popular in the hobbyist and academic space, as it is easy to construct with low-cost components.^102–104 However, the reconstruction of the datacube using CTIS requires using inverse projection, which can be computational-consuming. Another problem is that CTIS records both the spatial and spectral data on a single sensor, which requires a tradeoff between spatial and spectral resolution. Coded-aperture methods use a patterned filter (mask/coded aperture) at the location of the aperture, which will “encode” the incoming wavelengths by lossy compression. The compressed data can record more light compared with CTIS, but this also means that some of the wavelengths will be lost.¹⁰⁵ Speckle-based systems reconstruct the datacube through correlating the speckle data and the wavelengths. Spectrally resolved detector arrays (SRDA)^39,53–55 use interference filters manufactured on top of imaging sensor to capture snapshot images. For example, color filter arrays, the most popular of which are Bayer filters, are used in consumer digital cameras to simulate human color vision.¹⁰⁶ Snapshot cameras that use SRDA are fast, compact, lightweight, and require no additional movement mechanics. However, they downsample the spatial dimension, which can result in aliasing of the spatial data if the Nyquist limit is not obeyed. There is also a tradeoff between the number of filters and the spatial pixel resolution: the higher the number of different filters are, the lower the light throughput.¹⁰⁰ Readers who are interested in snapshot imaging implementations should consult a literature review by Hagen and Kudenov¹⁰⁰

4.5.

Conversion from RGB to Spectral Images

There existed a separate line of research that generates spectral images without using spectral cameras. Regular color vision cameras use three color filters in the red, blue, and green (RGB) wavelengths to simulate human color vision. However, these filters are broadband filters with a large amount of overlap, and transformation algorithms can be used to extrapolate multispectral or hyperspectral data. Furthermore, hyperspectral data are often sparse, making recovery of spectral data more feasible.¹⁰⁷ All conversion methods generated a mapping function that transform three-wavelengths data to multiwavelengths data. The function was approximated by minimizing the differences between generated spectral data and expected spectral images. One of the first methods used for this was Wiener estimation.^42,108,109 Arad and Ben-Shahar¹⁰⁷ used match pursuit algorithm to construct a mapping from natural RGB to hyperspectral images. In recent years, machine learning algorithms, including neural networks, were investigated to learn the mapping. Koundinya et al.¹¹⁰ proposed 2D and 3D convolutional neural networks (CNN) systems that mapped RGB images to 31-waveband images in the wavelengths 400 to 700 nm. Alvarez-Gila et al.¹¹¹ used generative adversarial networks (GAN) to produce spectral images from RGB. All these methods still faced significant constraints. As noted by Signoroni et al.¹¹² in a review of neural networks for spectral images analysis, several of these generation methods could only generate outputs in the visible light range. Furthermore, they required extensive training inputs of RGB images and corresponding spectral images, which by itself needs a spectral camera to capture. Most input images used for training were outdoor images, which can make the mapping unsuitable for indoor or laboratory settings.

4.6.

Comparisons of Acquisition Methods

The rule of thumb has been that spatial scanning cameras can achieve high spatial and spectral resolutions at the cost of acquisition time, snapshot cameras are fast, and spectral cameras achieve high spatial resolution while trading between acquisition time and spectral resolution.⁶ Considering the state-of-the-art available, it is important to re-examine this convention. We start by comparing two key spectral variables: the spectral resolution (defined as the full-width half-maximum or FWHM value at center wavelength⁸) and the spectral range. In this paper, we use the term “bandwidth” interchangeably with FWHM. Some authors such as Hagen and Kudenov¹⁰⁰ found the term “resolution” in digital spectral cameras to be confusing as it can also be used to describe the number of spectral samples. Point scan and line scan systems have reliable spectral resolution: many commercial systems have FWHM in the range of 2.5 to 5 nm and are capable of capturing more than 100 spectral bands at once.⁸¹ One line scan system for remote sensing reported FWHM as narrow as 1.85 nm.¹¹³ Spectral scanning systems that use filter wheels typically achieved 3 to 10 bands^38,114–116 and each individual filter had bandwidth ranging anywhere from 30 to 250 nm.¹¹⁷ On the other hand, tunable filters can achieve very fine spectral resolution. AOTF could achieve a bandwidth of 0.1 nm;¹¹⁸ however, commercially available AOTF filters often had a minimum FWHM in the 2 to 6 nm range.^91,119 LCTF systems often had larger spectral resolution compared with AOTF, ranging from 4 to 30 nm.⁹¹ Interferometers typically achieve good spectral resolution: FPI that were driven by piezoelectric had spectral bandwidth in the range of 10 to 20 nm.¹²⁰ The FWHM also affects the light throughput, as interference filters with smaller bandwidths have lower light throughput. Some filters’ specification uses the term “optical density” to quantify the amount of energy blocked. Applications that need low signal such as autofluorescence imaging require an optical density of 6 or greater. However, low signal also means low signal-to-noise ratio (SNR) and affects the image quality.

In terms of functional ranges, the wavelength ranges of these devices were more dependent on the sensor types than the acquisition method. The functional range of silicon-based sensors is constrained in the range 400 to 1000 nm because of silicon’s photoelectric properties. If researchers wanted to investigate wavelengths in the short-wave infrared (SWIR) and higher (1000 to 12,000 nm), an alternative semiconductor material such as indium gallium arsenide (InGaAs, also named GaInAs) should be used. In terms of spectral wavelength selection, LCTF, AOTF, and FPIs provide an advantage over spatial scanning, snapshot scanning, and filter wheels. That is because the former can tune to arbitrary wavelengths while the latter have discrete wavelength selection. In many cases, specific selection of center wavelengths was not a major concern, so this advantage was not often used.

The spatial pixel resolution in spatial scanning and spectral scanning system is dependent mostly on the sensor resolution and the binning (grouping of several pixels into one) used. In snapshot cameras, the optic architecture determines the maximum pixel resolution available. If the snapshot architecture captures the entire datacube representation upon a single detector plane as in the case of CTIS, IMS, and SRDA architectures, then the spatial resolution is often poor. Ford et al.¹² used a $2048\times 2048\text{pixels}$ detector to capture a datacube of dimension $203\times 203\times 55$ using CTIS. A recent CTIS system captured up to 315 wavebands but has a pixel resolution of $116\times 110\text{pixels}$.¹⁰² Many other commercial snapshot cameras could capture up to 100 or more wavebands but had spatial pixel resolution of not more than $100\times 100$.⁸¹ The acquisition time of point scanning, line scanning, and spatial–spectral scanning systems can be described as rapid. A recent series of line scanning cameras to be used in industrial settings were capable of scanning $576\text{spatial line}/\mathrm{s}$ or 2880 spatial–spectral elements/s.¹²¹ Because spatial and spatial–spectral scanning systems are used in many industrial and remote sensing environments, the readout speed of these systems also depends on the translation speed of the camera or the samples. The acquisition time of spatial scanning systems relies on the switching time of the filters. Filter wheels achieve switching time in orders of seconds, LCTF in orders of 50 to 500 ms, and AOTF in orders of 10 to 50 μs.^93,100 As for FPI, the switching time ranges from 5 to 50 ms,¹²² although $<3\mathrm{ms}$ switching time had been reported.¹²³ The fact that snapshot cameras can capture the datacube in one exposure does not necessarily mean that they have low acquisition time. Snapshot systems using mosaic filters can be very fast: one system was capable of capturing a $256\times 256\times 32\text{pixels}$ datacube at a rate of 340 images per second.¹²¹ The processing time was part of the consideration for a very long time. Unlike spatial scanning, spectral and snapshot scanning methods all required postprocessing to stitch together the spectrum.⁷⁴ Some methods required extensive processing to generate the datacube, such as CTIS (Fourier slice theorem) or Fourier transform spectroscopy (inverse Fourier transform). However, with advances in computer power, the processing time after acquisition to generate the datacube is becoming more similar across all platforms.¹⁰⁰

Light throughput, the amount of light in the datacube that the detector can measure, affects the SNR. Theoretically, it is been long known that some methods of spectroscopy acquisition have higher light throughput compared with others, which in turn increases the SNR.¹⁰⁰ However, among spatial and spatial-spectral scanning systems, the theoretical differences in SNR are not significant.¹⁴ Hagen et al.¹²⁴ argued for the snapshot advantage, which is the increased throughput that comes from the fact that snapshot cameras capture the entire datacube at once. However, the snapshot advantage is only available for a select number of snapshot architectures, such as CTIS, IMS, and CASSI. Realistically, throughput also depends on the filter transmission rate and the quantum efficiency (QE) of the sensors. Finally, some imaging applications are more suitable for certain methods of acquisitions. Confocal microscopy, for example, can only be coupled with point scanning imaging due to the fact that only a small section of light is imaged.¹⁰⁰

5.1.

Spatial Scanning

Miniature spatial scanning systems use two main classes of architectures: reflective grating and transmission grating/prism.⁷⁴ In both systems, after the light passes through the aperture, it gets collimated (made parallel), then dispersed, and then the individual wavelengths will be refocused onto the array detector.¹⁵ In the Czerny–Turner reflective grating configuration, a curved mirror acts as the collimating mirror and a second curved mirror refocuses the diffracted light onto the imaging sensor. In the Offner configuration, three concentric elements (collimating mirror, reflective grating mirror, and focusing mirror) make up the optical components [Fig. 4(a)]. Even though the Offner configuration provides better spectrographic ability, manufacturing them was not possible until the 1990s due to the lack of precision lithography technology.⁷⁴ Offner spectrometers are commonly used due to their low aberration.¹²⁷ Warren et al.¹²⁶ used a monolithic block of glass as the transmitting medium for the Offner relay, reducing the volume and weight of the imager down to 0.54 kg [Fig. 4(b)].

Fig. 4

Offner spectrograph is an example of a reflective grating configuration. (a) The working mechanism of an Offner spectrograph camera (adapted from Ref. 125). (b) An example of a device that uses Offner spectrograph (reproduced from Ref. 126).

Many low-cost and compact systems used prisms^128–130 or transmission grating^{70,85,102,131–133} as the monochromator. The problem was that these systems were prone to artifacts due to misalignments of the optical components.⁸⁵ Some key artifacts included chromatic aberration (lens not focusing all wavelengths properly), smile (bending of the spectral line), keystone (bending of the spatial data), and straylight (unwanted light caused by other sources).¹³⁴ Some optical architecture showed advantages in producing fewer artifacts. For example, Offner spectrometers have less smiles and keystones compared with Czerny–Turner and transmission grating architectures.^10,135 High quality optical components, such as achromatic lenses, also reduce aberration to a degree, but they can potentially increase the manufacturing cost of the device. Laboratory calibration can also be done to reduce smile and keystone.¹²⁷ In commercial spatial scanning systems, smile ranges between $-0.1$ and 0.1 pixels and keystone reaches a maximum of 3.5 pixels at 1000 nm.¹³⁶ Grism (prism and grating) can reduce some chromatic aberration and is also a common monochromator used in some compact spectral imagers.^134,137 Prism–grating–prism (PGP) was first seen in the works of Aikio⁷⁴ as a method of building ultracompact push broom imagers. In a PGP, two identical prisms sandwich a volume transmission grating [Fig. 5(a)]. Compared with prism and transmission grating alone, PGP disperses light linearly, has high throughput, and is extremely robust.⁷⁴ The greatest advantage of PGP, however, is the ability to disperse light with little space, allowing the development of miniaturized spectral imaging systems. Multiple compact spectral imagers used PGP as their dispersive element^{121,137,139–141} [Fig. 5(b)]. Some commercial systems also use high quality manufactured transmission holographic grating, such as the one reported by Wu et al.,⁷⁷ which demonstrated an ultracompact line scan imager using volume phase grating. Table 2 summarizes the different common methods of dispersion used in spatial scanning imaging cameras.

Fig. 5

PGP is an example of a transmission grating configuration, where no reflective device was used. (a) The working mechanism of a PGP imaging camera (adapted from Ref. 125). (b) An example of a device that uses PGP (reproduced from Ref. 138).

Table 2

Optical comparison among common types of spatial scanning imagers.

Due to their nature of acquisition, many laboratory-based spatial scanning imagers require either the stage to move or the camera to move. These methods of acquisition are not suitable for bioengineering applications such as surgical guidance or in vivo imaging.¹²⁹ To develop spectral imagers that can be compact and usable for live imaging, new techniques in spatial scanning acquisition have been devised to overcome the movement problem. These techniques use microelectronic internal devices such as digital micromirror devices (DMD)¹⁴² and piezoelectric motors^70,79,143 to move the imaging sensors or optical components. Several commercial systems, such as those shown by Wu et al.⁷⁷ and Behmann et al.,¹⁴⁴ are fast enough that they can capture accurate spatial data entirely handheld.

5.2.

Spectral Scanning

Of the compact spectral scanning systems that we surveyed, tunable filters were preferred over filter wheels due to their small size and narrow bandwidth. Some have succeeded in miniaturizing filter wheel systems. For example, Kim et al.²² applied a filter wheel with nine wavebands on top of a smartphone camera. While electrically tunable filters can achieve arbitrary waveband selection, efforts to miniaturize them were hindered by the lack of suitable compact configurations. Both AOTF and LCTF require large external power and large optical pathway, which makes them unsuitable candidates for ultracompact systems.⁹⁴ While the size of the filters in AOTF and LCTF themselves is compact and lightweight (often $<1\mathrm{kg}$),^{88,89,118,145,146} it is the weight of filter driver that adds up to the weight of these devices. Ishida et al.¹⁴⁷ were able to deploy an LCTF-based system on top of a UAV. However, the UAV was only able to fly for 10 min due to the high payload weight. On the other hand, interferometric imaging spectrometers are becoming increasingly compact and have found their way into many applications. We discussed two primary types of compact spectral cameras that use interferometers as their dispersive elements: Fourier-transform imaging spectrometers (FTIS) and FPIs (Figs. 6 and 7). All the previously discussed systems used filters to filter out broadband light sources. However, the light sources themselves can be used as mechanisms for spectral scanning. In such systems, there is an arrangement of narrow-band light sources that illuminate the subjects at different wavelengths. The reflected light measured by the electronics detector is analogous to the spectral response of the subjects. Light-emitting diodes (LEDs) are a common method to achieve variable lighting useful for multispectral or hyperspectral systems. Various LEDs are discussed in Sec. 5.6.1. Table 3 compares the common methods used to produce spectral scanning imagers.

Fig. 6

FTIS. (a) Working mechanism of an FTIS system (adapted from Ref. 148). (b) Image of a compact FTIS camera that uses a focal plane birefringent interferometer (FPBI in the figure) in front of the detector (reproduced from Ref. 149).

Fig. 7

FPI. (a) Working mechanism of an FPI system (adapted from Ref. 150). (b) A compact FPI chip that is driven by electrostatic actuation (reproduced from Ref. 151).

Table 3

Comparisons of several types of spectral scanning cameras. AOTF, acousto-optic tunable filter; LCTF, liquid crystal tunable filter; FTIS, Fourier transform imaging spectrometer; FPI, Fabry–Perot interferometer; FWHM, full-width at half-maximum (measured in nm). Wavelengths refer to the operating range of wavelengths. VIS, visible wavelengths (400 to 700 nm); NIR, near-infrared wavelengths (700 to 2000 nm). Switching speed refers to time to change from one wavelength to another one; ms stands for milliseconds (1/1000 s).

5.3.

Snapshot Scanning

The mechanisms used to acquire a spectral cube in a snapshot manner are numerous. Here, we describe the common methods used in compact spectral imaging, which were CTIS, SRDA, compressive sensing, image mapping spectrometer (IMS), and LVF.

5.3.1.

Computed tomographic imaging spectroscopy

The basis for CTIS was proposed in the early 1990s and was refined by Descour and Dereniak¹¹ and Johnson et al.¹⁰⁴ After passing through an objective lens, light enters the dispersive element of the CTIS, in order: an aperture, a collimator lens, and a grating/reflection dispersive device [Fig. 8(a)]. The aperture could be either a square or a slit depending on whether the desired purpose is imaging or line spectrometry. After passing through the dispersive device, a focusing lens focuses the light onto the staring sensor. What shows up on the sensor is a series of projections of the hypercube, arranged with the zeroth-order in the center and higher order further from the center. Dispersive devices were often transmissive devices, but reflective devices have been developed. Reconstruction from the projections slice is done using Fourier slice theorem. Reconstruction is more accurate with a higher number of projections. However, with limited sensor size, a higher number of projections also means that the reconstructed hypercube will have a lower spatial resolution. Due to advances in computing power, reconstruction-based systems such as CTIS were realized at lower costs. Habel et al.¹⁰³ demonstrated a CTIS camera that uses DSLR cameras and low-cost components. Salazar-Vazquez and Mendez-Vazquez¹⁰² demonstrated an entirely open-source CTIS system with 3D-printed housing and off-the-shelf optical systems. Their imager had a significantly lower cost yet achieved a higher number of wavebands compared with previous CTIS cameras.

Fig. 8

Optical layouts of some common snapshot imaging systems. (a) CTIS. (b) Coded-aperture imaging. (c) SRDA. (d) IMS (reproduced from Ref. 100).

5.4.

Mechanical Components

Mechanical systems refer to nonoptical systems that are responsible for driving the optical components and provide the mechanical housing. Here, we discuss MEMS, which are commonly used in compact spectral cameras. MEMS refers to small systems that are manufactured using semiconductor fabrication techniques. These devices have been hypothesized as theoretically possible since the early 1960s¹⁷⁷ and now encompass a large range of devices including RF switches, cantilever, piezoelectric, comb-drive actuator, and resonator.¹⁷⁸ MEMS are electrically reliable, low on heat dissipation, and can be scaled up to large-scale and low-cost manufacturing processes.⁸ Prior to spectral imagers, MEMS has been used in portable spectrometers.^179–181 The single most important process in the manufacturing of MEMS is photolithography, or the process of etching complex nanopatterns on a photosensitive polymer using lights.¹⁸² Photolithography enables MEMS to be produced in high volume and consistent quality. To integrate multiple MEMS components together, two main methods are used: surface micromachining (depositing the desired system on top of a sacrificial layer to be washed away) or bulk micromachining (directly shaping the substrate without the need for a sacrificial layer). In hyperspectral and multispectral imaging systems, both surface micromachining and bulk machining were used to produce MEMS.¹⁵⁷ In FPIs, MEMS is used as the actuator that drives the distances between the two reflective surfaces. This can be accomplished using either piezo-actuators or electrostatic actuators.¹⁸³ According to Trops et al.,¹⁸³ piezo-actuators FPI had larger optical apertures and higher SNR, and electrostatic actuators were mass-producible, have smaller optical apertures, and lower SNR. In terms of spectral ranges, piezo-actuated MEMS FPI have a wider tuning range compared with electrostatic actuated MEMS FPI. Rissanen et al.¹⁸⁴ and Näsilä et al.¹⁵⁹ used surface micromachining to produce MEMS FPI spectral cameras that work with mobile phone cameras. Näsilä et al.¹²³ used the same technique to produce a ultracompact ($<40\mathrm{g}$) spectral camera. Another type of MEMS is a digital micromirror device (DMD), which is an array of microscopic mirrors on a chip that can be individually activated and rotated. A common usage of DMD is in push-broom imaging, where DMD replaces the slit translation mechanism. The DMD selects the narrow section of the image and reflects that section onto the grating mechanism. This mechanism was seen in works by Arablouei et al.¹⁴² and Dong et al.^185,186 A comb-drive MEMS was used by Wang et al.¹⁸⁷ to rotate a mirror within their handheld spectral imaging camera. There are notable limitations to the range of movement in MEMS. On small scales, the mechanical strain and stress behavior is much different compared with their macrocounterparts. Pull-in phenomenon is seen when the electrostatic forces between MEMS elements are greater than the mechanical forces, which “pulls in” the MEMS components into each other and potentially causes a breakdown.¹⁷⁸ In spectral imagers, this is most applicable to MEMS-enabled FPI imagers, where the air gap between the mirrors cannot be smaller than two-thirds of the initial unactuated air gap size to avoid pull-in phenomena.^159,183,188

For compact devices that are planned to be used in outdoor situations, mechanical housing is as important as every other component. Crocombe⁸ discussed the importance of rigid and durable housing in portable spectrometers used in manufacturing. Even though devices in a clinical setting are not subject to harsh environmental factors the same ways devices in industrial or earth science fields do, mechanical housing is still an important factor in biomedical imaging devices. Many housing systems, if they are used in noncommercial devices, are likely to be highly customized using 3D printing. 3D printing has many advantages over traditional manufacturing, such as the ability to go from designs to prints within a short amount of time, the low cost of plastic, the lack of screws and adhesion, and the fact that when 3D printing different designs, the printer configuration remains largely similar.^85,189 The rise in 3D printing of optical instruments is due to not just the large availability of 3D printers but also because of the open-source movement.¹⁹⁰ For a review of 3D printing technologies and additive manufacturing, see Ngo et al.¹⁹¹ 3D printing can be used for many versatile components of spectral imaging. Ghassemi et al.¹⁹² and Cavalcanti et al.⁵⁹ used 3D printing to develop biological phantom models for hyperspectral imaging. Ortega et al.⁶⁰ created custom 3D printed gears to mechanically move the sample in a push broom microscopic imager. However, the most common use of 3D printing in spectral imager is for the housing of optical and dispersive elements.^{85,102,131,159,187,193} When researchers and hobbyists had access to 3D printers, design and manufacturing became an iterative process due to the speed and low-cost that 3D printing brings. Design was often done with the help of computer aided design (CAD) software. The most common 3D printing method used was fused deposition modeling (FSM), which uses a heated nozzle to fill semiliquid filaments in a layer-by-layer manner. Depending on the printer chosen, 3D printing using FSM can achieve high printing resolution and is capable of fitting optical lenses without much calibration necessary.^131,189 However, FSM methods can have weak mechanical properties and poor aesthetic appearance compared with more advanced methods of additive manufacturing, such as stereolithography or power bed fusion.¹⁹¹ The most common material used was polylactic acid, commonly abbreviated as PLA.¹⁹¹ PLA is notable for having a low melting point and good biocompatibility. This aspect means that it is possible for researchers to use PLA to build biomedical components in their spectral cameras.^51,59 However, PLA has some structural downsides. PLAs are known to shrink during printing; Sigernes et al.⁸⁵ suggested that the actual design is 1% to 2% larger than intended so the optical components can fit. If the optical alignment is critical, 3D-printed housing can potentially produce optical misalignment and imaging artifacts. Despite shrinkage being a common problem in polymer-based printing, not many research studies discuss the solutions to counter this.¹⁹⁴ Beyond accounting for the shrinkage in the initial design, Pearre et al.¹⁹⁵ suggested bounding the printed components with solid structures. Alternatively, inkjet technology using plastic powder and cyanoacrylate adhesive was used by Wang et al.¹⁸⁷ to print housing for their handheld spectral camera.

5.5.

Electronics Components

Electronics is the driving component of all spectral imaging systems. “Electronics systems” refers to systems that provide illumination, capture images, control hardware, and transfer data. Systems that are expected to perform their work remotely, such as UAV-based, require a means to store power and to store data as well. Here, we discuss the use of illumination with a special focus on LEDs, the use of sensors, the use of microcontrollers, and the expected power consumption of the system.

5.6.

Illumination

For most biomedical imaging applications, imaging was performed indoors using artificial illuminations. This posed various challenges for acquisitions. It is important to differentiate between luminance and radiance. Radiance refers to the quantity of radiant energy per emitted solid angle per receiving surface area and is applied across all wavelengths, whereas luminance is a human vision-centric measurement and is the radiance weighted by the response curve of the human eye. With spectral imaging devices, radiance is the more appropriate measurement of light sources whereas with regular cameras and human activities, luminance is more appropriate. Unlike outdoor illumination, which can exceed 100,000 lux in luminance, indoor systems using incandescent light only illuminate around 10,000 to 20,000 lux. Spectral imaging requires more illumination than similar RGB or monochrome cameras, because spectral cameras need to capture energy associated with a narrow range of wavelengths. Furthermore, many spectral imaging systems have additional filters that reduce the incoming light quantity. Some systems require wavelengths bandpass, lowpass, or high pass filters to block out unnecessary wavelengths.⁵⁴ Some systems use beam splitters to either achieve snapshot imaging or to capture both live images and spectral images.²⁰⁰ Additional illuminations are often needed in spectral imaging acquisition. The additional components can increase the footprints of the system. The addition of light sources should be balanced with the applications. If the intensity of the light is too large, damage to the tissues can be irreversible. The most vulnerable organ is the eyes. Permissible exposure limit to the human retina is dependent on both the wavelength and exposure time. Rees and Dobre²⁰⁴ calculated the maximum exposure power to the eye at 0 deg to be $\sim 180\mu \mathrm{W}$ at 5 s and $\sim 120\mu \mathrm{W}$ at 30 s. For thermal light sources, Yan et al.²⁰⁵ evaluated maximum exposure for a direct angle to be $1.8\times {10}^{-3}\times {\mathrm{t}}^{0.25}\mathrm{J}/{\mathrm{cm}}^2$ with time t in second and is $<3\times {10}^4\mathrm{s}$. These values are only for humans; for animals with retina diameter different than humans, the maximum permitted exposure needs to be varied. There is no definitive guidelines on animals’ retinal’ light exposure limits. For other organs, the limits are more forgiving and often orders of magnitude larger than many illuminations needs. Surgical lighting, for example, has a suggested luminance ranging from 40,000 to 160,000 lux,²⁰⁶ which is often enough for acquisitions in the visible light range.

There are different geometries to arrange the light source, the subject, and the imaging camera. Amigo et al.²⁰⁷ detailed several laboratory setups, which are frontal, lateral, contrast-transmission, diffuse, co-axial, and dark field. In each case, illumination should be as uniform as possible. Sawyer et al.²⁰⁸ detailed three types of nonuniformity: spatial uniformity, which refers to differences in illumination of incident light across object; angular uniformity, which refers to differences in incident and shadowed areas; and spectral uniformity, which refers to spatial uniformity across all wavelengths. They compared three different light setups for wide-field imaging: an LED ring, a fiber halogen ring, and a diffusion dome. They found that all systems achieved similar spatial and spectral uniformity, but the diffuse scattering dome achieved the highest angular uniformity.

For the illumination type, we covered the three commonly used types: halogen incandescent, gas discharged lamp, and LED. Halogen light operates in the ultraviolet, visible, and NIR regions. It offers a continuous spectrum suitable for acquisition with many bands. However, halogen light has a low color temperature (3200 to 5000 K) and can appear yellow at low illumination.²⁰⁶ Halogen and LED were the most often light sources in microscopes, so many spectral imaging applications that used microscopy also used halogen or LED as default illumination. Gas-discharged lamps, which include xenon lamps, mercury lamps, and mercury–argon lamps, often have higher color temperature. Xenon lamps have color temperatures ranging from 4000 to 6000 K, which makes them closer to outdoor lighting. Gas-discharged lamps often have energy spikes in the NIR regions, which can affect acquisitions. Due to these energy spikes, gas-discharged lamps are not used in retinal illumination and retinal surgeries. Both gas-discharged lamps and incandescent lamps emit a large amount of heat. If this is a concern, a fiber optic guide is needed to direct the light far away from the light source. Tunable laser is also seen in spectral imaging devices,^209–211 although not in compact systems due to the size and power consumption of the laser components.

5.6.1.

Light-emitting diode

The advantages of LED in illumination scanning systems are numerous: they are low-cost, fast, have low power consumption, low heat dissipation, and long life.²⁰⁶ A common way to use LED for spectral imaging is to use different LEDs to illuminate the object, each LED emits light within a specific waveband. Alignment of LED is important to achieve uniform illumination. Figure 9 shows different methods of arranging LEDs in a compact spectral imaging system. A common layout is the ring layout, where all LEDs are arranged in a circle.^22,24,25,38 Li et al.²¹² proposed a different setup where the LEDs were arranged in a mosaic fashion, however warned that this arrangement can lead to nonuniform lighting for different wavelengths. Bolton et al.³⁷ arranged the 16-LED setup into a $4\times 4$ rectangular array. Light uniformity was ensured by choosing LEDs with very wide illumination angles. If the LEDs were arranged in a circle, it is better to have multiple of each type that are opposite to each other. Delpueyo et al.²⁴ and Rey-Barroso et al.²⁵ proposed a system using 32 LEDs, where four of each type of LED were arranged 90 deg apart [Fig. 9(a)]. For the acquisition camera, Shrestha and Hardeberg²¹³ recommended that monochrome camera might perform better than commercial RGB cameras. The reason being that RGB cameras need to perform demosaicing operations, which altered the actual spectral readings. To control the LED, a separated power source and controller was needed to synchronize the LEDs with the acquisition camera. LEDs do not require a high-power source; a configuration built by Kim et al.^22,38 only needed a 3.7 V battery to drive a multi-LEDs system [Fig. 9(b)]. If the LEDs emit different wavelengths, the activation voltage could be different for each LED.³⁷ For Bolton et al.,³⁷ different resistors values were required to produce different activation voltage using the same input voltage [Fig. 9(c)]. The biggest downside to LEDs is the limitations of wavelengths. The spectral imaging systems described can acquire only as many wavelengths as the number of different LEDs are available. For many researchers, this may not be a problem. Some applications only focus on specific excitation and absorption wavelengths and only need three to four different types of LEDs. Well-chosen LEDs are very beneficial. Another potential engineering problem comes from the quality of the LEDs. If the LEDs are bought from commercial vendors, then their bandwidth can range anywhere from 20 to 70 nm. If this is a problem, the interference filters should be applied to achieve a narrower bandwidth.²¹⁴

Fig. 9

Different uses of LEDs in a compact imaging system. (a) A system that used 32 LEDs arranged in a circle. There were four of each types that are spaced 90 deg apart (reproduced from Ref. 24). (b) A system that used a singular white LED and a rotating spectral filter (reproduced from Ref. 22). (c) A system that arranged LEDs into a four-by-four rectangular plane (reproduced from Ref. 37).

6.1.

Commercial Off-the-Shelf Components

COTS, or commercial off-the-shelf, refers to components of a spectral camera that can be bought and assembled into a functional unit. Assembling COTS requires the researchers to have knowledge in building an optical system, electronics, control software, and hardware housing. The total price of all components added together is often much lower than the unit price of one commercial spectral camera of similar capability. Furthermore, COTS components allow researchers a high degree of customization. As mentioned in the previous section, components of a spectral imager include the optical system, the spectral dispersion system, the digital image detector, the control module/electronics, and the mechanical elements. It is more common for researchers to buy each of these components individually and assemble them together, because of the wide availability of low-cost options to choose from (Fig. 10). Alternatively, some researchers opted for portable spectrometers and digital cameras as the acquisition module and built the rest of their system surrounding those.

Fig. 10

Spectral devices enabled using commercial off-the-shelf components. (a) A spatial scanning camera built using customized optical components and 3D printed housing (reproduced from Ref. 85). (b) A spatial scanning camera that uses portable spectrometers as the spectral acquisition device (reproduced from Ref. 193). (c) A smartphone-powered spectral imaging systems (reproduced from Ref. 215).

6.2.

Optical Setup and Calibration

In the past, to conduct optical research, optical systems needed to be built on a rigid optical table and required large footprints. However, it is now possible to build optical systems with COTS components that are lightweight and have small spatial dimensions. Sigernes et al.⁸⁵ [Fig. 10(a)] and Fortuna and Johansen²¹⁶ built systems that were carried by UAVs. Their systems were spatial acquisition systems using transmission gratings. Kim et al.²² built a customized optical system to complement smartphone image acquisition. The system used a planoconcave lens, a reflective flat mirror, linear polarizers, and bandpass filters. The linear polarizers and the bandpass filters filtered the LED light sources of the smartphone, whereas the flat mirror and the planoconcave lens made the illumination broader and more uniform. Jeon et al.²¹⁷ introduced a new type of diffraction optical element using photolithography and reactive-ion etching techniques. They used the fact that Fresnel diffraction is dependent on the wavelength to produce a diffractive element that produces spectral-varying point spread functions. 3D printed optical elements, such as reflective mirrors, focusing lenses, and prisms, despite already existing,¹⁹⁴ have not been seen in any spectral imaging system as of yet. Customized mechanical housings are required to house these optical components. For that, 3D printing is the preferred option.

Even though COTS optical components are vastly cheaper than their commercial counterparts, lack of quality control can harm the acquisition quality. Smile, keystone, and chromatic aberration are all optical problems these customized systems face. We discussed smile and keystone in Sec. 5.1 in relation to spatial scanning systems. Chromatic aberration is a property of optical systems that makes the focus of different wavelengths not aligned. The point-spread function of the image depends on the wavelengths. Chromatic aberration makes focusing difficult. For spectral scanning systems, one method of dealing with chromatic aberration is to change the focus for each wavelength. However, this method could not be replicated with other types of spectral acquisition. A more robust engineering solution is to use achromatic lenses, which come at a higher cost than regular optical components. Mirror-based grating systems also reduce aberration; we discussed the Offner imaging spectrometer, which reduces aberration. If low-cost optical systems are to be expected to perform accurately, extensive calibration processes are necessary. Sigernes et al.⁸⁵ discussed the steps to perform chromatic and sensitivity calibration in their customized camera. Riihiaho et al.²¹⁸ performed smile correction by calculating the matrix that transforms curved spectral lines into the straight lines for each spectrum. Henriksen et al.²¹⁹ proposed an algorithm that could perform real-time correction of smile and keystone based on matrix operations of calibrated results.

6.3.

Portable Spectrometers

Compact and portable spectrometers have been developed and used for commercial, health, and scientific purposes.⁸ Many custom imagers in fact incorporated one or multiple commercially available spectrometers into their designs. By incorporating spectrometers, researchers did not need to worry about optical architecture and could focus more on the scanning methods. Spectrometers also have a much finer spectral resolution compared with line scanning spectral cameras. However, because the field of view of a spectrometer is typically small, the sampling scheme needs to be large with oversampling, otherwise the spatial resolution would be very low. Devices that used compact spectrometers were usually point-scanning or line scanning imagers. If only one spectrometer was used, the device would be like a point-scanning camera, and the researcher would devise an appropriate scanning path. Nevala and Baden¹⁹³ equipped a commercial spectrometer that has effective spectral ranges from 350 to 950 nm with two mirrors controlled by Arduino microcontrollers [Fig. 10(b)]. Because the scanner had a circular window, they proposed circular scanning paths that are based on Fermat’s spirals. In the system proposed by Stuart et al.,²²⁰ light entered a focusing lens before being redirected by two movable mirrors into a spectrometer. Commercial spectrometers were used in confocal microscopy by Frank et al.²²¹ who connected the confocal pinhole with a spectrograph using a fiber optic cable. If multiple spectrometers were used, then the device is a line scanning imager. Uto et al.²²² used a series of eight spectrometers for a line scan imager.

6.4.

Smartphones

Personal diagnostics with smartphones is gaining more attention in healthcare and bioengineering since smartphones are readily available, portable, offer high quality images, and are power efficient. This trend is reflected in spectroscopy and spectral imaging alike. Crocombe⁸ outlined four main usages of smartphones in spectrometry and spectral imaging: (1) to receive data from the spectrometer as a separated module; (2) to receive data; (3) to send command to the optical module; and (4) to be the optical module itself. The fourth usage is only applicable in the visible and near-infrared regions, as silicon detectors used in CMOS and CCD only operate within the 400 to 1000 nm wavelengths. Spectroscopic solutions have been developed for the smartphone for the purposes of food quality control,²²³ lab-on-a-chip diagnosis,^224,225 and fluorescence spectroscopy.²²⁶ Similarly, spectral imagers have been developed with smartphones in mind for biomedical, remote sensing, and field laboratory purposes. Kim et al.³⁸ developed a system that used smartphones as the controlling center for their MSI. In their system, the smartphone sent signals to a microcontroller unit that drove LEDs and received photos from a CMOS camera. Näsilä et al.¹⁵⁹ and Rissanen et al.¹⁸⁴ developed FPI-based spectral imagers with smartphone as the centerpiece, both as a controlling device that sent and received signals and as the optical camera itself. Smartphone cameras are extremely powerful and fast, which makes them the ideal candidate for spectral imaging modules. If the devices are used in personal healthcare settings, custom parts need to be developed for smartphone spectral imaging to fit the specific needs. Stuart et al.²¹⁵ built a spectrograph with the smartphone camera as the acquisition device [Fig. 10(c)]. Bolton et al.⁵¹ and Cavalcanti et al.⁵⁹ used 3D printing to create custom smartphone-based spectral colposcope and otoscope, respectively. Their systems used LEDs driven by microcontrollers to provide narrowband illuminations. These designs also used the smartphone’s ability to transfer signals over Bluetooth or Wi-Fi to construct an internet of things (IoT) system for spectral images acquisition and analysis. In some applications, smartphone can also be used as a geo-tagger, providing GPS data along with images to register images together.²²⁷

6.5.

Open-Source Design

Open-source design refers to the creation of software and hardware with the intention of freely sharing those designs for others to use, study, modify, and even commercialize without concerns of copyright or patent infringement. This allowance is performed through many different forms of open-source licenses. Whereas the nature of software makes open-source software easily distributed, open-source hardware still requires usages of tangible components to manufacture and assemble. Ideally, open-source hardware designers would want to use low-cost components, 3D-printed components, or other open-source hardware components and run free open-source software. Some spectral imaging researchers published systems that were open source with the goal of easily sharing research progress. As an example of the open-source model, Riihiaho et al.²¹⁸ built a 3D-printed spectral imaging camera with designs from Sigernes et al.⁸⁵ Some further modifications were made to reduce smile and aberration, which were publicly shared along with 3D CAD files. They also built software that performed aberration correction and shared it publicly. Nevala and Baden¹⁹³ published the entire hardware and software design of their system, along with raw spectral data captured by the camera. Low-cost spectral imaging was driven not just by open-source hardware but also through open-source software. Many researchers published hyperspectral processing software as free and open source. Berisha et al.²²⁸ published a framework for processing large-scale biomedical hyperspectral images at a faster speed using GPU. The entire software and all its algorithms were published under a license that allowed inclusion in both free and commercialized systems.

7.1.

Diagnosis and Monitoring of Diseases

The primary diagnostic targets for spectral imaging researchers were the skin, the oral cavity, and the retina. These organs are the most accessible organs for in vivo diagnosis. Using endoscopic systems, the luminal organs of the lower abdominal regions could also be imaged with spectral imagers. However, research in this field in vivo was scarce because in vivo research of lower abdominal regions requires systems with fast acquisition time and low illumination. For ex vivo applications, most spectral imaging research was done on pathology slides or tissues. Ex vivo tissues were imaged either in a tabletop setting using reflectance data or through a microscope, which can be reflectance or transmission data.

7.1.1.

Skin cancer

Both melanoma and nonmelanoma skin cancer (MNSC), which include Bowen’s disease, Kaposi’s sarcoma, basal cell carcinoma (BCC), and squamous cell carcinoma, are on the rise.²³⁴ While MNSC is more common, melanoma has a higher mortality rate.²³⁵ Early diagnosis and removal of both melanoma and MNSC are necessary. However, many forms of skin cancer appear similar to benign neoplasm.²³⁶ Dermatoscopy can be used to differentiate benign from malignant skin lesions; however, dermatoscopy requires clinician input. Spectral imaging has been used to improve automatic classification with high sensitivity.²³⁶ In early 2000, Hewett et al.^21,237 developed a portable imaging station to determine tumor margin of MNSC lesions. They used 5-aminolevulinic acid (5-ALA) to induce the fluorescing molecule protoporphyrin IX (ppIX). After application of 5-ALA to the skin, fluorescence imaging of MNSC lesions was taken at the 400, 540, 600, and 635 nm wavelengths. Their results showed a clear outline of the lesion at the 635 nm wavelength, which corresponds with maximum ppIX fluorescence. In the same time period, Elbaum et al.²³⁶ developed a handheld spectral imaging system to automatically classify melanoma and melanocytic nevi (pigmented nevi, moles). From a set of 10 spectral images obtained from 430 to 950 nm wavelengths, 822 candidate parameters were extracted in wavelet domain and spatial domain. The features were supposed to represent morphological and textural elements of the lesions. Through training an expert system using a dataset of 63 melanoma images and 183 melanocytic nevi images, they achieved 100% sensitivity at 85% specificity. A similar handheld system [Fig. 11(a)] was developed by Delpueyo et al.²⁴ to classify melanoma and BCC from pigmented nevi [Fig. 11(b)]. First-order statistics were used to extract morphological features. Their systems achieved 91.3% sensitivity at 54.5% specificity for both melanoma and BCC. Rey-Barroso et al.²⁵ continued the study with a second handheld camera that can image in the NIR region (995 to 1613 nm). Higher wavelengths penetrated deeper through the skin and showed more features pertinent to skin cancer diagnosis. With the same set of features, they produced a sensitivity of 85.7% at specificity 76.9%, which was an improvement in specificity over the results obtained by Delpueyo et al. without much loss in sensitivity.

Fig. 11

(a) A compact and handheld spectral imaging system that used LEDs to provide spectral scanning. The system weighed 500 g and could capture images in eight wavelengths. (b) Using the system described in (a), different spectral signatures could be seen for nevi, melanoma, and basal cell carcinoma (reproduced from Ref. 24).

Many melanoma diagnostic tools that use spectral imaging exist on the market. However, they have high cost.²³⁸ Several researchers developed smartphone solutions for identification of malignant melanoma that were low-cost and accessible. Kim et al.²² built a smartphone spectral imaging system that photographed in 10 wavebands from 440 to 690 nm. They also created a software platform that identified lesions margin and graded lesions severity from spectral images. Ding et al.²⁷ built a similar snapshot system that used smartphone camera. They photographed nevus lesions and identified elevated optical density of the nevus region in the 550 to 640 nm wavelengths; these wavelengths correspond to the peak absorption wavelengths of melanin and oxygenated hemoglobin. Uthoff et al.²⁸ developed a smartphone spectral imaging system that mapped oxygen concentration, melanin concentration, and erythema measurement onto images of squamous cell carcinoma. However, these methods were only able to visualize the outline of skin lesions. To show that these smartphone-based methods can classify between malignant and benign melanoma lesions, further clinical studies are necessary.

7.1.2.

Head and neck cancers

HNC refers to cancers that originate from the nasal and oral cavity, nasopharynx, oropharynx, hypopharynx, larynx, and esophagus. Up to 90% of cases of HNC are squamous cell carcinoma,²³⁹ so the majority of studies reviewed focus on head and neck squamous cell carcinoma (HNSCC). Liu et al.⁴⁵ used an AOTF camera to perform tongue tumor pixel segmentation. For classification, they used the sparse representation method. Their system achieved 96.5% pixel-accuracy on a dataset of 65 tumor images. Bedard et al.⁴⁶ built a snapshot imaging system for imaging both autofluorescence and reflectance data. They used the system to photograph images of the oral cavity in healthy individuals and in patients with oral cancer. Using spectral unmixing, they were able to (1) highlight vasculatures in the lip region and (2) determine boundaries of tumors in the oral cavity. To image further into the oral cavity, spectral imaging systems used endoscopes. Often, in in vivo diagnosis, perfusion is a feature of interest since higher perfusion could be an indication of neoplasm. Köhler et al.⁷⁰ created a spectral laparoscopic system for imaging the esophagus. Using customized metrics, they were able to visualize the hemoglobin index of a resected esophagus from a patient with Barrett’s syndrome. Our research lab contributed new research in identification of HNSCC ex vivo with the use of a compact spectral camera (Fig. 12). Ma et al.⁴⁷ imaged histologic slides resected from the laryngeal and hypopharyngeal regions of patients with HNSCC [Fig. 12(a)]. They proposed two different methods of classification: (1) a support-vector machine (SVM) method that uses spectra of segmented nuclei as input data and (2) a convolutional neural network method that uses small image patches as input data [Fig. 12(b)]. They found that the CNN classifier had better performance compared with the SVM classifier. However, they also found that classifiers trained on spectral images did not outperform classifiers trained on RGB images.

Fig. 12

(a) A compact hyperspectral Snapscan camera on top of a microscopic system (reproduced from Ref. 47). (b) Using the system described, a series of hyperspectral digital pathology images were acquired. The slides were from patients diagnosed with squamous head and neck carcinoma. A machine learning system was used to produce a probability heat map of cancer occurrence (reproduced from Ref. 240).

7.1.5.

Healing from burns, scars, and wounds

External factors such as age, lifestyle, smoking, and diet influence the healing process. Homeostatic imbalances such as ischemia (low perfusion), hypoxia, edema, and necrotic tissues also directly affect healing. To gauge healing process from burns, scars, and wounds, constant monitoring of perfusion, oxygenation rates, and tissue formation must be performed along with visual analysis.²⁴³ Methods used for wound monitoring included angiography with indocyanine green dye,^244–246 LSCI,^247–249 optical coherence tomography,^250,251 laser Doppler flowmetry,²⁵² and high-resolution ultrasound imaging.^253,254 While no method offers distinct advantages over others, reflectance spectral imaging offers compact hardware and low-cost, noninvasive imaging.

Marotz et al.²⁵⁵ and Holmer et al.²⁵⁶ used a compact push-broom camera that operated in the VIS-NIR region for the purpose of assessing skin transplantation wounds. Reflectance data in the visible region were used to calculate hemoglobin relative concentration and oxygenation rate in the superficial dermis layer [Figs. 13(b) and 13(c)]. On the other hand, near-infrared reflectance data revealed deeper circulation in the subcutaneous layers and were used for estimating deep perfusion [Fig. 13(e)]. These data were mapped over the wound to assess the presence of ischemia [Fig. 13(f)]. Kulcke et al.³⁴ later used the same commercial spectral system to image wound healing over a period of 2 weeks and showed that the perfusion and tissue oxygenation rates reduced to normal level after 2 weeks. Rutkowski et al.³³ used compact spectral camera to monitor wound healing treatment with cold atmospheric plasma. They showed improved angiogenesis in both the dermis and subcutaneous tissue layers through cold atmospheric plasma treatment in vivo.

Fig. 13

Estimation of physiological values from a wound photograph. (a) RGB image, (b) and (c) relative and segmented tissue oxygenation mapping (${\mathrm{S}}_{\mathrm{t}}{\mathrm{O}}_2$ in the figure), (d) reconstructed RGB image from hyperspectral data, (e) near-infrared-based perfusion data (NIR in the figure), and (f) relative tissue hemoglobin index (THI in the figure) (reproduced from Ref. 257).

7.1.8.

Diseases of the central nervous system

An exciting recent application of spectral imaging in bioengineering is the monitoring and diagnosis of neurological diseases through retinal imaging. It is known that the retina contains optical neurons that are directly linked to the brain, so it follows that biomarkers for many neurological diseases, such as Alzheimer’s disease (AD), Parkinson’s, and multiple sclerosis can be seen through retinal imaging.²⁷³ Amyloid beta (Ab) peptide has been identified in the brains of people with AD and is a known biomarker. Similarly, autopsy of patients with AD shows high concentration of Ab in the retina.²⁷⁴ Through spectral imaging of transgenic Alzheimer’s mice using a compact endoscope, More et al.⁵⁶ showed that in the wavelengths from 450 to 700 nm, there existed a marked difference in the optical spectra between wild-type mice and transgenic mice (Fig. 14). They also showed that changes in optical spectra strongly correlated with the accumulation of Ab in the retina and progression of AD over time.

Fig. 14

(a) A compact spectral imaging system used to image the retina of mice. (b) Using the system described, the retina of wild-type mice (WT) and transgenic mice for Alzheimer’s disease (APP1/PS1) were imaged. The result showed a significant difference in the spectral signature between the two (reproduced from Ref. 56).

7.2.

Surgical Guidance

Lu et al.⁷ identified the four key benefits of using spectral imaging in surgical guidance: (1) visualization of microsurgery features, (2) hyperspectral tumor segmentation during resection, (3) monitoring of tissue oxygenation rate, and (4) visualization of large organs. The use of compact spectral imager benefits surgical guidance greatly because they free up limited space in the surgical room. Because spectral imaging is good at identifying blood oxygenation status, many researchers used them in the surgical room to monitor blood flow. Anastomotic insufficiency is a break or leak in a surgical suture and is among the most serious complications in colorectal surgery. Jansen-Wilken et al.⁶⁹ used a spectral camera to detect the anastomosis site during small bowel surgery. Many other researchers explored similar surgical complications, most commonly ischemia.^63,67,275 Akbari et al.⁶⁷ used two cameras that operated in the 400 to 1000 nm and 900 to 1700 nm wavelength for bowel surgery. They found that the highest contrast between normal and ischemic regions in the intestine were seen in the 765 to 830 nm wavelength range, and they used SVM to evaluate ischemia progression over time.

7.2.1.

Spectral endoscopy

Many researchers investigated the use of spectral imaging in combination of surgical visualization tools. Endoscopic tools, which aid minimally invasive surgery, are often combined with compact spectral cameras. Kumashiro et al.⁴⁸ attached a mobile spectral camera to a colonoscope. They directly observed colon lesions in vivo during biopsy. However, because the camera had a long acquisition time, scanning time was limited to 5 s and scanning resolution was limited to $200\times 200\text{pixels}$. Nevertheless, they found significant differences in absorption between normal mucosa and adenocarcinoma regions at the wavelengths 525 nm. Laparoscopy is an operation performed through small incisions with the aid of cameras and is minimally invasive. Clancy et al.⁶³ built a custom laparoscope and measured bowel oxygenation rate during clamping in vivo. While their system was built for minimally invasive surgeries, imaging was performed during open surgery. Zhang et al.⁶⁵ used a similar system to identify different types of tissues. They identified an improvement in classification accuracy using multispectral images over using RGB images. However, their system was tested on ex vivo tissues and not during live surgery. Some researchers built laparoscopic systems with dual channels, such that both live video and spectral images can be simultaneously captured. This was commonly done with the use of beam splitters. Köhler et al.⁷⁰ developed a spatial laparoscopic camera for aiding esophagus surgery that captured monochromatic video and spectral images. To validate their system, they compared it with a commercial spectral camera developed for surgical settings. The specimen used was ex vivo human esophagus with adenocarcinoma. They found that their own compact laparoscopic system showed carcinoma classification result consistent with that of commercial devices. A similar dual camera endoscopic system that showed both spectral image and real-time video was developed by Yoon et al.²³¹ They demonstrated the system using an ex vivo pig esophagus and used linear unmixing to estimate concentration of staining solution (see Fig. 15). They stained the esophagus with methylene blue (MB) solution and used measured absorbance to calculate the MB concentration throughout the tissue. Yoon et al.²⁷⁶ later improved the system and used it for clinical testing on 10 patients that underwent colonoscopy. They used spectral angle mapping to extract features and $k$-nearest neighbors to classify between normal mucosa and polyps.

Fig. 15

Images taken from an ex vivo pig esophagus by a compact spectral endoscope. (a)–(c) The esophagus taken before staining. (d)–(f) The esophagus after MB staining. (g) and (h) The estimation map of MB concentration using two different linear unmixing methods. (i) The spectral absorbance plot for the shaded areas in (h). In (c) and (f), the color corresponds to the color segment in the corresponding image to the left (reproduced from Ref. 231).

8.1.

Hardware Limitations and Potential Solutions

Currently, there is still a large tradeoff in spatial scanning, spectral scanning, and snapshot imaging cameras. The tradeoff has three main components: spatial raster resolution, spectral number resolution, and acquisition time. The spatial and spectral scanning systems are optimized for spatial and spectral resolution, whereas the snapshot scanning systems are being optimized for acquisition time. If there are any “dream” spectral imaging systems that can achieve large spatial and spectral resolution within a short amount of time, several engineering barriers must be overcome. First, what should the acquisition mechanism be? If the system is snapshot, it needs to record the entire datacube onto the image sensor. The sensor for such a system would be much larger than any counterpart RGB imaging sensors; this already can impact the size of the camera. If the system is spatial or spectral, then the mechanism for translating or moving the filter must be fast enough to scan through the entire field in a reasonable time. Second, how will data recording work? Due to the enormous amount of data spectral cameras capture, most systems, even commercial ones, send raw data and receive commands to and from another computer. Behmann et al.¹⁴⁴ described a commercial system that performed live calibrations without user input. New generations of spectral camera systems are heading toward on-the-spot image processing within the system hardware, which can make analysis much more seamless.

We briefly touched upon the power issues of existing cameras. For now, power consumption is not important because many laboratory-based systems draw power from the grid. However, in a future where biomedical spectral imaging systems will be used in low-resource situations, remote power sources and low power consumption must be a research priority. Another potential problem with compact spectral cameras is system cooling. Thermal radiation emitted by the instrument itself is negligible in the visible-NIR range but can affect the imagers working in the long wave infrared (LWIR, 8000 to 14,000 nm) range.²⁸⁰ To minimize this effect, LWIR spectral cameras need a specific cooling apparatus, which increases the size and weight of the system.

8.2.

Quest for Lower Costs

A barrier toward popularization of spectral imaging research is the high cost of commercialized systems. As of now, a commercial spectral imaging camera can cost tens of thousands of dollars, many orders of magnitude more than their commercialized RGB counterparts. The reason for this is due to the small market size and complex manufacturing process. Manufacturing accurate spectral imaging systems requires precise spectral components and elaborate calibration processes, similar to the manufacturing process of other precision instruments. To counter this, many research facilities and hobbyists have produced customized spectral cameras using off-the-shelf custom instruments. We reviewed the use of open-source hardware and software, 3D printing, smartphone, and low-cost spectroscopy, and off-the-shelf optical systems. One common thread that enables all of them is the rise of personal simulation and modeling software. Ray-tracing software, optic simulation software, and 3D modeling software make customization possible. Still, a wide gap exists between the quality of COTS systems and that of commercialized systems. Many commercialized methods used high precision manufacturing standards. For example, SRDA, MEMS, and Offner spectrometers used lithographic manufacture. These methods were limited to research labs and clean room operations.^121,166 If systems are built using low-cost components, extensive optical calibrations are needed, typically using a secondary spectral camera or a known light source. Improper calibration severely distorts the experiment outcomes, especially in applications that require high spectral and spatial precision.

Progress in making more low-cost customized spectral cameras will come from engineering using off-the-shelf components and from the open-source movement. Researchers should share new systems designs through open-access journals and invite collaboration and improvements. While open-source design is becoming increasingly accepted, it faces significant challenges in medical research. Many new designs did not undergo the strict regulation necessary for medical devices. They also had lackluster business models, which discouraged further developments. Copyright laws that cover open-source designs differ between hardware and software. Software is “created work” that is often legally protected under copyright, whereas hardware is “invention” that is protected under patent. Many companies that manufacture open-source hardware still protect their product under a trademark, which acts as a form of quality assurance.²⁸¹ 3D printing and open-source movements have intertwined roots; software used to design and model in 3D are free and have large exchange forums on the internet. The majority of individuals involved in the 3D printing community are also involved in open-source projects, which shows a mindset of collaboration.²⁸² We summarized the use of 3D printing for the manufacturing of customized parts and housing of compact spectral imagers. However, the biggest issue with 3D-printed housing is the mechanical durability. PLA, the most commonly used material for 3D printing, is cheap and accurate but has low heat tolerance and outdoor tolerance. It also has the potential to shrink after cooling. Future researchers interested in developing 3D-printed housing for hyperspectral cameras should study different materials and their durability; many materials exist such as acrylonitrile butadiene styrene, carbon fiber filaments, or metal-filled filaments that can serve as an alternative to PLA.

We believe that smartphones will be an important component of future compact low-cost spectral imagers. Smartphones are already engineered to be extremely compact with good processing and low power consumption. They are also relatively low-cost and widely available. Smartphone cameras have achieved considerable progress in the last decades. Researchers should use smartphones not just as a camera substitution but also as a control unit. For this, open-source phone operating systems should be preferred, because they allow control software to be written and shared freely. Smartphones are also interconnected to the communication network, which means that they can be used in a larger IoT system. We expected more smartphone-based systems to be used in low resource settings or in coordinated clinical trials. A potential problem is that smartphones have a variety of different configurations short-term technical support, so long-term development and collaboration can be difficult.

8.3.

Future of Spectral Cameras in Bioimaging

In the second part of the paper, we reviewed the use of compact spectral imagers in the biomedical imaging field. Building spectral imaging biomedical systems in many cases were similar to “plug-and-play” systems. We believe that compact imaging systems will become much more common in clinical settings. The smaller size means that there will be less space, which means more room for other instruments. We predict that compact spectral cameras in biomedical imaging will have two types of applications. First, they will replace existing bulky spectral imaging systems. Surgery and diagnostic devices will be the main beneficiary of this change. In surgery, smaller cameras can be fitted on top of endoscopes, surgical microscopes, and operating robots without much interference to the surgical process. Already, spectral imaging has enabled the visualization of key features during surgery but it has not been widely adopted because of bulky sizes.⁷ Compact spectral cameras can allow for widespread adoption of spectral imaging technologies in the surgery room. We believe that snapshot cameras will be the dominant system in these situations because of their rapid acquisition time. Compact spectral systems will be used for more diagnosis of skin and retinal diseases in vivo. The strength of spectral devices comes from the fact that they are noninvasive and fast, which means that sensitive organs such as the retina and skin are prime candidates for diagnosis. These organs also contain well-known spectral-sensitive information, most notably blood oxygenation rate. The lighter weights will enable some systems to go from tabletop to handheld, which means that they can be much more convenient for clinical settings.

Second, compact spectral cameras will be used to research new physiological processes. If we want to see more research being done in biomedical imaging using compact cameras, we must understand more about the spectral signatures of physiological processes. Knowledge of hemoglobin spectra and skin physiology has already helped researchers to construct elaborate models to diagnose diseases such as melanoma, burns, wounds, ulcer, diabetic foot, and erythema. Large pathology datasets have been used to construct classification algorithms for digital staining, cancerous cell diagnosis, and cellular segmentations.²⁸³ We expected similar progress will be made in the retina, as it is currently linked to many complex diseases of the central nervous system.²⁸⁴ Recently, new understanding of how amyloid beta affects the scattering profile led to the development of hyperspectral imaging for Alzheimer’s diagnosis.^56,274,57 We advocate for creations and sharing of hyperspectral imaging databases. Currently, most spectral imaging databases are satellite images used for remote sensing. Only a handful of hyperspectral databases are for biomedical purposes,^285,286 and they are of specific diseases. Creation of new databases is difficult: acquisition requires a hyperspectral camera system and many patients or specimens. They also require a large hosting space, which can be hundreds of gigabytes. However, the scientific contribution of such database will be invaluable if they advance our understanding of human physiology.

We also predict the use of compact spectral imaging alongside other imaging modalities. Spectral imaging is not always the superior imaging modality. New applications are limited by the penetration depth of light through the skin tissues. Light in the VIS-NIR region has a penetration depth ranging from 0.48 mm at 550 nm to 3.57 mm at 850 nm. For comparison, photoacoustic multispectral imaging can achieve a penetration depth of up to 5 cm with a handheld system.²⁰ To circumvent this shortcoming, imaging in the SWIR wavelengths has been proposed to provide greater penetration depths.⁸⁸ By combining compact spectral imagers with other imaging modality, the multimodal system was constructed to provide point-of-care analysis to the patients.^108,225 There are several different imaging modalities that pair well with hyperspectral imaging systems. Optical coherent tomography (OCT) captures images with depth of several millimeters and micrometer resolution. Combination of OCT and spectral imaging provided both surface chemical information and depth information.^287,288 LSCI provided dynamic movements information of blood vessels, which paired well with spectral imaging ability to resolve blood oxygenation contents.²⁸⁹ Raman spectroscopy provided detailed chemical information in a small spatial scale, which complemented spectral imaging.²⁹⁰ Compact devices exist for many of these modalities, which means that multimodal systems can still be compact and convenient.