1.

Introduction

Regulation and coordination of cell shapes are central to native physiology during all stages of organismal existence. Recent advances in optical imaging have provided mechanistic insights into such phenomena by revealing cellular features and processes with previously unimagined detail.^1,2,3 Central to the accurate analysis of such intricate biological processes is the precise segmentation of cellular images. Quantifying cellular morphology, such as shape, area, circularity, aspect ratio, and others, starts with first segmenting the cells in a given field of view. Due to its unquestionable significance, much work has been done to standardize the process. There are well-developed open-source software suites, notably CellProfiler⁴ and CellPose,⁵ to perform such segmentation tasks with great accuracy. The recent update to CellProfiler includes the functionality of three-dimensional (3D) image segmentation and is currently the most widely used tool to perform such tasks. However, since the current workhorse for biological imaging is fluorescence microscopy, all standard segmentation software are optimized for and targeted to the analysis of fluorescence images.

Yet, contrast-agent-free microscopy is highly desirable to study the dynamics and physiological activity of various structures in living cells. Quantitative phase imaging (QPI) measures optical field images using laser-based interferometry and has rapidly emerged as a viable imaging alternative because it offers an objective measure of morphology and dynamics in a label-free manner.¹ In addition to the amplitude images provided by conventional intensity-based microscopy techniques, QPI measures optical phase delay maps governed by the refractive index (RI) distribution of a sample. Since the endogenous RI distribution is strongly related to the structural and biochemical characteristics of the cell type, the acquired field images can be analyzed for systematic discovery of cell type-specific morphological and biophysical fingerprints encoded in the images.

Over the past two decades, QPI has provided important insights into diverse biological phenomena, ranging from membrane dynamics of red blood cells⁶ to neuronal activity⁷ and cell-nanoparticle interactions,^8,9 and cell-drug interactions.¹⁰ Recently, it has also been shown that QPI images can be mapped to fluorescence images using deep learning techniques, a concept coined as image-to-image translation. Prediction of stains (i.e., where specific fluorophores/stains would bind in an unlabeled specimen) using a combination of QPI and machine learning has been successfully demonstrated,^11–13 and gradually, more stains are being added to the library. Indeed, phase imaging with computational specificity has allowed precise measurements of the growth of nuclei and cytoplasm independently, over many days, without loss of viability.

Central to many of the aforementioned and other emergent applications is QPI’s intrinsic ability to measure single-cell volume and mass non-destructively and ultra-sensitively over arbitrary periods of time in both adherent and flowing cell populations.¹ A critical step in undertaking such analysis is the accurate segmentation of the tomographic images of the cell populations. Since QPI imaging is still a relatively new technique in the field of cell biology, the analysis pipeline is not as developed as it is for fluorescence images. The toolbox developed for fluorescence image segmentation does not work well with QPI images, as the fluorescence contrasts are much sharper than the RI contrast. Also, in some segmentation procedures, the stained nucleus is used as a fiducial marker to define the respective cytoplasm boundary, and consequently, such algorithms cannot be directly implemented in QPI imaging. This has motivated researchers to develop segmentation algorithms tailored for QPI images,¹⁴ yet their applicability has been limited to two-dimensional images thus far.

The state-of-the-art method used for 3D QPI cell segmentation is an Otsu-based 3D watershed algorithm¹⁵ (hereafter referred to as the Otsu threshold algorithm in this work). This algorithm works very well for isolated cell images; however, it is challenging to draw boundaries when the cells are clumped. This process is also memory intensive since the processing requires computation on a 3D stack of images. As a point of reference, using the current state-of-the-art software module, this cell segmentation takes ca. 10 seconds for a 3D stack of images ($484\times 484\times 208$) on a workstation equipped with an 8^th generation i7 processor running at 3.7 GHz, 64 GB RAM, and Nvidia GeForce GTX 1080 - 8 GB graphics card. Another recently developed method for 3D QPI cell segmentation is an AI-based cell segmentation tool.¹⁶ While this should, in principle, be able to overcome the limitations of the Otsu threshold algorithm, such implementation would require an extensive training dataset with enough complexity to handle all the edge cases. The preparation of such training datasets would also require manual annotations, besides needing separate datasets for different cell types. While an AI-trained system can be implemented on a modest system with a GPU, the training for an AI-based segmentation tool would necessitate a state-of-the-art computational resource.

To overcome these drawbacks, we report here a fast and light-weight algorithm for 3D QPI image-based cell segmentation. Our cell segmentation algorithm is inspired by the process of gemstone extraction, where a stonemason starts by making a coarse extrusion cut along a given axis and then gradually carves away near the surface of the gemstone to reveal the final form [Fig. 1(a)]. The algorithm first extrudes a 3D image from a two-dimensional (2D) segmented mask, which can be seen as creating a rough shape of the cell structure. A 2D image is generated through maximum intensity projection (MIP) of the 3D image, and a 2D cell segmentation algorithm is used to make a 2D mask. This allows us to identify the boundary region for segmentation in the $x\text{-}y$ plane. Next, we take advantage of the continuity of cells and the absence of abrupt changes in consecutive $z$-stacks of the 3D image to perform the complete 3D segmentation. This step can be likened to the chiseling process in gemstone extraction, where the stone mason uses finer tools to carve away at the surface. The implication for developing such a method is vast as it can speed up high-throughput single-cell analysis and reduce the computational burden on the hardware. This also lowers the barrier to the adoption of QPI in biological sciences. The growing advancements in hardware development for high-throughput imaging in QPI^17,18 also offer significant utility to this method.

2.

Materials and Methods

2.3.

Design Principle and Workflow

• Step 1: The first stage involves gathering and exporting all unprocessed 3D image stacks from the imaging software. Our code is tailored to operate with a tiff stack file, which contains matrix values represented by RI, saved in uint16 format. Nevertheless, the algorithm is compatible with any file format provided that we are utilizing a scalar multiplication of the RI value as the matrix value.

• Step 2: We produce an MIP image of the 3D tiff image stack along the $z$-axis. MIP is a technique used to visualize 3D data along a visualization axis, where only the voxels with maximum intensity are projected onto the image. Originally developed for use in Nuclear Medicine by Jerold Wallis, it has since been applied to various tomographic imaging modalities, such as CT scans and X-ray imaging.²¹ Most 3D imaging software offers the option to export 2D MIP images. A standard library can also be utilized for this image-processing step.

• Step 3: In this step, we take advantage of already well-built 2D image analysis libraries, which works wonderfully on 2D images for cell segmentation. Specifically, we use CellProfiler¹ to perform 2D cell segmentation on MIP images. The image mask files were saved in a separate folder. While CellProfiler was our tool of choice because of its ease of usage and flexibility, any 2D segmentation tool can be used as long as they have the ability to generate mask images.

• Step 4: The 2D mask generated from CellProfiler is then extruded to 3D, where the number of image stacks for the mask is equal to the number of the $z$-stack planes in the original RI image.

• Step 5: The process begins by selecting the mask for each cell within the field of view individually and setting the matrix value for that mask to 1, and the background and the mask of all other cells are set to 0. The resulting matrix is then multiplied by the original RI image. This process is repeated iteratively for all cells in the given field of view. In addition, all pixels with RI values equal to or lower than the background RI value are set to zero. The resulting matrix, referred to as the rough_segmentation_matrix, shows a rough outline of the cell with noisy pixels around it, as shown in Fig. 1.

• Step 6: In this step, we use the continuity feature of the 3D cell image and implement 2D and 3D connectivity matrices to eliminate noisy pixels around the cell. First, we take the rough_segmentation_matrix and plot the number of non-zero pixels (normalized) against the number of $z$ slices. We set a hard threshold at the $z$-slice where the normalized non-zero-pixel value is 0.5. Instead of hard-coding the $z$-stack number to identify the bottom of the cell, we used a threshold condition because our microscope system has an axial resolution of 300 nm, which is lower than the surface roughness of commonly used petri dishes. Next, for each 2D plane, we label the connected components using the 2D connectivity matrix algorithm, which is achieved using the bwlabel function in MATLAB. We only retain the label corresponding to t highest number of pixels in each 2D slice plane to remove extra noise from cell debris [Fig. 2(a)]. After 2D noise removal, we use the bwlabeln function on the 3D image to eliminate all the noise in the 3D structure, removing all disconnected structures in the process, resulting in perfectly segmented cells [Fig. 2(b)].

• Step 7: The dry mass is calculated using the following equation:

  $$\begin{gathered} \text{drymass}=\frac{1}{\alpha }{\iint }_s\mathrm{OPD}(x,y)\mathrm{d}x\mathrm{d}y,\text{where} \\ \mathrm{OPD}(x,y)={\int }_0^h[n(x,y)-n_{\text{medium}}]\mathrm{d}z\text{and},\alpha =0.19\mu {\mathrm{m}}^3{\mathrm{pg}}^{-1} \end{gathered}$$

Volume is calculated by counting the number of non-zero pixels in the 3D segmentation mask.

Fig. 1

Our method draws inspiration from gemstone extraction from stone, where a stone block resembles a 3D raw image matrix. Coarse segmentation along the $z$-plane corresponds to step 4 in (b), whereas fine segmentation aligns with steps 5 and 6 in (b). (b) Step by step workflow for cell segmentation using QPI images.

Fig. 2

(a) The segmentation process involves utilizing a 2D connectivity matrix to label segments in each 2D slice, followed by the selection of the segment with the highest pixel count. Small segments, depicted in different colors within each slice, are subsequently discarded. (b) Analogously, the 3D segmentation process involves labeling distinct segments using a 3D connectivity matrix and selecting the segment with the highest voxel count.

3.

Results and Discussion

We tested our QPI cell segmentation algorithm against the current gold standard of the Otsu-based 3D watershed algorithm provided by the manufacturer with their microscopy software suite. We also used the recent AI-powered segmentation tool²² for cases where the watershed algorithm did not provide satisfactory segmentation. The AI-powered segmentation tool is currently available for a limited number of cell types, including macrophages. Our comparative study, therefore, is with U937-derived macrophages M1 and M2 cells. Since label-free volume and dry mass measurements are unique to QPI images, these two metrics act as suitable benchmarking metrics to compare the performance of our algorithm. Cell experiments were conducted in sparsely populated cell populations in a petri dish, with a confluency of ∼65% [Fig. 5(a)]. This enabled us to obtain high-quality images featuring a single cell within a given field of view. As a result, we could compare our algorithm to the current gold standard with minimal concern for its accuracy, as the latter is plagued by robustness issues in the presence of multiple cells within a given field of view. We observed a dry mass difference of ∼5% for both M1 and M2 cells [Figs. 5(b) and 5(c)], whereas the difference in volume was slightly higher, with a mean value of 8% for M1 and 6% for M2 cells [Figs. 5(d) and 5(e)]. Since the methods used to perform cell segmentation differed significantly, it is encouraging to note that the mean differences in dry mass and volume were both $<8\%$ [Figs. 5(f) and 5(g)]. In addition, we have included a table (Table 1) that compares the computation time required by each of these algorithms, demonstrating that our algorithm significantly outperforms the current gold standard even on a platform with modest computational hardware.

To put our study in the context of cell segmentation literature, we performed a comparison of our method against the absolute gold standard of manual segmentation. Each 3D cell image in our analysis consisted of a high-resolution stack comprising 208 slices. To establish a ground truth dataset, we manually created annotated masks for 10 cells from M1 ($\sim 2080$ images) and 10 cells from M2 ($\sim 2080$ images). Subsequently, we conducted a segmentation accuracy comparison using our CellSNAP algorithm. To ensure thoroughness, we also assessed performance against Cellpose, a currently popular human-in-the-loop-based deep learning tool for cell segmentation. The segmentation accuracy was determined by calculating the ratio of correctly classified pixels to the total number of pixels inside the manually segmented cells. We notice that CellSNAP has a consistent accuracy of above 90% for most of the cells. One likely explanation for the high accuracy observed is that human annotators base their decision to include a particular area as part of the cell on its continuous presence. In cases where certain regions appear fragmented, they are typically classified as cell debris. In the CellSNAP algorithm, we employ a connectivity matrix to facilitate precise segmentation by capturing the connectivity patterns within the cell. The performance of the Cellpose algorithm varies, succeeding in certain cases while exhibiting significant failures in others. This discrepancy arises from the fact that the network has not been specifically trained on our dataset, and we are utilizing the default weights. Although retraining the Cellpose network tailored to our requirements is certain to enhance its performance, this approach proves time-consuming, as each cell image necessitates manual annotation of 208 image slices. In the Supplementary Material, we have provided a comparison based on other metrics, such as intersection over union (Fig. S1 in the Supplementary Material), Dice coefficient (Fig. S2 in the Supplementary Material), and max Hausdorff distance (Fig. S3 in the Supplementary Material) for the cells shown in Fig. 3.

Fig. 3

Segmentation accuracy comparison against the manual annotated and segmented data: (a) M1 cells and (b) M2 cells for Cellpose and CellSNAP algorithm.

We embarked on developing a cell segmentation algorithm due to the limitations of the current method in segmenting cells when they are clumped together within a field of view. Moreover, the current method’s performance is hampered in suboptimal imaging conditions. Given that QPI imaging is widely used for longitudinal studies, temperature drifts, stage drifts, and calibration drifts are inevitable, rendering the current segmentation algorithms unsuitable for such scenarios, with resulting imaging data being either unusable or requiring extensive manual processing. In this regard, we present two instances where our algorithm’s robustness is evident. In cases where multiple cells are clumped together and share boundaries, the existing Otsu thresholding fails to distinguish between the cells, resulting in a single unit being segmented [Fig. 4(a)]. Although the AI segmentation tool offers some promise, as evidenced by the accurate segmentation of one cell in Fig. 4(a), it fails to accurately segment the remaining cells. By contrast, our segmentation algorithm accurately segments the cells and outperforms existing methods in terms of computational speed. The segmentation accuracy for each image slice can be viewed in Video 1.

Fig. 4

The efficacy of our algorithm in handling challenging scenarios encountered during QPI imaging. (a) Multiple cells in proximity pose a segmentation challenge for conventional methods, such as Otsu threshold and AI-based segmentation. However, our algorithm can accurately segment individual cells even in such a clumped condition. (b) In suboptimal imaging conditions, interferogram noise impedes accurate segmentation by traditional techniques. In contrast, our algorithm is resilient to such noise and can accurately segment cells (Video 1, MP4, 17.6 MB [URL: https://doi.org/10.1117/1.JBO.29.S2.S22706.s1]).

Fig. 5

(a) Cell experiments with sparsely populated cells in the petridish and a sample cell image obtained using the QPI microscope. (b) Comparison of dry mass values for M1 cells. (c) Comparison of dry mass values for M2 cells. (d) Comparison of volume for M1 cells. (e) Comparison of volume for M2 cells. For a statistical comparison between the two algorithms, the difference in the percentage of dry mass and volume was calculated for (f) M1 and (g) M2 cells with the current gold standard and our proposed algorithm.

Table 1

Speed comparison for different methods.

In another example, the image captured has interferogram patterns on the image. This typically arises from calibration drift when performing time-lapse over extended periods. Removing this error with just an Otsu threshold proves challenging, given the similarity between pixel values in the interferogram shadows and the cell. A pattern recognition conditional statement may be used to remove this error along with the Otsu threshold, but we are not aware of any implementation of such an algorithm. While AI segmentation tools can be trained with datasets featuring interferogram artifacts, such implementations are computationally intensive and demanding. Moreover, such errors are prevalent for tools based on trained models when presented with unique cases.²³ Introducing enough variations in the training dataset and retraining the algorithm can overcome this, but this process has limitations, as the machine learning algorithm seldom provides details on the methods.²⁴ Debugging is also difficult if the trained model influences other cases. Nevertheless, CellSNAP offers a possibility of segmenting such cell images as well [Fig. 4(b)].

While promising, CellSNAP’s straightforward approach also brings about certain limitations. One notable limitation is its inability to segment multiple cells along the $z$-axis, as the algorithm assumes the presence of only one cell in the $z$-stack. This restriction renders the algorithm unsuitable for scenarios where multiple cells are clustered along the $z$-axis, a situation commonly found in spheroids and organoids. To the best of our knowledge, the only way to perform 3D segmentation in spheroids or organoids with quantitative phase images is available via deep learning²⁵ where training dataset comprised of paired QPI and fluorescence images. The QPI images were then translated to fluorescence images through a U-Net architecture, which is then used to segment cells, skipping the need for manual segmentation of QPI images. Another limitation arises in cases of high clustering density, where the algorithm struggles to segment cells due to its assumption of a single cell along the extruded MIP volume. In instances of high-density clustering with interleaved cell structures along the $z$-stack, the segmentation becomes challenging. The third scenario where the algorithm faces challenges is when the continuity of dry components of cells is disrupted. Certain cell types, during differentiation or specific functional states, may experience water uptake, resulting in a significant increase in volume. In QPI images, the dark appearance of the water background leads to a discontinuous distribution of dry components, causing CellSNAP to fail in appropriately segmenting cells in such cases.

4.

Conclusion

QPI is a powerful imaging tool that is making major strides and has found numerous applications in basic biological studies and applied clinical research. As QPI utilizes the optical path length as intrinsic contrast, the imaging is noninvasive and, thereby, allows for monitoring live cell samples over several days without concerns of degraded viability. Therefore, significant recent attention has been focused on developing robust analysis pipelines for quantitative phase images, including the application of convolutional neural networks for computationally substituting chemical stains for cells, extracting biomarkers of interest, and enhancing imaging quality. Yet, 3D segmentation of cells, particularly clumped cells, presents a major challenge, as the existing methods work well only for isolated cells. In this work, we have shown that our cell segmentation algorithm for QPI images outperforms the existing gold standard both in terms of speed and robustness. Our algorithm takes about 2 s per cell on a single-core processor to perform the segmentation. This can easily be parallelized on a multi-core system for further improvement in speed. For the cases where segmentation is possible with the existing standard method, our algorithm has a mean error of 5% for dry mass and 8% for volume measurements. Further morphological analysis, such as the determination of surface area, aspect ratio, circularity, and others, can be done by standard function files on the segmented 3D mask images generated by our algorithm. Therefore, this work can lead to wider adoption of QPI imaging for high-throughput analysis, which was earlier stymied by a lack of suitable cell segmentation tools, and lower the barrier to the adoption of QPI imaging modality in biological sciences.