# Cell segmentation

Cell segmentation is a computational image-analysis method that identifies and outlines each individual cell in a microscopy image, producing a per-pixel label mask in which every pixel is assigned to a specific cell instance or to background. The output is an instance-level result, not merely a foreground map, because most downstream analyses require measurements attributed to individual cells rather than to tissue as a whole.<sup>[1](https://www.nature.com/articles/s41592-020-01018-x)</sup> Deep convolutional networks can perform this task on fluorescence images of nuclei, phase-contrast images of cytoplasms, and co-cultured cell types without a fluorescent cytoplasmic marker.<sup>[2](https://journals.plos.org/ploscompbiol/article?id=10.1371%2Fjournal.pcbi.1005177)</sup>

| Key fact | Detail |
|---|---|
| Output | A label image in which each cell is a distinct integer region (instance mask), plus optional per-object probabilities or flow fields<sup>[3](https://github.com/stardist/stardist/)</sup> |
| Dominant architecture | U-Net-style encoder–decoder networks, used in more than 70% of 198 surveyed cell instance segmentation papers from 2020 to 2024<sup>[4](https://link.springer.com/article/10.1007/s00521-025-11119-3)</sup> |
| Landmark result | U-Net reached 92% average IoU on the PhC-U373 phase-contrast dataset versus 83% for the second-best algorithm<sup>[5](https://doi.org/10.48550/arxiv.1505.04597)</sup> |
| Generalist models | Cellpose was trained on over 70,000 segmented objects and segments cells from a wide range of image types without retraining<sup>[1](https://www.nature.com/articles/s41592-020-01018-x)</sup> |
| Benchmark scale | The NeurIPS 2022 multimodality CellSeg benchmark spans more than 1,500 labeled images from more than 50 biological experiments across brightfield, fluorescence, phase-contrast, and DIC modalities<sup>[6](https://www.nature.com/articles/s41592-024-02233-6)</sup> |
| Current frontier | Foundation-model adaptations such as Cellpose-SAM substantially outperform inter-human agreement on cellular segmentation<sup>[7](https://www.biorxiv.org/content/10.1101/2025.04.28.651001v1)</sup> |

## How it works

Three related tasks are distinguished. [Semantic segmentation](https://www.edgepedia.org/semantic-segmentation) assigns every pixel a class (cell versus background) without separating individual cells; instance segmentation additionally gives each cell a unique label; cell detection localizes cells, for example as points or boxes, without full outlines. The distinction matters in practice: a pipeline that predicted a binary mask per cell produced boundary artifacts that discarded roughly 10% of cell area when cells touched, whereas uniquely labeled instance masks allow adjacent masks to be in contact.<sup>[8](https://journals.plos.org/ploscompbiol/article?id=10.1371%2Fjournal.pcbi.1013088)</sup>

Deep-learning methods represent cells in different ways. U-Net-style networks use a contracting path that captures context through repeated convolutions and max pooling, together with a symmetric expanding path whose skip connections combine high-resolution encoder features with upsampled output.<sup>[5](https://doi.org/10.48550/arxiv.1505.04597)</sup> StarDist represents each object as a star-convex polygon, regressing per pixel the distances to the object boundary along 32 radial directions with equidistant angles, together with an object probability, and resolving candidates by non-maximum suppression.<sup>[9](http://uweschmidt.org/research/pubs/miccai18schmidt.pdf)</sup> Cellpose encodes cells by vector fields pointing inward from the boundary, which are followed to instance centers.<sup>[10](https://openaccess.thecvf.com/content/ICCV2023/papers/Wolf_Unsupervised_Learning_of_Object-Centric_Embeddings_for_Cell_Instance_Segmentation_in_ICCV_2023_paper.pdf)</sup> Embedding-based methods such as Cellulus learn object-centric pixel embeddings and group them by mean-shift clustering.<sup>[10](https://openaccess.thecvf.com/content/ICCV2023/papers/Wolf_Unsupervised_Learning_of_Object-Centric_Embeddings_for_Cell_Instance_Segmentation_in_ICCV_2023_paper.pdf)</sup>

## How it is done

A typical workflow has five steps. First, images are prepared, sometimes with restoration; Cellpose3 added one-click image restoration to improve segmentation of degraded images.<sup>[11](https://doi.org/10.1038/s41592-025-02595-5)</sup> Second, a tool is chosen: pretrained generalist models, or a model trained on the user's data. Third, annotation: training a custom model requires label images in which every pixel of a cell region carries a distinct integer and background is 0.<sup>[3](https://github.com/stardist/stardist/)</sup> [Annotation](https://www.edgepedia.org/annotation) is the main labor cost, because delineating cell boundaries requires trained annotators.<sup>[4](https://link.springer.com/article/10.1007/s00521-025-11119-3)</sup> Fourth, training or zero-shot inference. Fifth, post-processing: label images are filtered, for example by object pixel count, and converted to region-of-interest objects for measurement.<sup>[12](https://py.imagej.net/en/stable/Cellpose-StarDist-Segmentation.html)</sup>

## Origin

The classical lineage includes thresholding, watershed, and Voronoi algorithms implemented in tools such as CellProfiler and Oufti, and supervised machine-learning tools such as Ilastik, which combine edge and texture filters with random-forest classification.<sup>[2](https://journals.plos.org/ploscompbiol/article?id=10.1371%2Fjournal.pcbi.1005177)</sup> Watershed-based segmentation and level-set methods were used to separate touching and overlapping nuclei, alongside active-contour and snake algorithms.<sup>[13](https://bmcbioinformatics.biomedcentral.com/articles/10.1186/s12859-018-2375-z)</sup>

The deep-learning era began with the fully convolutional network for semantic segmentation by Evan Shelhamer, Jonathan Long, and [Trevor Darrell](https://www.edgepedia.org/trevor-darrell) (2016, [IEEE Transactions on Pattern Analysis and Machine Intelligence](https://www.edgepedia.org/ieee-transactions-on-pattern-analysis-and-machine-intelligence)).<sup>[14](https://doi.org/10.1109/tpami.2016.2572683)</sup> U-Net, by Olaf Ronneberger, Philipp Fischer, and Thomas Brox (2015, arXiv), built on that design and won the ISBI cell tracking challenge 2015 on transmitted light microscopy by a large margin.<sup>[5](https://doi.org/10.48550/arxiv.1505.04597)</sup> StarDist was proposed by Uwe Schmidt and colleagues (2018, arXiv).<sup>[9](http://uweschmidt.org/research/pubs/miccai18schmidt.pdf)</sup> Cellpose was introduced by [Carsen Stringer](https://www.edgepedia.org/carsen-stringer) and colleagues (2020, Nature Methods).<sup>[1](https://www.nature.com/articles/s41592-020-01018-x)</sup>

## Variants

**StarDist family.** The 3D extension represents objects as star-convex polyhedra ([Martin Weigert](https://www.edgepedia.org/martin-weigert) and colleagues, 2019, arXiv).<sup>[15](https://doi.org/10.48550/arxiv.1908.03636)</sup> A histopathology version adds a classification head so each segmented nucleus is assigned a cell type (Martin Weigert and Uwe Schmidt, 2022, arXiv).<sup>[16](https://doi.org/10.48550/arxiv.2203.02284)</sup>

**Cellpose family.** Omnipose, by Kevin J. Cutler and colleagues, targets morphology-independent bacterial cell segmentation (2022, [Nature Methods](https://www.edgepedia.org/nature-methods)).<sup>[17](https://doi.org/10.1038/s41592-022-01639-4)</sup> Cellpose-SAM, by [Marius Pachitariu](https://www.edgepedia.org/marius-pachitariu), Michael Rariden, and Carsen Stringer (2025, bioRxiv), adapts the pretrained SAM transformer backbone to the Cellpose framework and is robust to channel shuffling, cell size, shot noise, downsampling, and blur.<sup>[7](https://www.biorxiv.org/content/10.1101/2025.04.28.651001v1)</sup>

**Other tools.** InstanSeg uses a modified Cellpose U-Net backbone of about four million parameters, roughly half the original Cellpose model, and clusters pixel embeddings around selected seed pixels (Thibaut Goldsborough and colleagues, 2024, arXiv).<sup>[18](https://doi.org/10.48550/arxiv.2408.15954)</sup> Mesmer performs whole-cell segmentation of tissue images trained on large-scale annotation (Noah F. Greenwald and colleagues, 2021, [Nature Biotechnology](https://www.edgepedia.org/nature-biotechnology)).<sup>[19](https://doi.org/10.1038/s41587-021-01094-0)</sup> nnU-Net self-configures preprocessing, architecture, and training for a given biomedical dataset (Fabian Isensee and colleagues, 2020, Nature Methods).<sup>[20](https://doi.org/10.1038/s41592-020-01008-z)</sup> Mask R-CNN provides a bounding-box-based instance segmentation baseline ([Kaiming He](https://www.edgepedia.org/kaiming-he) and colleagues, 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence).<sup>[21](https://doi.org/10.1109/tpami.2018.2844175)</sup>

## Applications

Cell segmentation supports live-cell imaging analysis, bacterial cytometry, histopathology nuclei classification, and tissue imaging. Input modalities include fluorescence, brightfield, phase contrast, and DIC; the NeurIPS 2022 benchmark deliberately spans all four.<sup>[6](https://www.nature.com/articles/s41592-024-02233-6)</sup> 3D stacks are supported: Cellpose's 3D extension reuses the 2D model without 3D-labeled data,<sup>[1](https://www.nature.com/articles/s41592-020-01018-x)</sup> and 3D U-Net learns dense volumetric segmentation from sparse annotation (Özgün Çiçek and colleagues, 2016, arXiv).<sup>[22](https://doi.org/10.48550/arxiv.1606.06650)</sup> Time-lapse bacterial pipelines such as OmniSegger convert raw images into dynamical cytometry data across phase-contrast, brightfield, and fluorescence modalities.<sup>[8](https://journals.plos.org/ploscompbiol/article?id=10.1371%2Fjournal.pcbi.1013088)</sup>

**Metrics.** A detected object counts as a true positive when its intersection over union (IoU) with a ground-truth object exceeds a threshold τ; under the DSB2018 convention adopted by StarDist, the score is then \( AP_{\tau} = TP_{\tau}/(TP_{\tau}+FN_{\tau}+FP_{\tau}) \), which differs from the standard average precision computed as the area under the precision–recall curve as the confidence threshold varies.<sup>[9](http://uweschmidt.org/research/pubs/miccai18schmidt.pdf)</sup> Panoptic quality is the product of detection quality (F1) and segmentation quality (average IoU of correct matches).<sup>[23](https://warwick.ac.uk/fac/cross_fac/tia/data/conic/stardist.pdf)</sup> Reported results include U-Net's 92% IoU on PhC-U373 and 77.5% on DIC-HeLa versus 46% for the runner-up,<sup>[5](https://doi.org/10.48550/arxiv.1505.04597)</sup> and StarDist outperforming all methods including [Mask R-CNN](https://www.edgepedia.org/mask-r-cnn) on DSB2018 for IoU thresholds below 0.75 with about 1.4 million parameters versus Mask R-CNN's roughly 45 million.<sup>[9](http://uweschmidt.org/research/pubs/miccai18schmidt.pdf)</sup>

**Foundation models.** The field has moved toward foundation models, and virtually all cell-segmentation foundation models are extensions of the Segment Anything Model adapted to microscopy; Anwai Archit and Constantin Pape evaluated CellPoseSAM, CellSAM, and μSAM alongside SAM, SAM2, and SAM3 on diverse light-microscopy datasets, and introduced automatic prompt generation, which consistently improves μSAM results.<sup>[24](https://proceedings.mlr.press/v315/archit26a.html)</sup> Large multimodal benchmarks matured: the NeurIPS 2022 CellSeg challenge, published in Nature Methods in March 2024, framed segmentation as a weakly supervised task with limited labeled patches and many unlabeled images, including a whole-slide test image of roughly 10,000 × 10,000 pixels, and was won by MEDIAR, a Transformer-based algorithm applicable across platforms without manual parameter adjustment.<sup>[6](https://www.nature.com/articles/s41592-024-02233-6)</sup> In spatial transcriptomics, Halo reconstructs whole-cell boundaries from DAPI-only staining by converting RNA transcript coordinates into molecular density maps processed with a Cellpose-SAM architecture.<sup>[25](https://academic.oup.com/bib/article/27/5/bbag515/8812206)</sup>

## Limitations and alternatives

**Cross-modality generalization is the clearest failure mode.** In a systematic evaluation of 18 methods, all methods failed when trained on cell nuclei images and applied to whole-cell images, and most lost performance in the reverse direction; only Cascade Mask R-CNN seesaw and RF-Next maintained nearly identical performance across modalities.<sup>[26](https://pmc.ncbi.nlm.nih.gov/articles/PMC10862744/)</sup>

**Touching cells and boundaries.** All models evaluated on CellBinDB obtained lower panoptic quality than F1, indicating difficulty separating closely adjacent instances; grayscale conversion and color inversion substantially improved H&E segmentation for fluorescence-designed models.<sup>[27](https://pmc.ncbi.nlm.nih.gov/articles/PMC12206155/)</sup> U-Net addressed touching cells with a weighted loss giving large weight to separating background labels.<sup>[5](https://doi.org/10.48550/arxiv.1505.04597)</sup>

**Which method class wins depends on the benchmark.** One systematic evaluation found that general-purpose attention-based methods (Centermask2, Swin, ResNeSt) performed best overall, with Centermask2 improving whole-cell accuracy over Cellpose by 96.9% on LIVECell and 76.5% on TissueNet, while Cellpose was the best biomedical-specific method;<sup>[26](https://pmc.ncbi.nlm.nih.gov/articles/PMC10862744/)</sup> on CellBinDB, however, Cellpose3 was the most highly recommended of the evaluated models.<sup>[27](https://pmc.ncbi.nlm.nih.gov/articles/PMC12206155/)</sup> Manual segmentation remains labor-intensive, susceptible to human bias, and dependent on annotator proficiency.<sup>[28](https://www.mdpi.com/2313-433X/10/12/311)</sup>

## References

1. [Cellpose: a generalist algorithm for cellular segmentation (Nature Methods)](https://www.nature.com/articles/s41592-020-01018-x)
2. [Deep Learning Automates the Quantitative Analysis of Individual Cells in Live-Cell Imaging Experiments (Van Valen et al., PLOS Computational Biology)](https://journals.plos.org/ploscompbiol/article?id=10.1371%2Fjournal.pcbi.1005177)
3. [stardist/stardist, StarDist official repository and documentation](https://github.com/stardist/stardist/)
4. [A survey of deep learning methods on cell instance segmentation (Neural Computing and Applications, 2025)](https://link.springer.com/article/10.1007/s00521-025-11119-3)
5. [Ronneberger, Olaf, Fischer, Philipp, Brox, Thomas (2015). U-Net: Convolutional Networks for Biomedical Image Segmentation. arXiv (Cornell University).](https://doi.org/10.48550/arxiv.1505.04597)
6. [The multimodality cell segmentation challenge: toward universal solutions (Nature Methods 2024)](https://www.nature.com/articles/s41592-024-02233-6)
7. [Cellpose-SAM: superhuman generalization for cellular segmentation (bioRxiv 2025)](https://www.biorxiv.org/content/10.1101/2025.04.28.651001v1)
8. [OmniSegger: A time-lapse image analysis pipeline for bacterial cells (PLOS Computational Biology)](https://journals.plos.org/ploscompbiol/article?id=10.1371%2Fjournal.pcbi.1013088)
9. [Cell Detection with Star-convex Polygons (StarDist, MICCAI 2018, author-hosted PDF)](http://uweschmidt.org/research/pubs/miccai18schmidt.pdf)
10. [Unsupervised Learning of Object-Centric Embeddings for Cell Instance Segmentation in Microscopy Images (Cellulus, ICCV 2023)](https://openaccess.thecvf.com/content/ICCV2023/papers/Wolf_Unsupervised_Learning_of_Object-Centric_Embeddings_for_Cell_Instance_Segmentation_in_ICCV_2023_paper.pdf)
11. [Carsen Stringer, Marius Pachitariu (2025). Cellpose3: one-click image restoration for improved cellular segmentation. Nature Methods.](https://doi.org/10.1038/s41592-025-02595-5)
12. [Cellpose/StarDist Segmentation (PyImageJ notebook)](https://py.imagej.net/en/stable/Cellpose-StarDist-Segmentation.html)
13. [A deep learning-based algorithm for 2-D cell segmentation in microscopy images (Al-Kofahi et al., BMC Bioinformatics 2018)](https://bmcbioinformatics.biomedcentral.com/articles/10.1186/s12859-018-2375-z)
14. [Evan Shelhamer, Jonathan Long, Trevor Darrell (2016). Fully Convolutional Networks for Semantic Segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence.](https://doi.org/10.1109/tpami.2016.2572683)
15. [Weigert, Martin and colleagues (2019). Star-convex Polyhedra for 3D Object Detection and Segmentation in Microscopy. arXiv (Cornell University).](https://doi.org/10.48550/arxiv.1908.03636)
16. [Weigert, Martin, Schmidt, Uwe (2022). Nuclei instance segmentation and classification in histopathology images with StarDist. arXiv (Cornell University).](https://doi.org/10.48550/arxiv.2203.02284)
17. [Kevin J. Cutler and colleagues (2022). Omnipose: a high-precision morphology-independent solution for bacterial cell segmentation. Nature Methods.](https://doi.org/10.1038/s41592-022-01639-4)
18. [Goldsborough, Thibaut and colleagues (2024). InstanSeg: an embedding-based instance segmentation algorithm optimized for accurate, efficient and portable cell segmentation. arXiv (Cornell University).](https://doi.org/10.48550/arxiv.2408.15954)
19. [Noah F. Greenwald and colleagues (2021). Whole-cell segmentation of tissue images with human-level performance using large-scale data annotation and deep learning. Nature Biotechnology.](https://doi.org/10.1038/s41587-021-01094-0)
20. [Fabian Isensee and colleagues (2020). nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature Methods.](https://doi.org/10.1038/s41592-020-01008-z)
21. [Kaiming He and colleagues (2018). Mask R-CNN. IEEE Transactions on Pattern Analysis and Machine Intelligence.](https://doi.org/10.1109/tpami.2018.2844175)
22. [Çiçek, Özgün and colleagues (2016). 3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation. arXiv (Cornell University).](https://doi.org/10.48550/arxiv.1606.06650)
23. [Nuclei Instance Segmentation and Classification in Histopathology Images with StarDist (CoNIC challenge, ISBIC 2022)](https://warwick.ac.uk/fac/cross_fac/tia/data/conic/stardist.pdf)
24. [Revisiting foundation models for cell instance segmentation (Archit & Pape, MIDL/PMLR v315)](https://proceedings.mlr.press/v315/archit26a.html)
25. [Halo: a pretrained model for whole-cell segmentation from nuclei images in spatial transcriptomics (Briefings in Bioinformatics)](https://academic.oup.com/bib/article/27/5/bbag515/8812206)
26. [A systematic evaluation of computational methods for cell segmentation (Briefings in Bioinformatics)](https://pmc.ncbi.nlm.nih.gov/articles/PMC10862744/)
27. [CellBinDB: a large-scale multimodal annotated dataset for cell segmentation with benchmarking of universal models](https://pmc.ncbi.nlm.nih.gov/articles/PMC12206155/)
28. [State-of-the-Art Deep Learning Methods for Microscopic Image Segmentation (Journal of Imaging)](https://www.mdpi.com/2313-433X/10/12/311)

---
*Topic: Encyclopedia › Life and health › Biological foundations*

*Initially written Sep 29, 2026 · Edited: Oct 15, 2026*

*Copyright 2026 EdgeChat AI, a subsidiary of Biostate AI.*

License: Edgepedia Community License 1.0, https://www.edgepedia.org/edgepedia/license