{ "id": "ep4xctda7f", "slug": "cell-segmentation", "title": "Cell segmentation", "updated": "2026-10-15", "topic_path": [ { "id": "life", "label": "Life and health", "api_url": "https://www.edgepedia.org/api/v1/topics/life" }, { "id": "life.biology", "label": "Biological foundations", "api_url": "https://www.edgepedia.org/api/v1/topics/life.biology" } ], "geo": [ { "id": "geo.nongeo.t1946.life.biology", "label": "Non-geographic · 1946 to 2000: Biological foundations", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1946.life.biology", "path": [ { "id": "geo.nongeo", "label": "Non-geographic", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo" }, { "id": "geo.nongeo.t1946", "label": "Non-geographic · 1946 to 2000", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1946" }, { "id": "geo.nongeo.t1946.life", "label": "Life and health", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1946.life" }, { "id": "geo.nongeo.t1946.life.biology", "label": "Biological foundations", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1946.life.biology" } ] }, { "id": "geo.nongeo.t2001.life.biology", "label": "Non-geographic · 2001 to 2020: Biological foundations", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.life.biology", "path": [ { "id": "geo.nongeo", "label": "Non-geographic", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo" }, { "id": "geo.nongeo.t2001", "label": "Non-geographic · 2001 to 2020", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001" }, { "id": "geo.nongeo.t2001.life", "label": "Life and health", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.life" }, { "id": "geo.nongeo.t2001.life.biology", "label": "Biological foundations", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.life.biology" } ] } ], "excerpt": "Cell segmentation is a computational image-analysis method that identifies and outlines each individual cell in a microscopy image, assigning every pixel to a specific cell or background.", "snippet": "Cell segmentation is a computational image-analysis method that identifies and outlines each individual cell in a microscopy image, assigning every pixel to a specific cell or background.", "node": "life.biology", "markdown": "# Cell segmentation\n\nCell segmentation is a computational image-analysis method that identifies and outlines each individual cell in a microscopy image, producing a per-pixel label mask in which every pixel is assigned to a specific cell instance or to background. The output is an instance-level result, not merely a foreground map, because most downstream analyses require measurements attributed to individual cells rather than to tissue as a whole.[1](https://www.nature.com/articles/s41592-020-01018-x) Deep convolutional networks can perform this task on fluorescence images of nuclei, phase-contrast images of cytoplasms, and co-cultured cell types without a fluorescent cytoplasmic marker.[2](https://journals.plos.org/ploscompbiol/article?id=10.1371%2Fjournal.pcbi.1005177)\n\n| Key fact | Detail |\n|---|---|\n| Output | A label image in which each cell is a distinct integer region (instance mask), plus optional per-object probabilities or flow fields[3](https://github.com/stardist/stardist/) |\n| Dominant architecture | U-Net-style encoder–decoder networks, used in more than 70% of 198 surveyed cell instance segmentation papers from 2020 to 2024[4](https://link.springer.com/article/10.1007/s00521-025-11119-3) |\n| Landmark result | U-Net reached 92% average IoU on the PhC-U373 phase-contrast dataset versus 83% for the second-best algorithm[5](https://doi.org/10.48550/arxiv.1505.04597) |\n| Generalist models | Cellpose was trained on over 70,000 segmented objects and segments cells from a wide range of image types without retraining[1](https://www.nature.com/articles/s41592-020-01018-x) |\n| Benchmark scale | The NeurIPS 2022 multimodality CellSeg benchmark spans more than 1,500 labeled images from more than 50 biological experiments across brightfield, fluorescence, phase-contrast, and DIC modalities[6](https://www.nature.com/articles/s41592-024-02233-6) |\n| Current frontier | Foundation-model adaptations such as Cellpose-SAM substantially outperform inter-human agreement on cellular segmentation[7](https://www.biorxiv.org/content/10.1101/2025.04.28.651001v1) |\n\n## How it works\n\nThree related tasks are distinguished. [Semantic segmentation](https://www.edgepedia.org/semantic-segmentation) assigns every pixel a class (cell versus background) without separating individual cells; instance segmentation additionally gives each cell a unique label; cell detection localizes cells, for example as points or boxes, without full outlines. The distinction matters in practice: a pipeline that predicted a binary mask per cell produced boundary artifacts that discarded roughly 10% of cell area when cells touched, whereas uniquely labeled instance masks allow adjacent masks to be in contact.[8](https://journals.plos.org/ploscompbiol/article?id=10.1371%2Fjournal.pcbi.1013088)\n\nDeep-learning methods represent cells in different ways. U-Net-style networks use a contracting path that captures context through repeated convolutions and max pooling, together with a symmetric expanding path whose skip connections combine high-resolution encoder features with upsampled output.[5](https://doi.org/10.48550/arxiv.1505.04597) StarDist represents each object as a star-convex polygon, regressing per pixel the distances to the object boundary along 32 radial directions with equidistant angles, together with an object probability, and resolving candidates by non-maximum suppression.[9](http://uweschmidt.org/research/pubs/miccai18schmidt.pdf) Cellpose encodes cells by vector fields pointing inward from the boundary, which are followed to instance centers.[10](https://openaccess.thecvf.com/content/ICCV2023/papers/Wolf_Unsupervised_Learning_of_Object-Centric_Embeddings_for_Cell_Instance_Segmentation_in_ICCV_2023_paper.pdf) Embedding-based methods such as Cellulus learn object-centric pixel embeddings and group them by mean-shift clustering.[10](https://openaccess.thecvf.com/content/ICCV2023/papers/Wolf_Unsupervised_Learning_of_Object-Centric_Embeddings_for_Cell_Instance_Segmentation_in_ICCV_2023_paper.pdf)\n\n## How it is done\n\nA typical workflow has five steps. First, images are prepared, sometimes with restoration; Cellpose3 added one-click image restoration to improve segmentation of degraded images.[11](https://doi.org/10.1038/s41592-025-02595-5) Second, a tool is chosen: pretrained generalist models, or a model trained on the user's data. Third, annotation: training a custom model requires label images in which every pixel of a cell region carries a distinct integer and background is 0.[3](https://github.com/stardist/stardist/) [Annotation](https://www.edgepedia.org/annotation) is the main labor cost, because delineating cell boundaries requires trained annotators.[4](https://link.springer.com/article/10.1007/s00521-025-11119-3) Fourth, training or zero-shot inference. Fifth, post-processing: label images are filtered, for example by object pixel count, and converted to region-of-interest objects for measurement.[12](https://py.imagej.net/en/stable/Cellpose-StarDist-Segmentation.html)\n\n## Origin\n\nThe classical lineage includes thresholding, watershed, and Voronoi algorithms implemented in tools such as CellProfiler and Oufti, and supervised machine-learning tools such as Ilastik, which combine edge and texture filters with random-forest classification.[2](https://journals.plos.org/ploscompbiol/article?id=10.1371%2Fjournal.pcbi.1005177) Watershed-based segmentation and level-set methods were used to separate touching and overlapping nuclei, alongside active-contour and snake algorithms.[13](https://bmcbioinformatics.biomedcentral.com/articles/10.1186/s12859-018-2375-z)\n\nThe deep-learning era began with the fully convolutional network for semantic segmentation by Evan Shelhamer, Jonathan Long, and [Trevor Darrell](https://www.edgepedia.org/trevor-darrell) (2016, [IEEE Transactions on Pattern Analysis and Machine Intelligence](https://www.edgepedia.org/ieee-transactions-on-pattern-analysis-and-machine-intelligence)).[14](https://doi.org/10.1109/tpami.2016.2572683) U-Net, by Olaf Ronneberger, Philipp Fischer, and Thomas Brox (2015, arXiv), built on that design and won the ISBI cell tracking challenge 2015 on transmitted light microscopy by a large margin.[5](https://doi.org/10.48550/arxiv.1505.04597) StarDist was proposed by Uwe Schmidt and colleagues (2018, arXiv).[9](http://uweschmidt.org/research/pubs/miccai18schmidt.pdf) Cellpose was introduced by [Carsen Stringer](https://www.edgepedia.org/carsen-stringer) and colleagues (2020, Nature Methods).[1](https://www.nature.com/articles/s41592-020-01018-x)\n\n## Variants\n\n**StarDist family.** The 3D extension represents objects as star-convex polyhedra ([Martin Weigert](https://www.edgepedia.org/martin-weigert) and colleagues, 2019, arXiv).[15](https://doi.org/10.48550/arxiv.1908.03636) A histopathology version adds a classification head so each segmented nucleus is assigned a cell type (Martin Weigert and Uwe Schmidt, 2022, arXiv).[16](https://doi.org/10.48550/arxiv.2203.02284)\n\n**Cellpose family.** Omnipose, by Kevin J. Cutler and colleagues, targets morphology-independent bacterial cell segmentation (2022, [Nature Methods](https://www.edgepedia.org/nature-methods)).[17](https://doi.org/10.1038/s41592-022-01639-4) Cellpose-SAM, by [Marius Pachitariu](https://www.edgepedia.org/marius-pachitariu), Michael Rariden, and Carsen Stringer (2025, bioRxiv), adapts the pretrained SAM transformer backbone to the Cellpose framework and is robust to channel shuffling, cell size, shot noise, downsampling, and blur.[7](https://www.biorxiv.org/content/10.1101/2025.04.28.651001v1)\n\n**Other tools.** InstanSeg uses a modified Cellpose U-Net backbone of about four million parameters, roughly half the original Cellpose model, and clusters pixel embeddings around selected seed pixels (Thibaut Goldsborough and colleagues, 2024, arXiv).[18](https://doi.org/10.48550/arxiv.2408.15954) Mesmer performs whole-cell segmentation of tissue images trained on large-scale annotation (Noah F. Greenwald and colleagues, 2021, [Nature Biotechnology](https://www.edgepedia.org/nature-biotechnology)).[19](https://doi.org/10.1038/s41587-021-01094-0) nnU-Net self-configures preprocessing, architecture, and training for a given biomedical dataset (Fabian Isensee and colleagues, 2020, Nature Methods).[20](https://doi.org/10.1038/s41592-020-01008-z) Mask R-CNN provides a bounding-box-based instance segmentation baseline ([Kaiming He](https://www.edgepedia.org/kaiming-he) and colleagues, 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence).[21](https://doi.org/10.1109/tpami.2018.2844175)\n\n## Applications\n\nCell segmentation supports live-cell imaging analysis, bacterial cytometry, histopathology nuclei classification, and tissue imaging. Input modalities include fluorescence, brightfield, phase contrast, and DIC; the NeurIPS 2022 benchmark deliberately spans all four.[6](https://www.nature.com/articles/s41592-024-02233-6) 3D stacks are supported: Cellpose's 3D extension reuses the 2D model without 3D-labeled data,[1](https://www.nature.com/articles/s41592-020-01018-x) and 3D U-Net learns dense volumetric segmentation from sparse annotation (Özgün Çiçek and colleagues, 2016, arXiv).[22](https://doi.org/10.48550/arxiv.1606.06650) Time-lapse bacterial pipelines such as OmniSegger convert raw images into dynamical cytometry data across phase-contrast, brightfield, and fluorescence modalities.[8](https://journals.plos.org/ploscompbiol/article?id=10.1371%2Fjournal.pcbi.1013088)\n\n**Metrics.** A detected object counts as a true positive when its intersection over union (IoU) with a ground-truth object exceeds a threshold τ; under the DSB2018 convention adopted by StarDist, the score is then \\( AP_{\\tau} = TP_{\\tau}/(TP_{\\tau}+FN_{\\tau}+FP_{\\tau}) \\), which differs from the standard average precision computed as the area under the precision–recall curve as the confidence threshold varies.[9](http://uweschmidt.org/research/pubs/miccai18schmidt.pdf) Panoptic quality is the product of detection quality (F1) and segmentation quality (average IoU of correct matches).[23](https://warwick.ac.uk/fac/cross_fac/tia/data/conic/stardist.pdf) Reported results include U-Net's 92% IoU on PhC-U373 and 77.5% on DIC-HeLa versus 46% for the runner-up,[5](https://doi.org/10.48550/arxiv.1505.04597) and StarDist outperforming all methods including [Mask R-CNN](https://www.edgepedia.org/mask-r-cnn) on DSB2018 for IoU thresholds below 0.75 with about 1.4 million parameters versus Mask R-CNN's roughly 45 million.[9](http://uweschmidt.org/research/pubs/miccai18schmidt.pdf)\n\n**Foundation models.** The field has moved toward foundation models, and virtually all cell-segmentation foundation models are extensions of the Segment Anything Model adapted to microscopy; Anwai Archit and Constantin Pape evaluated CellPoseSAM, CellSAM, and μSAM alongside SAM, SAM2, and SAM3 on diverse light-microscopy datasets, and introduced automatic prompt generation, which consistently improves μSAM results.[24](https://proceedings.mlr.press/v315/archit26a.html) Large multimodal benchmarks matured: the NeurIPS 2022 CellSeg challenge, published in Nature Methods in March 2024, framed segmentation as a weakly supervised task with limited labeled patches and many unlabeled images, including a whole-slide test image of roughly 10,000 × 10,000 pixels, and was won by MEDIAR, a Transformer-based algorithm applicable across platforms without manual parameter adjustment.[6](https://www.nature.com/articles/s41592-024-02233-6) In spatial transcriptomics, Halo reconstructs whole-cell boundaries from DAPI-only staining by converting RNA transcript coordinates into molecular density maps processed with a Cellpose-SAM architecture.[25](https://academic.oup.com/bib/article/27/5/bbag515/8812206)\n\n## Limitations and alternatives\n\n**Cross-modality generalization is the clearest failure mode.** In a systematic evaluation of 18 methods, all methods failed when trained on cell nuclei images and applied to whole-cell images, and most lost performance in the reverse direction; only Cascade Mask R-CNN seesaw and RF-Next maintained nearly identical performance across modalities.[26](https://pmc.ncbi.nlm.nih.gov/articles/PMC10862744/)\n\n**Touching cells and boundaries.** All models evaluated on CellBinDB obtained lower panoptic quality than F1, indicating difficulty separating closely adjacent instances; grayscale conversion and color inversion substantially improved H&E segmentation for fluorescence-designed models.[27](https://pmc.ncbi.nlm.nih.gov/articles/PMC12206155/) U-Net addressed touching cells with a weighted loss giving large weight to separating background labels.[5](https://doi.org/10.48550/arxiv.1505.04597)\n\n**Which method class wins depends on the benchmark.** One systematic evaluation found that general-purpose attention-based methods (Centermask2, Swin, ResNeSt) performed best overall, with Centermask2 improving whole-cell accuracy over Cellpose by 96.9% on LIVECell and 76.5% on TissueNet, while Cellpose was the best biomedical-specific method;[26](https://pmc.ncbi.nlm.nih.gov/articles/PMC10862744/) on CellBinDB, however, Cellpose3 was the most highly recommended of the evaluated models.[27](https://pmc.ncbi.nlm.nih.gov/articles/PMC12206155/) Manual segmentation remains labor-intensive, susceptible to human bias, and dependent on annotator proficiency.[28](https://www.mdpi.com/2313-433X/10/12/311)\n\n## References\n\n1. [Cellpose: a generalist algorithm for cellular segmentation (Nature Methods)](https://www.nature.com/articles/s41592-020-01018-x)\n2. [Deep Learning Automates the Quantitative Analysis of Individual Cells in Live-Cell Imaging Experiments (Van Valen et al., PLOS Computational Biology)](https://journals.plos.org/ploscompbiol/article?id=10.1371%2Fjournal.pcbi.1005177)\n3. [stardist/stardist, StarDist official repository and documentation](https://github.com/stardist/stardist/)\n4. [A survey of deep learning methods on cell instance segmentation (Neural Computing and Applications, 2025)](https://link.springer.com/article/10.1007/s00521-025-11119-3)\n5. [Ronneberger, Olaf, Fischer, Philipp, Brox, Thomas (2015). U-Net: Convolutional Networks for Biomedical Image Segmentation. arXiv (Cornell University).](https://doi.org/10.48550/arxiv.1505.04597)\n6. [The multimodality cell segmentation challenge: toward universal solutions (Nature Methods 2024)](https://www.nature.com/articles/s41592-024-02233-6)\n7. [Cellpose-SAM: superhuman generalization for cellular segmentation (bioRxiv 2025)](https://www.biorxiv.org/content/10.1101/2025.04.28.651001v1)\n8. [OmniSegger: A time-lapse image analysis pipeline for bacterial cells (PLOS Computational Biology)](https://journals.plos.org/ploscompbiol/article?id=10.1371%2Fjournal.pcbi.1013088)\n9. [Cell Detection with Star-convex Polygons (StarDist, MICCAI 2018, author-hosted PDF)](http://uweschmidt.org/research/pubs/miccai18schmidt.pdf)\n10. [Unsupervised Learning of Object-Centric Embeddings for Cell Instance Segmentation in Microscopy Images (Cellulus, ICCV 2023)](https://openaccess.thecvf.com/content/ICCV2023/papers/Wolf_Unsupervised_Learning_of_Object-Centric_Embeddings_for_Cell_Instance_Segmentation_in_ICCV_2023_paper.pdf)\n11. [Carsen Stringer, Marius Pachitariu (2025). Cellpose3: one-click image restoration for improved cellular segmentation. Nature Methods.](https://doi.org/10.1038/s41592-025-02595-5)\n12. [Cellpose/StarDist Segmentation (PyImageJ notebook)](https://py.imagej.net/en/stable/Cellpose-StarDist-Segmentation.html)\n13. [A deep learning-based algorithm for 2-D cell segmentation in microscopy images (Al-Kofahi et al., BMC Bioinformatics 2018)](https://bmcbioinformatics.biomedcentral.com/articles/10.1186/s12859-018-2375-z)\n14. [Evan Shelhamer, Jonathan Long, Trevor Darrell (2016). Fully Convolutional Networks for Semantic Segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence.](https://doi.org/10.1109/tpami.2016.2572683)\n15. [Weigert, Martin and colleagues (2019). Star-convex Polyhedra for 3D Object Detection and Segmentation in Microscopy. arXiv (Cornell University).](https://doi.org/10.48550/arxiv.1908.03636)\n16. [Weigert, Martin, Schmidt, Uwe (2022). Nuclei instance segmentation and classification in histopathology images with StarDist. arXiv (Cornell University).](https://doi.org/10.48550/arxiv.2203.02284)\n17. [Kevin J. Cutler and colleagues (2022). Omnipose: a high-precision morphology-independent solution for bacterial cell segmentation. Nature Methods.](https://doi.org/10.1038/s41592-022-01639-4)\n18. [Goldsborough, Thibaut and colleagues (2024). InstanSeg: an embedding-based instance segmentation algorithm optimized for accurate, efficient and portable cell segmentation. arXiv (Cornell University).](https://doi.org/10.48550/arxiv.2408.15954)\n19. [Noah F. Greenwald and colleagues (2021). Whole-cell segmentation of tissue images with human-level performance using large-scale data annotation and deep learning. Nature Biotechnology.](https://doi.org/10.1038/s41587-021-01094-0)\n20. [Fabian Isensee and colleagues (2020). nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature Methods.](https://doi.org/10.1038/s41592-020-01008-z)\n21. [Kaiming He and colleagues (2018). Mask R-CNN. IEEE Transactions on Pattern Analysis and Machine Intelligence.](https://doi.org/10.1109/tpami.2018.2844175)\n22. [Çiçek, Özgün and colleagues (2016). 3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation. arXiv (Cornell University).](https://doi.org/10.48550/arxiv.1606.06650)\n23. [Nuclei Instance Segmentation and Classification in Histopathology Images with StarDist (CoNIC challenge, ISBIC 2022)](https://warwick.ac.uk/fac/cross_fac/tia/data/conic/stardist.pdf)\n24. [Revisiting foundation models for cell instance segmentation (Archit & Pape, MIDL/PMLR v315)](https://proceedings.mlr.press/v315/archit26a.html)\n25. [Halo: a pretrained model for whole-cell segmentation from nuclei images in spatial transcriptomics (Briefings in Bioinformatics)](https://academic.oup.com/bib/article/27/5/bbag515/8812206)\n26. [A systematic evaluation of computational methods for cell segmentation (Briefings in Bioinformatics)](https://pmc.ncbi.nlm.nih.gov/articles/PMC10862744/)\n27. [CellBinDB: a large-scale multimodal annotated dataset for cell segmentation with benchmarking of universal models](https://pmc.ncbi.nlm.nih.gov/articles/PMC12206155/)\n28. [State-of-the-Art Deep Learning Methods for Microscopic Image Segmentation (Journal of Imaging)](https://www.mdpi.com/2313-433X/10/12/311)\n\n---\n*Topic: Encyclopedia › Life and health › Biological foundations*\n\n*Initially written Sep 29, 2026 · Edited: Oct 15, 2026*\n\n*Copyright 2026 EdgeChat AI, a subsidiary of Biostate AI.*\n\nLicense: Edgepedia Community License 1.0, https://www.edgepedia.org/edgepedia/license\n", "same_as": [], "url": "https://www.edgepedia.org/cell-segmentation", "markdown_url": "https://www.edgepedia.org/cell-segmentation.md", "license": { "name": "Edgepedia Community License 1.0", "url": "https://www.edgepedia.org/edgepedia/license", "summary": "Free with credit, commercial use included. AI training is open to everyone. For other uses, organizations over USD 100M in revenue or 100M monthly users license separately.", "spdx": "LicenseRef-Edgepedia-Community-1.0" }, "credit": "\"Cell segmentation\", Edgepedia (EdgeChat), https://www.edgepedia.org/cell-segmentation. Edgepedia Community License 1.0.", "credit_md": "\"[Cell segmentation](https://www.edgepedia.org/cell-segmentation)\", Edgepedia (EdgeChat), [https://www.edgepedia.org/cell-segmentation](https://www.edgepedia.org/cell-segmentation). [Edgepedia Community License 1.0](https://www.edgepedia.org/edgepedia/license).", "credit_html": "\"Cell segmentation\", Edgepedia (EdgeChat), https://www.edgepedia.org/cell-segmentation. Edgepedia Community License 1.0.", "speakable": "Cell segmentation is a computational image-analysis method that identifies and outlines each individual cell in a microscopy image, assigning every pixel to a specific cell or background." }