{ "id": "ep8py38hs5", "slug": "backdoor-detection-machine-learning", "title": "Backdoor detection (machine learning)", "updated": "2026-10-15", "topic_path": [ { "id": "technology", "label": "Technology and the built world", "api_url": "https://www.edgepedia.org/api/v1/topics/technology" }, { "id": "technology.computing", "label": "Computing and digital systems", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing" }, { "id": "technology.computing.ai", "label": "Artificial intelligence and data", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.ai" }, { "id": "technology.computing.ai.ml_computation", "label": "Machine learning and neural computation", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.ai.ml_computation" } ], "geo": [ { "id": "geo.nongeo.t2001.technology.computing.ai.ml_computation", "label": "Non-geographic · 2001 to 2020: Machine learning and neural computation", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology.computing.ai.ml_computation", "path": [ { "id": "geo.nongeo", "label": "Non-geographic", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo" }, { "id": "geo.nongeo.t2001", "label": "Non-geographic · 2001 to 2020", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001" }, { "id": "geo.nongeo.t2001.technology", "label": "Technology and the built world", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology" }, { "id": "geo.nongeo.t2001.technology.computing", "label": "Computing and digital systems", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology.computing" }, { "id": "geo.nongeo.t2001.technology.computing.ai", "label": "Artificial intelligence and data", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology.computing.ai" }, { "id": "geo.nongeo.t2001.technology.computing.ai.ml_computation", "label": "Machine learning and neural computation", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology.computing.ai.ml_computation" } ] } ], "excerpt": "Backdoor detection is a set of security techniques for determining whether a trained neural network carries a hidden trigger that forces attacker-chosen labels, by offline scanning or run-time analysis.", "snippet": "Backdoor detection is a set of security techniques for determining whether a trained neural network carries a hidden trigger that forces attacker-chosen labels, by offline scanning or run-time analysis.", "node": "technology.computing.ai.ml_computation", "markdown": "# Backdoor detection (machine learning)\n\nBackdoor detection is a set of security techniques for determining whether a trained neural network carries a hidden trigger, a pattern planted during training that forces the model to an attacker-chosen label while leaving normal behavior intact. A detector typically outputs a verdict (backdoored or clean), the suspected target label, and often a reconstructed trigger; some methods also suggest mitigation. The problem matters because practitioners routinely deploy third-party models, and a backdoor can survive later retraining.[1](https://doi.org/10.48550/arxiv.1708.06733) Detection is done either offline, by scanning a model before deployment, or at run time, by analyzing suspicious inputs.[2](http://proceedings.mlr.press/v139/shen21c/shen21c.pdf)\n\n| Key fact | Detail |\n|---|---|\n| What a backdoor is | A hidden pattern trained into a DNN that produces unexpected behavior if and only if a specific trigger is added to an input.[3](https://people.cs.vt.edu/vbimal/publications/backdoor-sp19.pdf) |\n| How attacks implant it | A random subset of training images is stamped with a trigger (pixels and color intensities) and relabeled to the target label before training.[1](https://doi.org/10.48550/arxiv.1708.06733) |\n| Detector outputs | A verdict, the target label, a reverse-engineered trigger, and optionally mitigation (input filters, neuron pruning, unlearning).[3](https://people.cs.vt.edu/vbimal/publications/backdoor-sp19.pdf) • [4](https://tao.aisec.world/assets/pdf/CCS19.pdf) |\n| Access assumptions | Methods range from reverse engineering (Neural Cleanse) to black-box, run-time input analysis (STRIP) and black-box meta-classification (MNTD).[3](https://people.cs.vt.edu/vbimal/publications/backdoor-sp19.pdf) • [5](https://doi.org/10.48550/arxiv.1902.06531) • [6](http://seclab.illinois.edu/wp-content/uploads/2021/05/xu2020detecting.pdf) |\n| Headline detection numbers | STRIP reports a false acceptance rate below 1% at a preset 1% false rejection rate; DL-TND and DF-TND report 0.99 average AUROC.[5](https://doi.org/10.48550/arxiv.1902.06531) • [7](https://ar5iv.labs.arxiv.org/html/2007.15802) |\n| Standard metrics | Clean accuracy (C-Acc), attack success rate (ASR), robust accuracy (R-Acc), and defense effectiveness rate (DER) on benchmarks such as BackdoorBench and TrojAI.[8](https://papers.nips.cc/paper_files/paper/2022/file/4491ea1c91aa2b22c373e5f1dfce234f-Paper-Datasets_and_Benchmarks.pdf) • [9](https://www.cs.purdue.edu/homes/shen447/files/paper/cvpr22_exray.pdf) |\n| Why scanning matters | A US traffic-sign BadNet retrained for Swedish signs retained its backdoor and caused a 25% accuracy drop.[1](https://doi.org/10.48550/arxiv.1708.06733) |\n\n## How it works\n\nThe attack mechanism explains the detection signal. An attacker poisons the training data: a subset of images is stamped with a trigger pattern and relabeled to the target label, so the trained model maps anything carrying the trigger to that label. On the MNIST benchmark this yields over 99% attack success without hurting clean accuracy.[3](https://people.cs.vt.edu/vbimal/publications/backdoor-sp19.pdf)\n\nDetectors exploit three kinds of anomaly. First, reverse engineering: in an infected model, far smaller modifications are needed to force misclassification into the target label than into any other label, so optimizing a minimal trigger per label and comparing their sizes exposes the backdoor.[3](https://people.cs.vt.edu/vbimal/publications/backdoor-sp19.pdf) Second, internal activation anomalies: ABS stimulates inner neurons and flags those that elevate one output label regardless of input,[4](https://tao.aisec.world/assets/pdf/CCS19.pdf) and backdoored networks share a property called the spectral signature, which is a detectable trace in a neural network's learned feature representations that can help identify and remove poisoned training examples.[10](https://doi.org/10.48550/arxiv.1811.00636) • [20](https://proceedings.neurips.cc/paper/2018/file/280cf18baf4311c92aa5a042336587d3-Paper.pdf) Third, run-time input suspicion: STRIP superimposes strong perturbations on incoming inputs and measures the entropy of predicted classes; trojaned inputs keep predicting the target label, so their entropy is low.[5](https://doi.org/10.48550/arxiv.1902.06531)\n\n## How it is done\n\nA practitioner scanning a third-party model chooses a regime first: offline model scanning before deployment, or on-the-fly detection of stamped inputs at run time.[2](http://proceedings.mlr.press/v139/shen21c/shen21c.pdf) For offline scanning with Neural Cleanse, the procedure has three steps. Step 1 treats each of the \\( N \\) labels in turn as a potential target and optimizes the minimal trigger that misclassifies all other samples into it. Step 2 measures each candidate trigger's size by the number of pixels it replaces. Step 3 runs outlier detection: a trigger candidate significantly smaller than the rest indicates a real backdoor and identifies the target label.[3](https://people.cs.vt.edu/vbimal/publications/backdoor-sp19.pdf) The same paper offers mitigation through input filters, neuron pruning, and unlearning, so a confirmed backdoor can be repaired rather than merely discarded.[3](https://people.cs.vt.edu/vbimal/publications/backdoor-sp19.pdf) ABS follows a different sequence: stimulate neurons, flag suspicious ones, then reverse-engineer the trigger by optimization to confirm the neuron is truly compromised.[4](https://tao.aisec.world/assets/pdf/CCS19.pdf)\n\n## Origin\n\nThe attack side came first. Gu, Dolan-Gavitt, and Garg reported the BadNets attack in 2017 on arXiv, building backdoored networks with state-of-the-art performance on training and validation data that behave badly on attacker-chosen inputs.[1](https://doi.org/10.48550/arxiv.1708.06733) Their demonstration that a backdoor persists through retraining for a new task, causing a 25% accuracy drop on Swedish traffic signs, gave a concrete reason to scan models rather than trust their clean accuracy.[1](https://doi.org/10.48550/arxiv.1708.06733) On the defense side, Tran, Li, and Madry published the spectral-signature analysis in 2018 on arXiv,[10](https://doi.org/10.48550/arxiv.1811.00636) and Gao and colleagues published STRIP in 2019 on arXiv.[5](https://doi.org/10.48550/arxiv.1902.06531) Neural Cleanse is a method for robust and general detection and mitigation against backdoor attacks on DNNs.[3](https://people.cs.vt.edu/vbimal/publications/backdoor-sp19.pdf) • [11](https://github.com/bolunwang/backdoor)\n\n## Variants\n\nThe named methods differ mainly in what they inspect and what they assume.\n\n**Neural Cleanse** reverse-engineers minimal triggers per label and requires many input samples and small triggers to work well.[3](https://people.cs.vt.edu/vbimal/publications/backdoor-sp19.pdf) • [4](https://tao.aisec.world/assets/pdf/CCS19.pdf) **ABS** stimulates inner neurons instead, achieving over 90% detection in most cases with only one input sample per label.[4](https://tao.aisec.world/assets/pdf/CCS19.pdf) **STRIP** works at run time in a black-box setting, blending test images with clean images and flagging low average entropy of the blended predictions; it uses only inputs and softmax outputs of an already-deployed model, is independent of its architecture, and is insensitive to trigger size.[5](https://doi.org/10.48550/arxiv.1902.06531) • [12](https://ar5iv.labs.arxiv.org/html/2112.03350) **MNTD** trains a meta-classifier on many \"jumbo\" models, both trojaned and clean, to learn detection features transferable to an unknown target model, with no assumptions on attack strategy; it likewise needs only black-box access to the target model.[6](http://seclab.illinois.edu/wp-content/uploads/2021/05/xu2020detecting.pdf) **DL-TND and DF-TND** detect trojaned CNNs with one clean sample per class, or with random noise when no clean data exists, reaching 0.99 average AUROC in the data-free case.[7](https://ar5iv.labs.arxiv.org/html/2007.15802) **K-Arm scanning** optimizes the trigger search to cut scanning time while achieving top accuracy.[2](http://proceedings.mlr.press/v139/shen21c/shen21c.pdf) **ABS+EX-RAY** extends ABS with symmetric feature differencing for complex backdoors.[9](https://www.cs.purdue.edu/homes/shen447/files/paper/cvpr22_exray.pdf) A test-time trigger detector requires neither the training set nor any fine-tuning, uses about 100 clean images per class, and is computationally cheap.[12](https://ar5iv.labs.arxiv.org/html/2112.03350) The TrojAI competition also evaluated ULP, DeepInspect, SCAn, noise analysis, and attribution-based detection, though their mechanisms are not detailed in the comparisons cited here.[9](https://www.cs.purdue.edu/homes/shen447/files/paper/cvpr22_exray.pdf)\n\n## Applications\n\nDetection applies wherever third-party or externally trained models are deployed, including image classifiers and, more recently, language models. BackdoorBench standardizes five metrics for evaluating attacks and defenses: clean accuracy (C-Acc), attack success rate (ASR, accuracy of poisoned samples on the target class), robust accuracy (R-Acc, accuracy of poisoned samples on the original class, with \\( \\mathrm{ASR} + \\mathrm{R\\text{-}Acc} \\leq 1 \\)), plus two defense metrics including the defense effectiveness rate (DER), which weighs drops in both ASR and C-Acc.[8](https://papers.nips.cc/paper_files/paper/2022/file/4491ea1c91aa2b22c373e5f1dfce234f-Paper-Datasets_and_Benchmarks.pdf)\n\nHeadline results vary by setting. STRIP reports false acceptance below 1% at a preset 1% false rejection rate across trigger types on MNIST, CIFAR-10, and GTSRB.[5](https://doi.org/10.48550/arxiv.1902.06531) On the IARPA TrojAI leaderboard, ABS+EX-RAY reached top performance in 2 of 4 image-classification rounds and was the only technique to beat the 0.3465 CE target in all 4 rounds.[9](https://www.cs.purdue.edu/homes/shen447/files/paper/cvpr22_exray.pdf) K-Arm scanning achieved the best accuracy and lowest scanning time per model on the TrojAI rounds 1-4 training sets (3,231 models).[2](http://proceedings.mlr.press/v139/shen21c/shen21c.pdf) For poisoned-data detection, ASSET detects poisoned training samples across semi-supervised and transfer learning paradigms and is reported as the only method in its evaluation able to detect the state-of-the-art clean-label attack, where poisoned samples carry correct labels.[13](https://www.usenix.org/system/files/usenixsecurity23-pan.pdf)\n\nDetection has also extended to language models. A NAACL 2024 Findings paper addresses the under-explored problem of detecting whether an NLP model has been backdoored, focusing on insertion-based attacks with a task-agnostic detector.[14](https://aclanthology.org/2024.findings-naacl.179.pdf) An EMNLP 2025 paper rethinks how backdoor detection for language models is evaluated, motivated by attacker-specified triggers posing a security risk for practitioners who depend on publicly released models.[15](https://aclanthology.org/2025.emnlp-main.318.pdf) A survey extends backdoor attack and defense analysis to large language models.[16](https://www.sciencedirect.com/science/article/pii/S1674862X25000278) Detection for diffusion models and multimodal backdoors is not covered by the comparisons cited here.\n\n## Limitations and alternatives\n\nA critical evaluation of model-inspection defenses documents sharp failure modes. Neural Cleanse is formally proven non-applicable to binary-classification models and empirically ineffective on deeper networks such as ResNet-101. ABS fails to detect a backdoor at a common poisoning rate of about 11% where attack success is nearly 100%, and fails when backdoored weights are constrained with small perturbations. MNTD can assess a single target model by training a meta-classifier to predict whether that model is trojaned, but is sensitive to its meta-classifier hyperparameters, and appears the least robust and most compute-intensive of the three; Neural Cleanse is the most robust within its threat model.[17](https://arxiv.org/pdf/2204.06273v1.pdf) • [21](https://arxiv.org/abs/1910.03137) These verdicts conflict with ABS's own evaluation, which reports it substantially outperforming Neural Cleanse with one sample per label;[4](https://tao.aisec.world/assets/pdf/CCS19.pdf) published comparisons do not settle which holds under which conditions, so detector choice should follow the attacker model and data budget at hand. In head-to-head comparisons at roughly 5% false positive rate, the test-time trigger detector identified nearly all trigger images on all tested attacks, while Neural Cleanse detected no trigger images in a single-class attack using the WB pattern and B3D and STRIP performed poorly for ResNet18 on CIFAR-10.[12](https://ar5iv.labs.arxiv.org/html/2112.03350)\n\nTwo structural limits remain. Even if a countermeasure declares a model backdoor-free within its threat model, the model may still carry a defense-targeted trigger or backdoor type designed against that countermeasure.[17](https://arxiv.org/pdf/2204.06273v1.pdf) A 2026 USENIX Security study provides the first large-scale evaluation of data-free detection methods on pre-trained models, benchmarking more than 30,000 models covering common backdoor attacks.[18](https://www.usenix.org/system/files/usenixsecurity26-zhao-quan.pdf) Surveys describe the overall situation as an escalating arms race between backdoor defenses and adaptive attacks that embed hidden functionality triggered by specific inputs.[19](https://link.springer.com/chapter/10.1007/978-981-96-8183-9_24)\n\n## References\n\n1. [Gu, Tianyu, Dolan-Gavitt, Brendan, Garg, Siddharth (2017). BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain. arXiv (Cornell University).](https://doi.org/10.48550/arxiv.1708.06733)\n2. [Backdoor Scanning for Deep Neural Networks through K-Arm Optimization (ICML 2021; arXiv 2102.05123 is the same paper)](http://proceedings.mlr.press/v139/shen21c/shen21c.pdf)\n3. [Neural Cleanse: Identifying and Mitigating Backdoor Attacks in Neural Networks (IEEE S&P 2019)](https://people.cs.vt.edu/vbimal/publications/backdoor-sp19.pdf)\n4. [ABS: Scanning Neural Networks for Back-doors by Artificial Brain Stimulation (CCS 2019)](https://tao.aisec.world/assets/pdf/CCS19.pdf)\n5. [Gao, Yansong and colleagues (2019). STRIP: A Defence Against Trojan Attacks on Deep Neural Networks. arXiv (Cornell University).](https://doi.org/10.48550/arxiv.1902.06531)\n6. [Detecting AI Trojans Using Meta Neural Analysis (MNTD, IEEE S&P 2020)](http://seclab.illinois.edu/wp-content/uploads/2021/05/xu2020detecting.pdf)\n7. [Practical Detection of Trojan Neural Networks: Data-Limited and Data-Free Cases (DL-TND / DF-TND)](https://ar5iv.labs.arxiv.org/html/2007.15802)\n8. [BackdoorBench: A Comprehensive Benchmark of Backdoor Learning (NeurIPS 2022 Datasets and Benchmarks; arXiv 2407.19845 is the same project)](https://papers.nips.cc/paper_files/paper/2022/file/4491ea1c91aa2b22c373e5f1dfce234f-Paper-Datasets_and_Benchmarks.pdf)\n9. [Complex Backdoor Detection by Symmetric Feature Differencing (ABS+EX-RAY, CVPR 2022)](https://www.cs.purdue.edu/homes/shen447/files/paper/cvpr22_exray.pdf)\n10. [Tran, Brandon, Li, Jerry, Madry, Aleksander (2018). Spectral Signatures in Backdoor Attacks. arXiv (Cornell University).](https://doi.org/10.48550/arxiv.1811.00636)\n11. [bolunwang/backdoor, official Neural Cleanse code](https://github.com/bolunwang/backdoor)\n12. [Test-Time Detection of Backdoor Triggers for Poisoned Deep Neural Networks](https://ar5iv.labs.arxiv.org/html/2112.03350)\n13. [ASSET: Robust Backdoor Data Detection Across a Multiplicity of Deep Learning Paradigms (USENIX Security 2023)](https://www.usenix.org/system/files/usenixsecurity23-pan.pdf)\n14. [Task-Agnostic Detector for Insertion-Based Backdoor Attacks (NAACL 2024 Findings)](https://aclanthology.org/2024.findings-naacl.179.pdf)\n15. [Rethinking Backdoor Detection Evaluation for Language Models (EMNLP 2025)](https://aclanthology.org/2025.emnlp-main.318.pdf)\n16. [A survey of backdoor attacks and defences: From deep neural networks to large language models](https://www.sciencedirect.com/science/article/pii/S1674862X25000278)\n17. [Towards A Critical Evaluation of Robustness for Backdoor Countermeasures (model inspection)](https://arxiv.org/pdf/2204.06273v1.pdf)\n18. [Unveiling the Pitfalls of Data-Free Backdoor Detection Against Pre-Trained Models (USENIX Security 2026)](https://www.usenix.org/system/files/usenixsecurity26-zhao-quan.pdf)\n19. [Arms Race in Deep Learning: A Survey of Backdoor Defenses and Adaptive Attacks (Springer proceedings)](https://link.springer.com/chapter/10.1007/978-981-96-8183-9_24)\n20. [proceedings.neurips.cc](https://proceedings.neurips.cc/paper/2018/file/280cf18baf4311c92aa5a042336587d3-Paper.pdf)\n21. [Detecting AI Trojans Using Meta Neural Analysis](https://arxiv.org/abs/1910.03137)\n\n---\n*Topic: Encyclopedia › Technology and the built world › Computing and digital systems › Artificial intelligence and data › Machine learning and neural computation*\n\n*Initially written Sep 29, 2026 · Edited: Oct 15, 2026*\n\n*Copyright 2026 EdgeChat AI, a subsidiary of Biostate AI.*\n\nLicense: Edgepedia Community License 1.0, https://www.edgepedia.org/edgepedia/license\n", "same_as": [], "url": "https://www.edgepedia.org/backdoor-detection-machine-learning", "markdown_url": "https://www.edgepedia.org/backdoor-detection-machine-learning.md", "license": { "name": "Edgepedia Community License 1.0", "url": "https://www.edgepedia.org/edgepedia/license", "summary": "Free with credit, commercial use included. AI training is open to everyone. For other uses, organizations over USD 100M in revenue or 100M monthly users license separately.", "spdx": "LicenseRef-Edgepedia-Community-1.0" }, "credit": "\"Backdoor detection (machine learning)\", Edgepedia (EdgeChat), https://www.edgepedia.org/backdoor-detection-machine-learning. Edgepedia Community License 1.0.", "credit_md": "\"[Backdoor detection (machine learning)](https://www.edgepedia.org/backdoor-detection-machine-learning)\", Edgepedia (EdgeChat), [https://www.edgepedia.org/backdoor-detection-machine-learning](https://www.edgepedia.org/backdoor-detection-machine-learning). [Edgepedia Community License 1.0](https://www.edgepedia.org/edgepedia/license).", "credit_html": "\"Backdoor detection (machine learning)\", Edgepedia (EdgeChat), https://www.edgepedia.org/backdoor-detection-machine-learning. Edgepedia Community License 1.0.", "speakable": "Backdoor detection is a set of security techniques for determining whether a trained neural network carries a hidden trigger that forces attacker-chosen labels, by offline scanning or run-time analysis." }