{ "id": "epbr7mfsa8", "slug": "self-distillation", "title": "Self-distillation", "updated": "2026-10-15", "topic_path": [ { "id": "technology", "label": "Technology and the built world", "api_url": "https://www.edgepedia.org/api/v1/topics/technology" }, { "id": "technology.computing", "label": "Computing and digital systems", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing" }, { "id": "technology.computing.ai", "label": "Artificial intelligence and data", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.ai" }, { "id": "technology.computing.ai.ml_computation", "label": "Machine learning and neural computation", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.ai.ml_computation" }, { "id": "technology.computing.ai.neural_deep", "label": "Neural networks and deep learning", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.ai.neural_deep" }, { "id": "technology.computing.ai.neural_deep.alpha-i-to-z", "label": "Titles I to Z", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.ai.neural_deep.alpha-i-to-z" } ], "geo": [ { "id": "geo.nongeo.t2001.technology.computing.ai.neural_deep", "label": "Non-geographic · 2001 to 2020: Neural networks and deep learning", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology.computing.ai.neural_deep", "path": [ { "id": "geo.nongeo", "label": "Non-geographic", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo" }, { "id": "geo.nongeo.t2001", "label": "Non-geographic · 2001 to 2020", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001" }, { "id": "geo.nongeo.t2001.technology", "label": "Technology and the built world", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology" }, { "id": "geo.nongeo.t2001.technology.computing", "label": "Computing and digital systems", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology.computing" }, { "id": "geo.nongeo.t2001.technology.computing.ai", "label": "Artificial intelligence and data", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology.computing.ai" }, { "id": "geo.nongeo.t2001.technology.computing.ai.ml_computation", "label": "Machine learning and neural computation", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology.computing.ai.ml_computation" }, { "id": "geo.nongeo.t2001.technology.computing.ai.neural_deep", "label": "Neural networks and deep learning", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology.computing.ai.neural_deep" } ] } ], "excerpt": "Self-distillation, also known as policy self-distillation, is a machine learning method in which a neural network imitates its own predictions, introduced as Born-Again Networks in 2018.", "snippet": "Self-distillation, also known as policy self-distillation, is a machine learning method in which a neural network imitates its own predictions, introduced as Born-Again Networks in 2018.", "node": "technology.computing.ai.neural_deep.alpha-i-to-z", "markdown": "# Self-distillation\n\nSelf-distillation is a variant of knowledge distillation in which a neural network is trained to imitate its own predictions, or the predictions of an identical-architecture clone, rather than those of a separate, larger teacher model. It was introduced for multi-generational training as Born-Again Networks (BAN) by Furlanello and colleagues in 2018, who showed that students parameterized identically to their teachers can outperform them on computer vision and language modeling tasks.[1](https://proceedings.mlr.press/v80/furlanello18a/furlanello18a.pdf) The method builds on knowledge distillation, in which a cumbersome model's class probabilities serve as soft targets for a smaller student, with the softmax temperature raised until the teacher produces suitably soft targets.[2](https://arxiv.org/abs/1503.02531) The surprising claim is that same-capacity students improve at all: BANs based on DenseNets reached 3.5% validation error on CIFAR-10 and 15.5% on CIFAR-100.[1](https://proceedings.mlr.press/v80/furlanello18a/furlanello18a.pdf)\n\n| Key fact | Detail |\n|---|---|\n| Definition | Distillation where teacher and student share architecture (or are the same model), removing the external teacher[1](https://proceedings.mlr.press/v80/furlanello18a/furlanello18a.pdf) |\n| Headline result | BAN DenseNets: 3.5% (CIFAR-10) and 15.5% (CIFAR-100) validation error; a 6.5M-parameter student matches a teacher with roughly 8x more parameters[1](https://proceedings.mlr.press/v80/furlanello18a/furlanello18a.pdf) |\n| First theory | Mobahi, Farajtabar, and Bartlett (NeurIPS 2020): self-distillation progressively limits the basis functions representing the solution, acting as amplified regularization[3](https://proceedings.neurips.cc/paper_files/paper/2020/file/2288f691b58edecadcc9a8691762b4fd-Paper.pdf) |\n| Typical loss | \\( \\mathcal{L}_{\\mathrm{KD}} = \\alpha \\mathcal{L}_{\\mathrm{CE}}(z_s, y) + (1-\\alpha) \\mathcal{L}_{\\mathrm{KL}}(z_s, z_t) \\) with temperature-scaled KL divergence[4](https://arxiv.org/pdf/2206.08491) |\n| Known failure mode | A few rounds reduce overfitting, but further rounds underfit and the solution can collapse to zero[3](https://proceedings.neurips.cc/paper_files/paper/2020/file/2288f691b58edecadcc9a8691762b4fd-Paper.pdf) |\n| LLM-era use | On-policy self-distillation (OPSD) matches GRPO on competition math benchmarks with better token efficiency[5](https://arxiv.org/abs/2601.18734v3) |\n\n## How it works\n\nThe conventional objective combines ordinary supervised loss with a distillation term: \\( \\mathcal{L}_{\\mathrm{KD}} = \\alpha \\mathcal{L}_{\\mathrm{CE}}(z_s, y) + (1-\\alpha) \\mathcal{L}_{\\mathrm{KL}}(z_s, z_t) \\), where the KL divergence is computed on temperature-scaled logits.[4](https://arxiv.org/pdf/2206.08491) In self-distillation the teacher logits \\( z_t \\) come from the model itself, a previous generation of the same architecture, or an auxiliary deep head. Hinton, Vinyals, and Dean's original recipe uses a weighted average of cross-entropy with soft targets at high temperature \\( T \\) and cross-entropy with correct labels at temperature 1; because soft-target gradients scale as \\( 1/T^2 \\), they must be multiplied by \\( T^2 \\).[2](https://arxiv.org/abs/1503.02531) In the high-temperature limit, assuming the logits are zero-centered, distillation is equivalent to minimizing the squared difference between student and teacher logits.[2](https://arxiv.org/abs/1503.02531)\n\nWhy identical-capacity imitation helps has several published explanations. Mobahi, Farajtabar, and Bartlett's analysis, in an \\( \\ell_2 \\)-regularized Hilbert-space regression setting, shows each distillation round modifies regularization by progressively limiting the number of basis functions that can represent the solution.[3](https://proceedings.neurips.cc/paper_files/paper/2020/file/2288f691b58edecadcc9a8691762b4fd-Paper.pdf) Zhang and Sabuncu interpret self-distillation as instance-specific label smoothing, casting teacher-student training as amortized MAP estimation of softmax outputs.[6](https://papers.nips.cc/paper/2020/file/1731592aca5fb4d789c4119c65c10b4b-Paper.pdf) Empirically, a single round of self-distillation produces students with flatter minima, lower Hessian trace, and lower \\( \\lambda_{\\max} \\), than the teacher.[4](https://arxiv.org/pdf/2206.08491) Under label noise, multi-round self-distillation acts as label averaging within ground-truth classes.[7](https://proceedings.iclr.cc/paper_files/paper/2025/file/9d4824d834b5fe8e6b53dcfe42cab8d2-Paper-Conference.pdf)\n\nThe student does not collapse into trivial logit copying because it never achieves it: even when the student has the capacity to match the teacher, a large discrepancy between their predictive distributions remains, and for a ResNet-56 self-distilled on CIFAR-100 the best test agreement falls far below 99%.[8](https://papers.neurips.cc/paper_files/paper/2021/file/376c6b9ff3bedbbea56751a84fffc10c-Paper.pdf) Indeed, in self-distillation the student can exceed the teacher only by failing at the distillation procedure; a student that matched the teacher perfectly could not outperform it.[8](https://papers.neurips.cc/paper_files/paper/2021/file/376c6b9ff3bedbbea56751a84fffc10c-Paper.pdf)\n\n## How it is done\n\n In the original BAN setup, \\( \\alpha \\) and \\( T \\) are set to 0 and 1 throughout training, and generations are trained sequentially.[6](https://papers.nips.cc/paper/2020/file/1731592aca5fb4d789c4119c65c10b4b-Paper.pdf) In BYOT, training uses three loss sources: cross-entropy with labels on all classifiers, KL divergence between each shallow classifier's softmax and the deepest classifier's output (distillation temperature normally 1), and an \\( L_{2} \\) hint loss between feature maps, balanced by hyperparameters \\( \\alpha \\) and \\( \\lambda \\).[9](https://openaccess.thecvf.com/content_ICCV_2019/papers/Zhang_Be_Your_Own_Teacher_Improve_the_Performance_of_Convolutional_Neural_ICCV_2019_paper.pdf) In the label-noise formulation, the per-sample objective is \\( \\xi \\cdot \\ell(\\text{teacher predictions}, \\text{student predictions}) + (1-\\xi) \\cdot \\ell(\\text{given labels}, \\text{student predictions}) \\), with \\( \\xi \\) the imitation parameter.[10](https://ar5iv.labs.arxiv.org/html/2301.13304) Practical stabilizers include weighting ground-truth targets with \\( \\alpha > 0 \\), where \\( \\alpha = 0.25 \\) reduced imposed regularization and increased stability in ResNet-50/CIFAR-10 experiments[11](https://proceedings.neurips.cc/paper/2021/file/2adcefe38fbcd3dcd45908fbab1bf628-Paper.pdf), and a random model restart before the second training stage, worth 1.90% accuracy on ResNet-32/CIFAR-100.[12](https://ar5iv.labs.arxiv.org/html/1811.07598)\n\n## Origin\n\nKnowledge distillation itself was reported by Hinton, Vinyals, and Dean in \"Distilling the Knowledge in a Neural Network\" (2015).[2](https://arxiv.org/abs/1503.02531) Same-size self-distillation across generations was introduced as Born-Again Networks by Furlanello and colleagues at ICML 2018[1](https://proceedings.mlr.press/v80/furlanello18a/furlanello18a.pdf); their paper cites earlier work on born-again trees and on model compression as precursors.[1](https://proceedings.mlr.press/v80/furlanello18a/furlanello18a.pdf) The first theoretical analysis is due to Hossein Mobahi, Mehrdad Farajtabar, and Peter L. Bartlett, \"Self-Distillation Amplifies Regularization in Hilbert Space\", NeurIPS 2020.[3](https://proceedings.neurips.cc/paper_files/paper/2020/file/2288f691b58edecadcc9a8691762b4fd-Paper.pdf) Priority is disputed for the LLM-era sense: the SDPO authors credit Snell et al. (2022) with first proposing self-distillation as sampling with extra context and training the same model to mimic those predictions without the context[13](https://arxiv.org/abs/2601.20802), a claim that conflicts with the 2018 BAN lineage and is not settled in the literature.\n\n## Variants\n\n**Born-again networks** train a sequence of identical students, each distilling from the previous generation.[1](https://proceedings.mlr.press/v80/furlanello18a/furlanello18a.pdf) **BYOT** distills from the deepest portion of a network into shallow sections equipped with auxiliary classifiers[9](https://openaccess.thecvf.com/content_ICCV_2019/papers/Zhang_Be_Your_Own_Teacher_Improve_the_Performance_of_Convolutional_Neural_ICCV_2019_paper.pdf); the TPAMI extension adds attention modules and shallow classifiers at different depths, distilling from the deepest classifier to shallower ones.[14](https://archive.ymsc.tsinghua.edu.cn/pacm_download/654/12353-baocl8.pdf) **SRDL** distills the in-training model's self-discovered knowledge back into itself.[12](https://ar5iv.labs.arxiv.org/html/1811.07598) **CS-KD** distills predictive distributions between different samples of the same label within a single network.[15](https://openaccess.thecvf.com/content_CVPR_2020/papers/Yun_Regularizing_Class-Wise_Predictions_via_Self-Knowledge_Distillation_CVPR_2020_paper.pdf) **SKD** for NLP distills from the model's own word-embedding space, with \\( q_n = \\min\\{\\exp\\{-\\sigma \\| w_t - w_n \\|_2\\}, 0.5\\} \\) clipped so the predicted class is never treated as more correct than the true target, and a distillation weight \\( \\alpha \\) ramped up from 0.[16](https://ar5iv.labs.arxiv.org/html/1908.01851) Teacher-free methods such as USKD generate customized soft labels without any teacher, smoothing the student's target logit and using intermediate-feature rank with [Zipf's law](https://www.edgepedia.org/zipfs-law) for non-target labels.[17](https://ar5iv.labs.arxiv.org/html/2303.13005) Conditions exist under which the \\( t \\)-th distilled model achieves 100% population accuracy on label-noisy data, and a single-round Partial Label Learning variant refines the teacher's softmax into a two-hot vector, replicating multi-round distillation in one round.[7](https://proceedings.iclr.cc/paper_files/paper/2025/file/9d4824d834b5fe8e6b53dcfe42cab8d2-Paper-Conference.pdf) For LLMs, a related method with an external teacher, **GKD**, trains a smaller student on its self-generated sequences using a separate teacher's feedback rather than a self-distillation variant[18](https://arxiv.org/pdf/2306.13649.pdf); **OPSD** uses a single LLM as both teacher and student with different contexts, the teacher conditioning on privileged information such as verified reasoning traces while gradients flow only through the student's logits[5](https://arxiv.org/abs/2601.18734v3); **u-OPSD** replaces ground-truth solutions with majority-vote pseudo-solutions[19](https://arxiv.org/html/2608.06296v1); **OISD** uses the model's final layer as a detached internal teacher for intermediate layers[20](https://arxiv.org/html/2605.29089); **SDPO** distills feedback-informed next-token predictions from the current model into the policy with no external teacher or reward model[13](https://arxiv.org/abs/2601.20802); and **SPD** extracts a low-rank capability subspace from the model's own gradients via SVD.[21](https://arxiv.org/html/2605.22675v1)\n\n## Applications\n\nReported gains cluster in three settings. In image classification, BYOT gives an average accuracy enhancement of 2.65% on CIFAR100 and 2.02% on ImageNet[9](https://openaccess.thecvf.com/content_ICCV_2019/papers/Zhang_Be_Your_Own_Teacher_Improve_the_Performance_of_Convolutional_Neural_ICCV_2019_paper.pdf), and the TPAMI extension reports average boosts of 3.49% on CIFAR100 and 2.32% on ImageNet.[14](https://archive.ymsc.tsinghua.edu.cn/pacm_download/654/12353-baocl8.pdf) On language modeling, BAN-LSTM decreases PTB test perplexity from 71.87 to 68.56, but only when trained with a combination of teacher outputs and label loss.[1](https://proceedings.mlr.press/v80/furlanello18a/furlanello18a.pdf)\n\nGains depend on conditions. A single round improves students even over strong teachers (97.16% to 97.40% on CIFAR-10 with Cutout and AutoAugment), but multi-round self-distillation fluctuates rather than improving.[4](https://arxiv.org/pdf/2206.08491) Self-distillation helps when the teacher is trained with advanced data augmentation, but the benefit is limited otherwise[4](https://arxiv.org/pdf/2206.08491), and \\( \\xi = 1 \\) yields no gains over the teacher with zero label corruption while gains grow with corruption.[10](https://ar5iv.labs.arxiv.org/html/2301.13304) On regression benchmarks, 2-step self-distillation reduced MSE by 47.2% versus optimal ridge on an Air Quality dataset, but gave no improvement on the AEP dataset.[22](https://proceedings.neurips.cc/paper_files/paper/2024/file/0eb1ac7551ddbae575415aa5183a88be-Paper-Conference.pdf) In the LLM setting, OPSD performs on par with or better than GRPO with significantly better sample efficiency[5](https://arxiv.org/abs/2601.18734v3), and SDPO reaches GRPO's accuracy 6x faster in wall-clock time on chemistry questions with Olmo3-7B-Instruct.[13](https://arxiv.org/abs/2601.20802)\n\n## Limitations and alternatives\n\n**Failure modes.** A few rounds of self-distillation may reduce overfitting, but further rounds lead to underfitting, and the solution can eventually collapse to zero[3](https://proceedings.neurips.cc/paper_files/paper/2020/file/2288f691b58edecadcc9a8691762b4fd-Paper.pdf); with \\( \\alpha = 0 \\) the procedure eventually over-regularizes, while fixed \\( \\alpha > 0 \\) converges to kernel ridge regression with regularization amplified by \\( \\alpha^{-1} \\).[11](https://proceedings.neurips.cc/paper/2021/file/2adcefe38fbcd3dcd45908fbab1bf628-Paper.pdf) Under label noise, the optimal imitation parameter \\( \\xi \\) is surprisingly greater than 1, and \\( \\xi > 1 \\) beats \\( \\xi \\le 1 \\) on datasets with 30–50% label corruption.[10](https://ar5iv.labs.arxiv.org/html/2301.13304) As rounds progress, softmax confidence for the true label decreases in clean samples while increasing in noisy samples.[7](https://proceedings.iclr.cc/paper_files/paper/2025/file/9d4824d834b5fe8e6b53dcfe42cab8d2-Paper-Conference.pdf) For LLMs, on-policy self-distillation fails in tested math-reasoning settings because privileged information is instance-specific: a Qwen3-14B teacher at 62.1% on GPQA-Diamond drops to 46.0% when conditioned on student prefixes.[23](https://arxiv.org/html/2605.11182v2)\n\n**Comparisons.** Born-Again Networks underperform a straightforward ensemble of independently trained models for all tested \\( \\alpha \\) values (0.2, 0.5, 0.8).[4](https://arxiv.org/pdf/2206.08491) Self-distillation outperforms label smoothing in Zhang and Sabuncu's experiments at equal effective smoothing, with much smaller expected calibration error.[6](https://papers.nips.cc/paper/2020/file/1731592aca5fb4d789c4119c65c10b4b-Paper.pdf) Against standard teacher-student distillation, BYOT reached 81.04% versus 79.33% on ResNet50/CIFAR100 while cutting training time from 26.98 to 5.87 hours, a 4.6x speedup, by avoiding a separate over-parameterized teacher.[9](https://openaccess.thecvf.com/content_ICCV_2019/papers/Zhang_Be_Your_Own_Teacher_Improve_the_Performance_of_Convolutional_Neural_ICCV_2019_paper.pdf)\n\n## References\n\n1. [Born-Again Neural Networks (Furlanello et al., ICML 2018)](https://proceedings.mlr.press/v80/furlanello18a/furlanello18a.pdf)\n2. [Distilling the Knowledge in a Neural Network (Hinton et al.)](https://arxiv.org/abs/1503.02531)\n3. [Self-Distillation Amplifies Regularization in Hilbert Space (Mobahi et al., NeurIPS 2020)](https://proceedings.neurips.cc/paper_files/paper/2020/file/2288f691b58edecadcc9a8691762b4fd-Paper.pdf)\n4. [Revisiting Self-Distillation / Towards Understanding Why Distillation Helps (arXiv 2206.08491)](https://arxiv.org/pdf/2206.08491)\n5. [Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models (OPSD)](https://arxiv.org/abs/2601.18734v3)\n6. [Self-Distillation as Instance-Specific Label Smoothing (Zhang & Sabuncu, NeurIPS 2020)](https://papers.nips.cc/paper/2020/file/1731592aca5fb4d789c4119c65c10b4b-Paper.pdf)\n7. [Self-Distillation Under Label Noise (ICLR 2025)](https://proceedings.iclr.cc/paper_files/paper/2025/file/9d4824d834b5fe8e6b53dcfe42cab8d2-Paper-Conference.pdf)\n8. [Does Knowledge Distillation Really Work? (NeurIPS 2021)](https://papers.neurips.cc/paper_files/paper/2021/file/376c6b9ff3bedbbea56751a84fffc10c-Paper.pdf)\n9. [Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation (Zhang et al., ICCV 2019)](https://openaccess.thecvf.com/content_ICCV_2019/papers/Zhang_Be_Your_Own_Teacher_Improve_the_Performance_of_Convolutional_Neural_ICCV_2019_paper.pdf)\n10. [Understanding Self-Distillation in the Presence of Label Noise (arXiv 2301.13304)](https://ar5iv.labs.arxiv.org/html/2301.13304)\n11. [Even your Teacher Needs Guidance: Ground-Truth Targets Dampen Regularization Imposed by Self-Distillation (NeurIPS 2021)](https://proceedings.neurips.cc/paper/2021/file/2adcefe38fbcd3dcd45908fbab1bf628-Paper.pdf)\n12. [Self-Referenced Deep Learning (SRDL) (arXiv 1811.07598)](https://ar5iv.labs.arxiv.org/html/1811.07598)\n13. [Reinforcement Learning via Self-Distillation (SDPO)](https://arxiv.org/abs/2601.20802)\n14. [Self-Distillation: Towards Efficient and Compact Neural Networks (Yang et al., IEEE TPAMI)](https://archive.ymsc.tsinghua.edu.cn/pacm_download/654/12353-baocl8.pdf)\n15. [Regularizing Class-Wise Predictions via Self-Knowledge Distillation (CS-KD, CVPR 2020)](https://openaccess.thecvf.com/content_CVPR_2020/papers/Yun_Regularizing_Class-Wise_Predictions_via_Self-Knowledge_Distillation_CVPR_2020_paper.pdf)\n16. [Self-Knowledge Distillation in Natural Language Processing (Kim, Kim, Kim, 2019)](https://ar5iv.labs.arxiv.org/html/1908.01851)\n17. [From Knowledge Distillation to Self-Knowledge Distillation: A Unified Approach with Normalized Loss and Customized Soft Labels (NKD/USKD, ICCV 2023)](https://ar5iv.labs.arxiv.org/html/2303.13005)\n18. [On-policy Distillation of Language Models: Learning from Self-Generated Mistakes (GKD, Agarwal et al.)](https://arxiv.org/pdf/2306.13649.pdf)\n19. [On-Policy Self-Distillation without Any Supervision (u-OPSD)](https://arxiv.org/html/2608.06296v1)\n20. [OISD: On-Policy Internal Self-Distillation of Language Models](https://arxiv.org/html/2605.29089)\n21. [Self-Policy Distillation via Capability-Selective Subspace Projection (SPD)](https://arxiv.org/html/2605.22675v1)\n22. [Understanding the Gains from Repeated Self-Distillation (NeurIPS 2024)](https://proceedings.neurips.cc/paper_files/paper/2024/file/0eb1ac7551ddbae575415aa5183a88be-Paper-Conference.pdf)\n23. [The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes](https://arxiv.org/html/2605.11182v2)\n\n---\n*Topic: Encyclopedia › Technology and the built world › Computing and digital systems › Artificial intelligence and data › Machine learning and neural computation › Neural networks and deep learning › Titles I to Z*\n\n*Initially written Sep 29, 2026 · Reviewed: Sep 30, 2026 · Edited: Oct 15, 2026 · Last review: Sep 30, 2026*\n\n*Copyright 2026 EdgeChat AI, a subsidiary of Biostate AI.*\n\nLicense: Edgepedia Community License 1.0, https://www.edgepedia.org/edgepedia/license\n", "same_as": [], "url": "https://www.edgepedia.org/self-distillation", "markdown_url": "https://www.edgepedia.org/self-distillation.md", "license": { "name": "Edgepedia Community License 1.0", "url": "https://www.edgepedia.org/edgepedia/license", "summary": "Free with credit, commercial use included. AI training is open to everyone. For other uses, organizations over USD 100M in revenue or 100M monthly users license separately.", "spdx": "LicenseRef-Edgepedia-Community-1.0" }, "credit": "\"Self-distillation\", Edgepedia (EdgeChat), https://www.edgepedia.org/self-distillation. Edgepedia Community License 1.0.", "credit_md": "\"[Self-distillation](https://www.edgepedia.org/self-distillation)\", Edgepedia (EdgeChat), [https://www.edgepedia.org/self-distillation](https://www.edgepedia.org/self-distillation). [Edgepedia Community License 1.0](https://www.edgepedia.org/edgepedia/license).", "credit_html": "\"Self-distillation\", Edgepedia (EdgeChat), https://www.edgepedia.org/self-distillation. Edgepedia Community License 1.0.", "speakable": "Self-distillation, also known as policy self-distillation, is a machine learning method in which a neural network imitates its own predictions, introduced as Born-Again Networks in 2018." }