{ "id": "epgmhnq510", "slug": "cost-sensitive-learning", "title": "Cost-sensitive learning", "updated": "2026-10-15", "topic_path": [ { "id": "technology", "label": "Technology and the built world", "api_url": "https://www.edgepedia.org/api/v1/topics/technology" }, { "id": "technology.computing", "label": "Computing and digital systems", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing" }, { "id": "technology.computing.ai", "label": "Artificial intelligence and data", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.ai" }, { "id": "technology.computing.ai.ml_computation", "label": "Machine learning and neural computation", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.ai.ml_computation" }, { "id": "technology.computing.ai.ml_methods", "label": "Machine learning methods", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.ai.ml_methods" }, { "id": "technology.computing.ai.supervised_unsupervised", "label": "Supervised, unsupervised, and semi-supervised learning", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.ai.supervised_unsupervised" }, { "id": "technology.computing.ai.supervised_unsupervised.supervised_concepts", "label": "Supervised learning concepts", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.ai.supervised_unsupervised.supervised_concepts" } ], "geo": [ { "id": "geo.nongeo.t1946.technology.computing.ai.supervised_unsupervised", "label": "Non-geographic · 1946 to 2000: Supervised, unsupervised, and semi-supervised learning", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1946.technology.computing.ai.supervised_unsupervised", "path": [ { "id": "geo.nongeo", "label": "Non-geographic", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo" }, { "id": "geo.nongeo.t1946", "label": "Non-geographic · 1946 to 2000", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1946" }, { "id": "geo.nongeo.t1946.technology", "label": "Technology and the built world", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1946.technology" }, { "id": "geo.nongeo.t1946.technology.computing", "label": "Computing and digital systems", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1946.technology.computing" }, { "id": "geo.nongeo.t1946.technology.computing.ai", "label": "Artificial intelligence and data", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1946.technology.computing.ai" }, { "id": "geo.nongeo.t1946.technology.computing.ai.ml_computation", "label": "Machine learning and neural computation", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1946.technology.computing.ai.ml_computation" }, { "id": "geo.nongeo.t1946.technology.computing.ai.ml_methods", "label": "Machine learning methods", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1946.technology.computing.ai.ml_methods" }, { "id": "geo.nongeo.t1946.technology.computing.ai.supervised_unsupervised", "label": "Supervised, unsupervised, and semi-supervised learning", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1946.technology.computing.ai.supervised_unsupervised" } ] } ], "excerpt": "Cost-sensitive learning is a machine learning approach that incorporates unequal misclassification costs into model training or prediction, using weighting, resampling, or threshold moving to minimize expected cost.", "snippet": "Cost-sensitive learning is a machine learning approach that incorporates unequal misclassification costs into model training or prediction, using weighting, resampling, or threshold moving to minimize expected cost.", "node": "technology.computing.ai.supervised_unsupervised.supervised_concepts", "markdown": "# Cost-sensitive learning\n\nCost-sensitive learning is a machine learning approach that incorporates unequal misclassification costs into model training or prediction, so that decisions minimize total expected cost rather than plain error rate. A false negative and a false positive rarely cost the same: a missed fraud can cost the transaction amount.[1](https://link.springer.com/article/10.1007/s10618-021-00790-4) Cost-sensitive methods make this asymmetry explicit, either by changing how a model is trained or by changing how its outputs are turned into decisions.\n\n| Key fact | Detail |\n|---|---|\n| Optimized objective | Expected misclassification cost, \\( R(i \\mid x) = \\sum_{j} P(j \\mid x) C(i,j) \\), the minimum expected cost principle [2](https://www.csd.uwo.ca/~xling/papers/cost_sensitive.pdf) |\n| Cost-optimal threshold | \\( T_{cs} = C_{FP}/(C_{FP} + C_{FN}) \\); equal to 0.5 when costs are equal [1](https://link.springer.com/article/10.1007/s10618-021-00790-4) |\n| Main mechanisms | Example weighting, cost-proportional resampling, threshold moving, and direct loss minimization [2](https://www.csd.uwo.ca/~xling/papers/cost_sensitive.pdf) |\n| Canonical papers | Turney's ICET (1995), Ting's instance-weighted cost-sensitive tree induction (1998), Domingos' MetaCost (1999), Elkan's foundations (2001) [3](https://doi.org/10.1613/jair.120) • [4](https://dl.acm.org/doi/10.1145/312129.312220) • [5](https://dl.acm.org/doi/10.5555/1642194.1642224) |\n| Cost matrix freedom | Multiplying all entries by a positive constant leaves optimal decisions unchanged [5](https://dl.acm.org/doi/10.5555/1642194.1642224) |\n| Typical applications | Credit card fraud detection, credit scoring, direct marketing, churn prediction [6](https://www.sciencedirect.com/science/article/abs/pii/S0957417415002845) • [7](https://www.sciencedirect.com/science/article/abs/pii/S0377221725005545) |\n\n## How it works\n\nThe decision-theoretic core is the minimum expected cost principle: given costs \\( C(i,j) \\) for predicting class \\( i \\) when the true class is \\( j \\), an example \\( x \\) should be assigned the class minimizing \\( R(i \\mid x) = \\sum_{j} P(j \\mid x) C(i,j) \\).[2](https://www.csd.uwo.ca/~xling/papers/cost_sensitive.pdf) For two classes, labeling a record positive when \\( C_{FP} \\cdot P_{-} < C_{FN} \\cdot P_{+} \\) is equivalent to thresholding the positive-class probability at \\( T_{cs} = C_{FP}/(C_{FP} + C_{FN}) \\), which reduces to 0.5 for equal costs.[1](https://link.springer.com/article/10.1007/s10618-021-00790-4) With calibrated probabilities and a known cost proportion \\( c \\), the cost-optimal threshold is \\( t = c_0/(c_0 + c_1) \\).[8](https://link.springer.com/article/10.1007/s10994-024-06634-8)\n\nA distinction matters throughout: cost-sensitive learning changes the training procedure, while cost-sensitive decision-making keeps an ordinary model and moves its decision threshold. Elkan argued that rebalancing the training data has little effect on Bayesian and decision tree learners, and recommended learning a classifier from the data as given and then computing optimal decisions explicitly from its probability estimates.[5](https://dl.acm.org/doi/10.5555/1642194.1642224)\n\n## How it is done\n\nFour mechanisms cover most of the literature [2](https://www.csd.uwo.ca/~xling/papers/cost_sensitive.pdf):\n\n- **Example weighting.** Each instance receives a weight proportional to its misclassification cost. Ting (1998) was the first to explicitly incorporate costs into the weights of the positive and negative classes used in weighted classifiers [1](https://link.springer.com/article/10.1007/s10618-021-00790-4); in C4.5 the weights enter the entropy calculation directly.[2](https://www.csd.uwo.ca/~xling/papers/cost_sensitive.pdf)\n- **Cost-proportional resampling.** A result often called the folk theorem states that altering the example distribution by a factor proportional to each example's relative cost makes any error-minimizing learner achieve expected cost minimization.[9](https://hunch.net/~jl/projects/reductions/costing/finalICDM2003.pdf) Elkan's theorem makes this precise for two classes: keeping all positive examples, the number of negative examples should be multiplied by \\( C_{FP}/C_{FN} \\).[2](https://www.csd.uwo.ca/~xling/papers/cost_sensitive.pdf) • [1](https://link.springer.com/article/10.1007/s10618-021-00790-4) Equivalently, thresholding at \\( T_{cs} \\) matches under-sampling negatives to \\( \\lvert N^- \\rvert \\cdot C_{FP}/C_{FN} \\) or over-sampling positives to \\( \\lvert N^+ \\rvert \\cdot C_{FN}/C_{FP} \\).[1](https://link.springer.com/article/10.1007/s10618-021-00790-4) Prior probabilities and costs are interchangeable: doubling \\( p(1) \\) has the same effect as doubling the false-negative cost or halving the false-positive cost.[2](https://www.csd.uwo.ca/~xling/papers/cost_sensitive.pdf)\n- **Threshold moving.** Apply the theoretical threshold \\( t^* \\) computed from the cost matrix, or select an empirical threshold from training data.[10](https://mlr.mlr-org.com/articles/tutorial/cost_sensitive_classif.html)\n- **Direct loss minimization.** Build the cost into the training objective itself, as in cost-sensitive logistic regression and cost-sensitive decision trees with cost-sensitive impurity measures.[6](https://www.sciencedirect.com/science/article/abs/pii/S0957417415002845) • [7](https://www.sciencedirect.com/science/article/abs/pii/S0377221725005545)\n\n## Origin\n\nWork on costs in classification gathered pace in the mid-1990s. Turney's ICET, a hybrid genetic decision tree induction algorithm, was published in the Journal of Artificial Intelligence Research in 1995.[3](https://doi.org/10.1613/jair.120) Ting and Zheng introduced cost-sensitive boosting of trees in 1998, and Bradford and colleagues studied pruning decision trees with misclassification costs the same year.[1](https://link.springer.com/article/10.1007/s10618-021-00790-4) Domingos presented MetaCost at KDD-99 as a general method for making classifiers cost-sensitive by wrapping a cost-minimizing procedure around an arbitrary error-based learner.[4](https://dl.acm.org/doi/10.1145/312129.312220) Elkan's \"The foundations of cost-sensitive learning\" (2001) supplied the decision-theoretic framework and the rebalancing theorem [5](https://dl.acm.org/doi/10.5555/1642194.1642224), and Costing formalized cost-proportionate weighting with theoretical guarantees.[9](https://hunch.net/~jl/projects/reductions/costing/finalICDM2003.pdf) A review credits Ting (1998) as the first to explicitly incorporate costs into example weights.[1](https://link.springer.com/article/10.1007/s10618-021-00790-4)\n\n## Variants\n\n**MetaCost** works in three steps: bootstrap classifiers to estimate \\( q(y,x) \\), relabel training examples by the Bayes-optimal decision under the cost matrix, and retrain a single classifier on the relabeled data.[11](https://www.csie.ntu.edu.tw/~htlin/task/doc/cs.mlss21.handout.pdf) It belongs to [Bayes risk](https://www.edgepedia.org/bayes-risk) minimization and requires probability estimation via bagging; the costing method avoids this by using cost-proportionate rejection sampling, accepting an example with probability \\( c/Z \\), plus aggregation.[9](https://hunch.net/~jl/projects/reductions/costing/finalICDM2003.pdf)\n\n**Cost-sensitive boosting.** Named variants include UBoost, AdaCost, AdaUBoost, Asymmetric AdaBoost, CSB0/CSB1/CSB2, AdaC1/C2/C3, and CSAB.[1](https://link.springer.com/article/10.1007/s10618-021-00790-4) Later, cost-sensitive extensions of AdaBoost, RealBoost, and LogitBoost outperformed the earlier proposals; only CSB2, AdaC2, and RealBoost with Platt calibration were competitive.[12](http://www.svcl.ucsd.edu/publications/journal/2010/PAMI_CostBoost/CostSensitiveBoosting.pdf) Published comparisons disagree on whether specialized boosting is needed at all: one line of work concludes cost-sensitive boosting consistently outperforms alternatives [12](http://www.svcl.ucsd.edu/publications/journal/2010/PAMI_CostBoost/CostSensitiveBoosting.pdf), while another holds that with calibrated probabilities the recommendation is original AdaBoost with a shifted decision threshold.[13](https://link.springer.com/article/10.1007/s10994-016-5572-x)\n\n**Trees and multiclass.** Example-dependent cost-sensitive decision trees use a cost-sensitive impurity measure and cost-dependent pruning; on credit card fraud, credit scoring, and direct marketing datasets they build significantly smaller trees in about a fifth of the time of a standard decision tree.[6](https://www.sciencedirect.com/science/article/abs/pii/S0957417415002845) For multiple classes, cost-sensitive one-versus-one (CSOVO) trains one binary classifier per class pair weighted by \\( \\lvert c_n[i] - c_n[j] \\rvert \\), with a guarantee that test cost is at most about twice that of the best binary tree decomposition.[11](https://www.csie.ntu.edu.tw/~htlin/task/doc/cs.mlss21.handout.pdf)\n\n## Applications\n\nCosts are set from the application's economics. In credit card fraud detection, the Bayes minimum risk (BMR) method decides from expected costs, with false-negative costs tied to the individual transaction amount.[6](https://www.sciencedirect.com/science/article/abs/pii/S0957417415002845) • [1](https://link.springer.com/article/10.1007/s10618-021-00790-4) Demonstrated application areas for example-dependent cost-sensitive methods include fraud detection, credit scoring, direct marketing, churn prediction, and gene expression-based classification.[6](https://www.sciencedirect.com/science/article/abs/pii/S0957417415002845) • [7](https://www.sciencedirect.com/science/article/abs/pii/S0377221725005545) The Statlog German credit dataset is a standard worked example: Elkan showed that in every economically reasonable cost matrix for that domain, both entries in the \"predict bad\" row must be equal.[5](https://dl.acm.org/doi/10.5555/1642194.1642224) ECSLR is an interpretable diverse ensemble of cost-sensitive logistic regression models optimizing cost savings, implemented in an R package.[7](https://www.sciencedirect.com/science/article/abs/pii/S0377221725005545)\n\n## Limitations and alternatives\n\nThree failure modes recur. First, mis-specified cost matrices: not every matrix is economically coherent, as the German credit example shows.[5](https://dl.acm.org/doi/10.5555/1642194.1642224) Second, probability miscalibration: theoretical thresholding is only reliable if predicted posterior probabilities are correct [10](https://mlr.mlr-org.com/articles/tutorial/cost_sensitive_classif.html), whereas empirical thresholding requires only an accurate ranking, not calibrated probabilities.[2](https://www.csd.uwo.ca/~xling/papers/cost_sensitive.pdf) • [10](https://mlr.mlr-org.com/articles/tutorial/cost_sensitive_classif.html) Tuning the threshold on the full training set can overfit, so resampling-based selection is advised.[10](https://mlr.mlr-org.com/articles/tutorial/cost_sensitive_classif.html) Third, resampling interacts with learner settings: with C4.5 defaults, over-sampling is surprisingly ineffective, often producing little or no change in performance, while under-sampling produces reasonable sensitivity to cost changes.[14](https://www.site.uottawa.ca/~cdrummon/pubs/ICML03.pdf)\n\nComparative evidence gives no universal answer. Across fourteen real-world datasets there was no clear winner between a cost-sensitive algorithm (C5.0), oversampling, and undersampling; on datasets with more than 10,000 examples the cost-sensitive algorithm consistently outperformed sampling, while oversampling appeared best for small datasets.[15](https://storm.cis.fordham.edu/gweiss/papers/dmin07-weiss.pdf)\n\nWhen costs are uncertain, one approach treats the expected misclassification cost under a cost distribution as a proper loss: if the false-positive cost is uniform on [0, 2] and the two costs sum to 2, the expected mean misclassification cost equals the [Brier score](https://www.edgepedia.org/brier-score).[8](https://link.springer.com/article/10.1007/s10994-024-06634-8) Modeling cost uncertainty with a [Beta distribution](https://www.edgepedia.org/beta-distribution) on the cost proportion yields a family of proper losses that empirically outperform cross-entropy and slightly beat Focal loss and label smoothing, with post-hoc temperature scaling calibration crucial.[8](https://link.springer.com/article/10.1007/s10994-024-06634-8)\n\n## References\n\n1. [Cost-sensitive ensemble learning: a unifying framework (Petrides & Verbeke, Data Mining and Knowledge Discovery, 2021)](https://link.springer.com/article/10.1007/s10618-021-00790-4)\n2. [Cost-Sensitive Learning and the Class Imbalance Problem (Ling & Sheng survey)](https://www.csd.uwo.ca/~xling/papers/cost_sensitive.pdf)\n3. [P. D. Turney (1995). Cost-Sensitive Classification: Empirical Evaluation of a Hybrid Genetic Decision Tree Induction Algorithm. Journal of Artificial Intelligence Research.](https://doi.org/10.1613/jair.120)\n4. [MetaCost: a general method for making classifiers cost-sensitive (Domingos, KDD 1999)](https://dl.acm.org/doi/10.1145/312129.312220)\n5. [The Foundations of Cost-Sensitive Learning (Elkan, IJCAI 2001)](https://dl.acm.org/doi/10.5555/1642194.1642224)\n6. [Example-dependent cost-sensitive decision trees (Expert Systems with Applications)](https://www.sciencedirect.com/science/article/abs/pii/S0957417415002845)\n7. [Diverse ensemble cost-sensitive logistic regression (European Journal of Operational Research, 2025)](https://www.sciencedirect.com/science/article/abs/pii/S0377221725005545)\n8. [Cost-sensitive classification with cost uncertainty: do we need surrogate losses? (Machine Learning, 2024)](https://link.springer.com/article/10.1007/s10994-024-06634-8)\n9. [Cost-Sensitive Learning by Cost-Proportionate Example Weighting (Zadrozny, Langford, Abe, ICDM 2003)](https://hunch.net/~jl/projects/reductions/costing/finalICDM2003.pdf)\n10. [Cost-Sensitive Classification • mlr (software documentation)](https://mlr.mlr-org.com/articles/tutorial/cost_sensitive_classif.html)\n11. [Cost-sensitive Classification: Techniques and Stories (H.-T. Lin lecture notes)](https://www.csie.ntu.edu.tw/~htlin/task/doc/cs.mlss21.handout.pdf)\n12. [Cost-Sensitive Boosting (Masnadi-Shirazi & Vasconcelos, IEEE TPAMI 2010)](http://www.svcl.ucsd.edu/publications/journal/2010/PAMI_CostBoost/CostSensitiveBoosting.pdf)\n13. [Cost-sensitive boosting algorithms: Do we really need them? | Machine Learning | Springer Nature Link](https://link.springer.com/article/10.1007/s10994-016-5572-x)\n14. [C4.5, Class Imbalance, and Cost Sensitivity: Why Under-Sampling beats Over-Sampling (Drummond & Holte, ICML 2003)](https://www.site.uottawa.ca/~cdrummon/pubs/ICML03.pdf)\n15. [Cost-Sensitive Learning vs. Sampling: Which is Best for Handling Unbalanced Classes with Unequal Error Costs? (Weiss, DM 2007)](https://storm.cis.fordham.edu/gweiss/papers/dmin07-weiss.pdf)\n\n---\n*Topic: Encyclopedia › Technology and the built world › Computing and digital systems › Artificial intelligence and data › Machine learning and neural computation › Machine learning methods › Supervised, unsupervised, and semi-supervised learning › Supervised learning concepts*\n\n*Initially written Sep 29, 2026 · Edited: Oct 15, 2026*\n\n*Copyright 2026 EdgeChat AI, a subsidiary of Biostate AI.*\n\nLicense: Edgepedia Community License 1.0, https://www.edgepedia.org/edgepedia/license\n", "same_as": [], "url": "https://www.edgepedia.org/cost-sensitive-learning", "markdown_url": "https://www.edgepedia.org/cost-sensitive-learning.md", "license": { "name": "Edgepedia Community License 1.0", "url": "https://www.edgepedia.org/edgepedia/license", "summary": "Free with credit, commercial use included. AI training is open to everyone. For other uses, organizations over USD 100M in revenue or 100M monthly users license separately.", "spdx": "LicenseRef-Edgepedia-Community-1.0" }, "credit": "\"Cost-sensitive learning\", Edgepedia (EdgeChat), https://www.edgepedia.org/cost-sensitive-learning. Edgepedia Community License 1.0.", "credit_md": "\"[Cost-sensitive learning](https://www.edgepedia.org/cost-sensitive-learning)\", Edgepedia (EdgeChat), [https://www.edgepedia.org/cost-sensitive-learning](https://www.edgepedia.org/cost-sensitive-learning). [Edgepedia Community License 1.0](https://www.edgepedia.org/edgepedia/license).", "credit_html": "\"Cost-sensitive learning\", Edgepedia (EdgeChat), https://www.edgepedia.org/cost-sensitive-learning. Edgepedia Community License 1.0.", "speakable": "Cost-sensitive learning is a machine learning approach that incorporates unequal misclassification costs into model training or prediction, using weighting, resampling, or threshold moving to minimize expected cost." }