气候变化领域集成服务门户(CCPortal): Stable reliability diagrams for probabilistic classifiers

CCPortal

DOI	10.1073/pnas.2016191118
	Stable reliability diagrams for probabilistic classifiers
	Dimitriadis T.; Gneiting T.; Jordan A.I.
发表日期	2021
ISSN	00278424
卷号	118 期号:8
英文摘要	A probability forecast or probabilistic classifier is reliable or calibrated if the predicted probabilities are matched by ex post observed frequencies, as examined visually in reliability diagrams. The classical binning and counting approach to plotting reliability diagrams has been hampered by a lack of stability under unavoidable, ad hoc implementation decisions. Here, we introduce the CORP approach, which generates provably statistically consistent, optimally binned, and reproducible reliability diagrams in an automated way. CORP is based on nonparametric isotonic regression and implemented via the pool-adjacent-violators (PAV) algorithm—essentially, the CORP reliability diagram shows the graph of the PAV-(re)calibrated forecast probabilities. The CORP approach allows for uncertainty quantification via either resampling techniques or asymptotic theory, furnishes a numerical measure of miscalibration, and provides a CORP-based Brier-score decomposition that generalizes to any proper scoring rule. We anticipate that judicious uses of the PAV algorithm yield improved tools for diagnostics and inference for a very wide range of statistical and machine learning methods. © 2021 National Academy of Sciences. All rights reserved.
英文关键词	Calibration; Discrimination ability; Probability forecast; Score decomposition; Weather prediction
语种	英语
scopus关键词	algorithm; article; calibration; classifier; decomposition; prediction; probability; reliability; uncertainty; weather
来源期刊	Proceedings of the National Academy of Sciences of the United States of America
文献类型	期刊论文
条目标识符	http://gcip.llas.ac.cn/handle/2XKMVOVA/180570
作者单位	Alfred Weber Institute of Economics, Heidelberg University, Heidelberg, 69115, Germany; Computational Statistics Group, Heidelberg Institute for Theoretical Studies, Heidelberg, 69118, Germany; Institute for Stochastics, Karlsruhe Institute of Technology, Karlsruhe, 76131, Germany
推荐引用方式 GB/T 7714	Dimitriadis T.,Gneiting T.,Jordan A.I.. Stable reliability diagrams for probabilistic classifiers[J],2021,118(8).
APA	Dimitriadis T.,Gneiting T.,&Jordan A.I..(2021).Stable reliability diagrams for probabilistic classifiers.Proceedings of the National Academy of Sciences of the United States of America,118(8).
MLA	Dimitriadis T.,et al."Stable reliability diagrams for probabilistic classifiers".Proceedings of the National Academy of Sciences of the United States of America 118.8(2021).