气候变化领域集成服务门户(CCPortal): Relative model score: a scoring rule for evaluating ensemble simulations with application to microbial soil respiration modeling

CCPortal

DOI	10.1007/s00477-018-1592-3
	Relative model score: a scoring rule for evaluating ensemble simulations with application to microbial soil respiration modeling
	Elshall, Ahmed S.; Ye, Ming; Pei, Yongzhen; Zhang, Fan; Niu, Guo-Yue; Barron-Gafford, Greg A.
通讯作者	Ye, M (通讯作者)
发表日期	2018
ISSN	1436-3240
EISSN	1436-3259
起始页码	2809
结束页码	2819
卷号	32 期号:10
英文摘要	This paper defines a new scoring rule, namely relative model score (RMS), for evaluating ensemble simulations of environmental models. RMS implicitly incorporates the measures of ensemble mean accuracy, prediction interval precision, and prediction interval reliability for evaluating the overall model predictive performance. RMS is numerically evaluated from the probability density functions of ensemble simulations given by individual models or several models via model averaging. We demonstrate the advantages of using RMS through an example of soil respiration modeling. The example considers two alternative models with different fidelity, and for each model Bayesian inverse modeling is conducted using two different likelihood functions. This gives four single-model ensembles of model simulations. For each likelihood function, Bayesian model averaging is applied to the ensemble simulations of the two models, resulting in two multi-model prediction ensembles. Predictive performance for these ensembles is evaluated using various scoring rules. Results show that RMS outperforms the commonly used scoring rules of log-score, pseudo Bayes factor based on Bayesian model evidence (BME), and continuous ranked probability score (CRPS). RMS avoids the problem of rounding error specific to log-score. Being applicable to any likelihood functions, RMS has broader applicability than BME that is only applicable to the same likelihood function of multiple models. By directly considering the relative score of candidate models at each cross-validation datum, RMS results in more plausible model ranking than CRPS. Therefore, RMS is considered as a robust scoring rule for evaluating predictive performance of single-model and multi-model prediction ensembles.
关键词	MARGINAL LIKELIHOOD DECOMPOSITION UNCERTAINTY MULTIMODEL RELIABILITY MECHANISMS PREDICTION SELECTION IMPROVE
英文关键词	Scoring rule; Continuous ranked probability score; Bayes factor; Log-score; Dispersion; Reliability
语种	英语
WOS研究方向	Engineering ; Environmental Sciences & Ecology ; Mathematics ; Water Resources
WOS类目	Engineering, Environmental ; Engineering, Civil ; Environmental Sciences ; Statistics & Probability ; Water Resources
WOS记录号	WOS:000446064000003
来源期刊	STOCHASTIC ENVIRONMENTAL RESEARCH AND RISK ASSESSMENT
来源机构	中国科学院青藏高原研究所
文献类型	期刊论文
条目标识符	http://gcip.llas.ac.cn/handle/2XKMVOVA/259330
推荐引用方式 GB/T 7714	Elshall, Ahmed S.,Ye, Ming,Pei, Yongzhen,et al. Relative model score: a scoring rule for evaluating ensemble simulations with application to microbial soil respiration modeling[J]. 中国科学院青藏高原研究所,2018,32(10).
APA	Elshall, Ahmed S.,Ye, Ming,Pei, Yongzhen,Zhang, Fan,Niu, Guo-Yue,&Barron-Gafford, Greg A..(2018).Relative model score: a scoring rule for evaluating ensemble simulations with application to microbial soil respiration modeling.STOCHASTIC ENVIRONMENTAL RESEARCH AND RISK ASSESSMENT,32(10).
MLA	Elshall, Ahmed S.,et al."Relative model score: a scoring rule for evaluating ensemble simulations with application to microbial soil respiration modeling".STOCHASTIC ENVIRONMENTAL RESEARCH AND RISK ASSESSMENT 32.10(2018).