气候变化领域集成服务门户(CCPortal): Impact of deep learning-based dropout on shallow neural networks applied to stream temperature modelling

CCPortal

DOI	10.1016/j.earscirev.2019.103076
	Impact of deep learning-based dropout on shallow neural networks applied to stream temperature modelling
	Piotrowski A.P.; Napiorkowski J.J.; Piotrowska A.E.
发表日期	2020
ISSN	0012-8795
卷号	201
英文摘要	Although deep learning applicability in various fields of earth sciences is rapidly increasing, shallow multilayer-perceptron neural networks remain widely used for regression problems. Despite many clear distinctions between deep and shallow neural networks, some techniques developed for deep learning may help improve shallow models. Dropout, a simple approach to avoid overfitting by randomly skipping some nodes in a net during each training iteration, is among methodological features that made deep learning networks successful. In this study we give a review of dropout methods and empirically show that, when used together with early-stopping, dropout and its variant dropconnect could improve performance of shallow multi-layer perceptron neural networks. Shallow neural networks are applied to streamwater temperature modelling problems in six catchments, based on air temperature, river discharge and declination of the Sun. We found that when training of a particular neural network architecture that includes at least a few hidden nodes is repeated many times, dropout reduces the number of models that perform poorly on testing data, and hence improves the mean performance. If the number of inputs or hidden nodes is very low, dropout only disturbs training. However, nodes need to be dropped out with a much lower probability than in the case of deep neural networks (about 1%, instead of 10–50% for deep learning), due to a much smaller number of nodes in the network. Larger probabilities of dropping out nodes hinder convergence of the training algorithm and lead to poor results for both calibration and testing data. Dropconnect turned out to be slightly more effective than dropout. © 2019 The Authors
英文关键词	Atmosphere-hydrosphere interactions; Deep learning; Dropout; Shallow artificial neural networks; Stream temperature modelling
语种	英语
scopus关键词	artificial neural network; atmosphere-hydrosphere interaction; machine learning; regression analysis; river discharge; streamwater; temperature gradient
来源期刊	EARTH-SCIENCE REVIEWS
文献类型	期刊论文
条目标识符	http://gcip.llas.ac.cn/handle/2XKMVOVA/209724
作者单位	Institute of Geophysics, Polish Academy of Sciences, Ks. Janusza 64, Warsaw, 01-452, Poland; Faculty of Polish Studies, University of Warsaw, Krakowskie Przedmiescie 26/28, Warsaw, 00-927, Poland
推荐引用方式 GB/T 7714	Piotrowski A.P.,Napiorkowski J.J.,Piotrowska A.E.. Impact of deep learning-based dropout on shallow neural networks applied to stream temperature modelling[J],2020,201.
APA	Piotrowski A.P.,Napiorkowski J.J.,&Piotrowska A.E..(2020).Impact of deep learning-based dropout on shallow neural networks applied to stream temperature modelling.EARTH-SCIENCE REVIEWS,201.
MLA	Piotrowski A.P.,et al."Impact of deep learning-based dropout on shallow neural networks applied to stream temperature modelling".EARTH-SCIENCE REVIEWS 201(2020).