Introduction
Empirical models are based on strong statistical relationships existing between two or more physical properties, usually with the goal of forecasting some physical property into the future based on other related observations. The benefits of empirical models are that, in many cases, the results of simple statistically-based models are often as robust as those obtained from much more complex, physically-based models (Elsner and Schmermann, 1994).
For example, a common empirical model, the positive degree day model, is based on the strong statistical relationships between measured air temperature on a glacier and the rate of melt of the snow or ice surface (Braithwaite, 1995). As air temperature and surface melt are both the integrated result of the total surface energy balance, the success of temperature-based models is not surprising. Temperature-based methods are frequently employed because temperature data are easily obtained, the parameterization schemes are simple yet relatively accurate, and temperature data are easily interpolated between sites (Ohmura, 2001).
When creating a statistical or empirical model from observations or from a set of historical data, it is important to accurately understand how well the model performs in forecast mode. Determining the true forecast skill is one of the most critical aspects of developing such a model. Some of the common techniques for assessing forecast skill, such as the standard regression skill estimate, are systematically biased towards higher skills. One method, which is less biased than the hindcast skill estimates that are typically used, is the cross-validation method (Michaelson, 1987).
Overview of the cross-validation method
The cross-validation method is a statistical procedure for accurately assessing the forecast skill of empirical models. In cross-validation, cases from the dataset are systematically deleted, a new forecast model is derived from the remaining data, and then that model is used to “hindcast” the deleted data (Elsner and Schmermann, 1994). One strength of this method is the ability to diagnose individual datum which are especially influential on the results (Michaelson, 1987).
Potential pitfalls
To accurately assess model skill with the cross-validation method, there are three important conditions which must be met. First, hindcasts must be made using only out-of-sample data. Using in-sample data underestimates the model prediction error by only considering the fit of the prediction rule to the data. Secondly, the prediction rule for each hindcast should be made from only the remaining data and not include the datum that is being predicted. This is an essential condition of true cross-validation. Finally, there may be no statistical auto-correlation, or serial correlation, between the omitted data and the remaining data used for the prediction.
Concluding remarks
Models provide an ideal testing ground for our understanding of important physical processes (Cassano et al., 2001). They range widely in their complexity and application. Empirical models are becoming more common as the algorithms and technology to implement them are becoming more readily available. Accordingly, consistent and accurate means of assessing these models need to be established.
Cassano, J. J., et al. (2001), Evaluation of Polar MM5 simulation of Greenland's atmospheric circulation, Journal of Geophysical Research, 106, 33,867-833,889.
Elsner, J. B., and C. P. Schmertmann (1994), Assessing forecast skill through cross validation, Weather and Forecasting, 9, 619-624.
Michaelson, J. (1987), Cross-validation in statistical climate forecast models, Journal of Climate and Applied Meteorology, 26, 1589-1600.
Ohmura, A. (2001), Physical basis for the temperature-based melt-index method, Journal of Applied Meteorology, 40, 753-761.