Publications
Following publications have been announced by our department Regional Land and Atmosphere Modeling. For further information please contact the marked authors:
Geyer, B., Ludwig, T., & von Storch, H. (2021): Limits of reproducibility and hydrodynamic noise in atmospheric regional modelling. Commun Earth Environ 2, 17, doi:10.1038/s43247-020-00085-4
Abstract:
Reproducibility of research results is a fundamental quality criterion in science; thus, computer architecture effects on simulation results must be determined. Here, we investigate whether an ensemble of runs of a regional climate model with the same code on different computer platforms generates the same sequences of similar and dissimilar weather streams when noise is seeded using different initial states of the atmosphere. Both ensembles were produced using a regional climate model named COSMO-CLM5.0 model with ERA-Interim forcing. Divergent phase timing was dependent on the dynamic state of the atmosphere and was not affected by noise seeded by changing computers or initial model state variations. Bitwise reproducibility of numerical results is possible with such models only if everything is fixed (i.e., computer, compiler, chosen options, boundary values, and initial conditions) and the order of mathematical operations is unchanged between program runs; otherwise, at best, statistically identical simulation results can be expected.
Menard, C.B., Essery, R., Krinner, G., Arduini, G., Bartlett, P., Boone, A., Brutel-Vuilmet, C., Burke, E., Cuntz, M., Dai, Y., Decharme, B., Dutra, E., Fang, X., Fierz, C., Gusev, Y., Hagemann, S., Haverd, V., Kim, H., Lafaysse, M., Marke, T., Nasonova, O., Nitta, T., Niwano, M., Pomeroy, J., Schädler, G., Semenov, V.A., Smirnova, T., Strasser, U., Swenson, S., Turkov, D., Wever, N., & Yuan, H. (2021): Scientific and Human Errors in a Snow Model Intercomparison. Bulletin of the American Meteorological Society, 102(1), E61-E79, doi:10.1175/BAMS-D-19-0329.1
Abstract:
Twenty-seven models participated in the Earth System Model–Snow Model Intercomparison Project (ESM-SnowMIP), the most data-rich MIP dedicated to snow modeling. Our findings do not support the hypothesis advanced by previous snow MIPs: evaluating models against more variables and providing evaluation datasets extended temporally and spatially does not facilitate identification of key new processes requiring improvement to model snow mass and energy budgets, even at point scales. In fact, the same modeling issues identified by previous snow MIPs arose: albedo is a major source of uncertainty, surface exchange parameterizations are problematic, and individual model performance is inconsistent. This lack of progress is attributed partly to the large number of human errors that led to anomalous model behavior and to numerous resubmissions. It is unclear how widespread such errors are in our field and others; dedicated time and resources will be needed to tackle this issue to prevent highly sophisticated models and their research outputs from being vulnerable because of avoidable human mistakes. The design of and the data available to successive snow MIPs were also questioned. Evaluation of models against bulk snow properties was found to be sufficient for some but inappropriate for more complex snow models whose skills at simulating internal snow properties remained untested. Discussions between the authors of this paper on the purpose of MIPs revealed varied, and sometimes contradictory, motivations behind their participation. These findings started a collaborative effort to adapt future snow MIPs to respond to the diverse needs of the community.
