Spatial machine-learning model diagnostics: a model-agnostic distance-based approach

Abstract

While significant progress has been made towards explaining black-box\nmachine-learning (ML) models, there is still a distinct lack of diagnostic\ntools that elucidate the spatial behaviour of ML models in terms of predictive\nskill and variable importance. This contribution proposes spatial prediction\nerror profiles (SPEPs) and spatial variable importance profiles (SVIPs) as\nnovel model-agnostic assessment and interpretation tools for spatial prediction\nmodels with a focus on prediction distance. Their suitability is demonstrated\nin two case studies representing a regionalization task in an\nenvironmental-science context, and a classification task from remotely-sensed\nland cover classification. In these case studies, the SPEPs and SVIPs of\ngeostatistical methods, linear models, random forest, and hybrid algorithms\nshow striking differences but also relevant similarities. Limitations of\nrelated cross-validation techniques are outlined, and the case is made that\nmodelers should focus their model assessment and interpretation on the intended\nspatial prediction horizon. The range of autocorrelation, in contrast, is not a\nsuitable criterion for defining spatial cross-validation test sets. The novel\ndiagnostic tools enrich the toolkit of spatial data science, and may improve ML\nmodel interpretation, selection, and design.\n