Missing Values Imputation on Multivariate Time Series in the field of Agriculture

Abstract

Missing values are ubiquitous in time series data due to various reasons, such as sensor malfunctions and communication errors. Missing values can cause serious problems by introducing bias to the data and hindering analytics. Therefore, it is crucial to provide a sufficient strategy to handle missing data. Imputation is the process of replacing missing data with substituted values. However, there is no generic solution to treat every case of missingness, as imputation performance depends on the nature of the data. In this dissertation, we investigate several imputation techniques to recover missing values in multivariate time series in the field of agriculture. This study is implemented in a demonstrated smart farming scenario of the LoRa IoT network of the American Farm School of Thessaloniki, which consists of multiple nodes and gateways. The data used in this study were collected by the sensors of the IoT nodes, that measure environmental conditions such as air temperature, air humidity, air pressure, soil temperature and soil moisture. This study, summarizes the most popular imputation methods classified into different categories, but also points out a promising direction to address this problem with the use of deep learning models and more specifically the Transformers’ architecture. At first, we apply data profiling to unfold crucial characteristics of the considered time series and then multiple experiments are conducted considering two different case studies. In this first case, we treat each node individually as an isolated entity, while in the second case, each node is identified as a member of a larger neighborhood of adjacent nodes. We experiment with both of these cases, by investigating the performance of both baseline and state-of-the-art approaches regarding the metric of mean absolute error. Furthermore, we draw important conclusions by studying each variable separately in order to suggest the most sufficient recovery strategies depending on the nature of the data within each series.