Time Series Forecasting using Sequence Models with Attention

Abstract

Due to the growing penetration of renewable energy sources, accurate energy forecasts are required to support their effective integration. In this field, deep learning methods are currently demonstrating successful results, but there is still a room for improvement that may eventually lead to their wide deployment. We suggest that sequence processing applications from the field of natural language processing (NLP) can be adopted to forecasting. Some recent advances in deep learning that brought NLP to near human performance include sequence models and an attention mechanism. Considering that these methods apply well to univariate time series such as language, they can be potentially extended to multivariate time series forecasting. This may be applicable, for instance, for prediction of photovoltaic (PV) power generation with added weather features it depends on. Current approaches popular in PV forecasting include statistical and machine learning methods as well as basic deep learning models such as feedforward neural networks and recurrent long short-term memory based architectures. Although deep learning has claimed some success in this field, there has been no wide adoption of sequence to sequence models yet. This insight inspired an exploration of adapting some NLP techniques to multivariate time series forecasting. We propose a sequence to sequence architecture with attention as a model superior with respect to the baseline architectures. Overall, this thesis recommends an adoption of sequence attention models in PV generation forecasting and validates this proposal by improving the quality of the forecasts. The model leverages the high resolution multivariate signal by extracting features from the numerical weather predictions and historical information to produce a binned probabilistic forecast. Sequence to sequence models benefit from a more expressive probabilistic forecast due to their recursive structure. The attention mechanism further aids context extraction. The proposed sequence to sequence model with attention outperforms common models, such as long short-term memory networks and a classic sequence attention model, for photovoltaic generation forecasting. k-fold cross-validation provides additional insights on the influence of a dataset's arrangement on the model’s performance and confirms the validity of the design decisions and architecture choices of the proposed model.