Efficient parallel viterbi algorithm for big data in a spark cloud computing environment

Abstract

The aim of this paper is to present a parallel distributed version of Viterbi algorithm that combines the advantages of Spark, the big data framework, and hidden Markov models to solve the decoding problem for large scale multidimensional data. The scope of the paper includes a review of hidden Markov models, a study of decoding problem, a presentation of related work, and a discussion of previously proposed implementations. The main part of the paper consists of a description of development and implementation of a parallel distributed Viterbi algorithm in a cloud computing environment, followed by a description of evaluation experiments of the presented algorithm. The results showed that the proposed algorithm is faster, with high scalability and no deterioration in forecast accuracy is observed.