Context-based Multi-stage Offline Handwritten Mathematical Symbol Recognition using Deep Learning

Abstract

We propose a context-based multi-stage machine learning (ML) architecture for offline handwritten mathematical symbol recognition. In the absence of context information, the first stage of the architecture acts as a generalized method of training a Multi-Column Deep Neural Network (MCDNN) model for isolated symbol recognition. The second stage trains a deep convolutional neural network that further classifies ambiguous symbols based on each symbols context information. To further improve the classification accuracy, we develop a set of rules in the third stage to classify ambiguity symbols that would avoid violating some mathematical syntax rules. The proposed method is evaluated using the Competition on Recognition of Online Handwritten Mathematical Expressions (CROHME) dataset. Experiments show that the proposed architecture outperforms all other previous approaches, and results the state-of-the-art accuracy on both the CROHME 2013 and 2016 datasets in offline handwritten mathematical symbol recognition. We believe the proposed multi-stage context-based ML architecture would have wide applications on handwritten symbol recognition.