GENERALIZATION IN RELATION-AWARE TRANSFORMERS FOR SEMANTIC PARSING

Abstract

In recent years, the advancement of natural language processing has significantly influenced the development of semantic parsing, particularly in the transformation of natural language queries into SQL commands. This thesis investigates various approaches to improve generalization in relation-aware transformers for text-to-SQL tasks.Chapter 1 presents the methodology for addressing the challenges of semantic parsing, outlining the specific problems encountered in current models and articulating the central hypothesis: the introduction of additional processing mechanisms for relations in RAT-SQL can lead to improved model performance and generalizability in text-to-SQL tasks by enabling the model to capture more complex and contextual relationships between tokens. Chapter 2 presents the background necessary for understanding the evolution of NLP models and their applications. In Chapter 3, a comprehensive literature review highlights the state-of-the-art methodologies and frameworks in the field of text-to-SQL, emphasizing the need for enhanced relation processing.Chapter 4 discusses the Light RAT-SQL model, which effectively reduces the number of pre-existing relations from over 50 to 7, presenting the first algorithm designed to specialize these relations. This reduction aims to simplify the model while maintaining high parsing accuracy.In Chapter 5, we introduce the Topological Relation-Aware Transformer (T-RAT), a novel approach that leverages topological structures to enhance the understanding of relationships between input tokens, leading to improved accuracy in text-to-SQL generation.Chapter 6 delves into the Spectral Relation-Aware Transformer (S-RAT), which incorporates spectral position encoding. This additional layer of processing aims to create more robust relation embeddings, capturing complex and contextual relationships that improve the model's performance and generalizability. Chapter 7 concludes the thesis by reiterating our hypothesis: the introduction of additional processing mechanisms for relations in RAT-SQL can lead to improved model performance and generalizability in text-to-SQL tasks. The findings indicate that by enhancing relation processing, we can significantly advance the capabilities of relation-aware transformers, paving the way for more effective semantic parsing solutions.