Knowledge-enhanced natural language processing

Abstract

Language is a fundamental part of being human. It allows people to communicate thoughts, express ideas, create memories, and build mutual comprehension. Developing machines capable of understanding language automatically has been a long-standing dream in artificial intelligence. Over the past decade, dramatic and unprecedented breakthroughs in natural language processing have been driven by transformer-based networks scaled on progressively larger parameter counts and training data. Large language models have recently become prominent, revealing early signs of general intelligence and ushering in a new era of human-machine interaction. However, although pre-trained language models could effectively store numerous facts encountered at training time, their knowledge awareness is still far from satisfactory. Current models grapple with lexical superficiality, human reporting bias, hallucination, poor reasoning, black-box behavior, and computational complexity. These open issues have led researchers to question the adequacy of scaling as the sole condition for scientific progress. A growing number of influential thinkers recognize that the key direction for substantial advancements lies in the incorporation of semantic-level abstractions and pertinent context derived from external knowledge sources. In this dissertation, we study knowledge-enhanced natural language processing, presenting multiple techniques for knowledge extraction, representation, and injection. We show that semantic parsing, graph representation learning, retrieval augmented generation, multi-modal signals, and back-propagation through combinatorial solvers are integral components of the solution. Overall, this work represents a significant step toward delineating a new generation of language models that prioritize meaning, reasoning, and interpretability.