ReAugKD: Retrieval-Augmented Knowledge Distillation For Pre-trained Language Models

Jianyi Zhang; Aashiq Muhamed; Aditya Anantharaman; Guoyin Wang; Changyou Chen; Kai Zhong; Qingjun Cui; Yi Xu; Belinda Zeng; Trishul Chilimbi; Yiran Chen

doi:10.18653/v1/2023.acl-short.97

JOURNAL ARTICLE

ReAugKD: Retrieval-Augmented Knowledge Distillation For Pre-trained Language Models

Jianyi Zhang Aashiq Muhamed Aditya Anantharaman Guoyin Wang Changyou Chen Kai Zhong Qingjun Cui Yi Xu Belinda Zeng Trishul Chilimbi Yiran Chen

Year: 2023 Pages: 1128-1136

DOI: 10.18653/v1/2023.acl-short.97

Get Full-Text PDF Get Analytical Report

Abstract

Jianyi Zhang, Aashiq Muhamed, Aditya Anantharaman, Guoyin Wang, Changyou Chen, Kai Zhong, Qingjun Cui, Yi Xu, Belinda Zeng, Trishul Chilimbi, Yiran Chen. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). 2023.

Keywords:

Chen Zhàng Computer science Natural language processing Association (psychology) Artificial intelligence Volume (thermodynamics) Distillation Philosophy Chemistry History Epistemology Physics Chromatography Archaeology Geology

Metrics

Cited By

2.81

FWCI (Field Weighted Citation Impact)

Refs

0.89

Citation Normalized Percentile

Is in top 1%

Is in top 10%

Citation History

Topics

Topic Modeling

Physical Sciences → Computer Science → Artificial Intelligence

Natural Language Processing Techniques

Physical Sciences → Computer Science → Artificial Intelligence

ReAugKD: Retrieval-Augmented Knowledge Distillation For Pre-trained Language Models

Abstract

Metrics

Citation History

Topics

Related Documents

Dynamic Knowledge Distillation for Pre-trained Language Models

Dynamic Knowledge Distillation for Pre-trained Language Models

Dynamic Knowledge Distillation for Pre-trained Language Models

Knowledge Base Grounded Pre-trained Language Models via Distillation

Improved Knowledge Distillation for Pre-trained Language Models via Knowledge Selection