Neural Architecture Search for Parameter-Efficient Fine-tuning of Large Pre-trained Language Models
Abstract
Parameter-efficient tuning (PET) methods fit pre-trained language models (PLMs) to downstream tasks by either computing a small compressed update for a subset of model parameters, or appending and fine-tuning a small number of new model parameters to the pre-trained network. Hand-designed PET architectures from the literature perform well in practice, but have the potential to be improved via automated neural architecture search (NAS). We propose an efficient NAS method for learning PET architectures via structured and unstructured pruning. We present experiments on GLUE demonstrating the effectiveness of our algorithm and discuss how PET architectural design choices affect performance in practice.
Metrics
Citation History
Topics
Related Documents
Parameter-efficient fine-tuning of large-scale pre-trained language models
Ning DingYujia QinGuang YangFuchao WeiZonghan YangYusheng SuShengding HuYulin ChenChi-Min ChanWeize ChenJing YiWeilin ZhaoXiaozhi WangZhiyuan LiuHai-Tao ZhengJianfei ChenYang LiuJie TangJuanzi LiMaosong Sun
Parameter-Efficient Fine-Tuning of Pre-trained Large Language Models for Financial Text Analysis
Kelly LangaHairong WangOlaperi Okuboyejo
Debiasing Pre-Trained Language Models via Efficient Fine-Tuning
Michael GiraRuisu ZhangKangwook Lee
Knowledge Distillation Framework of Pre-Trained Language Models Combined with Parameter Efficient Fine-Tuning
Point-PEFT: Parameter-Efficient Fine-Tuning for 3D Pre-trained Models
Yiwen TangRay ZhangZoey GuoXianzheng MaBin ZhaoZhigang WangDong WangXuelong Li