Length Extrapolation of Transformers: A Survey from the Perspective of Positional Encoding

Liang Zhao; Xiachong Feng; Xiaocheng Feng; Weihong Zhong; Dongliang Xu; Qing Yang; Hongtao Liu; Bing Qin; Tianxiang Liu

doi:10.18653/v1/2024.findings-emnlp.582

ScienceGate Book Chapters

JOURNAL ARTICLE

Length Extrapolation of Transformers: A Survey from the Perspective of Positional Encoding

Liang Zhao Xiachong Feng Xiaocheng Feng Weihong Zhong Dongliang Xu Qing Yang Hongtao Liu Bing Qin Tianxiang Liu

Year: 2024 Pages: 9959-9977

DOI: 10.18653/v1/2024.findings-emnlp.582

Get Full-Text PDF Get Analytical Report

Keywords:

Extrapolation Encoding (memory) Computer science Transformer Perspective (graphical) Artificial intelligence Mathematics Statistics Engineering Electrical engineering Voltage

Metrics

Cited By

3.60

FWCI (Field Weighted Citation Impact)

Refs

0.88

Citation Normalized Percentile

Is in top 1%

Is in top 10%

Citation History

Topics

Surface Roughness and Optical Measurements

Physical Sciences → Engineering → Computational Mechanics

Industrial Vision Systems and Defect Detection

Physical Sciences → Engineering → Industrial and Manufacturing Engineering

Blind Source Separation Techniques

Physical Sciences → Computer Science → Signal Processing

Length Extrapolation of Transformers: A Survey from the Perspective of Positional Encoding

Metrics

Citation History

Topics

Related Documents

DAPE: Data-Adaptive Positional Encoding for Length Extrapolation

POSITIONAL ENCODING FOR TRANSFORMERS

Applying Positional Encoding to Enhance Vision-Language Transformers

A Simple and Effective Positional Encoding for Transformers

Randomized Positional Encodings Boost Length Generalization of Transformers