Length Generalization of Causal Transformers without Position Encoding

Jie Wang; Tao Ji; Yuanbin Wu; Hang Yan; Tao Gui; Qi Zhang; Xuanjing Huang; Xiaoling Wang

doi:10.18653/v1/2024.findings-acl.834

ScienceGate Book Chapters

JOURNAL ARTICLE

Length Generalization of Causal Transformers without Position Encoding

Jie Wang Tao Ji Yuanbin Wu Hang Yan Tao Gui Qi Zhang Xuanjing Huang Xiaoling Wang

Year: 2024 Pages: 14024-14040

DOI: 10.18653/v1/2024.findings-acl.834

Get Full-Text PDF Get Analytical Report

Keywords:

Encoding (memory) Computer science Transformer Generalization Position (finance) Position paper Artificial intelligence Mathematics Electrical engineering Engineering Voltage

Metrics

Cited By

0.71

FWCI (Field Weighted Citation Impact)

Refs

0.59

Citation Normalized Percentile

Is in top 1%

Is in top 10%

Citation History

Topics

Blind Source Separation Techniques

Physical Sciences → Computer Science → Signal Processing

Length Generalization of Causal Transformers without Position Encoding

Metrics

Citation History

Topics

Related Documents

Position Coupling: Improving Length Generalization of Arithmetic Transformers Using Task Structure

Randomized Positional Encodings Boost Length Generalization of Transformers

Learning Syntax Without Planting Trees: Understanding Hierarchical Generalization in Transformers

Improving position encoding of transformers for multivariate time series classification

Causal Domain Generalization