Randomized Positional Encodings Boost Length Generalization of Transformers

Anian Ruoss; Grégoire Delétang; Tim Genewein; Jordi Grau-Moya; Róbert Csordás; Mehdi Bennani; Shane Legg; Joel Veness

doi:10.18653/v1/2023.acl-short.161

ScienceGate Book Chapters

JOURNAL ARTICLE

Randomized Positional Encodings Boost Length Generalization of Transformers

Anian Ruoss Grégoire Delétang Tim Genewein Jordi Grau-Moya Róbert Csordás Mehdi Bennani Shane Legg Joel Veness

Year: 2023 Pages: 1889-1903

DOI: 10.18653/v1/2023.acl-short.161

Get Full-Text PDF Get Analytical Report

Abstract

Anian Ruoss, Grégoire Delétang, Tim Genewein, Jordi Grau-Moya, Róbert Csordás, Mehdi Bennani, Shane Legg, Joel Veness. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). 2023.

Keywords:

Transformer Generalization Computer science Library science Artificial intelligence Mathematics Engineering Electrical engineering Mathematical analysis

Metrics

Cited By

3.83

FWCI (Field Weighted Citation Impact)

Refs

0.92

Citation Normalized Percentile

Is in top 1%

Is in top 10%

Citation History

Topics

Natural Language Processing Techniques

Physical Sciences → Computer Science → Artificial Intelligence

Topic Modeling

Physical Sciences → Computer Science → Artificial Intelligence

Speech Recognition and Synthesis

Physical Sciences → Computer Science → Artificial Intelligence

Randomized Positional Encodings Boost Length Generalization of Transformers

Abstract

Metrics

Citation History

Topics

Related Documents

Incorporating Noisy Length Constraints into Transformer with Length-aware Positional Encodings

Algebraic Positional Encodings

Why Are Positional Encodings Nonessential for Deep Autoregressive Transformers? A Petroglyph Revisited

WinStat: A Family of Trainable Positional Encodings for Transformers in Time Series Forecasting

Length Generalization of Causal Transformers without Position Encoding