ScienceGate Book Chapters

JOURNAL ARTICLE

Problematic Tokens: Tokenizer Bias in Large Language Models

Jin Yang Zhiqiang Wang Yanbin Lin Zunduo Zhao

Year: 2024 Pages: 6387-6393

DOI: 10.1109/bigdata62323.2024.10825615

Get Full-Text PDF Get Analytical Report

Keywords:

Computer science Natural language processing Language model

Metrics

2

Cited By

1.28

FWCI (Field Weighted Citation Impact)

8

Refs

0.80

Citation Normalized Percentile

Is in top 1%

Is in top 10%

Citation History

Topics

Natural Language Processing Techniques

Physical Sciences → Computer Science → Artificial Intelligence

Topic Modeling

Physical Sciences → Computer Science → Artificial Intelligence

Related Documents

JOURNAL ARTICLE

Acquiescence Bias in Large Language Models

Year: 2025 Pages: 11341-11355

JOURNAL ARTICLE

Evaluating Tokenizer Adaptation Methods for Large Language Models on Low-Resource Programming Languages

Georgy Andryushchenko Vladimir Ivanov

Year: 2025 Pages: 823-833

JOURNAL ARTICLE

Memory Tokens: Large Language Models Can Generate Reversible Sentence Embeddings

Ignacio Sastre Aiala Rosá

Year: 2025 Pages: 183-189

JOURNAL ARTICLE

Layered Bias: Interpreting Bias in Pretrained Large Language Models

Nirmalendu Prakash Roy Ka-Wei Lee

Year: 2023 Pages: 284-295

JOURNAL ARTICLE

Upcycling Candidate Tokens of Large Language Models for Query Expansion

Jinseok Kim Sukmin Cho Soyeong Jeong Sangyeop Kim Sungzoon Cho

Year: 2025 Pages: 4889-4894