Today's fact21 November Archive

15×

biggest gap found between languages

Not every language costs the same in tokens

Chatbots read text as tokens, small chunks of words. Tokenizers are built mostly from English text, so the same sentence in another language often splits into more tokens. A 2023 study found gaps of up to 15 times between languages. More tokens means higher cost and less room in the AI’s context window.

Source: Petrov et al., arXiv (2023)