15×
biggest gap found between languages
Not every language costs the same in tokens
Chatbots read text as tokens, small chunks of words. Tokenizers are built mostly from English text, so the same sentence in another language often splits into more tokens. A 2023 study found gaps of up to 15 times between languages. More tokens means higher cost and less room in the AI’s context window.
Source: Petrov et al., arXiv (2023)