K-Culture GlossaryㅌTechnical words in the news
Tokenizer
The component that splits text into tokens. Because the cutting pattern differs by language, it's at the heart of the debate over Korean-language AI costs.
In plain words
A tokenizer is the part that chops text into token pieces a model can actually consume. Exactly where and how it cuts is designed differently for each model.
What matters is that this cutting is more or less efficient depending on the language — the same content often ends up as more tokens in Korean than in English, which is the root of the argument that "users get charged more for saying the same thing in Korean." News about Korean-optimized models or improved tokenizers is usually about reducing this inefficiency.
See also
Stories using this term
No story has used this term yet. New ones attach here automatically.