What is Token?
Also called LLM token
Definition
A token is a unit of content that a language model processes. For text, tokens can represent words, word fragments, or other character sequences, and the mapping depends on the model’s tokenizer.
The operation of Token is explained.
Token counts determine how much material fits within model limits and contribute to usage accounting. They are not interchangeable with word counts. Anthropic provides token-counting tools for estimating the size of model inputs.
The relevance of Token is explained.
A retrieval application for SaaS documentation must fit instructions, retrieved evidence, and conversation history into its available context. Redundant text consumes space that could otherwise hold relevant implementation details.
The evaluation of Token is explained.
Token estimates use the tokenizer or counting endpoint for the deployed model. Shortened content retains necessary caveats because minimizing tokens at the expense of meaning can make an answer less useful.
Frequently asked questions about Token
Is one token always one word?
A token can be a word fragment or another content unit rather than a complete word.
Do models tokenize identically?
Tokenization can differ across models and tokenizer versions.
Are token counts a visibility metric?
Token counts measure input or output size, not whether a brand appears in AI answers.