Thinking in Tokens and Choosing a Model
Thinking in Tokens and Choosing a Model
Tokens are the unit the model prices and counts. One English word is roughly one and a half tokens, but that varies wildly by language and content. Understanding tokens explains why long prompts cost more and why some outputs get cut mid-sentence.
Estimate Tokens Before You Build
Budget the context: system prompt, conversation history, and the answer you need. A classic failure is feeding an entire book and asking for a summary, then watching the input bill exceed the value of the summary. Trim history and summarize old turns instead.
Count Tokens in Python
import tiktoken
enc = tiktoken.encoding_for_model("gpt-4o")
text = "Hello world, this is a token demo"
print(len(enc.encode(text)))The tiktoken library shows the exact count for a model. Use it to cap prompt length before making the call instead of paying for it after.
Model Families and Trade-offs
The platform offers several model families. Small models are cheap, fast, and fine for classification, extraction, and short answers. Large models handle complex reasoning, long documents, and tricky code but cost more and respond slower. Pick the smallest model that passes your test cases; revisit the choice as your workload changes.
Sampling Makes Outputs Non-Deterministic
Models pick the next token from a probability distribution, so identical prompts can yield different answers. Lower the temperature to make output steadier, or fix a seed for reproducible testing. For anything that matters, assert on structure and behavior, not on exact text.
Key Points
- Tokens drive cost and context limits.
- Trim the history you send to the API.
- Use the smallest capable model.
- Test on structure and behavior, not exact strings.
