Choosing the Right Model

Step 1 - Fit the Hardware

A model needs roughly 1 GB of RAM or VRAM per billion parameters at full precision, less when quantized. Start with a small model, then move up until quality or speed stops improving.

• 1-3B - Laptops, quick tasks, autocomplete-style work.
• 7-8B - Everyday chat and coding with 8-16 GB RAM.
• 14B and up - Desktops with a good GPU or lots of RAM.

Step 2 - Consider Quantization

Quantization stores weights in fewer bits. A q4 model runs in about half the memory of full precision with a small quality dip, which is usually the best tradeoff for local use.

Step 3 - Match the Task

• General chat - llama3.2 or qwen2.5.
• Code - A code-focused model or a strong general one.
• Reasoning - A model flagged for reasoning steps.
• Embeddings - nomic-embed-text.

Step 4 - Measure, Don't Guess

Watch tokens per second and answer quality on your real prompts. Model rankings shift quickly, so re-test when new tags appear.

Key Points

• Fit the model size to your RAM or VRAM.
• Quantized q4 weights halve memory use.
• Pick a model per task, not one model for everything.
• Measure speed and quality on your own prompts.