Choosing Models & Tools (a decision framework)
The AI landscape changes monthly. New models, new tools, new leaderboards. If you learn "use tool X, model Y," your knowledge expires the moment they're dethroned. So this lesson teaches something durable: how to choose, not what to use.
1️⃣ Pick the right kind of model: fast vs. reasoning
Modern providers ship (at least) two flavors of model, and picking the wrong one wastes time or money:
| Fast models | Reasoning models | |
|---|---|---|
| Good for | Autocomplete, quick edits, boilerplate, simple Q&A, high-volume calls | Hard bugs, architecture, multi-step refactors, tricky algorithms |
| Latency | Low | Higher (they "think" first) |
| Cost | Cheap | More expensive |
| Prompting | Benefits from explicit steps | Needs less hand-holding — they reason internally, so heavy "think step by step" scaffolding is often redundant |
💡 Rule of thumb: reach for a fast model by default; escalate to a reasoning model when a task is genuinely hard or has failed once. Don't pay reasoning prices to rename a variable.
When building AI into a product, default to the latest, most capable models from your provider and downgrade only if cost/latency demands it — capability improves fast, and yesterday's flagship is often today's mid-tier price.
2️⃣ Pick the right kind of tool: IDE vs. CLI vs. cloud
The delivery form matters as much as the model:
- 🖥️ IDE assistants/agents (Cursor, Copilot, Windsurf, Cline) — tight feedback loop, you watch every diff. Best for interactive, in-flow work.
- ⌨️ CLI agents (Claude Code, Codex CLI, Gemini CLI, Aider) — scriptable, composable, great for terminal-native workflows, CI, and running several in parallel (e.g. via git worktrees).
- ☁️ Cloud/async agents (Cursor background, Copilot coding agent, Codex cloud tasks) — fire-and-forget long jobs off your critical path. Powerful, but the review burden is higher, not lower.
Most teams run several at once. That's expected — they share the same primitives, so switching is cheap once you think in categories.
3️⃣ Weigh the trade-offs that actually bite
For any choice, sanity-check four axes:
- Cost — per-token or per-seat; multiply by how often you'll call it.
- Latency — interactive work needs snappy; batch/background can be slow.
- Privacy / data residency — can this codebase legally leave your machine or region? Some providers offer zero-retention or self-hosted options; check before piping proprietary code to a cloud agent.
- Ecosystem fit — does it read
AGENTS.md? Support MCP? Integrate with your CI and review flow?
4️⃣ How to know what's actually good (without trusting marketing)
Two durable signals, used together:
- Public benchmarks — execution-based ones like SWE-bench Verified measure whether a model's patch actually makes real tests pass. Useful as a signal — but treat leaderboards as signal, not gospel: they can be gamed or contaminated by training data. (In Feb 2026 OpenAI publicly stepped back from SWE-bench Verified over exactly this.)
- Your own tiny eval set — collect 5–20 real tasks from your codebase with known-good outcomes, and run candidate models/tools against them. Nothing predicts performance on your work like your work. This is the seed of an evals practice.
🎯 Takeaway: don't memorize a ranking that expires next month. Learn the axes (fast vs. reasoning, IDE vs. CLI vs. cloud, cost/latency/privacy/fit) and a repeatable way to measure (benchmarks + your own evals). That skill never goes stale.
Next: now that you can pick a tool, let's give it the project context it needs to be useful → Project Context (AGENTS.md).
Member discussion