Fine-tune when you have 1,000+ labelled examples and the task is narrow, repetitive, and stable — a fixed classification schema, a consistent output format, a single well-defined transformation. Prompt when your task is variable, your data is scarce, or the requirements are still moving. Most teams get this backwards: they reach for fine-tuning because it feels more "real" than prompt engineering, and they end up maintaining a training pipeline for a problem that a better system prompt would have solved in an afternoon.
The common wrong approach
A team ships a feature with a decent prompt. It's 80% accurate. Instead of digging into the 20% of failures, someone suggests "let's just fine-tune it." They scrape together 200 examples from support tickets, fine-tune a model, and get... 82% accuracy, now locked to a specific model version, with a training pipeline nobody wants to own six months later.
This happens because fine-tuning looks like a shortcut around the hard work of prompt iteration. It isn't. Fine-tuning bakes in whatever patterns are in your training data — including the noise, the edge cases you didn't think to exclude, and the labeling inconsistencies from whoever tagged ticket #4,891 at 6pm on a Friday. If your underlying task definition is fuzzy, fine-tuning makes it fuzzily consistent, not correct. And now every prompt change requires retraining instead of a git commit.
The better approach
Start with the decision tree, in order:
1. Do you have 1,000+ labelled examples, and is the task bounded? Bounded means: same input shape every time, same output schema every time, task definition unlikely to change in the next six months. Email Triage, one of our internal tools, hit this bar — routing incoming email into a fixed set of categories based on years of labeled examples. Fine-tuning a smaller model there beat prompting a larger one on cost and latency, because the task never varies.
2. If you don't have the data or the task shifts, prompt — and treat prompting as engineering, not guesswork. That means: a versioned prompt file in your repo, not a string in a Slack thread. A small eval set (even 30-50 examples) that you run against every prompt change. Structured output via tool calling or JSON schema instead of hoping the model formats things correctly. For Ghost Writer, our content-drafting tool, the brief changes every time — different client voice, different format, different constraints. There's no fixed task to fine-tune against. The system prompt plus a retrieval step pulling in the client's style guide does the work that fine-tuning couldn't.
3. Compare real costs, not vibes. Fine-tuning costs: data collection and labeling (usually the biggest line item — budget weeks, not days), training runs, evaluation, and — the one teams forget — versioning. When the base model updates, do you retrain? When your task definition shifts, do you relabel and retrain? Prompting costs: iteration time up front, and marginally higher per-token cost at inference if you need a bigger model to hit the same accuracy a fine-tuned small model would. For most startups shipping a feature with fewer than a few hundred thousand monthly requests, prompt iteration time is cheaper than a training pipeline's maintenance burden, full stop.
4. Consider the middle ground. Few-shot examples in the prompt handle a surprising amount of what people think they need fine-tuning for. If 5-10 examples in-context get you to 90%+ accuracy, you don't have a fine-tuning problem. We've had clients come to us assuming they needed a custom model when what they actually needed was better examples and a stricter output schema — that's a conversation worth having before you commit engineering time to either path.
Where this breaks
The decision tree isn't absolute. Sometimes you have 1,000+ examples but the task still isn't bounded — customer support tickets, for instance, where the category taxonomy itself keeps growing. Fine-tuning there gives you a model confidently wrong about categories that didn't exist when you trained it. And sometimes you have a genuinely bounded task with only 200 examples — not enough to fine-tune reliably, but few-shot prompting with those same 200 examples as a retrieval-augmented example bank gets you most of the benefit without the training cost. Data volume and task boundedness both have to hold. One without the other, and you're better off prompting.
Practical next step
This week, before you fine-tune anything: build a 30-50 example eval set for the task you're considering, and run your current best prompt against it. If you're above 85% and the failures are random rather than systematic, you have a prompting problem, not a training-data problem — iterate on the prompt. If you're below that and the failures cluster around specific patterns you have hundreds of labeled examples for, that's your fine-tuning signal. If you want a second opinion on which side of the line your project falls on, start a project with us and we'll look at your eval numbers directly.