The rule we use#
Prompt first, always: it is fast, reversible and tells you whether the task is feasible at all. Switch to fine-tuning when any of three things is true: the prompt has grown into a manual of instructions and examples, accuracy has plateaued below what the business needs despite prompt iterations, or the per-request cost of the long prompt is material at your volume.
The hidden cost of long prompts#
Every token of instruction and few-shot example is paid on every request. If your prompt overhead is tokens, volume requests a month and input price per token, the monthly cost of the prompt alone is:
A 3,000-token prompt (a typical size once a team has added rules, edge cases and eight examples) at 2 million requests a month and $3 per million input tokens is $18,000 a month before the actual input and output. A fine-tuned model internalizes those instructions, so its prompt shrinks to a line or two — and it typically runs on a model a fraction the size. Long prompts also add latency: prefill time grows with prompt length on every call.
| Prompt overhead | 1M requests / month | 5M requests / month | 20M requests / month |
|---|---|---|---|
| 500 tokens | $1,500 | $7,500 | $30,000 |
| 1,500 tokens | $4,500 | $22,500 | $90,000 |
| 3,000 tokens | $9,000 | $45,000 | $180,000 |
| 6,000 tokens | $18,000 | $90,000 | $360,000 |
Prompt-overhead cost per month at $3 per million input tokens. Our break-even model covers the full comparison including GPU costs.
The accuracy ceiling#
Prompting can only select behaviour the model already has. When a task depends on conventions the base model doesn’t know — your loss categories, your codes, which of two dates matters — prompting plateaus. In our extraction experiment, a carefully instructed 0.5B model got individual fields like policy numbers 99% right but all five fields right only 18% of the time; fine-tuning on 800 examples took exact match to 100%. Larger models plateau higher, but the shape is the same: conventions are learned, not prompted.
Side by side#
| Prompt engineering | Fine-tuning | |
|---|---|---|
| Time to first result | hours | weeks |
| Upfront cost | near zero | $24k–$180k with data and evaluation |
| Per-request cost | grows with prompt length | short prompts, smaller model |
| Accuracy on conventions | plateaus | learns them from examples |
| Consistency of format | good, not perfect | near-perfect |
| Changing requirements | edit the prompt | retrain (hours to days) |
| Needs labelled data | a handful of examples | hundreds to thousands |
| Vendor dependence | high | you own the model |
Five signals it’s time to switch#
- The system prompt is over 1,500 tokens and still growing.
- Each new fix to the prompt breaks something that used to work.
- Accuracy has moved less than a point across the last several prompt iterations.
- Prompt overhead exceeds about $5,000 a month at current volume.
- You have, or can cheaply produce, a few hundred correct examples of the task.
Two or more of these and a two-week feasibility sprint will usually pay for itself.
It isn’t either/or#
Fine-tuned models are still prompted, just briefly. And the work done during prompt engineering is not wasted: the rules you wrote become the annotation guidelines, the failure cases become evaluation items, and the few-shot examples become seed training data. Teams that kept good notes during prompting start fine-tuning projects weeks ahead.
References#
- Hu, E. J. et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685.