1. Task definition#
- The task can be stated in one sentence with a defined input and output.
- There is an agreed definition of a correct output, written down.
- Domain experts agree with each other on most cases (measure it: ≥ 85% agreement on 50 items).
2. Value#
- Volume is at least tens of thousands of requests a month, or each decision is high-value.
- You know the current accuracy, cost per request and latency.
- A target metric has a business owner who will sign off.
3. Data#
- You have, or can produce, 500+ correct examples.
- Historical decisions or corrections exist in logs you can access.
- Data can legally be used for training (consent, contracts, licences checked).
- Sensitive fields can be redacted or de-identified before training.
4. Alternatives tried#
- A strong prompted baseline has been measured on representative inputs.
- If knowledge changes often, retrieval has been tried or ruled out.
- You know why prompting plateaued (format, conventions, judgement, cost).
5. Environment#
- The model’s runtime environment is decided (cloud account, on-prem, edge).
- GPU capacity for training and serving is available or budgeted.
- Security and risk reviewers are identified and engaged early.
6. Operations#
- Someone will own the model after launch.
- Production inputs and outcomes can be logged for monitoring and retraining.
- There is a rollback path to the previous system.
Scoring it#
Count the yes answers. 16–19: ready; a production engagement can start. 11–15: ready for a two-week feasibility sprint that closes the gaps. 10 or fewer: fix the gaps first, usually by measuring a baseline and writing down what “correct” means.
The three most common gaps: no agreed definition of correct (fix: write annotation guidelines with two experts and measure their agreement), no measured baseline (fix: score the current system on 300 representative items), and unclear data rights (fix: ask legal before, not after, data collection).
Next steps: the fine-tuning vs prompt engineering signals, the evaluation plan template, and our post-training guide.