Self-hosting a fine-tuned model vs paying per token: the break-even model

The model#

Monthly cost of a per-token API for $V$ requests of $t$ tokens at price $p$ per token:

Monthly cost of serving a fine-tuned model on $G$ GPUs at $h$ dollars per GPU-hour, with fixed operations cost $O$ and a one-off project cost $P$ amortised over $A$ months:

Here $\theta$ is sustained tokens per second per GPU at your latency target and $u$ is average utilisation — traffic has peaks, so you provision for the peak and pay for the average. $G_{\text{min}}$ is the minimum footprint for redundancy (two GPUs). Ignoring the ceiling, break-even volume is:

The step function in $G(V)$ means self-hosting cost is flat until you need another GPU, which is why the curves below cross only once over a wide range.

Base case#

With these inputs the break-even is 1.05M requests per month. Below it, keep paying per token; above it, the fine-tuned model is cheaper every month, before counting any accuracy gain.

Which assumption matters most?#

We moved each input across a plausible range while holding the rest at base values and recomputed the break-even volume:

The API price dominates: from 422k requests a month against a premium model to 25.31M against a budget one. Tokens per request comes second, because long-context workloads multiply API cost but barely change GPU count at low volume. GPU price, throughput and utilisation matter much less than intuition suggests, because at the break-even point you are usually still on the two-GPU minimum.

Three worked scenarios#

Plugging three common workloads into the same model (other inputs at base values):

The long-context support agent is the clearest case: 5.4 billion tokens a month at frontier prices is a large bill, while the fine-tuned model still fits on a handful of GPUs. The internal classifier is the opposite: against a cheap small model at low volume, self-hosting loses money every month, and the case for fine-tuning (if any) has to be accuracy.

Putting accuracy in the same equation#

Cost per request is only half the comparison. If each error costs $c_e$ downstream — a mis-routed claim, an analyst rework, a wrong refund — and the two systems have error rates $e_{\text{API}}$ and $e_{\text{FT}}$, the monthly value of the accuracy difference is:

For the claims scenario, an improvement from 86.9% to 94.7% field accuracy (the client case on our results page) at an assumed $2 of rework per error is worth $2\times 10^6\times 0.078\times 2\approx \$312,000$ a month — dwarfing the inference saving. In most engagements we see, accuracy pays for the project and cost savings pay for the next one. The break-even volume including accuracy becomes:

Where θ and u come from#

The throughput $\theta$ is not a fixed property of a GPU; it depends on batch size, which depends on your latency target. Serving engines batch concurrent requests: larger batches raise tokens per second per GPU but make each request wait longer. A useful sizing identity is Little’s law — the average number of requests in flight equals arrival rate times time in system:

At 2M requests a month (about 0.76 per second on average) and 2 seconds per request, fewer than two requests are in flight on average — far below the batch sizes at which a GPU is efficient. That is why low-volume workloads sit on the two-GPU minimum with utilisation well below 35%, and why $u$ barely affects break-even: you are paying for availability, not throughput. Above roughly ten million requests a month the GPU count starts to scale with volume and $\theta$ and $u$ begin to matter.

Measuring your own inputs#

Every input can be measured from a week of production logs. The minimum query, assuming a request log with token counts:

-- monthly volume, tokens per request, and peak-to-average ratio
SELECT
  COUNT(*) * 30.0 / 7                         AS requests_per_month,
  AVG(prompt_tokens + completion_tokens)      AS tokens_per_request,
  MAX(per_min) / AVG(per_min)                 AS peak_to_average
FROM (
  SELECT date_trunc('minute', ts) AS m, COUNT(*) AS per_min,
         AVG(prompt_tokens) AS prompt_tokens, AVG(completion_tokens) AS completion_tokens
  FROM llm_requests WHERE ts > now() - interval '7 days' GROUP BY 1
) t;

Utilisation is roughly the inverse of the peak-to-average ratio if you provision for the peak. Benchmark $\theta$ on your hardware with your real prompt-length distribution; it varies several-fold between short-in/short-out and long-context workloads.

What the model leaves out#

It treats the API as perfectly elastic and ignores rate limits, batch discounts and prompt caching, all of which lower $p$. It assumes the fine-tuned model meets the quality bar; the experiment and results pages address that separately. It ignores engineering time for on-call beyond the fixed $O$, and it ignores the option value of owning the model. The throughput figure $\theta$ is an assumption for an 8B model on a modern data-centre GPU and should be replaced with a benchmark on your hardware and traffic shape.

References#

1. Kaplan, J. et al. (2020). Scaling Laws for Neural Language Models. arXiv:2001.08361.
2. NVIDIA. H100 Tensor Core GPU datasheet (BF16 dense tensor throughput, SXM form factor: 989 TFLOPS).