When SRE best practices backfire on LLM Services
Self-hosted inference
When SRE best practices backfire on LLM Services
Scaling, retries, timeouts, load balancing and even load testing can each make an overloaded inference service worse.
Manoj Dobbali
Director, AI/ML Infrastructure · MasterControl
Where this comes from
I run a self-hosted inference platform, and I spent a year arguing with our VP of SRE about how to measure it.
The platform
The disagreement
The burden of proof
The case for self-hosting
It is a genuinely good trade
Control
Data residency
Economics
Tuning freedom
What rarely makes the business case
You also bought a scheduling problem the API was quietly handling on your behalf.
A provider’s endpoint is not just their hardware. It is a fleet large enough that your traffic is a rounding error, run by people whose entire job is absorbing the variance you are about to inherit.
You didn’t move a workload onto your own machines. You took on the hard part of it.
Which is why it becomes the strangest service in your stack
It falls over at loads it handled yesterday.
Latency graphs look fine until they suddenly don’t.
Adding capacity sometimes helps, and sometimes arrives after the incident is over.
A retry policy that improves every other service makes this one worse.
Not catastrophically broken. Unreliable in a way that resists explanation — which is the part that wastes the most engineering time.
The argument
Service reliability quietly rests on four assumptions about requests
01
Their cost is roughly estimable
02
They are mostly independent
03
Capacity can be added quickly
04
Overload degrades gracefully
LLM inference can violate all four at once.
Assumption 01
You can estimate how long a request will take
In inference
Prompt length is known. Maximum output may be bounded. But the number of tokens actually generated isn’t known until generation progresses — and the same endpoint returns twenty tokens for one request and two thousand for the next.
RPS measures how often work arrived. Not how much.
Same RPS — 10 requests
Same RPS — 10 requests
The same defect shows up three more times
Load balancing. Least-connections sees a replica holding three short generations and one holding three long ones, and calls them equally loaded.
Timeouts. A request past its deadline might be stuck, or might be a healthy generation that is simply longer than whatever request set the timeout.
Autoscaling. Scaling on RPS means scaling on a number that only loosely describes the work.
The deeper version
Shortest-job-first minimises average wait time. It needs to know job length.
That result is old, well understood and not in dispute. Exact generation length is not available when the request arrives, so the best answer scheduling theory has is out of reach by construction.
But you are not helpless, and this is the useful question:
How much of the benefit of knowing job size can you recover from imperfect predictions of request shape?
Route, prompt length, requested maximum output, workload class and historical generation length are all signals. Even coarse classification beats pretending every request costs the same.
Assumption 02
Requests are independent
In inference
Continuous batching makes several sequences share GPU execution. A request’s token generation speed depends on the other sequences beside it — their number, their prompt lengths, their generation lengths, their cache requirements.
Your p99 is not a property of your request. It’s a property of your neighbours.
And this is not a noisy-neighbour problem. Noisy neighbours are an isolation failure you fix with quotas and dedicated pools. Here the interference is the optimisation.
Two things this breaks that you rely on every week
Debugging
You take a slow production request and replay it in isolation. It’s fast. You replay it again. Still fast.
The pathological behaviour existed only in company.
Load testing
A synthetic test firing uniform prompts at a fixed rate reproduces your RPS while reproducing almost none of your workload.
You can pass at twice production volume and still fail at peak.
Traffic volume isn’t the workload. Traffic shape is.
Assumption 03
Capacity is elastic
In inference
A replica acquires GPU capacity, pulls large weights, initialises CUDA, allocates cache and warms up. Useful capacity arrives minutes after the signal that asked for it — and GPUs aren’t summoned, they’re procured.
A rolling update needs somewhere for the new pod to land
With a spare node
Node 1
Node 2
Old drains
Every node claimed
Node 1
blocked
New pod
So you do the thing every deployment system is designed to avoid: you delete the healthy replica first.
The expensive part isn’t the deploy. It’s the rollback.
Reverting assumes you can bring the previous version up before taking the bad one down. Under GPU constraint, rollback is another scheduling event — so a bad model version costs you three waits, not one.
01
Deployment delay
+
02
Detection delay
+
03
Rollback delay
Scheduler latency multiplies the blast radius of every deployment mistake.
The counterintuitive fix
Stop treating deployment capacity as something you request at deployment time
Before a planned rollout, raise the capacity floor. Bring the nodes online early enough that they are schedulable and warm before you need them. For a while, that hardware sits underutilised.
Elastic infrastructure was sold on paying for what you use.
Here you deliberately pay for what you don’t.
You are buying scheduling certainty with idle GPU hours — converting a risk you can’t control into a cost you can predict. That is a reasonable trade. It is just not one any autoscaling documentation prepares you to make.
Assumption 04
Overload degrades gracefully
In inference
KV-cache demand grows with the tokens resident across active sequences — not with how many requests you are serving. When pressure gets severe the engine preempts, evicts and recomputes, and recovery work consumes the resource that was already scarce.
The loop that turns pressure into an outage
Cache pressure
Preemption
Recomputation
Less capacity
… which feeds straight back into cache pressure. The recovery work consumes the resource that was already scarce.
You don’t always get a warning slope. Sometimes you get a cliff.
Three things that make a cliff worse than it sounds
01
Your alerts are calibrated for slopes
“Page if p99 exceeds X for five minutes” assumes a gradient you can catch partway down. You are not alerted early enough to intervene. You are alerted in time to watch.
02
Recovery is asymmetric
Dropping back below the load that broke you may not restore the system. Recovery can require going significantly lower. The path in and the path out don’t look alike.
03
Headroom is deceptive
Thirty percent spare CPU means roughly thirty percent more traffic. Thirty percent apparent inference headroom does not. A shift toward longer contexts can eat the margin with no change in RPS at all.
A walkthrough
Nothing about the traffic looks unusual
Volume is within ten percent of a normal Tuesday.
Two days earlier, a product change let one route return substantially longer responses. A product decision, not an infrastructure one. Nobody treated it as a capacity event.
Now p99 is climbing while p50 stays flat.
Flat median plus rising tail usually means a bad replica or a slow dependency. So that is where you look. There isn’t one.
Then you try the obvious things, in the obvious order
Requests start timing out. Clients retry. The retries do what retries do.
You do this
And here is why it doesn’t help
Scale out
New capacity needs several minutes before it serves traffic. The incident evolves faster than the autoscaler.
Raise the timeout
If the constraint is memory and scheduling pressure, longer timeouts keep expensive work alive longer.
Drain a replica
You briefly change the scheduling state. The remaining capacity fills again.
Let retries ride
Every retry submits another sequence into the same constrained pool.
Every one of these is the correct move on a conventional service. None of them addresses the actual constraint.
Then somebody opens the metric that was on a dashboard the whole time
KV-cache utilisation
Preemption counter
The fix
The retries weren’t a symptom of the incident. They became part of the workload causing it.
And GPU utilisation wasn’t a saturation signal. It was a comfort signal.
Why latency becomes so hard to predict
total latency ≈ TTFT + (TPOT × output tokens)
TTFT
prompt length, prefill scheduling, server state, cache behaviour
TPOT
decode performance, batch composition, concurrency — your neighbours
Output tokens
not known when the request begins
A slightly wrong TPOT estimate costs nothing on a ten-token answer. On a two-thousand-token generation that per-token error compounds two thousand times — and the output length is itself uncertain.
The tail
You’re estimating a product of uncertainties and treating it like a sum.
Latency estimation here isn’t adding independent sources of noise. One uncertain quantity is multiplying another, and the error accumulates in the tail.
Retry with backoff is excellent advice — when the retry can land on healthy capacity
Saturation
Timeout
Retry
More saturation
Your reliability mechanism is now generating load precisely when the service is least able to absorb it. Backoff changes the timing. It doesn’t change the direction of the loop.
Under true saturation, the correct response is to shed work, not duplicate it.
There is a difference between rejecting some requests deliberately and failing most of them accidentally. Admission control makes that trade explicit. Retries hide it until the system makes the decision for you.
The shift that matters
Stop thinking in requests
Conventional service
LLM inference
Request
Job or sequence
Requests per second
Token workload and request shape
Mostly independent requests
Coupled scheduled sequences
Fast elastic replicas
Expensive, slow-to-acquire GPU capacity
CPU utilisation
Compute, bandwidth and cache pressure
Queue everything
Admission control
Retry transient failures
Retry selectively, under known capacity
Autoscale after demand
Predict and pre-provision
Generic timeout
Workload-specific resource budget
Graceful slowdown
Potential nonlinear saturation
Nothing on the left is wrong. The abstraction underneath it changed.
A batch scheduler with an SLA bolted on
A fast-food line
Orders are standardised. Service times are short and predictable enough that staffing against arrival rate works.
That is your web service.
The other kitchen
Every order takes between two and forty minutes. You don’t know which when you accept it. Every dish shares one cooking surface. Another cook needs twenty minutes of prep before touching an order.
That is inference.
You wouldn’t run that kitchen by accepting every order, promising the same ticket time, and calling another cook once the queue explodes.
You’d control what enters the kitchen.
What actually helps
The effective interventions reduce variance and uncontrolled admission. They do not add theoretical capacity.
Cap output length per route
Segregate by shape, not tenant
Admission control over autoscaling
Measure TTFT and TPOT separately
Treat cache pressure as first-class
Make retry budgets saturation-aware
Pre-provision predictable demand
Predict request shape
The dashboard I actually want
Admission
- Active sequences
- Queued sequences
- Rejected or shed
- Admitted input tokens/s
- Admitted generation budget/s
Generation
- TTFT at p50, p95, p99
- TPOT at p50, p95, p99
- Prompt tokens per request
- Output tokens per request
- Generation-length distribution
Memory & scheduling
- KV-cache utilisation
- Resident tokens
- Queue depth
- Preemptions and evictions
- Prefix-cache hit rate
Capacity & reliability
- Ready replicas, warm spares
- Pending GPU pods
- Timeout and retry rate
- Load-shed rate
- Effective tokens per second
It should answer something more useful than “are my GPUs busy?”
What resource is preventing me from safely admitting the next unit of work?
The real lesson
None of this means the SRE playbook is wrong
Retries are useful. Autoscaling is useful. Load balancing, horizontal scaling, rolling deployments, RPS — all useful. They were developed around abstractions where their assumptions hold well enough. LLM inference changes several of those assumptions at once.
Stop asking
How do I make my inference service behave like a normal web service?
Start asking
How do I schedule an unpredictable, coupled workload on a fixed, expensive resource pool while still meeting an SLA?
The old playbook is not wrong. It is borrowed from a problem you don’t have.
If you remember nothing else
Three things to take away
01
Autoscale on tokens, not requests
02
Cap output length per route
03
Watch cache utilisation, not GPU utilisation
Thank you
Happy to argue about any of this — particularly if you have run into the deploy deadlock, or if your engine recovers from cache exhaustion better than mine does.
Manoj Dobbali
MasterControl · Hyderabad
Find me
← → to move · G for all slides · F for fullscreen