map / ai-hardware / training-vs-inference
AI Hardware · principle 5 of 7
Training vs inference
Training is a long, synchronous, throughput-bound batch job; inference is a latency-bound service — they fail and scale differently.
01 · Why it’s true
Training repeatedly runs data forward through the model, computes how wrong the output was, and runs backward to update every weight. It runs for days or weeks on many GPUs in lockstep, so throughput and surviving failures matter most.
Inference only runs forward, for one user request at a time (or a small batch). It has to answer quickly and reliably, around the clock, under unpredictable traffic — so it looks much more like running a web service: latency SLOs, autoscaling, and graceful degradation.
02 · Picture it
03 · In the interview
- Explaining how reliability practices differ for training jobs and serving.
- Defining SLOs for inference: time to first token, tokens per second, error rate.
- Capacity planning for each: scheduled jobs versus traffic-driven autoscaling.
04 · Practice question
“How would reliability goals differ for a training cluster and an inference service?”
A strong answer covers
- Training: maximize useful progress — fast failure detection, frequent checkpoints, quick replacement of bad nodes; a short pause is acceptable.
- Inference: user-facing SLOs — latency (such as time to first token), availability, and error rate; redundancy across replicas and regions.
- Inference needs autoscaling and load shedding for traffic spikes; training needs scheduling and capacity reservation.
- Both need GPU health monitoring, but they act on it differently: training restarts the job from a checkpoint, inference drains the replica.