map / ai-hardware / training-vs-inference

AI Hardware · principle 5 of 7

Training vs inference

Training is a long, synchronous, throughput-bound batch job; inference is a latency-bound service — they fail and scale differently.

01 · Why it’s true

Training repeatedly runs data forward through the model, computes how wrong the output was, and runs backward to update every weight. It runs for days or weeks on many GPUs in lockstep, so throughput and surviving failures matter most.

Inference only runs forward, for one user request at a time (or a small batch). It has to answer quickly and reliably, around the clock, under unpredictable traffic — so it looks much more like running a web service: latency SLOs, autoscaling, and graceful degradation.

02 · Picture it

TrainingForward and backward passesBatch jobs running days to weeksMany GPUs in lockstepGoal: throughput; checkpoint to survivefailuresInferenceForward pass onlyOnline, request by requestMany independent replicasGoal: latency, availability, cost perrequestvsTraining is run like a giant batch job; inference is run like a web service.
TrainingForward and backward passesBatch jobs running days to weeksMany GPUs in lockstepGoal: throughput; checkpoint tosurvive failuresInferenceForward pass onlyOnline, request by requestMany independent replicasGoal: latency, availability, costper requestTraining is run like a giant batchjob; inference is run like a webservice.
Same model, two very different operating problems.

03 · In the interview

  • Explaining how reliability practices differ for training jobs and serving.
  • Defining SLOs for inference: time to first token, tokens per second, error rate.
  • Capacity planning for each: scheduled jobs versus traffic-driven autoscaling.

04 · Practice question

“How would reliability goals differ for a training cluster and an inference service?”

A strong answer covers

  1. Training: maximize useful progress — fast failure detection, frequent checkpoints, quick replacement of bad nodes; a short pause is acceptable.
  2. Inference: user-facing SLOs — latency (such as time to first token), availability, and error rate; redundancy across replicas and regions.
  3. Inference needs autoscaling and load shedding for traffic spikes; training needs scheduling and capacity reservation.
  4. Both need GPU health monitoring, but they act on it differently: training restarts the job from a checkpoint, inference drains the replica.