map / ai-hardware / interconnects
AI Hardware · principle 4 of 7
NVLink & InfiniBand
Within a node GPUs talk over fast local links; across nodes they go over the network, and the slowest link sets the pace of a collective.
01 · Why it’s true
When one job spans many GPUs, they constantly exchange data, such as gradients in training. The links between them come in tiers: very fast links between GPUs in the same server (such as NVLink), slower PCIe to the host, and a network (InfiniBand or Ethernet with RDMA) between servers.
Each step out is slower and shared by more traffic. Synchronous jobs move at the pace of the slowest link they cross, so placing work that talks the most on the fastest links matters — and a single bad cable or flapping port can slow down a whole job.
02 · Picture it
03 · In the interview
- Explaining how parallelism strategies map to the network topology.
- Debugging a slow distributed job: one slow link or host slows everyone.
- Monitoring link errors, retransmits, and per-port throughput.
04 · Practice question
“A multi-node training job got 30% slower overnight with no code change. How do you investigate?”
A strong answer covers
- Synchronous training runs at the pace of the slowest participant, so look for one slow node or link.
- Compare per-node step times and communication time; check NIC and switch port error counters, link speed, and flapping.
- Check for a GPU with a problem (thermal throttling, errors) or a host with other load.
- Isolate by running communication benchmarks between node pairs, then drain the bad node or link.