map / ai-hardware / interconnects

AI Hardware · principle 4 of 7

NVLink & InfiniBand

Within a node GPUs talk over fast local links; across nodes they go over the network, and the slowest link sets the pace of a collective.

01 · Why it’s true

When one job spans many GPUs, they constantly exchange data, such as gradients in training. The links between them come in tiers: very fast links between GPUs in the same server (such as NVLink), slower PCIe to the host, and a network (InfiniBand or Ethernet with RDMA) between servers.

Each step out is slower and shared by more traffic. Synchronous jobs move at the pace of the slowest link they cross, so placing work that talks the most on the fastest links matters — and a single bad cable or flapping port can slow down a whole job.

02 · Picture it

1Inside one GPUHBM — fastest2GPU ↔ GPU, same serverNVLink-class links3GPU ↔ CPU / network cardPCIe4Server ↔ serverInfiniBand / RDMA Ethernet5Data center ↔ data centerslowest, highest latencyeach step down is slower and shared by more trafficSynchronous jobs move at the pace of the slowest link they cross — keep the chattiestcommunication on the fastest tier.
1Inside one GPUHBM — fastest2GPU ↔ GPU, same serverNVLink-class links3GPU ↔ CPU / network cardPCIe4Server ↔ serverInfiniBand / RDMA Ethernet5Data center ↔ data centerslowest, highest latencyeach step down is slower and shared bymore trafficSynchronous jobs move at the pace ofthe slowest link they cross — keep thechattiest communication on the fastesttier.
The tiers of links a multi-GPU job’s data has to cross.

03 · In the interview

  • Explaining how parallelism strategies map to the network topology.
  • Debugging a slow distributed job: one slow link or host slows everyone.
  • Monitoring link errors, retransmits, and per-port throughput.

04 · Practice question

“A multi-node training job got 30% slower overnight with no code change. How do you investigate?”

A strong answer covers

  1. Synchronous training runs at the pace of the slowest participant, so look for one slow node or link.
  2. Compare per-node step times and communication time; check NIC and switch port error counters, link speed, and flapping.
  3. Check for a GPU with a problem (thermal throttling, errors) or a host with other load.
  4. Isolate by running communication benchmarks between node pairs, then drain the bad node or link.