What is GPU interconnect?

Training or running a large AI model rarely fits on one chip. The work is split across many GPUs, and those GPUs must constantly swap results with each other. The GPU interconnect is the wiring and switching that carries that traffic.

Why it matters

Picture a kitchen with a dozen excellent cooks and one narrow hallway. If every dish has to pass through the hallway, the cooks spend more time waiting than cooking. A slow interconnect does the same to GPUs: they sit idle until the data they need arrives.

That is why the interconnect can matter nearly as much as the chips. Adding more GPUs only helps if the links between them can keep up.

Levels of connection

Interconnects work at several distances, each with its own technology:

• Within a server or rack: very fast, short links join GPUs directly. Nvidia's NVLink is a well-known example, and it relies mostly on copper over these short distances.
• Between servers: networking such as InfiniBand or Ethernet joins racks into a cluster. These links are slower per connection but reach much farther, and they use both copper cables and optical modules.
• Inside the chip package: chips and high-bandwidth memory are joined by extremely short connections on the same package.

Numbers vary by generation and by how vendors count them, so check the manufacturer's spec sheet before comparing figures.

Copper and light

Electrical signals in copper weaken and heat up as speeds and distances grow. Optical links send data as light through fiber, which can cover longer distances with less loss. As clusters grow, more of the connections are moving to optical parts such as lasers and transceivers.

What it means for you

You will not choose an interconnect when you call an AI API. But it shapes what you see indirectly:

• It affects how fast a provider can serve inference for a very large model, especially one split across many chips.
• It is part of the cost of compute, since networking is a real share of building a cluster.
• It is one reason a rack-scale system is sold as a unit rather than as separate chips.

If you ever compare cloud GPU offers, look past the chip name. Ask how the GPUs are connected, because eight GPUs on a fast shared link behave very differently from eight GPUs reached over a slower network.