Cross-Data-Center AI Training: Faster Networks or Fewer Synchronization Barriers?

Cross-Data-Center AI Training: Build a Faster Network or Synchronize Less?

Imagine training a large foundation model with two GPU clusters in different parts of a city. The eastern data center is approaching its power limit, while the western facility still has available electrical capacity and floor space.

Could you connect the two sites with high-speed fiber and use their GPUs to train the same model?

Yes. The networking technology exists, and research into distributed training across geographically separated clusters is advancing. But connecting two data centers is not the same as making them operate like one tightly coupled GPU system.

The difficult part is synchronization.

During conventional distributed training, GPUs repeatedly exchange gradients or other updates and coordinate before proceeding. If one site must wait for data from another, network latency and congestion can leave otherwise productive accelerators idle.

There are two broad ways to address this problem:

1. Improve the network: Reduce congestion and make cross-site communication faster and more predictable.
2. Reduce synchronization: Change the training algorithm so that sites can perform more local computation before exchanging updates.

NVIDIA Spectrum-XGS represents the first approach. Google’s Decoupled DiLoCo research illustrates the second.

These strategies are not mutually exclusive. In fact, combining them may be the most practical way to make geographically distributed AI infrastructure useful for large-scale training.

🌐 From Scale-Up to Scale-Out to Scale-Across

#

AI infrastructure is often described using three expansion dimensions: Scale-Up, Scale-Out, and Scale-Across. Each addresses a different part of the system.

Scale-Up: Connect more accelerators closely

#

Scale-Up increases the number of accelerators that can cooperate within a tightly coupled system.

GPUs communicate through high-bandwidth, low-latency interconnects, often using specialized links and switching fabrics. The objective is to make multiple accelerators behave like a larger computing unit, with efficient access to data and frequent communication.

This approach offers strong communication performance, but it is constrained by physical packaging, power delivery, cooling, and system cost. A failure in a tightly coupled system can also affect a substantial portion of a training job.

Scale-Out: Expand across servers and racks

#

Scale-Out connects additional servers and racks into a data-center cluster.

High-speed Ethernet or InfiniBand networks carry distributed training traffic between nodes. This architecture provides a more flexible way to add computing capacity than relying exclusively on larger individual systems.

However, communication remains subject to the network topology, congestion, and synchronization requirements of the cluster.

Scale-Across: Extend training across facilities

#

Scale-Across takes distributed computing beyond a single data center or campus.

Instead of assuming that all accelerators are connected through one low-latency local fabric, it allows multiple sites to contribute to the same training workload.

The distinction is important: a cross-site connection does not eliminate the physical and operational differences between facilities. It introduces additional propagation delay, independent failure domains, and potentially different bandwidth and congestion characteristics.

The three approaches can coexist in one training architecture:

• Scale-Up handles frequent communication among closely connected accelerators.
• Scale-Out connects servers and racks within a facility.
• Scale-Across coordinates work between geographically separated clusters.

The design challenge is deciding which communication belongs at each level.

⏱️ Why Cross-Data-Center Training Is So Difficult

#

Connecting two facilities with fiber is not new. The challenge is that large-scale AI training has a different traffic pattern from ordinary cloud applications.

Synchronous training makes the slowest participant matter

#

Many distributed training workloads use synchronous data-parallel operations. GPUs process different portions of a batch, calculate gradients, and exchange those gradients through collective operations such as AllReduce.

The next training step may depend on the completion of that collective operation.

A simplified iteration looks like this:

GPU computation
      |
      v
Gradient calculation
      |
      v
Collective communication
      |
      v
Synchronization
      |
      v
Next training iteration

If one group of GPUs finishes its work but must wait for a remote group to complete communication, the faster group cannot necessarily move ahead.

This creates a synchronization barrier. Its cost depends not only on bandwidth, but also on latency, congestion, collective algorithms, and the degree to which communication overlaps with computation.

Fiber latency cannot be engineered away

#

Light travels through optical fiber at approximately 200,000 kilometers per second. That translates to roughly 5 microseconds of one-way propagation delay per kilometer.

A 10-kilometer fiber path therefore introduces about 50 microseconds of one-way propagation delay before accounting for switches, transceivers, routing, and other processing.

The actual path may be considerably longer than the straight-line distance between two facilities.

For ordinary application traffic, this delay may be manageable. For tightly synchronized GPU collectives, repeated communication rounds can make it expensive.

Congestion and jitter make the problem harder

#

Bandwidth is only part of the story.

When many GPUs transmit simultaneously, traffic can create congestion at network links and switches. Queueing delays vary with traffic conditions, and uneven completion times can increase the time spent waiting at synchronization barriers.

Traditional data-center Ethernet can be adapted for AI workloads, but a network designed primarily for general-purpose cloud traffic may not provide the predictable behavior that large collective operations need.

The engineering objective is therefore broader than maximizing peak throughput:

A cross-data-center AI network must deliver predictable communication performance while handling the distance, congestion, and failure characteristics of a wide-area environment.

🛣️ Route 1: Build a Better Network and Reduce Communication Delay

#

The first approach keeps the basic training synchronization model but improves the network carrying the traffic.

NVIDIA Spectrum-XGS Ethernet is a prominent example of this direction.

NVIDIA Spectrum-XGS: Distance-aware Ethernet

#

Introduced in 2025, Spectrum-XGS extends NVIDIA’s AI networking strategy toward Scale-Across environments.

The goal is to connect geographically distributed data centers into a larger AI infrastructure system without treating long-distance links as ordinary extensions of an in-building network.

Distance-aware networking matters because the behavior of a cross-site fabric depends on physical propagation delay as well as congestion. Network algorithms can account for those conditions when managing traffic and coordinating communication.

Rather than focusing exclusively on theoretical link bandwidth, the design aims to improve the predictability and efficiency of distributed communication.

NVIDIA has published results showing that, in a 10-kilometer test scenario, Spectrum-XGS achieved up to approximately 1.9 times the NCCL AllReduce bandwidth of its reference general-purpose Ethernet baseline.

That figure needs context. It measures collective communication bandwidth under the specified test conditions. It does not mean that an entire AI training job runs 1.9 times faster.

The actual training benefit depends on several factors:

• How much of each iteration is spent in AllReduce.
• Whether communication overlaps with GPU computation.
• The number of synchronization rounds.
• The network topology and traffic pattern.
• The performance of the GPUs and other system components.

A network improvement has the greatest effect when communication is a significant bottleneck in the workload.

Cisco: Congestion management and optical transport

#

Cisco approaches the same broad problem through a combination of networking and optical technologies.

Coherent optics can transport high-speed signals over longer distances. Congestion awareness and path selection can help steer traffic away from overloaded links, while buffering handles short-term traffic bursts.

Cisco’s Intelligent Collective Networking is part of its effort to make network behavior better suited to distributed AI communication.

These capabilities address different layers of the problem:

The effectiveness of these features depends on the actual deployment. A network that performs well in a controlled benchmark may behave differently under a production training workload with multiple traffic classes and competing applications.

Google Virgo: Specialized networking for AI traffic

#

Google’s Virgo Network illustrates another way to approach large-scale AI networking.

Rather than relying on one general-purpose network architecture for every workload, Virgo separates network functions into specialized tiers. Its design includes a low-latency, high-bandwidth fabric for east-west AI communication.

This reflects a broader infrastructure principle: AI training traffic has different requirements from routine cloud application traffic, so the network should be designed around the communication patterns that dominate the workload.

However, a low-latency campus fabric and a cross-region WAN are not interchangeable.

As communication extends across distant facilities, propagation delay, link capacity, routing, and failure recovery become more significant. Network architecture must account for those differences rather than simply extending the local design over a longer distance.

🔄 Route 2: Change the Training Algorithm and Synchronize Less Often

#

The second approach attacks the problem from the software side.

If two sites must exchange updates after every small unit of computation, network latency becomes part of the critical path. Improving the network helps, but it does not remove the repeated synchronization requirement.

What if each site could perform more local training before communicating with the others?

That is the basic idea behind Google’s DeepMind’s Decoupled DiLoCo research.

From tightly synchronized workers to independent computing islands

#

Traditional synchronous training often requires participants to coordinate frequently. A slow or unavailable participant can delay the entire operation.

Decoupled DiLoCo divides a large training system into relatively independent computing groups, or islands.

Within each island, accelerators continue to use high-speed local networking for frequent communication. Between islands, the training process exchanges updates less frequently and allows local computation to continue while cross-site communication is in progress.

The simplified model looks like this:

             Computing Island A
        +---------------------------+
        | GPU  GPU  GPU  GPU        |
        |       Local training      |
        +---------------------------+
                    |
                    | Less frequent updates
                    |
                    v
              Cross-site WAN
                    |
                    v
        +---------------------------+
        |       Local training      |
        | GPU  GPU  GPU  GPU        |
        +---------------------------+
             Computing Island B

Instead of making every GPU wait for every remote update at every step, each island performs multiple local steps before exchanging information.

The WAN is no longer required to participate in every local synchronization event.

How local computation hides WAN latency

#

Consider a simplified comparison.

In a conventional synchronous arrangement:

Local computation
      |
      v
Cross-site synchronization
      |
      v
Wait for remote participants
      |
      v
Next computation

In a less frequently synchronized arrangement:

Local computation
      |
      +---- Local computation
      |
      +---- Local computation
      |
      +---- Local computation
      |
      v
Cross-site update exchange
      |
      v
Continue local training

The second arrangement gives local accelerators more useful work to perform while remote communication is underway.

This does not make communication free. It changes how often communication is required and how much computation can proceed independently.

The trade-off is algorithmic: local models can diverge between synchronization events, so the training method must manage the effects of delayed or less frequent updates.

Decoupled DiLoCo and fault tolerance

#

The DeepMind work also explores how independent computing islands respond to failures.

In conventional tightly synchronized training, the failure or disconnection of one participant can stall the entire job.

With a more decoupled architecture, surviving islands can continue local training while another island is unavailable. The disconnected group can rejoin after recovery, subject to the algorithm’s update and recovery rules.

This is not automatic fault tolerance. The system still needs mechanisms for tracking training progress, handling stale updates, recovering state, and maintaining convergence.

The benefit is that a temporary WAN interruption does not necessarily have to stop all useful computation.

Reported results across four US regions

#

DeepMind reported training a 12-billion-parameter model across four regions in the United States using a WAN with bandwidth in the 2–5 Gbps range.

Under the reported experimental conditions, the researchers achieved training speeds of up to 20 times those of traditional synchronization methods.

This is a research result, not a general performance guarantee. The comparison depends on the model, workload, network configuration, synchronization strategy, and baseline used in the experiments.

It nevertheless demonstrates an important point: changing the communication pattern can produce gains that would be difficult to obtain by increasing network bandwidth alone.

⚖️ Two Routes, Two Different Trade-Offs

#

Spectrum-XGS and Decoupled DiLoCo address different parts of the cross-data-center training problem.

The two strategies can also be combined.

For example, a distributed training system could use high-performance local fabrics within each facility, a distance-aware network between facilities, and an algorithm that exchanges model updates less frequently across sites.

This leads to a practical division of responsibilities:

• Keep latency-sensitive, high-frequency communication local whenever possible.
• Use the cross-site network for traffic that genuinely needs to move between facilities.
• Reduce the amount of cross-site communication through algorithmic techniques where the workload permits.
• Design explicit recovery and resynchronization mechanisms for network or site failures.

The result is not necessarily one giant low-latency cluster. It is a hierarchy of computing resources with different communication characteristics.

🏢 What Cross-Data-Center Training Means for AI Infrastructure

#

The implications extend beyond connecting more GPUs.

1. Compute capacity becomes less tied to one location

#

A training job may be able to use computing resources from facilities with different power, cooling, and capacity constraints.

This can make it easier to use available infrastructure without requiring every accelerator to be installed in the same building.

However, geographic flexibility comes with operational constraints. Data sovereignty, privacy requirements, network availability, power pricing, and facility scheduling all affect whether multiple sites can participate in one training job.

2. Failure domains become more independent

#

A multi-site architecture can reduce dependence on a single facility.

A power or cooling failure at one location does not necessarily have to stop all training activity if the algorithm and infrastructure support continued operation elsewhere.

But resilience requires more than geographic separation. Training systems need checkpointing, state recovery, update reconciliation, and a clear policy for handling disconnected or delayed participants.

3. The network becomes part of the computing architecture

#

In traditional infrastructure planning, networking can be treated as a way to connect servers.

For geographically distributed AI training, network behavior directly affects how much useful computation the system can sustain.

Propagation delay, congestion, bandwidth allocation, topology, and failure recovery become part of the training design itself.

The boundary of a distributed AI system is therefore determined not only by where its GPUs are installed, but also by how effectively its software and network coordinate them.