---
title: Optimizing the network for AI workloads
url: https://www.broadcom.com/company/news/articles/ai-infrastructure/optimizing-the-network-for-ai-workloads
description: Contributor Sameh Boujelbene from the Dell'Oro Group shares her perspective on AI workloads in the Broadcom blog.
---

# Optimizing the network for AI workloads

June 12, 2023  
6 min read  

## [Sameh Boujelbene](https://www.broadcom.com/company/news/author/sameh-boujelbene)  

![Sameh Boujelbene](https://www.broadcom.com/media/blt4ac44e0e6c6d8341/blt9d8a0ff346619f14/64874fadd40ad01098680763/Sameh-headshot-250x250ppi.jpg?width=150)

VP, Data Center and Campus Ethernet Switch Market Research, Dell'Oro Group  

- AI Infrastructure
- Networking
- Networking for AI
- Ethernet

### Switches

The AI industry is currently undergoing a significant inflection point that some industry pioneers have dubbed the “iPhone moment of AI.” This turning point is marked and defined by the rapid rise of emerging large language models and generative AI applications such as OpenAI's ChatGPT. What sets these emerging AI applications apart is the sheer number of parameters they have to manage. Some of them deal with billions — or even trillions — of parameters, necessitating the use of thousands or tens of thousands of GPUs, TPUs, or other types of accelerated processors. Dell'Oro Group predicts that by 2027, 50% of the server market will be propelled by accelerated servers (figure 1). Connecting these accelerated servers in large clusters requires a data center-scale fabric, known as the AI back-end network, which differs from the traditional front-end network used to connect general-purpose servers. Additionally, AI workloads have unique attributes and characteristics that are vastly different from traditional general-purpose compute workloads. These unique attributes have important implications for the type of network required to run these AI workloads.

![AI Workloads Diagram](https://www.broadcom.com/media/blt4ac44e0e6c6d8341/blt16cb931e9aa019fe/64875539616b142f3615cee9/AI-Servers-Total-Market-DellOroGroup-Diagram.jpg)

**AI workload characteristics**
The process of handling AI workloads involves a continuous loop of compute, synchronize, and communicate operations between the various process nodes. As this process continues, there is an exponential increase in the data exchanged between nodes. Presently, a single GPU can consume up to 800 Gbps. In the near future, it may require over 1 Tbps. Furthermore, not only is the amount of bandwidth required per GPU increasing, but the average size of clusters is skyrocketing. Previously, the average number of GPUs per cluster was 256. However, Google recently announced the deployment of some clusters consisting of 26,000 GPUs, surpassing all market expectations of the speed and size at which AI clusters can grow.

Moreover, in AI networks, any delayed flow can hold back the progress of all nodes. For instance, Meta demonstrated that, on average, 30% of the elapsed time in AI/ML is spent waiting for the network (figure 2). Considering that a single compute node can cost hundreds of thousands of dollars, having it idle for 30% of the time can result in significant expenses. Therefore, even small incremental enhancements in network performance can lead to billions of dollars in savings or can increase the number of workloads and applications that can be run on the AI infrastructure. Ideally, the AI back-end network should operate at a 100% utilization rate, which is notably different from traditional front-end networks that connect low-cost general-purpose servers, where network utilization rates can drop to 50%.

![Time Spent in Networking Diagram](https://www.broadcom.com/media/blt4ac44e0e6c6d8341/blt76aa7194e270d024/6487556974d50136ab014622/Time-Spent-in-Networking-Diagram.jpg)

**Getting the network out of the way**
To satisfy the unique characteristics of AI workloads, AI back-end networks require a high-speed, lossless, scalable, low-latency fabric. It's important to note that we are discussing network latency, not packet latency. While in High-Performance Networks (HPC), packet latency can have a huge impact on application performance, for AI/ML workloads, what matters most is network latency or tail latency. In other words, it’s not about the latency of the packet, but rather the time needed to complete the job. This is because, as mentioned earlier, any delayed flow can impede the progress of all nodes. Multiple factors can result in delayed flows and a longer time for job completion, including flow collisions and oversubscriptions. Therefore, implementing the following features in the AI fabric is critical for reducing the time spent in networking and getting the network out of the way:

- Load-balancing features 
- End-to-end scheduled fabric to prevent collisions 
- Zero Impact Failover (ZIF) to ensure minimal disruption to job completion
- High-radix fabric that allows to build massive flat networks

**AI networks at a major crossroads**
It's widely known that Ethernet has been the most dominant and pervasive fabric in the history of networking. At Dell'Oro Group, we have estimated that over the past two decades, the Ethernet industry has shipped almost 9 billion Ethernet switch ports, equivalent to almost 1,000 Ethernet ports shipping every minute during that time. Although InfiniBand has made inroads in certain niche applications such as High-Performance Computing, as well as in the initial wave of generative AI applications, we believe that two factors, in particular, may challenge the status quo and put pressure on InfiniBand's current share in AI networks:

- The scale of AI networks is rapidly expanding, and while deploying vertically integrated solutions may not be problematic in small-scale networks, vendor lock-in becomes a concern at a larger scale. Therefore, we anticipate that open standards promoting interoperability and vendor diversity will be preferred for large-scale AI deployments. Additionally, as hyperscalers develop their own in-house versions of accelerated compute nodes, they will likely seek a more open and interoperable type of network fabric for their AI workloads.
- Ethernet network chips have witnessed remarkable advancements and improvements specifically tailored for AI workloads. These latest enhancements enable Ethernet to achieve shorter job completion times compared to InfiniBand. Moreover, Ethernet is currently ahead of InfiniBand in terms of port speeds. In addition, the latest Ethernet network chips destined for AI deployments offer a high network radix, effectively reducing the number of tiers required for large cluster size. With recent and future introductions of new features and innovations in Ethernet network chips, we anticipate a substantial increase during the coming years in Ethernet's adoption within AI networks.

**The power of power**
As customers strive to optimize their networks for AI workloads, power consumption has become a critical metric to consider. This is particularly important for hyperscalers, as power consumption can limit their ability to build and expand their data centers. Pluggable optics currently account for over 50% of the switch system power consumption with 51.2 Tbps chips and beyond, an issue that will get further amplified as Cloud Service Providers (SPs) build their next-generation AI networks.

Co-packaged optics (CPOs) have emerged as a promising alternative to pluggable optics, with the potential to reduce both power consumption and cost. However, while progress is being made in developing CPOs, challenges related to serviceability, manufacturability, and testability remain unresolved. In the meantime, Linear Drive Pluggable Optics (LPOs) have also emerged as an alternative that may address some CPO-related challenges. The idea behind LPOs is to remove the DSP from pluggable optics, as the DSP is responsible for about half of the power consumption and a large portion of the cost of pluggable optics. However, this approach puts more stress on the electrical part of the switch system and requires a high-performance switch SERDES, as well as a careful motherboard signal integrity design. As Cloud SPs seek to optimize their networks for AI workloads, power consumption will undoubtedly be a key consideration in their decision-making process. Certainly, no stone will be left unturned in their pursuit of the most optimum network for running their AI workloads.

### Topics:
- [AI clusters](https://www.broadcom.com/company/news/articles?topic[]=AI%20clusters)
- [Ethernet](https://www.broadcom.com/company/news/articles?topic[]=Ethernet)
- [Switches](https://www.broadcom.com/company/news/articles?topic[]=Switches)

## Related Articles

- [Enabling AI Networking @ Scale with Multi-path Reliable Connections (MRC)](https://www.broadcom.com/company/news/articles/ai-infrastructure/enabling-ai-networking-scale-with-multi-path-reliable-connections-mrc)  
  ![Enabling AI Networking @ Scale with Multi-path Reliable Connections (MRC)](https://www.broadcom.com/media/blt4ac44e0e6c6d8341/blt25b537b296425507/69f8bdae9df29a3f4dc1d5c2/BRCM-BLOGHERO-1438203847-1920x455ppi_%281%29_%283%29.jpg)
- [OFC 2026: Broadcom Paves the Path for the 200T AI Era](https://www.broadcom.com/company/news/articles/innovation/ofc-2026-broadcom-paves-the-path-for-the-200t-ai-era)  
  ![OFC 2026: Broadcom Paves the Path for the 200T AI Era](https://www.broadcom.com/media/blt4ac44e0e6c6d8341/blt904d4195fdc1b754/69ceda3f3b53e13c51539f93/Blog_Hero_OFCrecap.jpg)
- [Tomahawk® 5: Revolutionizing Power Efficiency for AI Infrastructure](https://www.broadcom.com/company/news/articles/innovation/tomahawk-5-revolutionizing-power-efficiency-for-ai-infrastructure)  
  ![Tomahawk® 5: Revolutionizing Power Efficiency for AI Infrastructure](https://www.broadcom.com/media/blt4ac44e0e6c6d8341/blt95c1bcb2b4aa03e4/69be0957544b1e764f03e41d/wwwhero_%281%29.jpg)
