Posts

How SONiC 202611 Shows You Exactly Why Packets Are Dropped

Image
Packet loss in an AI factory is hard to troubleshoot. Distributed compute depends on the network across large clusters, and when packets go missing, engineers need to know three things: which packets, where they were dropped, and why. Interface counters only answer one question: did drops happen? SONiC 202611 changes that with per-packet drop notifications carried in sFlow. Why counters fall short Look at what a counter gives you on leaf-2: Ethernet12 rx_discards +4,112 over 5 min Now look at what a drop notification gives you for the same event: 4,112 records showing traffic from 10.1.4.7 to 10.1.9.2 on UDP 4791, ingress Ethernet12, group acl_drops , trap ingress_flow_action_drop . In plain terms, a RoCEv2 flow that a new ACL just blocked. The counter is a number. The notification is an answer. The problem with vendor-only tools Vendor drop telemetry has existed for years. NVIDIA’s What Just Happened is the best-known example. But it works on one ASIC family and one export path. SONi...
Post a Comment
Read more

What Does It Take to Run a Multi-Tenant GB200 or GB300 GPU Service Reliably?

Image
Introduction Once an operator decides to offer rack-scale GPUs as a shared service, the question becomes, "How do we ensure tenant isolation?" rather than, "Can we offer tenant isolation?" NVL72 systems interconnect 72 Blackwell GPUs using NVLink, and racks are interconnected using an InfiniBand scale-out fabric. Isolation is needed at both the interconnect and fabric level, and both have to agree. Here we focus on the interconnect side of this problem. Why Isn't a VLAN Enough Here? As NVLink doesn’t follow traditional Ethernet behaviors, methods used to isolate Ethernet tenants do not work here. Isolation needs to occur in the fabric. Full NVLink bandwidth is preserved in the hardware-level GPU partition. There is no connectivity to other partitions. How Do Single-Rack and Multi-Rack Tenants Differ? In a single rack, a tenant requires NVLink partitioning. A tenant spanning multiple racks requires NVLink partitions and a private InfiniBand partition spanning the...
Post a Comment
Read more

How Do You Operate Host-Based Networking Across Hundreds of DPUs?

Image
  Introduction Moving networking functions into each host addresses a scaling issue but creates an operations issue. Instead of a single leaf carrying the state for a rack, you have a router in every server. These routers need consistent BGP, EVPN, and tenant configuration and need to be in sync with the leaf switches they peer with. This document will address the operations side of the problem. We will demonstrate how the design and implementation of a fabric leads to a running HBN fabric and discuss where automation and validation prevent the problems that would occur with manual work. How Is HBN Turned On in ONES? Configuration is in the Network Setup section under N-S Network Settings. Enabling DPU HBN requires just one switch, but changes how ONES models the entire fabric. VTEP termination is now done at the DPU, not the leaf. ONES programs the DPUs directly, requiring DPU IPs and credentials instead of host entries. Five coupled settings define the N-S network model. The Oper...
Post a Comment
Read more

How Do You Keep a Network Reliable When an AI Factory Scales to 10,000 Nodes?

Image
Scaling AI systems mainly focuses on computational units. Networks are considered "plumbing" and assumed to work. In reality, there are significant differences in architecture and behavior of AI systems between hundreds of nodes and tens of thousands of nodes. Unfortunately, many AI projects remain "stuck" at the factory stage due to the inability to scale beyond hundreds of nodes. Why does scale change the problem instead of just making it bigger? Nicolas Kremer, CTO and Co-Founder of Polarise, put it plainly on a recent episode of The Human Voice in the Age of AI : "I have this great solution. But then I ask them, okay, how do I scale that across 10,000 different nodes? And then it gets interesting, right?" The interesting part is not just a bigger version of the same problem. The problem is multi-tenancy, predictable automation, and failure recovery all need to work together to support the main functions of the system. This is typically not an issue dur...
Post a Comment
Read more

AI Factory Networking: A Simple Day-0 to Day-2 Checklist

Image
Planning compute for an AI cluster is more interesting work, and, accordingly, gets more planning attention. After that, network planning can get short shrift. As a result, there are many “gotchas” waiting to bit the unwary. This checklist attempts to address the planning concerns across the network for an AI cluster. Day-0: Planning This requirements specification describes how tenants will be isolated at the application and at the network layers. You probably won't need to purchase an additional stack because a lot of tools that integrate well with your platform (like enterprise Linux) already exist. Identify your requirements for support of disconnected or air-gapped environments. Day-1: Deployment Integrate your network setup with your workload orchestration to provide new tenants with network configuration automatically. You should turn on telemetry from the start of your project. Adding it later will be harder. Test that one tenant's traffic can't affect another's...
Post a Comment
Read more

Bridging Workload Orchestration and Network Operations in AI Factories

Image
The transformation of enterprise infrastructure happens in two directions at the same time. In some cases, fast-changing and extremely distributed workloads need to be handled by enterprise infrastructures. On the other hand, enterprise infrastructures need to provide predictability and minimize operational costs. Managing multi-tenancy throughout the infrastructure and applications is vital in both cases. Also, constant visibility to application and network behaviors is required. With the increase in workload number and variety, quickly and efficiently resolving customer issues becomes increasingly difficult. Aviz Networks, together with Red Hat, provides solutions for workload and application lifecycle management with Red Hat OpenShift and Red Hat Enterprise Linux. For management of the network layer, they provide solutions at the Day-0 to Day-2 operations of AI fabrics. Aviz Service Node (ASN) provides a range of advanced capabilities such as deep packet inspection, packet-level dif...
Post a Comment
Read more

AI's Hidden Bottleneck: The Network

Image
Networking for data centers is adapting to AI. Since GPU clusters are scaling up in size, they require very synchronized communication that can handle huge bandwidth as well as ultra-low latency. An Aviz Networks podcast features Taylor Allison, NVIDIA Senior Product Marketing Manager, who talks about the trend of networking with the current AI wave. During this conversation Allison touches upon how communication is essential in the AI training process, where GPUs transfer gradients back and forth, making network a crucial part of total workload performance. NVIDIA's Spectrum-X Ethernet, along with their InfiniBand options, has the capabilities to maintain performance in high demanding environments and help manage network related to AI. The interview further delves into NVIDIA's new Air feature that helps in deploying and configuring services by leveraging a Digital Twin feature, this can ultimately limit risk in planning for day 0 operations. Day 0 operations are by far the be...
Post a Comment
Read more