Aviz Networks Blogs

  Introduction Moving networking functions into each host addresses a scaling issue but creates an operations issue. Instead of a single leaf carrying the state for a rack, you have a router in every server. These routers need consistent BGP, EVPN, and tenant configuration and need to be in sync with the leaf switches they peer with. This document will address the operations side of the problem. We will demonstrate how the design and implementation of a fabric leads to a running HBN fabric and discuss where automation and validation prevent the problems that would occur with manual work. How Is HBN Turned On in ONES? Configuration is in the Network Setup section under N-S Network Settings. Enabling DPU HBN requires just one switch, but changes how ONES models the entire fabric. VTEP termination is now done at the DPU, not the leaf. ONES programs the DPUs directly, requiring DPU IPs and credentials instead of host entries. Five coupled settings define the N-S network model. The Oper...

Scaling AI systems mainly focuses on computational units. Networks are considered "plumbing" and assumed to work. In reality, there are significant differences in architecture and behavior of AI systems between hundreds of nodes and tens of thousands of nodes. Unfortunately, many AI projects remain "stuck" at the factory stage due to the inability to scale beyond hundreds of nodes. Why does scale change the problem instead of just making it bigger? Nicolas Kremer, CTO and Co-Founder of Polarise, put it plainly on a recent episode of The Human Voice in the Age of AI : "I have this great solution. But then I ask them, okay, how do I scale that across 10,000 different nodes? And then it gets interesting, right?" The interesting part is not just a bigger version of the same problem. The problem is multi-tenancy, predictable automation, and failure recovery all need to work together to support the main functions of the system. This is typically not an issue dur...

Planning compute for an AI cluster is more interesting work, and, accordingly, gets more planning attention. After that, network planning can get short shrift. As a result, there are many “gotchas” waiting to bit the unwary. This checklist attempts to address the planning concerns across the network for an AI cluster. Day-0: Planning This requirements specification describes how tenants will be isolated at the application and at the network layers. You probably won't need to purchase an additional stack because a lot of tools that integrate well with your platform (like enterprise Linux) already exist. Identify your requirements for support of disconnected or air-gapped environments. Day-1: Deployment Integrate your network setup with your workload orchestration to provide new tenants with network configuration automatically. You should turn on telemetry from the start of your project. Adding it later will be harder. Test that one tenant's traffic can't affect another's...

The transformation of enterprise infrastructure happens in two directions at the same time. In some cases, fast-changing and extremely distributed workloads need to be handled by enterprise infrastructures. On the other hand, enterprise infrastructures need to provide predictability and minimize operational costs. Managing multi-tenancy throughout the infrastructure and applications is vital in both cases. Also, constant visibility to application and network behaviors is required. With the increase in workload number and variety, quickly and efficiently resolving customer issues becomes increasingly difficult. Aviz Networks, together with Red Hat, provides solutions for workload and application lifecycle management with Red Hat OpenShift and Red Hat Enterprise Linux. For management of the network layer, they provide solutions at the Day-0 to Day-2 operations of AI fabrics. Aviz Service Node (ASN) provides a range of advanced capabilities such as deep packet inspection, packet-level dif...

Networking for data centers is adapting to AI. Since GPU clusters are scaling up in size, they require very synchronized communication that can handle huge bandwidth as well as ultra-low latency. An Aviz Networks podcast features Taylor Allison, NVIDIA Senior Product Marketing Manager, who talks about the trend of networking with the current AI wave. During this conversation Allison touches upon how communication is essential in the AI training process, where GPUs transfer gradients back and forth, making network a crucial part of total workload performance. NVIDIA's Spectrum-X Ethernet, along with their InfiniBand options, has the capabilities to maintain performance in high demanding environments and help manage network related to AI. The interview further delves into NVIDIA's new Air feature that helps in deploying and configuring services by leveraging a Digital Twin feature, this can ultimately limit risk in planning for day 0 operations. Day 0 operations are by far the be...

The AI network doesn't just begin when the first GPU is plugged into a switch. A day zero in NVIDIA Spectrum-X involves topology definition, IP assignment, BGP setup, QoS settings, etc. At scale, the human effort that goes into configuring all of these components can turn into a substantial engineering effort. Intent-based automation can alleviate these difficulties. Starting with basic requirements such as Scale Unit and number of GPUs, IP ranges to utilize, Aviz ONES Fabric Designer can generate both the design of the fabric and configuration for the desired environment. This takes care of things on both ends of the system, from the IPCLOS underlay (leaf and spine setup, BGP configuration, ASN assignment, IP address, interfaces and MTU setup) all the way up to server-side setup (RoCEv2 parameters with PFC, and ECN, BlueField-3 settings, etc.). But setting up is only one hurdle. Validation is the other, so using NVIDIA AIR enables validation of the design, before rolling it out in...