Aviz Networks Blogs

Posts

How SONiC 202611 Shows You Exactly Why Packets Are Dropped

October 22, 2026

Packet loss in an AI factory is hard to troubleshoot. Distributed compute depends on the network across large clusters, and when packets go missing, engineers need to know three things: which packets, where they were dropped, and why. Interface counters only answer one question: did drops happen? SONiC 202611 changes that with per-packet drop notifications carried in sFlow. Why counters fall short Look at what a counter gives you on leaf-2: Ethernet12 rx_discards +4,112 over 5 min Now look at what a drop notification gives you for the same event: 4,112 records showing traffic from 10.1.4.7 to 10.1.9.2 on UDP 4791, ingress Ethernet12, group acl_drops , trap ingress_flow_action_drop . In plain terms, a RoCEv2 flow that a new ACL just blocked. The counter is a number. The notification is an answer. The problem with vendor-only tools Vendor drop telemetry has existed for years. NVIDIA’s What Just Happened is the best-known example. But it works on one ASIC family and one export path. SONi...

What Does It Take to Run a Multi-Tenant GB200 or GB300 GPU Service Reliably?

October 14, 2026

Introduction Once an operator decides to offer rack-scale GPUs as a shared service, the question becomes, "How do we ensure tenant isolation?" rather than, "Can we offer tenant isolation?" NVL72 systems interconnect 72 Blackwell GPUs using NVLink, and racks are interconnected using an InfiniBand scale-out fabric. Isolation is needed at both the interconnect and fabric level, and both have to agree. Here we focus on the interconnect side of this problem. Why Isn't a VLAN Enough Here? As NVLink doesn’t follow traditional Ethernet behaviors, methods used to isolate Ethernet tenants do not work here. Isolation needs to occur in the fabric. Full NVLink bandwidth is preserved in the hardware-level GPU partition. There is no connectivity to other partitions. How Do Single-Rack and Multi-Rack Tenants Differ? In a single rack, a tenant requires NVLink partitioning. A tenant spanning multiple racks requires NVLink partitions and a private InfiniBand partition spanning the...

How Do You Operate Host-Based Networking Across Hundreds of DPUs?

October 06, 2026

Introduction Moving networking functions into each host addresses a scaling issue but creates an operations issue. Instead of a single leaf carrying the state for a rack, you have a router in every server. These routers need consistent BGP, EVPN, and tenant configuration and need to be in sync with the leaf switches they peer with. This document will address the operations side of the problem. We will demonstrate how the design and implementation of a fabric leads to a running HBN fabric and discuss where automation and validation prevent the problems that would occur with manual work. How Is HBN Turned On in ONES? Configuration is in the Network Setup section under N-S Network Settings. Enabling DPU HBN requires just one switch, but changes how ONES models the entire fabric. VTEP termination is now done at the DPU, not the leaf. ONES programs the DPUs directly, requiring DPU IPs and credentials instead of host entries. Five coupled settings define the N-S network model. The Oper...

How Do You Keep a Network Reliable When an AI Factory Scales to 10,000 Nodes?

September 30, 2026

Scaling AI systems mainly focuses on computational units. Networks are considered "plumbing" and assumed to work. In reality, there are significant differences in architecture and behavior of AI systems between hundreds of nodes and tens of thousands of nodes. Unfortunately, many AI projects remain "stuck" at the factory stage due to the inability to scale beyond hundreds of nodes. Why does scale change the problem instead of just making it bigger? Nicolas Kremer, CTO and Co-Founder of Polarise, put it plainly on a recent episode of The Human Voice in the Age of AI : "I have this great solution. But then I ask them, okay, how do I scale that across 10,000 different nodes? And then it gets interesting, right?" The interesting part is not just a bigger version of the same problem. The problem is multi-tenancy, predictable automation, and failure recovery all need to work together to support the main functions of the system. This is typically not an issue dur...

AI Factory Networking: A Simple Day-0 to Day-2 Checklist

September 25, 2026

Planning compute for an AI cluster is more interesting work, and, accordingly, gets more planning attention. After that, network planning can get short shrift. As a result, there are many “gotchas” waiting to bit the unwary. This checklist attempts to address the planning concerns across the network for an AI cluster. Day-0: Planning This requirements specification describes how tenants will be isolated at the application and at the network layers. You probably won't need to purchase an additional stack because a lot of tools that integrate well with your platform (like enterprise Linux) already exist. Identify your requirements for support of disconnected or air-gapped environments. Day-1: Deployment Integrate your network setup with your workload orchestration to provide new tenants with network configuration automatically. You should turn on telemetry from the start of your project. Adding it later will be harder. Test that one tenant's traffic can't affect another's...

Bridging Workload Orchestration and Network Operations in AI Factories

September 17, 2026

The transformation of enterprise infrastructure happens in two directions at the same time. In some cases, fast-changing and extremely distributed workloads need to be handled by enterprise infrastructures. On the other hand, enterprise infrastructures need to provide predictability and minimize operational costs. Managing multi-tenancy throughout the infrastructure and applications is vital in both cases. Also, constant visibility to application and network behaviors is required. With the increase in workload number and variety, quickly and efficiently resolving customer issues becomes increasingly difficult. Aviz Networks, together with Red Hat, provides solutions for workload and application lifecycle management with Red Hat OpenShift and Red Hat Enterprise Linux. For management of the network layer, they provide solutions at the Day-0 to Day-2 operations of AI fabrics. Aviz Service Node (ASN) provides a range of advanced capabilities such as deep packet inspection, packet-level dif...

Search This Blog