Posts

From DPU to Fabric: Multi-Tenant Network Automation with NVIDIA NICo and Aviz ONES

Image
Multi-tenant AI infrastructure spans two networking domains. The first is the GPU host and DPU layer, where tenant networking begins. The second is the data center fabric, where that tenant needs to reach storage, shared services, and external networks. Both domains must work from the same tenant information. NVIDIA Infra Controller (NICo) and Aviz ONES handle this by splitting the work clearly. NICo manages GPU hosts and their BlueField-3 DPUs and sets the network identity for each tenant. ONES takes that identity and extends it across the AI fabric. NICo: The Source of Truth for Tenant Identity NICo works closest to the workload. When a GPU server is assigned to a tenant, NICo configures its BlueField-3 DPU with the details needed to identify and isolate that tenant: VLAN VNI VTEP information Since these values are created during host provisioning, NICo holds the authoritative tenant network identity. The fabric does not assign its own identifiers. It uses the ones NICo already a...
Post a Comment
Read more

ONES 4.4 Release: Key Features for AI Fabric Visibility and Automation

Image
Aviz Networks has released ONES 4.4, the latest version of its Open Networking Enterprise Suite. This release improves fabric visibility, access control, deployment options, and AI fabric automation for multi-tenant, multi-vendor environments. 1. Cumulus Buffer Histograms ONES 4.4 collects ASIC buffer histograms and shows them as heat maps. Two new widgets sit under Monitor → Traffic → Advanced: Egress Buffer Depth Histogram (per traffic class) and Ingress Buffer Depth Histogram (per priority group). Hovering over a cell shows the depth range, share of samples, and dwell time. 2. QoS Buffer Metrics Seven new widgets track ingress, egress, headroom, and pool shared buffer occupancy, plus shared buffer pool occupancy on the Capacity page. They help validate lossless (RoCE) configurations and catch buffer exhaustion early. 3. RADIUS Authentication ONES acts as a RADIUS client and hands credential checks to a RADIUS server such as FreeRADIUS, which validates users against OpenLDAP or Activ...
Post a Comment
Read more

FTAS 4.1: AI Factory Automation, Expanded FMCLI Coverage, and Large-Scale Validation

Image
We’re excited to announce FTAS 4.1. This release adds 142 new test cases and brings the framework to 1,072 test cases in total. Its headline feature is automated validation for AI Factory fabrics. Spotlight: AI Factory (N-S) automation FTAS 4.1 adds an ordered 39-case AI Factory test plan that checks the full ONES-driven fabric lifecycle: Design and planning: fabric types, scalable units, tenancy modes and network planes. Deployment and onboarding: fabric deployment and tenant orchestration. Control-plane parity: ONES API inventory compared with the switch CLI. Data-plane proof: Ixia traffic across the deployed fabric. By the numbers Category Total SONiC Framework 599 FMCLI 188 Telemetry 285 Total FTAS test cases 1072 What else is new FTAS with FMCLI: 54 new test cases bring FMCLI coverage to 188. BGP convergence at IPv6 scale: 18 tests at 32k, 64k and 128k routes. L3VNI multi-tenancy: 12 tests cover 3 VRFs with 128 VLANs. BFD session scale: 4 tests with 64 and 128 sessions. A...
Post a Comment
Read more

How SONiC 202611 Shows You Exactly Why Packets Are Dropped

Image
Packet loss in an AI factory is hard to troubleshoot. Distributed compute depends on the network across large clusters, and when packets go missing, engineers need to know three things: which packets, where they were dropped, and why. Interface counters only answer one question: did drops happen? SONiC 202611 changes that with per-packet drop notifications carried in sFlow. Why counters fall short Look at what a counter gives you on leaf-2: Ethernet12 rx_discards +4,112 over 5 min Now look at what a drop notification gives you for the same event: 4,112 records showing traffic from 10.1.4.7 to 10.1.9.2 on UDP 4791, ingress Ethernet12, group acl_drops , trap ingress_flow_action_drop . In plain terms, a RoCEv2 flow that a new ACL just blocked. The counter is a number. The notification is an answer. The problem with vendor-only tools Vendor drop telemetry has existed for years. NVIDIA’s What Just Happened is the best-known example. But it works on one ASIC family and one export path. SONi...
Post a Comment
Read more

What Does It Take to Run a Multi-Tenant GB200 or GB300 GPU Service Reliably?

Image
Introduction Once an operator decides to offer rack-scale GPUs as a shared service, the question becomes, "How do we ensure tenant isolation?" rather than, "Can we offer tenant isolation?" NVL72 systems interconnect 72 Blackwell GPUs using NVLink, and racks are interconnected using an InfiniBand scale-out fabric. Isolation is needed at both the interconnect and fabric level, and both have to agree. Here we focus on the interconnect side of this problem. Why Isn't a VLAN Enough Here? As NVLink doesn’t follow traditional Ethernet behaviors, methods used to isolate Ethernet tenants do not work here. Isolation needs to occur in the fabric. Full NVLink bandwidth is preserved in the hardware-level GPU partition. There is no connectivity to other partitions. How Do Single-Rack and Multi-Rack Tenants Differ? In a single rack, a tenant requires NVLink partitioning. A tenant spanning multiple racks requires NVLink partitions and a private InfiniBand partition spanning the...
Post a Comment
Read more

How Do You Operate Host-Based Networking Across Hundreds of DPUs?

Image
  Introduction Moving networking functions into each host addresses a scaling issue but creates an operations issue. Instead of a single leaf carrying the state for a rack, you have a router in every server. These routers need consistent BGP, EVPN, and tenant configuration and need to be in sync with the leaf switches they peer with. This document will address the operations side of the problem. We will demonstrate how the design and implementation of a fabric leads to a running HBN fabric and discuss where automation and validation prevent the problems that would occur with manual work. How Is HBN Turned On in ONES? Configuration is in the Network Setup section under N-S Network Settings. Enabling DPU HBN requires just one switch, but changes how ONES models the entire fabric. VTEP termination is now done at the DPU, not the leaf. ONES programs the DPUs directly, requiring DPU IPs and credentials instead of host entries. Five coupled settings define the N-S network model. The Oper...
Post a Comment
Read more

How Do You Keep a Network Reliable When an AI Factory Scales to 10,000 Nodes?

Image
Scaling AI systems mainly focuses on computational units. Networks are considered "plumbing" and assumed to work. In reality, there are significant differences in architecture and behavior of AI systems between hundreds of nodes and tens of thousands of nodes. Unfortunately, many AI projects remain "stuck" at the factory stage due to the inability to scale beyond hundreds of nodes. Why does scale change the problem instead of just making it bigger? Nicolas Kremer, CTO and Co-Founder of Polarise, put it plainly on a recent episode of The Human Voice in the Age of AI : "I have this great solution. But then I ask them, okay, how do I scale that across 10,000 different nodes? And then it gets interesting, right?" The interesting part is not just a bigger version of the same problem. The problem is multi-tenancy, predictable automation, and failure recovery all need to work together to support the main functions of the system. This is typically not an issue dur...
Post a Comment
Read more