How SONiC 202611 Shows You Exactly Why Packets Are Dropped



Packet loss in an AI factory is hard to troubleshoot. Distributed compute depends on the network across large clusters, and when packets go missing, engineers need to know three things: which packets, where they were dropped, and why.

Interface counters only answer one question: did drops happen? SONiC 202611 changes that with per-packet drop notifications carried in sFlow.

Why counters fall short

Look at what a counter gives you on leaf-2:

Ethernet12 rx_discards +4,112 over 5 min

Now look at what a drop notification gives you for the same event:

4,112 records showing traffic from 10.1.4.7 to 10.1.9.2 on UDP 4791, ingress Ethernet12, group acl_drops, trap ingress_flow_action_drop. In plain terms, a RoCEv2 flow that a new ACL just blocked.

The counter is a number. The notification is an answer.

The problem with vendor-only tools

Vendor drop telemetry has existed for years. NVIDIA’s What Just Happened is the best-known example. But it works on one ASIC family and one export path. SONiC runs on Marvell, Broadcom, Cisco, NVIDIA and Innovium silicon, so a solution tied to one of them isn’t a SONiC solution.

There’s a second gap. Vendor tools stop at the ASIC. A meaningful share of packet loss happens after the packet reaches the CPU, in the kernel receive path, the backlog queue or socket buffers. No ASIC feature can see those drops.

How the new feature works

SONiC 202611 uses one channel for both kinds of drops:

  • Hardware drops: The ASIC raises a drop event to its driver. The driver translates it into a standard Linux netlink message on the NET_DM family, tagged as hardware origin.

  • Software drops: The Linux drop_monitor module already publishes kernel drops on the same channel, tagged as software origin, with the drop reason and the kernel function where it happened.

  • Export: The hsflowd agent reads that one channel and sends each drop as an sFlow discard record, along with normal flow samples and counters.

The only platform-specific part is the ASIC driver. Everything else is open and shared.

Drop reasons you will see

Drop reasons follow the Linux kernel’s devlink trap names, so they mean the same thing on SONiC as on any Linux host.

Group

Example reasons

l2_drops

ingress_vlan_filter, vlan_tag_mismatch

l3_drops

blackhole_route, unresolved_neigh

tunnel_drops

decap_error

acl_drops

ingress_flow_action_drop

buffer_drops

tail_drop, early_drop

How to enable it

sflow collector add ncp-collector 192.0.2.129 --vrf mgmt

sflow sample-direction both

sflow drop-monitor-limit 50

sflow enable

show sflow

To verify it works end to end:

  1. Check STATE_DB shows MIRROR_ON_DROP_CAPABLE: true.

  2. Confirm /etc/hsflowd.conf contains dropmon { limit = 50 }.

  3. Create a known drop. An ACL deny on a test flow is the simplest.

  4. Decode the record at your collector. sflowtool prints it clearly.

If your platform doesn’t support the feature, the CLI rejects the config. There is no silent failure.

About the rate limit

The limit accepts 0 to 500 notifications per second per switch. Setting 0 turns it off. Below the limit, every drop is reported. Above it, extra notifications are discarded by a policer. Forwarding is never affected.

On a healthy fabric, drops are near zero, so even a low limit captures everything. Fifty is a good starting point.

Things to keep in mind

  • Software drop visibility is on by default, even on platforms without ASIC support yet.

  • Any config change restarts hsflowd, which causes a brief gap in all sFlow data.

  • Don’t edit /etc/hsflowd.conf by hand. It is generated automatically.

  • For RoCEv2, make sure header truncation is long enough to reach the BTH header if you want to classify by opcode.

What’s validated today

On Marvell Teralynx 6512, these drop types were confirmed at the collector: egress queue congestion, ACL drops, route drops, buffer drops and policer drops. Each test fault could be told apart by its trap group alone, before looking at any packet header.

What this means for operators

Flow samples show traffic that crossed the fabric. Drop notifications show traffic that didn’t. Put together, they help separate policy and routing faults from congestion before anyone opens a queue counter.

Aviz Networks’ Network Copilot uses these records with interface counters and flow samples to answer the questions that matter: which flows, which switch and port, which reason, and whether the issue is isolated or fabric-wide.

Comments

Post a Comment

Popular posts from this blog

Scaling Deep Network Observability for 5G: Reflections from a Real Deployment

Image
  I recently went through a detailed implementation showing how deep network observability enhances 5G mobile user experience while reducing operational complexity. What stood out was how grounded the deployment was in real traffic and production-scale constraints. Some practical observations: • Fragmented systems increase cost and operational overhead • Frequent infrastructure upgrades create instability and repair expenses • Optimised filtering significantly reduces DPI hardware requirements • Real-time KPI computation improves SLA monitoring and service assurance • Standardised data export through streaming pipelines simplifies integration One misconception addressed during the walkthrough was that deeper observability automatically means more hardware. In reality, efficient filtering and intelligent packet processing reduce hardware footprint while improving performance. My biggest takeaway is that scalable observability in 5G environments requires architectural efficiency,...
Read more

Evolving Packet Brokering for Modern Network Observability

Image
  I recently reviewed a technical overview describing how packet brokering platforms are evolving to support large-scale observability in modern data centers and service provider networks. The discussion focused on scalability, automated deployment, and operational efficiency. Some practical observations: • Network visibility platforms are expanding support for 400G switching hardware used in modern data center fabrics • GRE encapsulation enables mirrored traffic to move across Layer-3 networks while preserving packet metadata • Zero-Touch Provisioning simplifies onboarding for new or factory-reset switches • Packet brokers aggregate traffic from TAPs and SPAN ports and apply filtering and distribution policies • CLI improvements help operators manage large monitoring environments more efficiently One notable element was how GRE tunneling extends observability beyond Layer-2 boundaries. By encapsulating mirrored traffic into GRE tunnels, monitoring systems can receive full packet c...
Read more

How Network Copilot Uses Agentic AI to Correlate FortiGate and Splunk

Image
I recently went through a detailed implementation explaining how Network Copilot correlates FortiGate firewall telemetry with operational data indexed in Splunk . What stood out was how grounded the discussion was in real operational workflows rather than theoretical AI claims. Some practical observations: • Network and security logs are typically distributed across multiple platforms • Troubleshooting often requires manually correlating events between systems • Network Copilot ingests metrics, events, logs, and traces from firewall environments • Operators can interact with the system using natural language questions • Explainable root cause analysis improves operational confidence One misconception addressed during the walkthrough was that AI in operations means replacing engineers. In reality, the system augments teams by surfacing insights faster and correlating information across environments. Another interesting point was how the platform learns from real firewall deployments...
Read more