https://network-switch.com/pages/about-us
Quick take
This is not a buying guide. It's the rack-level deployment runbook we use when we have to bring up 100G-400G links in AI/HPC pods and keep them stable after the rack gets crowded, hot, and touched by real humans.
What this runbook is
- A repeatable method for deploying DAC / ACC / AEC / AOC side by side inside a pod without turning operations into constant firefighting.
What it is not
- A "link-up = success" checklist.
- A spec-table shootout.
The 3 non-negotiables we enforce (because we learned the hard way)
- One routing template per rack type - route first, then length.
We stopped letting "same BOM, different cable path" create different channel behavior. - Acceptance is based on counter trends under sustained load - not link-up.
If we can't show stability under realistic traffic, we treat it as not qualified. - Mixed media is allowed only with documentation + change control.
You can mix DAC/AEC/AOC in one pod, but only if the pod has a single source of truth and triggers for re-qualification.
Pod wiring blueprint: the map we standardize
1. Pod link taxonomy (by function, not distance)
We don't organize cabling decisions by "meters" first. We organize them by what the link is responsible for:
- Compute-facing links: NIC ↔ leaf/ToR (dense, hot, frequently touched)
- Fabric-facing links: leaf ↔ spine (repeatability matters more than cleverness)
- Storage-facing links: compute ↔ storage leaf / storage spine (sensitive to microbursts and consistency)
- Pod-edge links: egress / DCI / handoff (change control and documentation matter most)
This taxonomy helps because the failure modes are different. A compute-facing link fails from crowding and handling. A fabric-facing link fails from inconsistency and change drift.
2. Media intent per link type (why we place DAC/AEC/AOC where we do)
We choose media based on intent, risk, and operational cost, not because one label sounds "better."
Compute-facing links (NIC ↔ leaf/ToR)
- Intent: keep the rack manageable and stable under heat + bundling
- Risk: tight bends behind ports, dense bundles, maintenance touch
- Typical mix: DAC when routes are clean; ACC/AEC when we see margin drift; AOC when bulk/airflow dominates
Fabric-facing links (leaf ↔ spine)
- Intent: repeatability across racks and across rollout waves
- Risk: inconsistent routing templates or mixing batches without traceability
- Typical mix: the simplest medium that passes our gates; we avoid "creative variety" here
Storage-facing links
- Intent: stable behavior under sustained load patterns
- Risk: "looks fine until it doesn't" under real IO and congestion
- Typical mix: whatever survives sustained load + counters + bundling; we keep documentation tight
Pod-edge links
- Intent: clean change control across teams
- Risk: upgrades, audits, and drift in topology/ownership
- Typical mix: usually optics + fiber if reach/plant dominates; we treat these links like interfaces, not just cables
The bring-up incident that changed our method
We'll keep this generic (no customer details), but the story pattern is real-and common.
1. Timeline (what happened, in six bullets)
- Staging: Bench tests looked clean. Links came up instantly. Quick throughput tests passed.
- Wave-1 rollout: First racks went live and looked "fine" in basic checks.
- Under real load: Counters started drifting on a subset of links, especially during sustained traffic windows.
- Maintenance touch: After a routine tidy-up, a few links became movement-sensitive.
- Isolation: The behavior correlated with routing density and bend points more than with a single SKU label.
- Fix: We stopped treating cabling as a commodity and built a routing + acceptance system.
2. What we got wrong (three mistakes we now prevent)
-
We validated single links, not bundled reality.
We tested "a cable," not "a cable in a dense tray with real neighbors." -
We didn't pin port mode + lane mapping in one source of truth.
A mis-mapped breakout leg can look exactly like a hardware issue. -
We qualified without freezing firmware/NOS baselines.
We learned that upgrades change behavior-even when nothing "looks different."
3. What we changed (three changes that cut troubleshooting)
- Routing templates per rack type
- Acceptance data pack per rollout wave (so we can prove what changed)
- Batch sampling rules (so a single inconsistent batch doesn't scale into an incident)
The routing template system: how we stop "same BOM, different channel"
This is the part that makes the pod boring-in a good way.
1. Route-first length selection (how we measure)
We stopped ordering by guesswork and started ordering by path.
Our internal length logic is simple:
- tray path length (actual route, not straight-line)
- service loop policy (enough slack to service, not enough to create chaos)
- connector-side offsets (left/right port banks and top/bottom routes are not symmetric)
Why this matters: when people order "close enough" lengths, they compensate in the rack with tight bends and pull tension-exactly where copper assemblies become margin-sensitive.
2. Bundle discipline for AI racks (rules we enforce)
AI racks push density and heat. We treat bundling as a first-class engineering constraint.
We enforce three practical policies:
- Bundle density has a cap. If cables are pressed into a "brick," you're increasing coupling and stress points.
- Bend points get audited. The most dangerous bends are not in the middle of the tray-they're right behind the port where people force a turn.
- Airflow keep-out zones are real. We don't route heavy bundles across known hot flow paths just because it looks tidy.
3. A labeling schema that prevents breakout leg mistakes
Breakout mistakes are common because the cable looks "obvious" until it isn't.
We use a label format that works even at 2am:
RackID-Device-Port-Lane/Leg-MediaType-Batch
Example (just a pattern, not a mandate):
- R12-LEAF1-Eth1/49-L2-AEC-B2401
- R12-LEAF1-Eth1/49-L3-AEC-B2401
The point is not aesthetics. The point is that every cable is traceable back to:
- where it's supposed to land
- what leg/lane it represents
- what medium it is
- what batch it came from
Qualification & acceptance: what must be true before we scale
We treat qualification as a set of gates and a set of artifacts. If you can't prove it later, you didn't qualify it-you just got lucky.
1. Qualification gates (pass/fail)
Gate 1 - Workload run (under sustained load)
- If counter trends drift upward under realistic traffic, we treat the link as near-edge.
Gate 2 - Bundled-as-installed
- If a link is clean alone but drifts when bundled, the rack is part of the channel. We test it that way.
Gate 3 - Maintenance disturbance check
- We simulate a realistic touch/re-route. If a link becomes movement-sensitive, it's not rollout-safe.
Gate 4 - Strictest-platform replay
- If the pod mixes platform families, we replay the test on the strictest/touchiest platform first.
2. The acceptance data pack (what we record every rollout)
This is what stops arguments later.
Our "acceptance data pack" includes:
- firmware/NOS baselines
- port mode / channelization snapshot (especially for breakout)
- counter baseline + time-series trend during sustained load
- cabling map export (what goes where)
- batch IDs for traceability
We once lost hours because a rollout had no reliable record of "what changed." Now we don't scale a wave without this pack.
Mixed media without chaos: how we keep DAC/AEC/AOC coexisting safely
Mixing media is normal in AI/HPC pods. The mistake is mixing media without rules.
1. Boundaries: where we refuse to mix
We avoid mixing in places where heterogeneity creates hard-to-debug variability:
- critical fabric uplinks where repeatability is everything
- any link class where multiple teams touch it without shared documentation
Mixing is more acceptable on compute-facing links if:
- routing templates are enforced
- acceptance packs are captured
- change control triggers are followed
2. Change control triggers (what forces re-qualification)
These are the events that have burned teams repeatedly:
- firmware/NOS major update
- new platform family (different ASIC/PHY tolerance)
- batch change (new manufacturing lot)
- routing change (tray redesign, new bundle density, new bend points)
If any of these happen, we treat it as: re-qualify small, then scale.
3. Spare strategy: what we actually keep on hand
"Having spares" isn't a strategy. Having known-good spares is.
We keep:
- at least one known-good cable per link class that has passed our gates
- spares that match the same media type and, when possible, known batch behavior
This saves time because in an incident you want a swap that is a real control-not a new variable.
Fast isolation runbook: the 10-minute triage we actually use
This is the minimal loop that prevents endless guessing.
1. The 4-step isolation loop
-
Follow the cable vs follow the port
Move the suspect cable and observe whether the behavior follows it. -
Single vs bundled
If it fails only in a bundle, your "environment" is part of the failure. -
Known-good substitution
Swap in a known-good control and watch whether trends stabilize. -
Config sanity check first (especially breakout)
Confirm port mode and lane mapping before you blame hardware.
2. Symptoms likely cause (mini map)
- Counters drift only under sustained load margin/thermal/bundling pressure
- Movement-sensitive after re-route bend/strain near termination
- Only one rack affected routing template violation or tray density difference
- Works before upgrade, fails after validation/tuning policy change; re-qualify
FAQs
Q1: In pod qualification, what's the single most reliable signal that a link is near-edge?
A: A trend of rising FEC corrections under sustained load, especially if it correlates with bundling or peak temperature windows.
Q2: We once had a wave that passed bench tests but drifted in production-what did we fail to standardize?
A: The routing template; "same BOM" becomes different channels when tray path, bends, and bundle density vary rack to rack.
Q3: How do you prevent a breakout leg mix-up from becoming a "hardware incident"?
A: Freeze port mode + lane mapping in the acceptance pack and enforce a leg-aware label schema; verify mapping before any cable swaps.
Q4: We once passed a throughput snapshot but still saw application pain-what metric did we ignore?
A: Counter trends over time; short tests can hide retransmits/corrections that show up only under sustained workload patterns.
Q5: What's your minimum acceptance gate before scaling mixed media in the same pod?
A: Each link class must pass load stability, bundled-as-installed stability, and maintenance disturbance tolerance with documented baselines.
Q6: How do you keep batch variation from turning into a pod-wide issue?
A: Track batch IDs, qualify a small sample from each batch on the strictest platform, then release inventory in controlled waves.
Q7: What's the fastest way to separate "port tolerance" from "cable/channel" issues in a hot rack?
A: Swap a known-good cable on the same port, then move the suspect cable to a different port; follow-the-cable vs follow-the-port isolates quickly.
Q8: When do you stop trying to make copper work and move to AOC/optics?
A: When the dominant constraint is bulk/airflow/routing space or reach-at that point optics solves the operational problem more cleanly than tuning copper.
Summary
AI/HPC pods don't fail because teams didn't know what DAC or AEC means. They fail because teams couldn't keep behavior consistent across dense racks, rollout waves, and maintenance touches.
That's why our runbook is built around three things:
- routing templates (route first, then length)
- acceptance data packs (so changes are visible and debuggable)
- change control triggers (so upgrades and batches don't surprise you)
If your pod can pass sustained-load counter trends, bundled-as-installed reality, and maintenance disturbance checks-then you've earned the right to scale.
Did this article help you or not? Tell us on Facebook and LinkedIn . We’d love to hear from you!
