ECMP Not Load Balancing on a Layer 3 Switch: A Real H3C S6500 Troubleshooting Case

Published: August 27, 2026 | Last Technically Reviewed: August 27, 2026

Historical H3C S6500 Broadband Metro Network Field Case

Conclusion

In this historical H3C S6500 field incident, ECMP traffic sharing failed because the expected upstream route for one of two equal-cost Layer 3 uplinks was missing from the routing table, even though the interface was up, ARP was present, the peer responded to ping, and the configuration showed no obvious error. One uplink therefore carried very little traffic while most traffic used the other path. Resetting the affected interface restored the missing route and dual-uplink traffic sharing, but this was only a temporary recovery action. H3C R&D later identified a route-table programming/write abnormality as the immediate technical cause. High CPU load during bulk route writes was considered a possible deeper factor, but the surviving records do not establish it as the definitive root cause.

Introduction

During evening peak traffic, users connected through several H3C S6500 trial sites began reporting gaming lag. The affected GE access interfaces were up, interface checks showed no obvious errors, and metro-network speed tests were generally acceptable. What made the incident difficult was that the packet loss was intermittent rather than continuous.

The switch had two equal-cost Layer 3 uplinks configured, but repeated path observations and interface statistics eventually showed that only one uplink was carrying meaningful traffic.

The question was straightforward:

The answer was not visible at the port level. The decisive evidence appeared only after we compared ARP state, routing-table state and the forwarding path that traffic was actually using.

Historical Project Context

This incident came from the initial field-trial period of H3C's first-generation S6500 general-purpose mid-to-high-end Layer 3 switch.

At the end of 2004, the newly developed S6500 was deployed across 112 broadband metro-network trial sites at a friendly telecom operator. As real subscriber traffic was gradually migrated onto the platform, the trial shifted from laboratory validation into production-traffic testing.
To support the trial, the S6500 R&D organization rotated two development engineers on long-term assignment locally. They worked with local IP engineers and partner engineers as a joint field-support team, collecting fault information and feeding recurring issues back to R&D.

Historical Project Note: This case is reconstructed from surviving engineering records and first-hand project experience from the original S6500 trial. Where those records do not preserve an exact software version, CLI output, traffic value, packet-loss percentage, CPU percentage or definitive deeper root cause, this article does not recreate those details.

Where This Incident Fits in the Full S6500 Trial

The ECMP incident was not the only issue exposed after the 112 trial sites began carrying real broadband traffic.

The surviving project record documents several different problem classes during the trial, including:

1. ARP process hang

2. Interface VLAN scale limitation

3. Forwarding/FIB programming abnormality

4. Route-table programming abnormality - the incident covered here

5. High CPU and memory utilization under abnormal traffic

6. Port-chip hang or failure
   These were separate incidents.

In particular, the FIB programming abnormality described elsewhere in the trial must not be merged with the route-table programming abnormality in this article. They involved different symptoms and different technical findings.

This article focuses only on the route/ECMP incident because it provides a particularly useful Layer 3 troubleshooting example.

The Access Interfaces Looked Healthy

The first checks did not point toward a Layer 3 routing problem.

Field engineers inspected the affected subscriber-side GE port for:

• operating mode;

• interface state;

• error rate;

• packet-error counters.

No obvious abnormality was found. The team also visited the affected internet café during the evening peak period and checked the connected access equipment and port operating mode. Again, nothing obvious explained the gaming lag.

This matters because intermittent packet loss often pushes troubleshooting toward the physical interface first.

That is a reasonable place to start, but the absence of physical errors does not end the investigation.

The physical port can be operational while the forwarding state above it is incomplete.

Throughput Looked Normal, but Packet Loss Did Not

The team then tested the service from the user side.

The surviving project notes record three important checks:

• speed testing inside and outside the metro network;

• segmented packet-loss testing;

• segmented traceroute.

The throughput tests were basically acceptable.

But the packet-loss tests showed irregular, intermittent losses. Sometimes a single test packet was lost. At other times two or three packets were lost, and occasionally four or five were lost close together.

The historical record does not preserve a packet-loss percentage, throughput value, latency figure or jitter measurement, so none should be reconstructed.

The engineering lesson from this stage is simple:

A bulk transfer can appear acceptable while a periodic forwarding fault remains very noticeable to latency-sensitive applications such as online gaming.

Traceroute Gave Us the First Important Clue

The affected S6500 was configured with two equal-cost upstream routes.

During repeated segmented traceroute testing, however, the observed path from the user side toward the metro-network exit kept following the same upstream direction.

That observation had to be interpreted carefully.

Repeated traceroutes following the same path are not proof by themselves that ECMP is broken. Equal-cost forwarding behavior depends on the forwarding implementation and hashing behavior, and a set of probes can legitimately remain on the same path.

But in this deployment, the repeated path observation was enough to justify checking whether both configured uplinks were actually participating in forwarding.

It was the first strong clue-not the conclusion.

One Uplink Was Carrying Almost No Traffic

Back at the S6500 site, the team checked both upstream interfaces again.

This time, the inspection included:

• operating mode;

• interface state;

• traffic statistics.

The critical observation was that one of the two upstream interfaces showed very little traffic compared with the other.

At that point, the troubleshooting hypothesis became much stronger:

Two equal-cost uplinks configured
            
Both physical interfaces available
            
Only one carries meaningful traffic
            
Peak traffic concentrates on one path
            
Congestion
            
Intermittent packet loss
            
Gaming lag

This was still a working hypothesis.

The low traffic counter showed that traffic sharing was not functioning as expected, but it did not yet explain why.

The Configuration Still Looked Correct

The team continued upward through the protocol state.

They reviewed:

• VLAN configuration;

• interface VLAN configuration;

• routing protocol configuration.

No obvious configuration error was found.

That narrowed the problem further.

The intended topology existed in the configuration, yet the operational traffic pattern did not match it.

This distinction is fundamental in Layer 3 troubleshooting.

A switch can contain the expected configuration while one of the internal protocol or forwarding states required to make that configuration operational is missing.

ARP Was Present, but the Expected Route Was Missing

The next checks produced the most important evidence in the case.

The affected uplink had a normal ARP entry.

The peer IP could also be pinged successfully.

But when the routing table was checked, the expected upstream route associated with that uplink was missing.

Troubleshooting State Table

This combination explains why the incident was difficult to isolate.

Several individual checks looked healthy.

The physical interface was available.

The Layer 2/neighbor state existed.

Basic IP reachability worked.

The configuration appeared correct.

But the route needed for the second equal-cost path was absent.

The Successful Ping Was Misleading

The successful ping initially looked contradictory.

How could ARP be normal and the peer respond to ping if the expected route through that interface did not exist?

The working suspicion was that the ping traffic was not actually using the affected uplink.

The engineers therefore traced the peer IP itself. The path confirmed that the traffic was reaching the peer by going around through the other working upstream interface.

That resolved the apparent contradiction.

The peer was reachable.

But it was not being reached through the path the team was trying to validate.

That is one of the most transferable lessons from this entire trial.

A successful ping answers a reachability question. It does not necessarily validate path selection.

Resetting the Interface Restored the Route

Before making a recovery change, the field engineer collected the fault and debug information requested by the development team.

The affected uplink was then reset using a shutdown followed by an undo shutdown operation.

After the interface returned:

• ARP remained normal;

• the previously missing upstream route reappeared;

• dual-uplink traffic sharing was verified as restored.

This distinction is critical:

The incident should not be converted into a generic recommendation that an ECMP problem should be solved by bouncing an interface.

In this case, resetting the port restored the missing operational state and allowed the site to recover while the collected information was analyzed.

What the Development Team Found

Field teams collected fault and debug information from multiple affected sites and sent it back to the S6500 development team.

The R&D analysis ultimately identified the immediate technical problem as:

The project record further states that excessive CPU load during large batches of route writes was considered a possible explanation for why the route-table write could become abnormal.

But the wording of the historical source matters.

It does not establish high CPU load as the fully proven underlying root cause.

The defensible conclusion is:

That evidence boundary should remain intact.

What We Observed vs. What We Concluded

This distinction is important because the field symptom, the troubleshooting hypothesis, the immediate technical cause and the deeper root cause are not the same thing.

What This Case Taught Us About Layer 3 Troubleshooting

The most useful method from this incident is not a command sequence.

It is the order in which operational state was verified:

User symptom
      
Packet-loss test
      
Path observation
      
Uplink traffic counters
      
Configuration
      
ARP / neighbor state
      
Routing table
      
Actual forwarding path

When redundant Layer 3 uplinks appear healthy but traffic is not actually being shared, check the operational forwarding state, not only the physical interface and intended configuration.

The incident can be reduced to four states that should not be treated as equivalent:

The most misleading part of this case was that almost every isolated check looked healthy: the interface was up, ARP existed, the peer responded to ping and the configuration looked correct.

The problem became visible only when the expected forwarding path was compared with the traffic actually using the two uplinks.

Want the Full H3C S6500 Field Troubleshooting Story?

This article covers only one Layer 3 routing/ECMP incident from the broader 112-site S6500 trial.

The historical trial also exposed separate issues involving an ARP process hang, interface VLAN scale limitations, a distinct FIB programming abnormality, abnormal CPU/memory utilization, worm-traffic protection requirements and port-chip hang/failure. Field teams and R&D responded with troubleshooting, temporary mitigation, software changes and continued trial-site validation.

Download the complete H3C S6500 troubleshooting white paper to see how the full 112-site trial evolved and how the field and development teams responded to the other major issues encountered during deployment.

More Real-World Network Deployment and Troubleshooting Cases

The original S6500 trial was a historical carrier broadband project. Current enterprise, campus, hotel, retail and SMB networks operate with different equipment generations, software and traffic requirements.

The transferable value is the troubleshooting discipline.

Today, a project review may still require us to compare the actual topology, equipment role, port state, protocol state, routing, uplink capacity, optics, PoE, software version and interoperability rather than making a recommendation from a specification sheet alone.

You can explore our real-world network deployment and troubleshooting case studies for additional examples of design, migration and fault-isolation work.

The original S6500 in this article belongs to an earlier product generation. For current projects, current H3C enterprise switches should be evaluated according to present Layer 3 requirements, uplink capacity, redundancy, security features, supported software and the actual deployment topology.

The current products in that collection should not be interpreted as equipment used in the historical S6500 trial or as evidence that modern H3C switches share the same historical defect.

Resource

Internal-Link Anchor List

• complete H3C S6500 troubleshooting white paper
  https://network-switch.com/pages/white-paper-troubleshooting-the-h3c-s6500-layer-3-switch

• real-world network deployment and troubleshooting case studies
  https://network-switch.com/blogs/case-studies

• current H3C enterprise switches
  https://network-switch.com/collections/all-h3c-switches

Troubleshooting Timeline

Evidence vs. Hypothesis

Historical Project Numbers Used

No packet-loss percentage, Mbps throughput, latency, jitter, CPU percentage, exact software version or historical CLI output has been added because the source does not preserve those values.

Fact List

• Original S6500 platform and 112-site trial: H3C's newly developed S6500 was deployed at 112 broadband metro-network trial locations at the end of 2004.

• Trial support model: two R&D engineers rotated locally with IP and partner engineers to support production-traffic validation.

• User symptom and early testing: peak-hour gaming lag, normal access-port checks, acceptable speed testing and irregular packet loss.

• ECMP evidence: repeated same-path observation, one uplink with little traffic and no obvious configuration fault.

• Critical routing evidence: ARP present, ping successful, expected upstream route missing, and peer traffic verified through the other uplink.

• Temporary recovery: interface reset restored the missing route and equal-cost traffic sharing.

• R&D finding: route-table programming/write abnormality; high CPU during bulk route writes was considered possible but not definitively established.

• Separate FIB incident: the earlier forwarding/FIB programming abnormality was a different fault and must not be merged with this ECMP/routing-table incident.

Final Verification

• No historical CLI has been fabricated.

• No software version has been invented.

• No packet-loss percentage has been added.

• No throughput, latency or jitter value has been invented.

• No CPU percentage has been added.

• The FIB programming incident and route-table programming incident remain separate.

• shutdown / undo shutdown is described only as a temporary recovery action.

• High CPU is described only as a possible deeper factor, not the confirmed root cause.

• No historical S6500 defect is attributed to current H3C switches.

• Current H3C products are clearly separated from equipment used in the original historical project.