Fair, reproducible benchmarks for agentic network troubleshooting.

Website · Quickstart · Build Your Agent · Trace Dataset

NetOpsBench is an open benchmark arena for agentic network troubleshooting — run reproducible fault scenarios on live SONiC-VS / Containerlab topologies, plug in any troubleshooting agent, and score it across quality and efficiency dimensions.

Troubleshooting agents are difficult to compare when the network, incident, and evidence change from run to run. NetOpsBench turns those variables into a controlled live benchmark:

• Reproducible incidents — labeled faults run against repeatable SONiC-VS and Containerlab topologies.
• Interactive evidence — agents inspect live Pingmesh, BGP, gNMI, syslog, and switch state instead of static logs.
• Comparable outcomes — one evaluator measures detection, localization, efficiency, and tool use across agent strategies.

NetOpsBench provides: (1) an interactive and realistic environment mimicking production networks, with common tracing and telemetry tooling; (2) comprehensive and reproducible benchmarks covering a wide range of faults and failures; (3) an extensible architecture with an open SDK to readily integrate with various agent paradigms and observability tools, allowing users to try out their own agentic workflows.

It is built for researchers and engineers who want to compare LLM-backed, symbolic, heuristic, or hybrid troubleshooting strategies on the same operational benchmark, not just on static logs or hand-written prompts.

• 2026-08: 🚀 NetOpsBench v0.2.0 - Large-topology benchmark release.
  • Add Xlarge CLOS and Fat-tree K=8/K=12 profiles, with 70 generated cases per large topology.
  • Replace per-client Python Pingmesh and iperf processes with the native Rust client agent for Pingmesh and background traffic.
  • Harden large-topology fault injection, recovery, observability, and exact runtime teardown.
  • Publish a versioned DeepSeek validation snapshot across all seven supported scales. See the v0.2.0 release notes.
• 2026-05: 🎉 Initial Release - NetOpsBench is now available as an open arena for agentic network troubleshooting.
  • Provide public SDK with run_scenario() and run_suite() APIs to launch live network environments from Python.
  • Equip native MCP tools of complete observability utilities and pre-configured SONiC-VS network covering XS, Small, Medium and Large scales.
  • Offer fault scenario generation scripts and an expanding repository of reproducible fault cases with standard ground truth labels.
  • A full-fledged benchmark evaluator that accesses detection accuracy and token utilization efficiency.

NetOpsBench runtime execution requires Linux because Containerlab depends on Linux networking primitives.

git clone https://github.com/NetX-lab/NetOpsBench.git
cd NetOpsBench

python -m venv .venv
source .venv/bin/activate
pip install -e ".[agent]"
python -m netopsbench.platform.runtime.build_tools  # verified official Containerlab v0.77.0

netopsbench benchmark prepare --scales xs
export OPENAI_API_KEY=...
PYTHONPATH=. python examples/01_run_scenario.py --vendor openai

The first successful run produces a BenchmarkReport with case-level scores, timing, and artifact paths. For Docker, Containerlab, and runtime setup details, read Quickstart.

Runtime creation uses the official Containerlab v0.77.0 release, with Docker-managed networking and sequential SONiC startup. The installer verifies both the release archive and executable; it does not replace the system Containerlab or modify Docker. Prepare the private tools after installation and configure privileged operations as described in Quickstart. Build the current client image with docker build -f containers/client/Dockerfile -t netopsbench-client:pingmesh-v2 .. Pingmesh uses one immutable offline plan for execution, coverage checks, and observation windows. Xlarge has 256 clients and a 66-second minimum window; Fat-tree K12/K18 use 42 seconds. K18 has 405 switches and 324 clients. Regenerate scenario assets with netopsbench benchmark prepare --scales xlarge,fat-tree-k12,fat-tree-k18 and recreate runtimes after this refactor; previous artifact formats are rejected. NETOPSBENCH_CLIENT_IMAGE selects a matching custom image.

from examples.agents import MinimalDeepAgent
from netopsbench.sdk import NetOpsBench

scenario = "scenarios/generated/xs/generated_link_down_xs_001.yaml"

with NetOpsBench(workspace=".") as bench:
    agent = bench.agents.wrap(MinimalDeepAgent(vendor="openai"))
    run = bench.sessions.run_scenario(scenario=scenario, agent=agent)
    report = run.wait()

print(report.summary)

Scenario YAML files define the benchmark case: topology scale, traffic profile, fault type, target device, and interface-level ground truth when applicable. Use the Python API Guide for run_scenario(...), run_suite(...), and workers=N; see Custom Troubleshooting Agents when you are ready to replace MinimalDeepAgent with your own strategy.

NetOpsBench reports detection, fault type, device/interface localization, runtime, tool calls, and token usage so troubleshooting quality and operational cost can be compared together.

Diagnosis score is the mean end-to-end case score: healthy cases require the correct verdict, while fault cases receive localization credit only after the fault is detected. Fault detection F1 measures the fault-versus-healthy decision independently.

The largest validated Fat-tree profile provides a compact case-level view. Each square below is one K=12 case; detailed cross-topology observability analysis remains in the full results.

Read the v0.2.0 release notes, Benchmark Methodology, and Benchmark Results for the full validation snapshot and scoring definitions.

The public v0.2 trace release contains 319 validated DeepSeek Harbor/ATIF trajectories across XS through Fat-tree K=12. Aggregate metrics and immutable publication provenance are recorded in the v0.2 result snapshot.

• Global community: NetOpsBench Slack
• Chinese-language community: NetOpsBench Feishu group

NetOpsBench is released under the MIT License. See LICENSE.

If you use NetOpsBench in your research, please cite:

@software{netopsbench2026,
  author  = {Yang, Yitao and Xu, Hong},
  title   = {{NetOpsBench}: Open Arena for NetOps in AI Infrastructure},
  year    = {2026},
  url     = {https://github.com/netx-lab/NetOpsBench},
}