Auto-Router Benchmarks & Telemetry
Independently verifiable benchmark comparing SentioGate Auto Router against OpenRouter Jev Router (System-One decision layer) and OpenRouter Auto, plus real-time 24h edge telemetry and interactive failover testing.
What it proves: Across 16 genuine bug fixes from public Python repositories evaluated blindly with automated pytest suites (48 samples per arm), SentioGate routing achieves 100% Pass@3, 93.8% All-3-Passed consistency (beating OpenRouter Jev Router's 75% and OpenRouter Auto's 18.8%), and 97.9% Run Pass Rate while reducing cost per completed task to $0.000044 (vs $0.000145 on Jev Router and $0.007521 on OpenRouter Auto).
What it does not prove: This benchmark does NOT prove that SentioGate will save the exact same percentage on every conceivable prompt, open-ended human conversation, or architectural task; it provides a reproducible measurement on verified OSS test-fixing tasks.
Headline Benchmark Comparison (16 Tasks × 3 Runs = 48 Runs / Arm)
| Arm / Router | Pass@3 (Tasks) | All-3-Passed | Run Pass Rate | Cost / Completed Task | Total Billed Cost | Avg Latency (p50 / p95) | Models Chosen |
|---|---|---|---|---|---|---|---|
OpenRouter Jev Routertypesafe/jev-router | 16 / 16 (100%) | 12 / 16 (75%) | 42 / 48 (87.5%) | $0.000145 | $0.00608 | 6974ms (6143ms / 11649ms) | z-ai/glm-5.3-flash × 39deepseek/deepseek-v4.1-flash-1017 × 9 |
OpenRouter Autoopenrouter/auto | 13 / 16 (81.2%) | 3 / 16 (18.8%) | 24 / 48 (50%) | $0.007521 | $0.18049 | 16498ms (16290ms / 25195ms) | deepseek/deepseek-v4.1-flash-1017 × 48 |
| SentioGate Raw Byte-identical prompt (guidance/compression off) | 16 / 16 (100%) | 13 / 16 (81.2%) | 45 / 48 (93.8%) | $0.000046 | $0.00206 | 4921ms (4456ms / 8004ms) | z-ai/glm-5.3-flash × 48 |
| SentioGate Auto (Default Stack) Auto model selection + coding guidance + compression | 16 / 16 (100%) | 15 / 16 (93.8%) | 47 / 48 (97.9%) | $0.000044 | $0.00206 | 5030ms (4756ms / 7969ms) | z-ai/glm-5.3-flash × 48 |
*Cost per completed task = Total Billed Cost / Passing Runs. A cheap run that fails adds cost without adding completions, penalizing fragile routers.
Why SentioGate Wins: Intelligent Routing Architecture & Execution
The significant cost advantage and superior consistency over standard aggregators comes from SentioGate's intelligent routing and optimization pipeline:
- Dynamic Capability Matching: SentioGate analyzes incoming prompt complexity and semantic context in real time, intelligently selecting capable models suited for the domain while filtering out candidates that produce brittle or incomplete code.
- Intelligent Reasoning Optimization: SentioGate dynamically manages reasoning budgets across hybrid reasoning models, preventing runaway latency and completion limits while ensuring deep thinking is concentrated on complex edge cases.
- Context Optimization & Smart Caching: SentioGate eliminates redundant token overhead and duplicate upstream calls, maximizing throughput and reducing cost on every request.
The result: SentioGate delivers 100% Pass@3, 93.8% All-3-Passed consistency, and 97.9% Run Pass Rate while reducing latency to 5030ms and cost per completed task to $0.000044.
Per-Task Benchmark Matrix (16 Genuine OSS Bugs)
Showing 16 of 16 tasks. Each scored blindly across 3 attempts.| Task ID | Project & File | Bug Description | Jev Router | OpenRouter Auto | SentioGate Raw | SentioGate Auto |
|---|---|---|---|---|---|---|
TASK-01 | pallets/clickclick/_compat.py | Fix strip_ansi to strip OSC (Operating System Command) hyperlinks and sequences | 3 / 3 | 1 / 3 | 3 / 3 | 3 / 3 |
TASK-02 | encode/httpxhttpx/_urls.py | Fix QueryParams multi-value items, unique keys order, and blank/bool/None value encoding | 3 / 3 | 1 / 3 | 2 / 3 | 3 / 3 |
TASK-03 | Textualize/richrich/_wrap.py | Fix word wrapping when long word exceeds width and fold=True vs fold=False | 3 / 3 | 1 / 3 | 3 / 3 | 3 / 3 |
TASK-05 | marshmallow-code/marshmallowmarshmallow/utils.py | Fix get_value to handle literal dotted keys and plucking across lists of dicts | 1 / 3 | 1 / 3 | 3 / 3 | 3 / 3 |
TASK-06 | pydantic/pydanticpydantic/_internal/_utils.py | Fix deep_update so it recursively copies nested dicts without mutating original inputs | 3 / 3 | 2 / 3 | 3 / 3 | 3 / 3 |
TASK-07 | python-attrs/attrsattr/_funcs.py | Fix asdict with recursive nested collections (lists of tuples of dataclasses and dicts of lists) | 3 / 3 | 1 / 3 | 3 / 3 | 3 / 3 |
TASK-10 | pallets/jinjajinja2/utils.py | Fix LRUCache.__setitem__ evicting existing key when updating at capacity | 3 / 3 | 3 / 3 | 3 / 3 | 3 / 3 |
TASK-14 | pytest-dev/pytest_pytest/python_api.py | Fix ApproxScalar equality for infinities, NaN (nan_ok), and mixed float/sequence comparisons | 3 / 3 | 2 / 3 | 3 / 3 | 3 / 3 |
TASK-16 | python-jsonschema/jsonschemajsonschema/_pointer.py | Fix RFC 6901 JSON Pointer resolution: ~1 before ~0 unescaping, leading slash validation, and array '-' or leading zeros | 3 / 3 | 2 / 3 | 3 / 3 | 3 / 3 |
TASK-18 | jd/tenacitytenacity/rate_limiter.py | Fix TokenBucket refill drift when fractional tokens accumulate across short intervals | 3 / 3 | 3 / 3 | 3 / 3 | 3 / 3 |
TASK-19 | theskumar/python-dotenvdotenv/parser.py | Fix parse_env_line to handle '#' inside quoted strings, export prefix, and escaped characters in double quotes | 2 / 3 | 2 / 3 | 2 / 3 | 2 / 3 |
TASK-20 | jpadilla/pyjwtjwt/utils.py | Fix base64url_decode and base64url_encode to handle unpadded strings, URL-safe chars, and reject invalid length mod 4 == 1 | 3 / 3 | 3 / 3 | 3 / 3 | 3 / 3 |
TASK-22 | andialbrecht/sqlparsesqlparse/split.py | Fix split_sql_statements so semicolons inside single/double quotes and -- or /* */ comments do not split statements | 3 / 3 | 0 / 3 | 3 / 3 | 3 / 3 |
TASK-23 | prometheus/client_pythonprometheus_client/metrics.py | Fix Histogram.observe and cumulative bucket counts (+Inf bucket and NaN observations) | 2 / 3 | 0 / 3 | 3 / 3 | 3 / 3 |
TASK-29 | giampaolo/psutilpsutil/_common.py | Fix bytes2human unit rollover at 999.95 (e.g. 1023999 bytes rounding to 1000.0 KB instead of 0.98 MB) and negative values | 1 / 3 | 0 / 3 | 2 / 3 | 3 / 3 |
TASK-30 | pallets/werkzeugwerkzeug/http.py | Fix parse_content_range to validate start < stop < length and support unsatisfied range 'bytes */length' | 3 / 3 | 2 / 3 | 3 / 3 | 3 / 3 |
How the LLM leaderboard is measured
Every row comes from requests routed through SentioGate, the AI LLM router, not from vendor-reported figures. Time to first token (TTFT) and p50/p95 latency are a rolling average of recent requests per model. The 429 rate is the share of those requests that an upstream provider rejected as rate limited. Prices are provider list prices per million tokens, because SentioGate adds 0% markup on your own keys. A model with no recent traffic shows no row instead of an estimate.
Use it to choose which providers belong in your pool, then read the guides on OpenAI 429 fallback, Groq rate limits and Gemini 429 errors, or see how SentioGate compares on the comparison page.