Measured in Production, Not Synthetic Notebooks.
Metrics derived from a 4.1M request production workload across extraction, structured data generation, agentic tool calling, and multi-step reasoning.
8.4ms
P99 pre-inference routing overhead
70.2%
Median token cost reduction
51.6%
Median TTFT (time-to-first-token) reduction
99.99%
Schema execution success rate
Benchmark Methodology & Production Compliance Note
The figures published below represent actual production telemetry collected across production agent workloads. To comply with commercial non-disclosure agreements (NDAs) and customer data privacy guarantees, raw prompt/response payload datasets are strictly redacted. These benchmarks reflect real-world operational traffic evaluated using KEEL Core under identical prompt loads against un-proxied frontier model endpoints.
| Evaluation Metric | Raw GPT-4o (Direct API) | Standard Proxy / Router | TALON AI (KEEL Core) |
|---|---|---|---|
| P99 Overhead | Direct (0ms) | 40ms – 120ms | < 10ms (8.4ms avg) |
| Token Cost Efficiency | 100% Baseline | 95% of Baseline | 30% of Baseline (70% Savings) |
| Context Pruning | None | Partial (Static Caching) | Dynamic SIMD Strip-Mining |
| Deterministic Schema Fallback | No | No | Yes (Mid-Stream Retries) |
| Self-Hosted VPC (ZDR) | No | Partial | Yes (Full Helm Support) |
Understanding the Metrics
Where each number comes from, and why the architecture makes it reproducible.
How We Achieve 8.4ms P99 Overhead
Traditional routers add 50–100ms by calling external LLM evaluation endpoints or using slow Python proxies. KEEL Core executes in-memory inside a compiled Rust binary using vectorized C++ bindings for INT8 ONNX models, completing intent scoring before TCP buffers flush.
Where the 70.2% Cost Savings Originate
Savings are driven by two factors: (1) System Prompt Strip-Mining, which eliminates up to 60% of redundant middle-turn conversation context, and (2) Cognitive Model Cascading, which directs 68% of routine string extraction prompts to low-cost SLMs.
How TTFT Drops by 51.6%
Frontier models like Claude 3.5 Sonnet often suffer queue delays during peak hours. By routing low-complexity tasks to local or edge-hosted SLMs (e.g. Llama-3.1-8B), initial token generation begins in milliseconds.
Stop Paying Top-Tier Prices for Basic String Utilities.
Route intelligently, prune aggressively, and reserve frontier reasoning for the requests that actually need it. Two lines of code, sub-10ms overhead.