Kernel for Efficient Execution & Latency
KEEL Core is a sub-10ms memory-safe Rust proxy that treats prompt evaluation as an in-memory classification problem solved before inference starts.
02 · Classifier
Classifier
In-process INT8 ONNX intent classification in under 8ms. Reasoning depth, tool-use likelihood, and safety class are scored before a single token is billed.
Rust Runtime
Tokio Async Engine
Multi-threaded work-stealing scheduler with lock-free ring buffers.
Classifier
ONNX INT8 Model (22MB)
In-process execution with zero IPC network hops. P99 classification < 8ms.
Latency Budget
< 10ms Total Overhead
Hard-bounded execution budget before routing decisions commit.
Context Engine
SIMD-Accelerated Pruning
Content-hashes system prompts to eliminate redundant middle-turn tokens.
Telemetry Plane
Out-of-Band ClickHouse Pipeline
Asynchronous streaming logs that never block the client TTFT stream.
Deployment Matrix
Single Binary / Helm VPC
Deployable as a stateless Docker container across EKS, GKE, or AKS.
KEEL Core Low-Level Engineering Primitives
What the proxy actually does between accepting your socket and committing a routing decision.
Lock-Free Concurrency & Memory Safety
Built using Rust's async task model on Tokio. Memory allocations for request buffers are pre-allocated at startup. Zero heap re-allocation during hot routing execution paths.
In-Process Machine Learning Inference
Uses direct C++ bindings to the embedded ONNX Runtime. The 22MB INT8 quantized intent classifier executes directly in CPU memory cache without calling external API endpoints.
Deterministic Circuit Breaker & Fallbacks
Executes a lightweight streaming JSON schema parser. If an SLM outputs malformed tokens or exceeds TTFT thresholds, the connection transparently escalates to GPT-4o.
Walk the request lifecycle, step by step
Select a stage to inspect simulated latency, token reduction, and runtime memory behavior.
INT8 intent classification in CPU cache
The 22MB quantized model scores reasoning depth, tool-use likelihood, and safety class in-process — no IPC, no network hop.
P99 classification
7.8ms
Model size
22MB INT8
Tokens billed
0
Stop Paying Top-Tier Prices for Basic String Utilities.
Route intelligently, prune aggressively, and reserve frontier reasoning for the requests that actually need it. Two lines of code, sub-10ms overhead.