Seeing the Traffic: Why We Built Talon Keel as a Streaming Reverse Proxy

The First Problem: AI Traffic Was Invisible

The first problem to solve was how to see the important traffic that Talon Keel would build off of. In the naive setup, the application sends a request to an API, OpenAI processes it, and tokens start streaming back. But there was a lot in between that interaction worth measuring: how many tokens were sent, how many came back, how long the model took to start responding, how quickly it generated tokens, and how much the whole request ultimately cost. If Talon Keel was going to help developers optimize AI infrastructure, we first needed a way to actually see this traffic. That led to reverse proxies.

What Is a Reverse Proxy?

Conceptually, a reverse proxy is a pipeline between the application and OpenAI — a pipeline Talon Keel uses to see and control the model. It can observe the request before it reaches the model and the response as it comes back. That means we can measure what is happening without requiring the application itself to do any of that work.

[Application] → [TALON KEEL proxy] → [AI provider]
                     ↑            ↑            ↑
                 sees request   measures    sees response
                 before model   in-flight   as it streams back

Compatibility: A Drop-In, Not a Rewrite

Developers who already use the OpenAI SDK should only have to change where requests are sent — nothing else. So the proxy needed to understand OpenAI-formatted requests and pass them through without unnecessarily changing them.

# The only change required to adopt Talon Keel:
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:4000/v1",  # was https://api.openai.com/v1
    api_key="sk-...",
)

# Requests still speak OpenAI's format — the proxy sees each request
# before it reaches the model and each response as it comes back.

The 5-Millisecond Overhead Budget

Putting another server between the application and the AI provider will, for sure, make things slower — so the target was less than 5 milliseconds of processing overhead. The first idea that made that achievable was thinking of Talon Keel as a pipe. It shouldn't wait for the entire response, analyze it, store some information, and finally return it. Instead, it should act while data is flowing — a technique called streaming — and pass bytes through as they arrive. That shortens times drastically.

Telemetry While Streaming

Once the proxy could observe the request, collecting useful information became possible — and the response kept moving while the telemetry was processed asynchronously:

From Visibility to Cognitive Routing

This experiment started as a way of measuring AI traffic. But the more I worked through it, the more it became clear the reverse proxy is a foundation for everything Talon Keel is trying to build. Once you sit between an application and an AI provider, you're in a very interesting position: the proxy gives us visibility. Visibility gives us data. Data gives us the ability to make decisions. And those decisions eventually create the possibility for cognitive routing — instead of every prompt automatically going to whatever model the developer originally selected, the most appropriate path gets determined from real signals:

What Hides Under a Simple API Call

The most useful part of building this wasn't simply learning what a reverse proxy was. It was realizing how many infrastructure decisions hide underneath something as simple as an AI API call — and how much leverage you get from being the layer that sees them.

“The proxy gives us visibility. Visibility gives us data. Data gives us decisions. And decisions create cognitive routing.”