Overview
Kinference is an inference control plane open-sourced by Buoyant, the company behind Linkerd, in November 2026, built for self-hosted inference clusters that pair Kubernetes with vLLM. It selects among inference endpoints: it takes a request in OpenAI chat-completions format, applies policies, routing and load balancing, and returns the endpoint the request should go to along with an appropriately rewritten body.
Its decisions come from runtime signals rather than a static weight table. Kinference watches vLLM KV-cache events and engine metrics, parsing cache events through NVIDIA's dynamo-kv-router, and it monitors and sets the status of a set of Kubernetes CRDs that configure its behavior.
It is written in Rust, licensed Apache 2.0, and currently at v0.1.0-alpha.7. By design it does not forward traffic itself; it pairs with a proxy, with Buoyant Gateway as the reference example. The repository also ships kinfctl, a CLI for validating policies locally before they reach the cluster.
Key Features
- Endpoint selection and policy routing: Chooses among vLLM endpoints by policy and returns the target endpoint plus a rewritten body, so callers do not maintain their own endpoint table.
- KV-cache aware scheduling: Watches vLLM KV-cache events and engine metrics, folding cache state into routing so requests prefer instances that already hold the relevant context.
- Kubernetes CRD configuration: Policies and behavior are declared as custom resources, with the controller syncing their status so configuration fits existing GitOps flows.
- gRPC interface: Exposes endpoint selection over gRPC with request bodies compatible with the OpenAI chat-completions format, so existing clients keep their protocol.
- Pairs with a proxy: The control plane forwards no traffic itself; actual routing is handled by a proxy such as Buoyant Gateway, keeping responsibilities separate.
- kinfctl validation tool: Validates policy syntax and logic locally before deployment, cutting down trial and error inside the cluster.
Use Cases
- Running a self-hosted vLLM cluster on Kubernetes that needs request distribution across several model endpoints.
- Wanting cache hit rates factored into load balancing rather than connection counts and weights alone.
- Managing inference routing policies as declarative resources inside an existing GitOps workflow.
- Checking a policy locally before pushing it to the cluster.
Pros
- Routing decisions come from KV-cache and engine metrics, closer to real load than static weights.
- Written in Rust with a small per-process footprint, fitting for an always-on control plane.
- Apache 2.0 licensed, free to modify and use commercially.
- Works alongside an existing proxy, so the gateway layer stays in place.
Pricing
Free and open source under the Apache 2.0 license, with no per-call fees or seat limits. Commercial support and managed offerings are available from Buoyant.
Summary
Kinference suits teams already running vLLM on Kubernetes who have started to feel the pain of endpoint selection. It swaps weight tables for cache state, at the cost of adopting a set of CRDs and a companion proxy. The current release is alpha and the project states it has not seen sustained production exposure, so it is best explored in a test cluster first.
Version History
- Buoyant open-sources the Kinference inference control plane (2026-11-07): Buoyant released Kinference, an open-source inference control plane for Kubernetes and vLLM, currently at v0.1.0-alpha.7, written in Rust and licensed Apache 2.0. It watches vLLM KV-cache events and engine metrics, selects among endpoints by policy, and returns the target endpoint with a rewritten body over gRPC, designed to pair with a proxy such as Buoyant Gateway.