inferenceoss

Inference is where models become products, and it is the layer with the least shared law. Training has PyTorch and a foundation. Serving on Kubernetes has KServe. The runtimes themselves — vLLM, SGLang, llama.cpp, TGI, TensorRT-LLM, Ollama, llm-d, Dynamo, AIBrix — are excellent, and they do not speak a common dialect.

The OpenAI Chat Completions API became the IPv4 of this layer: everyone implements a subset, everyone extends it, nobody versions the folklore. Tool-call shapes drift. Structured output sometimes means constrained decoding and sometimes means “please JSON.” Prefix cache dies at a process boundary. KV layouts are private. Telemetry names are invented per dashboard. Benchmarks are blog posts.

KServe’s original Open Inference Protocol covers predictive well. Generative inference still means “speak OpenAI, mostly.” That is not a constitution. It is a habit.

Make inference engines interchangeable without making them identical.

InferenceOSS incubates protocols, interchange formats, conformance, and reference implementations. It does not incubate another engine. An engine that passes the suite is a citizen, whether it runs on an H100 cluster or a phone NPU.

Six constraints. Endorse the ones you will actually defend in a working group.

1. 01

   Specify interfaces, not winners

   The org exists so engines can be swapped. A spec that only one runtime can implement is a press release. If a proposal cannot be implemented independently by two engines, it does not ship.

   Endorse · 0
2. 02

   Interoperate with what already works

   vLLM, SGLang, llama.cpp, TGI, TensorRT-LLM, Ollama, llm-d, KServe, and Dynamo are not competitors to replace. They are citizens. We write adapters and contracts, not a thirteenth serving stack.

   Endorse · 0
3. 03

   No vendor majority

   No employer, cloud, or chip house may hold a majority of Technical Steering Committee seats. Hardware-specific extensions are allowed; hardware-specific lock-in in a core spec is not.

   Endorse · 0
4. 04

   Measure in public

   Benchmarks that cannot be reproduced on rented hardware, with published configs, are marketing. SPEC/I runs are signed, versioned, and boring on purpose.

   Endorse · 0
5. 05

   Cloud and edge are one problem

   A protocol that only works behind an H100 is half a protocol. EdgeABI and OIP-G must share a ticket shape so an agent can move from a phone NPU to a datacenter without rewriting the client.

   Endorse · 0
6. 06

   Apache 2.0, DCO, no CLA tax

   Default license is Apache 2.0 with an implicit patent grant. Contributions are Developer Certificate of Origin, not a lawyer-gated CLA. Trademarks stay with the org; copyright stays with authors.

   Endorse · 0

• Write another full serving engine.
• Operate a hosted inference cloud. The org is not a vendor.
• Bless a single accelerator, quantization, or kernel family in a core spec.
• Replace PyTorch, vLLM, KServe, or OpenTelemetry. We interoperate.
• Sell a leaderboard, or let a member-submitted number stand in for SPEC/I.
• Require a CLA, a contributor license assignment, or an employment gate.

Eight incubator projects. All start as drafts. Graduation requires two independent implementations and a conformance suite, not a slide.

1. 01 · OIP-G

   draft

   Open Inference Protocol — Generative

   The conversational, tool-using, structured, multimodal protocol engines should actually implement — with capability discovery, not folklore.

   Open the brief

2. 02 · CapSpec

   draft

   Engine capability manifest

   OpenAPI for runtimes. A JSON document that says, precisely, what this process can do on this hardware today.

   Open the brief

3. 03 · KVX

   draft

   KV cache interchange

   A portable prefix-cache format so a hit on one engine is a hit on another, and so disaggregated prefill/decode is not a vendor feature.

   Open the brief

4. 04 · InferTrace

   draft

   Inference telemetry conventions

   OpenTelemetry semantic conventions for the numbers that actually matter: TTFT, ITL, cache, batch, speculation, and abort.

   Open the brief

5. 05 · SPEC/I

   draft

   Standard Performance Evaluation — Inference

   A vendor-neutral, continuously run inference benchmark. Signed configs. Rented hardware. No vendor-submitted slides.

   Open the brief

6. 06 · Gate

   draft

   Inference-time policy sidecar

   A sidecar spec for schema enforcement, PII, allow/deny, and output constraints that is not baked into one engine.

   Open the brief

7. 07 · EdgeABI

   draft

   Portable edge runtime ABI

   A small contract so llama.cpp, MLX, NPU runtimes, and WASM builds look like the same citizen as a datacenter engine.

   Open the brief

8. 08 · Router

   draft

   Reference inference router

   A small, boring scheduler above engines: CapSpec-aware routing, disaggregated prefill/decode, and retry. Reference, not product.

   Open the brief

Six rooms. Join the ones you will attend.

• Specs

  OIP-G and CapSpec. RFCs, conformance, language-agnostic documents.

• Runtime

  Router, EdgeABI, and engine adapters. Reference code that other people can ignore.

• Interchange

  KVX. Portable prefix cache and block addressing.

• Observe

  InferTrace conventions and the SPEC/I harness.

• Safety

  Gate sidecar, constrained decode, operator policy packs.

• Community

  Charter, membership, docs, working sessions, the public memory of the org.

1. 01

   Charter and TSC

   Publish this document. Invite twelve interim TSC nominees from independent engine, academic, and operator backgrounds. Seat seven. Publish meeting notes.

2. 02

   CapSpec 0.1 RFC

   Freeze a JSON schema. Ship generators for vLLM, SGLang, and llama.cpp. One page of docs, no more.

3. 03

   InferTrace draft

   Name TTFT, ITL, queue, cache, batch, speculation, abort. Align with OpenTelemetry. Send to the OTel SIG.

4. 04

   SPEC/I harness

   Pin one model, one GPU type, four engines, three workloads. Publish the exact commands. Do not declare a winner.

5. 05

   Legal and identity

   Apache 2.0, DCO, trademark filing for InferenceOSS, domain, GitHub org. No CLA.

6. 06

   First working session

   A three-hour public session. RFCs only. No keynotes. Notes on the site within a day.

1. Phase 0 · Aug–Dec 2026

   Found

   • Ratify Charter 0.1 and stand up an interim TSC of seven, no employer majority.
   • Publish CapSpec v0.1 and the InferTrace attribute draft.
   • File trademarks, Apache 2.0 all repos, DCO on every commit.
   • First SPEC/I dry run: one public model, one instance type, four engines.
   • GitHub org, RFC process, monthly working session.

2. Phase 1 · 2027 H1

   Interfaces

   • OIP-G v0.2: tools, structured output, abort, capabilities.
   • KVX prototype: prefix round-trip between two independent engines on the same weights.
   • SPEC/I public run 1, signed configs.
   • Industry and academic membership program opens.

3. Phase 2 · 2027 H2

   Interchange

   • KVX adapters on four engines.
   • Gate sidecar spec + reference filters.
   • EdgeABI 0.1 (llama.cpp + one additional target).
   • Conformance suite used in at least one cloud and one edge product.

4. Phase 3 · 2028

   1.0

   • OIP-G 1.0. CapSpec and InferTrace graduate.
   • A datacenter engine and an edge runtime share a ticket shape in production.
   • SPEC/I cited in procurement language by at least one public buyer.

5. Phase 4 · 2029

   Default

   • “Speaks OIP-G” is a normal checkbox, the way “speaks OpenAPI” is for HTTP services.
   • Independent engines interchangeable for the majority of serving jobs that do not need a vendor kernel.
   • The org is boring. That is the point.

Form

Independent nonprofit (or unincorporated association on the way there), with a path to a Linux Foundation directed fund if that later serves the charter. The charter is the constraint, not the parent org.

TSC

Seven seats, two-year terms, odd number on purpose. No more than two seats from any one employer. TSC votes on incubation, graduation, and spec releases. Working groups propose; they do not ship alone.

Membership

Individuals sign the charter for free. Academic labs join at cost. Industry membership (three tiers) funds a tiny staff and SPEC/I hardware. Dues do not buy a TSC seat. One organization, one vote in TSC elections.

IP

Apache 2.0 default. DCO on every commit. Specs are dual-licensed as code. The InferenceOSS mark is held by the org and licensed to conforming implementations.

Why not LF / PyTorch / KServe

vLLM already incubates at LF AI & Data. PyTorch Foundation holds the training-adjacent stack. KServe is the Kubernetes inference platform. Those are the right homes for those projects. InferenceOSS exists because the interfaces between engines have no owner — and an owner that is also an engine is a conflict.

If these are not true in 2029, the org failed, however many stars the repos have.

• Three major engines publish a CapSpec and pass OIP-G conformance.
• KVX round-trips a prefix between two independent engines on the same weights.
• SPEC/I is cited in place of a vendor blog chart at least once in a serious venue.
• A cloud runtime and an edge runtime share an OIP-G ticket in production.
• No single employer holds a TSC majority, including in the fifth year.