Local compute / Featured system

Local AI lab

Making heterogeneous local models behave like dependable product infrastructure.

Case study / Local AI lab

01 / Context

Context

Local models expose different server lifecycles, APIs, context behavior, hardware assumptions, stream formats, and client expectations. A model that runs once from a terminal is not yet dependable infrastructure.

02 / Problem

Problem

Products and agent clients need consistent APIs, visible cold starts, constrained model switching, tool-call compatibility, useful telemetry, and reversible integration. Large local models add memory pressure and sometimes require unconventional execution such as SSD-streamed experts.

03 / System

System

A unified OpenAI-compatible router sits over MLX-family runtimes, llama.cpp-style servers, and DwarfStar / DS4. It translates Chat Completions and Responses behavior, manages backend lifecycle, emits SSE heartbeats, records performance telemetry, and supports isolated client-compatibility experiments.

04 / Architecture

See the connective layer.

Architecture mode

System architecture / Local AI lab

Product clients speak a stable OpenAI-style contract while the router selects, starts, health-checks, translates, measures, idles, and unloads heterogeneous backends.

  1. Agent or product client
  2. OpenAI-compatible compatibility router
  3. Alias, defaults, and tool-schema translation
  4. Backend lifecycle and health wait
  5. MLX, llama.cpp-style, or DwarfStar / DS4 runtime
  6. Normalized stream and telemetry

05 / Hard parts

Where the work actually was

  1. 01

    Translating between Chat Completions, Responses, tool schemas, and streamed lifecycle events without breaking clients.

  2. 02

    Making cold starts and large-model switching visible and serialized rather than presenting a hung connection.

  3. 03

    Validating unconventional SSD-streamed expert execution and long-context continuation on actual hardware.

  4. 04

    Experimenting with closed client integration in a reversible way while preserving upstream attribution.

06 / What changed or was learned

The field note

Reliable local inference is mostly lifecycle and compatibility engineering. The strongest result is a predictable contract around imperfectly consistent runtimes—plus the discipline to label upstream support, local execution proof, and focused contribution as different kinds of evidence.

07 / Evidence

Proof

Safe public evidence, translated from tests and runtime validation rather than raw internal logs.

TypeEvidenceWhat it establishes
runtimeCompatibility routing

Validated Chat Completions and Responses translation, streamed lifecycle events, tool-schema conversion, aliases, health waits, and cold-start heartbeats.

runtimeHardware-aware lifecycle

Exercised lazy backend spawning, serialized large-model switching, idle unload, and telemetry for time-to-first-token, prompt rate, decode rate, tokens, and requests.

runtimeSSD-streamed model experiment

Integrated GLM 5.2 with a Metal-based DwarfStar path, GGUF tooling, SSD-streamed experts, and continuation and long-context validation.

contributionFocused Osaurus fix

Contributed a targeted external-volume free-space preflight correction with regression coverage.

08 / Stack

Technology in service of the system

  • OpenAI-compatible APIs
  • Responses API
  • Chat Completions
  • SSE
  • MLX-family runtimes
  • llama.cpp-style servers
  • DwarfStar / DS4
  • Metal
  • GGUF
  • Open WebUI
  • macOS
  • Swift

10 / Status and boundaries

Built with the edges visible.