Diagrams — Module FT20: Serving Stacks

Module: FT20 — Serving Stacks Diagram count: 5 Tool: Mermaid (primary). Each diagram validated in Mermaid Live Editor.

Diagram 1 — The Serving Stack Decision Matrix

Type: Quadrant / routing matrix Purpose: The single diagram that tells you which runtime to pick. Three variables (audience, hardware, telemetry posture) route to one of the production / dev / air-gap / Mac cells. Reading the diagram: Top axis is the audience (single user → many concurrent). Side axis is the deployment constraint (open network → air-gap). Each quadrant names the canonical runtime. NVIDIA production (top-left) is vLLM; air-gap (right) is llama.cpp; Mac is MLX; dev is Ollama.

flowchart LR
  subgraph Matrix["ROUTING FUNCTION: 3 variables -> 1 runtime"]
    direction TB
    Q1["PRODUCTION · NVIDIA · open subnet\n-> vLLM\n(PagedAttention, continuous batching,\nOpenAI-compatible, OTel-native)"]
    Q2["DEV / SINGLE USER · any HW\n-> Ollama\n(wraps llama.cpp; collapses at ~5 concurrent)"]
    Q3["AIR-GAP / regulated · any HW\n-> llama.cpp server\n(single binary, no network deps,\nmax hardware breadth)"]
    Q4["MAC FLEET · Apple Silicon\n-> mlx-lm\n(unified-memory native, 4-bit,\nfederated local serving)"]
    Q1 --- Q2
    Q3 --- Q4
  end

  Vars["THREE VARIABLES (ask in order):\n1. Audience (1 user? team? hundreds?)\n2. Hardware (CUDA? Mac? CPU? ROCm?)\n3. Telemetry posture (open? regulated? air-gap?)"]
  Vars --> Matrix

  style Q1 fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style Q2 fill:#14141f,stroke:rgba(255,255,255,0.12),color:#9494a0
  style Q3 fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style Q4 fill:#14141f,stroke:#5eead4,color:#e4e4e8
  style Vars fill:#08080c,stroke:rgba(94,234,212,0.4),color:#5eead4
  style Matrix fill:#08080c,stroke:rgba(255,255,255,0.06),color:#e4e4e8

Diagram 2 — The No-Telemetry / Air-Gap Spectrum

Type: Spectrum / comparison Purpose: Which runtimes phone home, which never do, and the pre-load-then-sever recipe for a true air-gap. Reading the diagram: Left = "phones home / needs network." Right = "never reaches out by construction." llama.cpp is the rightmost anchor. Ollama is in the middle — needs the network only for ollama pull, then can be blocked. The bottom track is the air-gap recipe that works for any runtime.

flowchart LR
  Ollama["Ollama\n(needs net only for ollama pull;\nthen block + bind 127.0.0.1)"]
  vLLM["vLLM\n(OTel-native: emits to YOUR collector;\nno vendor phone-home)"]
  llama["llama.cpp server\n(single binary,\nNO network code path for telemetry)"]

  Ollama -.->|"telemetry posture: configurable"| vLLM
  vLLM -.->|"telemetry posture: yours to point"| llama

  Recipe["AIR-GAP RECIPE (any runtime):\n1. Pre-load on a connected staging machine\n2. Transfer via approved media (USB / diode)\n3. SHA256-verify on the isolated side\n4. Serve (llama.cpp preferred)\n5. Patch on a schedule via the same media"]
  Recipe -.->|"the network never returns"| llama

  style Ollama fill:#14141f,stroke:rgba(255,255,255,0.12),color:#9494a0
  style vLLM fill:#14141f,stroke:rgba(94,234,212,0.6),color:#e4e4e8
  style llama fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style Recipe fill:#08080c,stroke:rgba(94,234,212,0.4),stroke-dasharray: 4 2,color:#5eead4

Diagram 3 — The Ollama Concurrency Ceiling

Type: Throughput-vs-latency curve (schematic) Purpose: Why Ollama is a dev tool, not a production server. The ~5 concurrent user collapse, and what vLLM does instead. Reading the diagram: As concurrent users rise, Ollama's latency explodes past ~5 users (3s -> 60s+) because it serializes requests and duplicates context per request. vLLM on the same hardware holds stable latency via PagedAttention + continuous batching. The annotation names the root cause.

flowchart LR
  subgraph Load["LATENCY vs CONCURRENT USERS (schematic)"]
    direction TB
    L1["1 user\nOllama ~3s   |   vLLM ~1s"]
    L2["3 users\nOllama ~8s   |   vLLM ~1s"]
    L3["5 users  <-- THE CEILING\nOllama 30-60s+   |   vLLM ~2s"]
    L4["10+ users\nOllama: unusable   |   vLLM ~3s"]
    L1 --> L2 --> L3 --> L4
  end

  Root["ROOT CAUSE:\nOllama: serializes requests + duplicates context per request\n(no PagedAttention, no real continuous batching)\n\nvLLM: PagedAttention (paged KV cache, no fragmentation)\n+ continuous batching (insert requests at every token step)\n-> stable latency under load"]
  Root -.->|"why the curves diverge"| Load

  style L1 fill:#14141f,stroke:rgba(255,255,255,0.12),color:#9494a0
  style L2 fill:#14141f,stroke:rgba(255,255,255,0.12),color:#9494a0
  style L3 fill:#14141f,stroke:#f08080,stroke-width:1.5px,color:#f08080
  style L4 fill:#14141f,stroke:#f08080,stroke-width:1.5px,color:#f08080
  style Root fill:#08080c,stroke:rgba(94,234,212,0.4),color:#5eead4
  style Load fill:#08080c,stroke:rgba(255,255,255,0.06),color:#e4e4e8

Diagram 4 — Production vs Dev Deployment Patterns

Type: Side-by-side architecture Purpose: The pattern that separates a working dev setup from a service you can page on. Both speak the same OpenAI-compatible API; the prod path adds the reverse proxy, concurrency limits, and observability. Reading the diagram: Top = dev pattern (Ollama on loopback, single user, the app calls directly). Bottom = prod pattern (vLLM behind a reverse proxy with auth + rate limits, metrics to your own collector, the app sees the same /v1/chat/completions either way). The unifying property is the identical client interface.

flowchart TB
  subgraph Dev["DEV PATTERN (single user)"]
    direction LR
    AppD["Your app\n(OpenAI client)"]
    OllamaSrv["Ollama server\nOLLAMA_HOST=127.0.0.1:11434"]
    AppD -->|"base_url=localhost"| OllamaSrv
  end

  subgraph Prod["PRODUCTION PATTERN (multi-user)"]
    direction LR
    AppP["Your app\n(same OpenAI client)"]
    Proxy["Reverse proxy\nnginx / envoy\nauth + rate limits + timeouts"]
    vLLMSrv["vLLM server\nPagedAttention + continuous batching\n+ concurrency limits + queueing"]
    Metrics["Your collector\nOTel / Prometheus\nlatency p50/p95/p99"]
    AppP --> Proxy --> vLLMSrv
    vLLMSrv -.metrics.-> Metrics
  end

  Shared["UNIFYING PROPERTY:\nThe app sees the SAME /v1/chat/completions either way.\nSwap base_url between dev and prod with NO code change."]
  Dev -.-> Shared
  Prod -.-> Shared

  style AppD fill:#14141f,stroke:rgba(255,255,255,0.12),color:#9494a0
  style OllamaSrv fill:#14141f,stroke:rgba(255,255,255,0.12),color:#9494a0
  style AppP fill:#14141f,stroke:#5eead4,color:#e4e4e8
  style Proxy fill:#14141f,stroke:#5eead4,color:#e4e4e8
  style vLLMSrv fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style Metrics fill:#14141f,stroke:rgba(94,234,212,0.6),color:#9494a0
  style Shared fill:#08080c,stroke:rgba(94,234,212,0.4),stroke-dasharray: 4 2,color:#5eead4
  style Dev fill:#08080c,stroke:rgba(255,255,255,0.06),color:#e4e4e8
  style Prod fill:#08080c,stroke:rgba(255,255,255,0.06),color:#e4e4e8

Diagram 5 — The Apple Silicon / MLX Path

Type: Pipeline / federated architecture Purpose: When the fleet is Macs, the runtime is MLX — and the architecture is federated local serving (MLX on each workstation), not centralized serving (one big vLLM box). This is the privacy-by-construction pattern for healthcare/defense Mac fleets. Reading the diagram: Top = the MLX workflow (pull pre-quantized 4-bit from mlx-community, serve with mlx_lm.server, the app calls the OpenAI-compatible endpoint). Bottom = the federated pattern — each Mac runs its own MLX; no central server holds anyone's prompts. The privacy property is structural, not a policy.

flowchart TB
  subgraph Workflow["MLX WORKFLOW (per Mac)"]
    direction LR
    HF["mlx-community on HuggingFace\npre-quantized 4-bit models"]
    Pull["mlx_lm.server\npython -m mlx_lm.server\n--model mlx-community/...-4bit"]
    Local["127.0.0.1:8080\n/v1/chat/completions"]
    HF -->|"hf download"| Pull --> Local
  end

  subgraph Fleet["FEDERATED LOCAL SERVING (the Mac-fleet pattern)"]
    direction LR
    Mac1["Mac 1\nMLX + local data\n(workstation A)"]
    Mac2["Mac 2\nMLX + local data\n(workstation B)"]
    Mac3["Mac N\nMLX + local data\n(workstation N)"]
    Mac1 -. "no shared server" .-> Mac2
    Mac2 -. "no shared server" .-> Mac3
  end

  Property["STRUCTURAL PRIVACY:\nNo central server holds anyone's prompts.\n4-bit ~75% size reduction, minimal quality loss.\n~143 tok/s on Qwen3-VL-4B class on current M-series.\nWWDC25 session 298: Apple-endorsed local serving path."]
  Workflow -.-> Property
  Fleet -.-> Property

  style HF fill:#14141f,stroke:rgba(255,255,255,0.12),color:#9494a0
  style Pull fill:#14141f,stroke:#5eead4,color:#e4e4e8
  style Local fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style Mac1 fill:#14141f,stroke:rgba(94,234,212,0.6),color:#e4e4e8
  style Mac2 fill:#14141f,stroke:rgba(94,234,212,0.6),color:#e4e4e8
  style Mac3 fill:#14141f,stroke:rgba(94,234,212,0.6),color:#e4e4e8
  style Property fill:#08080c,stroke:rgba(94,234,212,0.4),stroke-dasharray: 4 2,color:#5eead4
  style Workflow fill:#08080c,stroke:rgba(255,255,255,0.06),color:#e4e4e8
  style Fleet fill:#08080c,stroke:rgba(255,255,255,0.06),color:#e4e4e8

Validation notes

# Diagrams — Module FT20: Serving Stacks

**Module**: FT20 — Serving Stacks
**Diagram count**: 5
**Tool**: Mermaid (primary). Each diagram validated in [Mermaid Live Editor](https://mermaid.live).

---

## Diagram 1 — The Serving Stack Decision Matrix

**Type**: Quadrant / routing matrix
**Purpose**: The single diagram that tells you which runtime to pick. Three variables (audience, hardware, telemetry posture) route to one of the production / dev / air-gap / Mac cells.
**Reading the diagram**: Top axis is the audience (single user → many concurrent). Side axis is the deployment constraint (open network → air-gap). Each quadrant names the canonical runtime. NVIDIA production (top-left) is vLLM; air-gap (right) is llama.cpp; Mac is MLX; dev is Ollama.

```mermaid
flowchart LR
  subgraph Matrix["ROUTING FUNCTION: 3 variables -> 1 runtime"]
    direction TB
    Q1["PRODUCTION · NVIDIA · open subnet\n-> vLLM\n(PagedAttention, continuous batching,\nOpenAI-compatible, OTel-native)"]
    Q2["DEV / SINGLE USER · any HW\n-> Ollama\n(wraps llama.cpp; collapses at ~5 concurrent)"]
    Q3["AIR-GAP / regulated · any HW\n-> llama.cpp server\n(single binary, no network deps,\nmax hardware breadth)"]
    Q4["MAC FLEET · Apple Silicon\n-> mlx-lm\n(unified-memory native, 4-bit,\nfederated local serving)"]
    Q1 --- Q2
    Q3 --- Q4
  end

  Vars["THREE VARIABLES (ask in order):\n1. Audience (1 user? team? hundreds?)\n2. Hardware (CUDA? Mac? CPU? ROCm?)\n3. Telemetry posture (open? regulated? air-gap?)"]
  Vars --> Matrix

  style Q1 fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style Q2 fill:#14141f,stroke:rgba(255,255,255,0.12),color:#9494a0
  style Q3 fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style Q4 fill:#14141f,stroke:#5eead4,color:#e4e4e8
  style Vars fill:#08080c,stroke:rgba(94,234,212,0.4),color:#5eead4
  style Matrix fill:#08080c,stroke:rgba(255,255,255,0.06),color:#e4e4e8
```

---

## Diagram 2 — The No-Telemetry / Air-Gap Spectrum

**Type**: Spectrum / comparison
**Purpose**: Which runtimes phone home, which never do, and the pre-load-then-sever recipe for a true air-gap.
**Reading the diagram**: Left = "phones home / needs network." Right = "never reaches out by construction." llama.cpp is the rightmost anchor. Ollama is in the middle — needs the network only for `ollama pull`, then can be blocked. The bottom track is the air-gap recipe that works for any runtime.

```mermaid
flowchart LR
  Ollama["Ollama\n(needs net only for ollama pull;\nthen block + bind 127.0.0.1)"]
  vLLM["vLLM\n(OTel-native: emits to YOUR collector;\nno vendor phone-home)"]
  llama["llama.cpp server\n(single binary,\nNO network code path for telemetry)"]

  Ollama -.->|"telemetry posture: configurable"| vLLM
  vLLM -.->|"telemetry posture: yours to point"| llama

  Recipe["AIR-GAP RECIPE (any runtime):\n1. Pre-load on a connected staging machine\n2. Transfer via approved media (USB / diode)\n3. SHA256-verify on the isolated side\n4. Serve (llama.cpp preferred)\n5. Patch on a schedule via the same media"]
  Recipe -.->|"the network never returns"| llama

  style Ollama fill:#14141f,stroke:rgba(255,255,255,0.12),color:#9494a0
  style vLLM fill:#14141f,stroke:rgba(94,234,212,0.6),color:#e4e4e8
  style llama fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style Recipe fill:#08080c,stroke:rgba(94,234,212,0.4),stroke-dasharray: 4 2,color:#5eead4
```

---

## Diagram 3 — The Ollama Concurrency Ceiling

**Type**: Throughput-vs-latency curve (schematic)
**Purpose**: Why Ollama is a dev tool, not a production server. The ~5 concurrent user collapse, and what vLLM does instead.
**Reading the diagram**: As concurrent users rise, Ollama's latency explodes past ~5 users (3s -> 60s+) because it serializes requests and duplicates context per request. vLLM on the same hardware holds stable latency via PagedAttention + continuous batching. The annotation names the root cause.

```mermaid
flowchart LR
  subgraph Load["LATENCY vs CONCURRENT USERS (schematic)"]
    direction TB
    L1["1 user\nOllama ~3s   |   vLLM ~1s"]
    L2["3 users\nOllama ~8s   |   vLLM ~1s"]
    L3["5 users  <-- THE CEILING\nOllama 30-60s+   |   vLLM ~2s"]
    L4["10+ users\nOllama: unusable   |   vLLM ~3s"]
    L1 --> L2 --> L3 --> L4
  end

  Root["ROOT CAUSE:\nOllama: serializes requests + duplicates context per request\n(no PagedAttention, no real continuous batching)\n\nvLLM: PagedAttention (paged KV cache, no fragmentation)\n+ continuous batching (insert requests at every token step)\n-> stable latency under load"]
  Root -.->|"why the curves diverge"| Load

  style L1 fill:#14141f,stroke:rgba(255,255,255,0.12),color:#9494a0
  style L2 fill:#14141f,stroke:rgba(255,255,255,0.12),color:#9494a0
  style L3 fill:#14141f,stroke:#f08080,stroke-width:1.5px,color:#f08080
  style L4 fill:#14141f,stroke:#f08080,stroke-width:1.5px,color:#f08080
  style Root fill:#08080c,stroke:rgba(94,234,212,0.4),color:#5eead4
  style Load fill:#08080c,stroke:rgba(255,255,255,0.06),color:#e4e4e8
```

---

## Diagram 4 — Production vs Dev Deployment Patterns

**Type**: Side-by-side architecture
**Purpose**: The pattern that separates a working dev setup from a service you can page on. Both speak the same OpenAI-compatible API; the prod path adds the reverse proxy, concurrency limits, and observability.
**Reading the diagram**: Top = dev pattern (Ollama on loopback, single user, the app calls directly). Bottom = prod pattern (vLLM behind a reverse proxy with auth + rate limits, metrics to your own collector, the app sees the same `/v1/chat/completions` either way). The unifying property is the identical client interface.

```mermaid
flowchart TB
  subgraph Dev["DEV PATTERN (single user)"]
    direction LR
    AppD["Your app\n(OpenAI client)"]
    OllamaSrv["Ollama server\nOLLAMA_HOST=127.0.0.1:11434"]
    AppD -->|"base_url=localhost"| OllamaSrv
  end

  subgraph Prod["PRODUCTION PATTERN (multi-user)"]
    direction LR
    AppP["Your app\n(same OpenAI client)"]
    Proxy["Reverse proxy\nnginx / envoy\nauth + rate limits + timeouts"]
    vLLMSrv["vLLM server\nPagedAttention + continuous batching\n+ concurrency limits + queueing"]
    Metrics["Your collector\nOTel / Prometheus\nlatency p50/p95/p99"]
    AppP --> Proxy --> vLLMSrv
    vLLMSrv -.metrics.-> Metrics
  end

  Shared["UNIFYING PROPERTY:\nThe app sees the SAME /v1/chat/completions either way.\nSwap base_url between dev and prod with NO code change."]
  Dev -.-> Shared
  Prod -.-> Shared

  style AppD fill:#14141f,stroke:rgba(255,255,255,0.12),color:#9494a0
  style OllamaSrv fill:#14141f,stroke:rgba(255,255,255,0.12),color:#9494a0
  style AppP fill:#14141f,stroke:#5eead4,color:#e4e4e8
  style Proxy fill:#14141f,stroke:#5eead4,color:#e4e4e8
  style vLLMSrv fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style Metrics fill:#14141f,stroke:rgba(94,234,212,0.6),color:#9494a0
  style Shared fill:#08080c,stroke:rgba(94,234,212,0.4),stroke-dasharray: 4 2,color:#5eead4
  style Dev fill:#08080c,stroke:rgba(255,255,255,0.06),color:#e4e4e8
  style Prod fill:#08080c,stroke:rgba(255,255,255,0.06),color:#e4e4e8
```

---

## Diagram 5 — The Apple Silicon / MLX Path

**Type**: Pipeline / federated architecture
**Purpose**: When the fleet is Macs, the runtime is MLX — and the architecture is federated local serving (MLX on each workstation), not centralized serving (one big vLLM box). This is the privacy-by-construction pattern for healthcare/defense Mac fleets.
**Reading the diagram**: Top = the MLX workflow (pull pre-quantized 4-bit from mlx-community, serve with mlx_lm.server, the app calls the OpenAI-compatible endpoint). Bottom = the federated pattern — each Mac runs its own MLX; no central server holds anyone's prompts. The privacy property is structural, not a policy.

```mermaid
flowchart TB
  subgraph Workflow["MLX WORKFLOW (per Mac)"]
    direction LR
    HF["mlx-community on HuggingFace\npre-quantized 4-bit models"]
    Pull["mlx_lm.server\npython -m mlx_lm.server\n--model mlx-community/...-4bit"]
    Local["127.0.0.1:8080\n/v1/chat/completions"]
    HF -->|"hf download"| Pull --> Local
  end

  subgraph Fleet["FEDERATED LOCAL SERVING (the Mac-fleet pattern)"]
    direction LR
    Mac1["Mac 1\nMLX + local data\n(workstation A)"]
    Mac2["Mac 2\nMLX + local data\n(workstation B)"]
    Mac3["Mac N\nMLX + local data\n(workstation N)"]
    Mac1 -. "no shared server" .-> Mac2
    Mac2 -. "no shared server" .-> Mac3
  end

  Property["STRUCTURAL PRIVACY:\nNo central server holds anyone's prompts.\n4-bit ~75% size reduction, minimal quality loss.\n~143 tok/s on Qwen3-VL-4B class on current M-series.\nWWDC25 session 298: Apple-endorsed local serving path."]
  Workflow -.-> Property
  Fleet -.-> Property

  style HF fill:#14141f,stroke:rgba(255,255,255,0.12),color:#9494a0
  style Pull fill:#14141f,stroke:#5eead4,color:#e4e4e8
  style Local fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style Mac1 fill:#14141f,stroke:rgba(94,234,212,0.6),color:#e4e4e8
  style Mac2 fill:#14141f,stroke:rgba(94,234,212,0.6),color:#e4e4e8
  style Mac3 fill:#14141f,stroke:rgba(94,234,212,0.6),color:#e4e4e8
  style Property fill:#08080c,stroke:rgba(94,234,212,0.4),stroke-dasharray: 4 2,color:#5eead4
  style Workflow fill:#08080c,stroke:rgba(255,255,255,0.06),color:#e4e4e8
  style Fleet fill:#08080c,stroke:rgba(255,255,255,0.06),color:#e4e4e8
```

---

## Validation notes

- All five diagrams use the course design system colors: `#14141f` panel fill, `#5eead4` accent for primary, `rgba(255,255,255,0.12)` for secondary borders, `#e4e4e8` / `#9494a0` for text. Danger callouts use `#f08080` (the Ollama ceiling diagram).
- Paste each into [Mermaid Live Editor](https://mermaid.live) to render. All use stable Mermaid syntax (`flowchart` with `subgraph`) supported in current Mermaid (v10.4+).
- For the slide deck (artifact 03), these are rendered as static captures from Mermaid Live, inlined into reveal.js.