Course: Course 3 — LLM Fine-Tuning Masterclass Module: FT20 — Serving Stacks Duration: 60 minutes Level: Senior Engineer and above Prerequisites: FT19 — Quantization Formats (you must arrive with a runnable quantized artifact)
After completing this module, you will be able to:
127.0.0.1, expose an OpenAI-compatible endpoint, add concurrency limits and queueing, and choose the right Apple Silicon path (mlx-lm, 4-bit) for Mac fleets in regulated domains.You have a quantized model from FT19. Now you must run it. The runtime you pick is a function of three variables: who calls it, what hardware it runs on, and how much it is allowed to talk to the network.
Every serving decision collapses to three questions, asked in this order:
The runtime landscape, mapped against the variables:
| Runtime | Best for | Hardware | Concurrency | Telemetry posture |
|---|---|---|---|---|
| vLLM | Production serving | NVIDIA (CUDA); limited CPU/Metal | High (PagedAttention, continuous batching) | OpenTelemetry-native; point exporters at your own collector |
| llama.cpp server | Air-gap, edge, max breadth | CUDA, Metal, ROCm, CPU, Vulkan | Low–medium | Never phones home; single binary, no network deps |
| Ollama | Local dev, single user | Wraps llama.cpp (CUDA/Metal/CPU) | Collapses at ~5 concurrent | Needs network only for ollama pull; plaintext history; 6 known CVEs |
| TGI | HF-ecosystem production | NVIDIA (CUDA) | High | Prometheus/Grafana; mature observability |
| localai | Ollama-compatible, broader backends | CUDA, Metal, CPU | Low–medium | Community project; varies by backend |
| mlx-lm | Apple Silicon native | Apple Silicon (Metal) | Single-user to small team | Local-only; no phone home |
| SGLang / TensorRT-LLM | High-end production | NVIDIA (CUDA) | Very high | Alongside vLLM; production telemetry |
Read that table as a routing function, not a ranking. There is no "best" runtime — there is the runtime that fits your three variables. vLLM is the production standard on NVIDIA. llama.cpp is the air-gap standard everywhere. Ollama is the dev standard for one person. MLX is the standard on Mac. Pick the one your variables point at.
vLLM is the runtime you reach for when the audience is "more than a handful of concurrent users" and the hardware is NVIDIA. Three engineering choices made it the standard:
--tensor-parallel-size N), not a recompile.The API is OpenAI-compatible — /v1/chat/completions and siblings. Any client written for OpenAI works against a local vLLM server by changing one base URL. This is why vLLM slots into existing agent stacks without code changes.
The telemetry posture matters here. vLLM is OpenTelemetry-native: it emits traces and metrics through the OTel protocol, but it does not phone home to any vendor. You point the OTel exporters at your own collector (Jaeger, a self-hosted Grafana Tempo) — or point them at nothing and the traces drop. No forced telemetry, no call-home check, no licensing server. This is what makes vLLM acceptable inside sensitive subnets: the network egress is yours to control.
The honest limitation: vLLM's Apple Silicon and CPU support is limited. It is built around CUDA primitives. If your fleet is Macs, you want MLX, not vLLM. If your fleet is mixed, run vLLM on the NVIDIA hosts and MLX on the Macs, behind the same OpenAI-compatible interface.
If vLLM is the production standard, llama.cpp is the substrate. The same C++ library that produces GGUF (FT19) ships a minimal HTTP server (llama-server). Its virtues are negative ones, and they are exactly the virtues an air-gap rewards:
This is why, when the deployment is a hospital subnet with no internet egress or a classified facility with no network at all, the answer is almost always llama.cpp server. It is the most self-contained option in the matrix.
The trade-off is concurrency. llama.cpp server handles a few concurrent requests respectably, but it lacks vLLM's mature multi-GPU tensor parallelism and battle-tested scheduler under heavy load. For single-user or small-team serving it is excellent. For hundreds of concurrent users, route to vLLM.
Ollama wraps llama.cpp in a polished CLI and a clean API. Its DX is the best in the field: ollama run llama3.1, and you are chatting. For a single developer iterating on a laptop, Ollama is unmatched.
Ollama's official privacy policy is direct: "We don't see your prompts or data when you run locally." That is true and it matters — when you run ollama run, your prompts stay on your machine. But "we don't see your data" is not the whole privacy story, and engineers who stop reading there get burned. Three caveats:
ollama run conversations and Modelfile system prompts persist on disk. On a shared laptop or fleet-managed Mac, that is a data-at-rest concern.OLLAMA_HOST exposes the API. The default bind is 127.0.0.1, which is correct. But setting OLLAMA_HOST=0.0.0.0 to "make it work" from another machine exposes an unauthenticated API to the network — and public-internet Ollama instances have been scraped repeatedly this way. Never bind Ollama to 0.0.0.0 without an auth layer in front.And then the production ceiling. Ollama collapses at roughly five concurrent users. This is a measured, reproducible finding (Towards AI's concurrency test, corroborated by Red Hat's benchmark deep dive). Response times go from ~3 seconds at one user to over a minute at five, because Ollama serializes requests and duplicates context per request — no PagedAttention, no real continuous batching. vLLM on the same hardware holds the workload at stable latency. Ollama is a dev tool, not a production server. The moment you have real concurrency, you move to vLLM or TGI. This is the single most expensive serving mistake teams make.
mlx_lm.server exposing the OpenAI-compatible API on Apple Silicon. The Mac-native path, covered in 20.3.The decision is rarely vLLM versus SGLang versus TensorRT-LLM. It is "one of these three on NVIDIA for production, llama.cpp for air-gap, Ollama for dev, MLX for Mac." Hold the matrix in your head and routing becomes obvious.
The single most underweighted variable in serving. Get it wrong in a regulated domain and the model never ships — or it ships and you fail the audit.
llama.cpp and vLLM genuinely never need to phone home. Ollama needs the network only for
ollama pull; once models are local, block the network and bindOLLAMA_HOST=127.0.0.1. For a true air-gap, pre-load the models, then sever the network.
That is the whole rule. Everything else is implementation detail.
llama.cpp is a single binary with no network code path for telemetry. Nothing to disable, no env var to set, no phone-home flag. The binary reads a local file and serves. You can prove this with a packet capture — start the server on a default-deny egress firewall and it works identically.
vLLM is slightly more nuanced but lands in the same place. vLLM is OpenTelemetry-native — it emits telemetry, but to whatever endpoint you configure via the standard OTel env vars (OTEL_EXPORTER_OTLP_ENDPOINT, etc.). Point them at your own collector on 127.0.0.1, or leave them unset and traces are no-ops. No vendor call-home, no license check. The egress story is entirely yours.
Ollama needs the network for exactly one thing in normal operation: ollama pull, which fetches models from the Ollama registry (which itself proxies HuggingFace and other sources). Once a model is local — once ollama list shows it and the blobs are on disk — Ollama no longer needs any network to serve it.
The recipe for an Ollama-on-a-restricted-network deploy:
ollama pull every model you need. Verify with ollama list.~/.ollama/models.OLLAMA_HOST=127.0.0.1:11434 so the API is reachable only from the local machine. If a remote machine needs it, put an authenticated reverse proxy in front — never expose raw Ollama.The mistake is assuming "Ollama is local therefore it is safe." It is safe once configured. The default install, pointed at 0.0.0.0 on an internet-facing box, is a known attack surface. The configuration is the security boundary.
For a true air-gap (no network egress at all, ever), the recipe is the same regardless of runtime:
This recipe is the bridge to Pillar 7. FT21 (HIPAA and BAA elimination) and FT22 (government/air-gap) both build on it. If you cannot execute steps 1–5 cleanly, you are not ready for a regulated deploy.
If your fleet is Macs — common in healthcare workstations, defense laptops, and design studios — the runtime choice is not vLLM. It is MLX.
Apple Silicon (M1/M2/M3/M4) has unified memory: the GPU and CPU share the same physical RAM. A model that "loads into VRAM" on a Mac is really loading into this unified pool. But CUDA-targeted runtimes (vLLM, TGI, TensorRT-LLM) cannot exploit this — they expect discrete NVIDIA GPUs. Running them on a Mac means falling back to CPU or experimental Metal paths that give up most of the optimizations.
MLX is Apple's answer: an array framework, designed by Apple's ML research team, targeting unified memory and the Metal backend natively. mlx-lm is the model-serving layer on top. It is to Apple Silicon what vLLM is to NVIDIA — built for the hardware, not ported to it.
Apple officially embraced local serving with MLX at WWDC 2025 (session 298). The platform vendor's message was explicit: local, on-device, private-by-construction serving is a supported path on macOS, not a hack.
For MLX, the quantization recommendation lines up with FT19: 4-bit is the sweet spot. Roughly 75% size reduction versus FP16, with minimal measurable quality loss for the model classes you deploy locally (sub-30B instruct and reasoning models).
The practical workflow: pull a pre-quantized model from mlx-community on HuggingFace (e.g., mlx-community/Qwen2.5-7B-Instruct-4bit), serve with python -m mlx_lm.server --model ..., point your app at the OpenAI-compatible endpoint. You rarely quantize yourself — the community hosts thousands already.
Performance is the draw. Benchmarks show ~143 tokens/sec on a Qwen3-VL-4B class model on current M-series hardware. A quantized 7B fits comfortably in 8GB of unified memory and serves at chat-speed on an M2 Pro.
MLX is the right answer when the deployment target is a Mac fleet and the constraint is privacy by construction. Healthcare workstations where patient data cannot leave the device. Defense laptops that operate disconnected. Design and legal studios where the IP is the conversation itself. In all of these, "the model runs locally on the Mac, the data never leaves the device" is not a feature — it is the compliance argument. MLX makes that argument technically true.
The honest limitation: MLX does not serve hundreds of concurrent users. It is a single-machine runtime. For a hospital with 500 Mac workstations, you do not stand up one MLX server and share it; you run MLX on each workstation. The architecture is federated local serving, not centralized serving. This is actually a privacy advantage — there is no central server holding everyone's prompts — but it is a different operational model than the vLLM-on-a-GPU-box pattern, and teams used to the latter need to adjust.
The patterns that separate a working dev setup from a service you can page on.
127.0.0.1, always, by default. The model serves to processes on the same machine. Exposing the raw API to the network is something you do deliberately, with auth, never by accident./v1/chat/completions with a configurable base_url; you swap base_url between local Ollama in dev and a vLLM cluster in prod with no code change. This is the single highest-leverage integration decision in the serving layer.The unifying pattern is the model is one OpenAI-compatible endpoint behind your reverse proxy, with metrics exported to your own stack. Whether the backing runtime is vLLM, llama.cpp, or MLX, the application sees the same interface and the operator sees the same dashboards. This lets you run vLLM on the NVIDIA box for high-traffic service and MLX on a Mac for a low-traffic internal tool, without retraining ops on two different stacks.
The cardinal serving mistake. Ollama collapses at ~5 concurrent users because it serializes requests and duplicates context per request. Response times go from seconds to minutes. Ollama is a dev tool. The moment you have real concurrency, move to vLLM or TGI.
OLLAMA_HOST=0.0.0.0 without authThe default 127.0.0.1 bind is correct. Setting 0.0.0.0 to "make it work from another machine" exposes an unauthenticated API to the network. Public-internet Ollama instances have been scraped repeatedly, and the 2025 Oligo CVEs make raw exposure actively dangerous. Never bind Ollama broadly without an auth layer in front.
"Ollama is fast on my laptop with one user" is not a production signal. You must benchmark under your realistic concurrent load before declaring a runtime production-ready. vLLM's continuous-batching advantage only shows up under load; Ollama's collapse only shows up under load. Single-request latency tells you nothing about either.
Treating "it runs locally" as equivalent to "it is private." It is not. Ollama stores chat history in plaintext; vLLM emits OTel traces to whatever you point it at; a misconfigured bind exposes the API. For HIPAA, defense, or air-gap deployments, the telemetry posture is a first-class variable — audit before shipping. (FT21, FT22.)
| Term | Definition |
|---|---|
| PagedAttention | vLLM's technique of storing the KV cache in paged blocks (like OS virtual memory), eliminating fragmentation and enabling much higher batch sizes in the same VRAM |
| Continuous batching | Inserting new requests into running batches at every token step (rather than waiting for the batch to drain), keeping GPUs saturated and bounding tail latency |
| Tensor parallelism | Splitting a single model's layers across multiple GPUs, each holding a slice, with synchronization at each attention block — how vLLM serves models too big for one GPU |
| OpenAI-compatible API | The /v1/chat/completions (and siblings) dialect that vLLM, llama.cpp, Ollama, TGI, MLX, and SGLang all speak — the lingua franca of the serving layer |
| OTel-native | A runtime that emits OpenTelemetry traces/metrics to a collector you configure, with no forced vendor phone-home — vLLM's telemetry posture |
| Air-gap | A deployment with no network egress at all; requires pre-loading models on a connected staging machine, transferring via approved media, validating checksums, and serving with a runtime that never reaches out |
| mlx-lm | Apple's model-serving layer on MLX, the native array framework for Apple Silicon unified memory; the Mac-native alternative to CUDA-targeted runtimes |
| OLLAMA_HOST | The environment variable controlling Ollama's bind address; default 127.0.0.1 is correct, 0.0.0.0 without auth is a known attack surface |
| The ~5 concurrent user ceiling | Ollama's measured production limit, beyond which latency explodes; the reason Ollama is a dev tool, not a production server |
See 07-lab-spec.md. The "Serve Locally" lab: take your quantized model from FT19, serve it via (a) Ollama for dev/single-user and (b) vLLM for simulated multi-user load, hit both with the same eval suite, and report latency under concurrency. The lab that makes the Ollama-vs-vLLM decision felt, not theoretical.
OLLAMA_HOST exposure risk.llama-server; the air-gap champion; maximum hardware breadth.# Module FT20 — Serving Stacks **Course**: Course 3 — LLM Fine-Tuning Masterclass **Module**: FT20 — Serving Stacks **Duration**: 60 minutes **Level**: Senior Engineer and above **Prerequisites**: FT19 — Quantization Formats (you must arrive with a runnable quantized artifact) --- ## Learning Objectives After completing this module, you will be able to: 1. Place each major serving runtime — vLLM, llama.cpp server, Ollama, TGI, localai, MLX, SGLang, TensorRT-LLM — on the decision matrix (production vs dev, hardware breadth vs throughput, telemetry posture) and justify a pick for a given target environment. 2. State the **no-telemetry rule of thumb**: which stacks genuinely never phone home (llama.cpp, vLLM), which need the network only for model pulls (Ollama), and the pre-load-then-sever recipe for a true air-gap. 3. Predict when Ollama collapses — the ~5 concurrent user ceiling — and explain *why* (no PagedAttention, request serialization, per-request context duplication), then route that workload to vLLM or TGI. 4. Configure a production deploy pattern: bind to `127.0.0.1`, expose an OpenAI-compatible endpoint, add concurrency limits and queueing, and choose the right Apple Silicon path (mlx-lm, 4-bit) for Mac fleets in regulated domains. --- # 20.1 — The Decision Matrix *You have a quantized model from FT19. Now you must run it. The runtime you pick is a function of three variables: who calls it, what hardware it runs on, and how much it is allowed to talk to the network.* ## The three variables Every serving decision collapses to three questions, asked in this order: 1. **Audience.** One developer on a laptop? A small team behind a VPN? Hundreds of concurrent end users? The audience determines whether you need a *runtime* (one process, one user) or a *server* (concurrent, queued, observable). 2. **Hardware.** NVIDIA CUDA? Apple Silicon? CPU-only edge? ROCm? Hardware determines which runtimes are even options — and on Apple Silicon, whether you use a CUDA-targeted stack or the native MLX path. 3. **Telemetry posture.** Open network? Regulated hospital subnet? Classified air-gap? This determines which runtimes you may operationally run, full stop. It is the variable most engineers underweight, and the one this module spends the most time on, because Pillar 7 (FT21–FT22) builds directly on it. The runtime landscape, mapped against the variables: | Runtime | Best for | Hardware | Concurrency | Telemetry posture | | --- | --- | --- | --- | --- | | **vLLM** | Production serving | NVIDIA (CUDA); limited CPU/Metal | High (PagedAttention, continuous batching) | OpenTelemetry-native; point exporters at your own collector | | **llama.cpp server** | Air-gap, edge, max breadth | CUDA, Metal, ROCm, CPU, Vulkan | Low–medium | Never phones home; single binary, no network deps | | **Ollama** | Local dev, single user | Wraps llama.cpp (CUDA/Metal/CPU) | Collapses at ~5 concurrent | Needs network only for `ollama pull`; plaintext history; 6 known CVEs | | **TGI** | HF-ecosystem production | NVIDIA (CUDA) | High | Prometheus/Grafana; mature observability | | **localai** | Ollama-compatible, broader backends | CUDA, Metal, CPU | Low–medium | Community project; varies by backend | | **mlx-lm** | Apple Silicon native | Apple Silicon (Metal) | Single-user to small team | Local-only; no phone home | | **SGLang / TensorRT-LLM** | High-end production | NVIDIA (CUDA) | Very high | Alongside vLLM; production telemetry | Read that table as a routing function, not a ranking. There is no "best" runtime — there is the runtime that fits your three variables. vLLM is the production standard *on NVIDIA*. llama.cpp is the air-gap standard *everywhere*. Ollama is the dev standard *for one person*. MLX is the standard *on Mac*. Pick the one your variables point at. ## vLLM — the production standard vLLM is the runtime you reach for when the audience is "more than a handful of concurrent users" and the hardware is NVIDIA. Three engineering choices made it the standard: - **PagedAttention.** The KV cache is stored in paged blocks (like an OS's virtual memory), not a contiguous tensor. This eliminates the fragmentation that wastes 60–80% of KV-cache memory in naïve implementations, letting vLLM pack many more sequences into the same VRAM. - **Continuous batching.** Naïve batching waits for the longest sequence in a batch to finish before starting the next batch. Continuous batching inserts new requests into running batches at every token step, so GPUs stay saturated and tail latency stays bounded. The throughput difference is often 5–10× under realistic load. - **Multi-GPU tensor parallelism.** A model too big for one GPU is split across several — each GPU holds a slice of every layer, synchronizing at each attention block. vLLM makes this a flag (`--tensor-parallel-size N`), not a recompile. The API is OpenAI-compatible — `/v1/chat/completions` and siblings. Any client written for OpenAI works against a local vLLM server by changing one base URL. This is why vLLM slots into existing agent stacks without code changes. The telemetry posture matters here. vLLM is **OpenTelemetry-native**: it emits traces and metrics through the OTel protocol, but it does *not* phone home to any vendor. You point the OTel exporters at your own collector (Jaeger, a self-hosted Grafana Tempo) — or point them at nothing and the traces drop. No forced telemetry, no call-home check, no licensing server. This is what makes vLLM acceptable inside sensitive subnets: the network egress is *yours* to control. The honest limitation: **vLLM's Apple Silicon and CPU support is limited.** It is built around CUDA primitives. If your fleet is Macs, you want MLX, not vLLM. If your fleet is mixed, run vLLM on the NVIDIA hosts and MLX on the Macs, behind the same OpenAI-compatible interface. ## llama.cpp server — the air-gap champion If vLLM is the production standard, llama.cpp is the *substrate*. The same C++ library that produces GGUF (FT19) ships a minimal HTTP server (`llama-server`). Its virtues are negative ones, and they are exactly the virtues an air-gap rewards: - **Single binary.** No Python interpreter, no container runtime, no transitive deps at startup. One statically-linked binary, copied onto the box, runs the model. - **No network dependencies.** The server does not reach out for models, license checks, telemetry, anything. You give it a local GGUF path; it serves. Cut the network cable and it keeps serving. - **Maximum hardware breadth.** CUDA, Metal, ROCm, Vulkan, pure CPU. The same code path. The only first-class runtime that treats CPU as a real target rather than a fallback. This is why, when the deployment is a hospital subnet with no internet egress or a classified facility with no network at all, the answer is almost always llama.cpp server. It is the most self-contained option in the matrix. The trade-off is concurrency. llama.cpp server handles a few concurrent requests respectably, but it lacks vLLM's mature multi-GPU tensor parallelism and battle-tested scheduler under heavy load. For single-user or small-team serving it is excellent. For hundreds of concurrent users, route to vLLM. ## Ollama — dev simplicity, not production Ollama wraps llama.cpp in a polished CLI and a clean API. Its DX is the best in the field: `ollama run llama3.1`, and you are chatting. For a single developer iterating on a laptop, Ollama is unmatched. Ollama's official privacy policy is direct: *"We don't see your prompts or data when you run locally."* That is true and it matters — when you run `ollama run`, your prompts stay on your machine. But "we don't see your data" is not the whole privacy story, and engineers who stop reading there get burned. Three caveats: 1. **Chat history is stored locally, in plaintext.** `ollama run` conversations and Modelfile system prompts persist on disk. On a shared laptop or fleet-managed Mac, that is a data-at-rest concern. 2. **Misconfigured `OLLAMA_HOST` exposes the API.** The default bind is `127.0.0.1`, which is correct. But setting `OLLAMA_HOST=0.0.0.0` to "make it work" from another machine exposes an unauthenticated API to the network — and public-internet Ollama instances have been scraped repeatedly this way. Never bind Ollama to `0.0.0.0` without an auth layer in front. 3. **Oligo found six vulnerabilities.** In 2025, Oligo Security disclosed six CVEs in Ollama (including CVE-2025-1975 DoS, CVE-2025-51471 cross-domain token exposure, and CVE-2025-63389 authentication bypass in versions ≤ 0.12.3). Patched in current releases — *if you keep Ollama updated*. An air-gapped Ollama that never patches is an air-gapped Ollama carrying known CVEs. And then the production ceiling. **Ollama collapses at roughly five concurrent users.** This is a measured, reproducible finding (Towards AI's concurrency test, corroborated by Red Hat's benchmark deep dive). Response times go from ~3 seconds at one user to over a minute at five, because Ollama serializes requests and duplicates context per request — no PagedAttention, no real continuous batching. vLLM on the same hardware holds the workload at stable latency. **Ollama is a dev tool, not a production server.** The moment you have real concurrency, you move to vLLM or TGI. This is the single most expensive serving mistake teams make. ## TGI, localai, MLX, SGLang, TensorRT-LLM — the rest of the field - **TGI (text-generation-inference).** HuggingFace's production server. CUDA-first, high concurrency, and the strongest out-of-box observability — Prometheus metrics and Grafana dashboards are first-class. Choose TGI when your org runs the HF stack and wants mature metrics without wiring OTel by hand. - **localai.** An Ollama-compatible API surface over a broader set of backends (diffusers, audio, vision). Community-maintained. Treat its telemetry and security posture as "varies by backend" — audit before deploying in regulated environments. - **mlx-lm.** Apple's native array framework, with an `mlx_lm.server` exposing the OpenAI-compatible API on Apple Silicon. The Mac-native path, covered in 20.3. - **SGLang.** A research-origin runtime that, alongside vLLM, defines the high end of production serving. RadixAttention (prefix caching across requests with shared prefixes) gives it an edge on agentic workloads where many requests share a long system prompt. - **TensorRT-LLM.** NVIDIA's own highly-optimized runtime, the lowest-latency option on H100/H200 class hardware. The cost is integration complexity — building engines, paged KV cache tuning, stricter model support. The decision is rarely vLLM *versus* SGLang *versus* TensorRT-LLM. It is "one of these three on NVIDIA for production, llama.cpp for air-gap, Ollama for dev, MLX for Mac." Hold the matrix in your head and routing becomes obvious. --- # 20.2 — The No-Telemetry Rule of Thumb (and the Air-Gap) *The single most underweighted variable in serving. Get it wrong in a regulated domain and the model never ships — or it ships and you fail the audit.* ## The rule of thumb > **llama.cpp and vLLM genuinely never need to phone home. Ollama needs the network only for `ollama pull`; once models are local, block the network and bind `OLLAMA_HOST=127.0.0.1`. For a true air-gap, pre-load the models, then sever the network.** That is the whole rule. Everything else is implementation detail. ### Why llama.cpp and vLLM are safe by construction llama.cpp is a single binary with no network code path for telemetry. Nothing to disable, no env var to set, no phone-home flag. The binary reads a local file and serves. You can prove this with a packet capture — start the server on a default-deny egress firewall and it works identically. vLLM is slightly more nuanced but lands in the same place. vLLM is OpenTelemetry-native — it *emits* telemetry, but to whatever endpoint *you* configure via the standard OTel env vars (`OTEL_EXPORTER_OTLP_ENDPOINT`, etc.). Point them at your own collector on `127.0.0.1`, or leave them unset and traces are no-ops. No vendor call-home, no license check. The egress story is entirely yours. ### Why Ollama is safe *if you configure it* Ollama needs the network for exactly one thing in normal operation: `ollama pull`, which fetches models from the Ollama registry (which itself proxies HuggingFace and other sources). Once a model is local — once `ollama list` shows it and the blobs are on disk — Ollama no longer needs any network to serve it. The recipe for an Ollama-on-a-restricted-network deploy: 1. **Pre-load on a connected machine.** `ollama pull` every model you need. Verify with `ollama list`. 2. **Copy the blob store to the target** (or pre-load directly on the target if it can briefly reach the registry, then cut over). The blobs live in `~/.ollama/models`. 3. **Bind loopback.** Set `OLLAMA_HOST=127.0.0.1:11434` so the API is reachable only from the local machine. If a remote machine needs it, put an authenticated reverse proxy in front — never expose raw Ollama. 4. **Sever the network** (for a true air-gap) or lock down egress (for a regulated subnet). Ollama will serve from the local blobs indefinitely. The mistake is assuming "Ollama is local therefore it is safe." It is safe *once configured*. The default install, pointed at `0.0.0.0` on an internet-facing box, is a known attack surface. The configuration is the security boundary. ### The air-gap recipe, generalized For a true air-gap (no network egress at all, ever), the recipe is the same regardless of runtime: 1. **Pre-load models on a staging machine that *does* have network.** Pull from HuggingFace, the Ollama registry, wherever your artifacts live. 2. **Transfer via approved media** — approved USB, one-way data diode, whatever your security office permits. Often a "sneakernet" hop. 3. **Validate checksums on the air-gapped side.** Models are large; transfers can corrupt. SHA256 on staging, re-hash on the isolated side, compare. 4. **Serve with llama.cpp server** (preferred — single binary, no runtime deps) **or a pre-loaded vLLM/Ollama** with egress firewalled. The model never leaves the box; the runtime never reaches out. 5. **Patch on a schedule via the same media.** Air-gap does not mean "frozen forever." It means updates come through the approved channel, not the live internet. Air-gapped != invulnerable, only isolated — known CVEs still apply, so plan the patch cadence. This recipe is the bridge to Pillar 7. FT21 (HIPAA and BAA elimination) and FT22 (government/air-gap) both build on it. If you cannot execute steps 1–5 cleanly, you are not ready for a regulated deploy. --- # 20.3 — Apple Silicon and MLX *If your fleet is Macs — common in healthcare workstations, defense laptops, and design studios — the runtime choice is not vLLM. It is MLX.* ## Why MLX exists Apple Silicon (M1/M2/M3/M4) has unified memory: the GPU and CPU share the same physical RAM. A model that "loads into VRAM" on a Mac is really loading into this unified pool. But CUDA-targeted runtimes (vLLM, TGI, TensorRT-LLM) cannot exploit this — they expect discrete NVIDIA GPUs. Running them on a Mac means falling back to CPU or experimental Metal paths that give up most of the optimizations. MLX is Apple's answer: an array framework, designed by Apple's ML research team, targeting unified memory and the Metal backend natively. `mlx-lm` is the model-serving layer on top. It is to Apple Silicon what vLLM is to NVIDIA — built for the hardware, not ported to it. Apple officially embraced local serving with MLX at WWDC 2025 (session 298). The platform vendor's message was explicit: local, on-device, private-by-construction serving is a supported path on macOS, not a hack. ## The 4-bit recommendation For MLX, the quantization recommendation lines up with FT19: **4-bit is the sweet spot.** Roughly 75% size reduction versus FP16, with minimal measurable quality loss for the model classes you deploy locally (sub-30B instruct and reasoning models). The practical workflow: pull a pre-quantized model from `mlx-community` on HuggingFace (e.g., `mlx-community/Qwen2.5-7B-Instruct-4bit`), serve with `python -m mlx_lm.server --model ...`, point your app at the OpenAI-compatible endpoint. You rarely quantize yourself — the community hosts thousands already. Performance is the draw. Benchmarks show ~143 tokens/sec on a Qwen3-VL-4B class model on current M-series hardware. A quantized 7B fits comfortably in 8GB of unified memory and serves at chat-speed on an M2 Pro. ## When MLX is the right answer MLX is the right answer when the deployment target is a Mac fleet and the constraint is *privacy by construction*. Healthcare workstations where patient data cannot leave the device. Defense laptops that operate disconnected. Design and legal studios where the IP is the conversation itself. In all of these, "the model runs locally on the Mac, the data never leaves the device" is not a feature — it is the compliance argument. MLX makes that argument technically true. The honest limitation: MLX does not serve hundreds of concurrent users. It is a single-machine runtime. For a hospital with 500 Mac workstations, you do not stand up one MLX server and share it; you run MLX on each workstation. The architecture is *federated local serving*, not centralized serving. This is actually a *privacy advantage* — there is no central server holding everyone's prompts — but it is a different operational model than the vLLM-on-a-GPU-box pattern, and teams used to the latter need to adjust. --- # 20.4 — Production Deployment Patterns *The patterns that separate a working dev setup from a service you can page on.* ## The four patterns 1. **Bind loopback for local-only.** `127.0.0.1`, always, by default. The model serves to processes on the same machine. Exposing the raw API to the network is something you do deliberately, with auth, never by accident. 2. **OpenAI-compatible API for app integration.** vLLM, llama.cpp server, Ollama, TGI, MLX, SGLang all speak the OpenAI chat-completions dialect. Standardize on it. Your application calls `/v1/chat/completions` with a configurable `base_url`; you swap `base_url` between local Ollama in dev and a vLLM cluster in prod with no code change. This is the single highest-leverage integration decision in the serving layer. 3. **Concurrency limits and queueing for multi-user.** Once you have more than a handful of concurrent users, you need explicit limits: max in-flight requests, a queue with bounded wait, graceful 429s when the queue fills. vLLM and TGI have these built in; Ollama does not. In front of any runtime, put a reverse proxy (nginx, envoy) enforcing per-client rate limits and timeouts. 4. **Observability wired to your own collector.** OTel for vLLM, Prometheus for TGI. The metrics come to *you*; you do not phone home to a vendor dashboard. Latency percentiles (p50/p95/p99), tokens/sec, queue depth, error rate. If you cannot see the latency tail, you cannot tune the concurrency limit. ## The pattern that ties them together The unifying pattern is **the model is one OpenAI-compatible endpoint behind your reverse proxy, with metrics exported to your own stack.** Whether the backing runtime is vLLM, llama.cpp, or MLX, the application sees the same interface and the operator sees the same dashboards. This lets you run vLLM on the NVIDIA box for high-traffic service and MLX on a Mac for a low-traffic internal tool, without retraining ops on two different stacks. --- ## Anti-Patterns ### Using Ollama for production multi-user The cardinal serving mistake. Ollama collapses at ~5 concurrent users because it serializes requests and duplicates context per request. Response times go from seconds to minutes. Ollama is a dev tool. The moment you have real concurrency, move to vLLM or TGI. ### Exposing `OLLAMA_HOST=0.0.0.0` without auth The default `127.0.0.1` bind is correct. Setting `0.0.0.0` to "make it work from another machine" exposes an unauthenticated API to the network. Public-internet Ollama instances have been scraped repeatedly, and the 2025 Oligo CVEs make raw exposure actively dangerous. Never bind Ollama broadly without an auth layer in front. ### Not benchmarking under realistic load "Ollama is fast on my laptop with one user" is not a production signal. You must benchmark under your *realistic concurrent load* before declaring a runtime production-ready. vLLM's continuous-batching advantage only shows up under load; Ollama's collapse only shows up under load. Single-request latency tells you nothing about either. ### Ignoring telemetry posture for sensitive deployments Treating "it runs locally" as equivalent to "it is private." It is not. Ollama stores chat history in plaintext; vLLM emits OTel traces to whatever you point it at; a misconfigured bind exposes the API. For HIPAA, defense, or air-gap deployments, the telemetry posture is a first-class variable — audit before shipping. (FT21, FT22.) --- ## Key Terms | Term | Definition | | --- | --- | | **PagedAttention** | vLLM's technique of storing the KV cache in paged blocks (like OS virtual memory), eliminating fragmentation and enabling much higher batch sizes in the same VRAM | | **Continuous batching** | Inserting new requests into running batches at every token step (rather than waiting for the batch to drain), keeping GPUs saturated and bounding tail latency | | **Tensor parallelism** | Splitting a single model's layers across multiple GPUs, each holding a slice, with synchronization at each attention block — how vLLM serves models too big for one GPU | | **OpenAI-compatible API** | The `/v1/chat/completions` (and siblings) dialect that vLLM, llama.cpp, Ollama, TGI, MLX, and SGLang all speak — the lingua franca of the serving layer | | **OTel-native** | A runtime that emits OpenTelemetry traces/metrics to a collector *you* configure, with no forced vendor phone-home — vLLM's telemetry posture | | **Air-gap** | A deployment with no network egress at all; requires pre-loading models on a connected staging machine, transferring via approved media, validating checksums, and serving with a runtime that never reaches out | | **mlx-lm** | Apple's model-serving layer on MLX, the native array framework for Apple Silicon unified memory; the Mac-native alternative to CUDA-targeted runtimes | | **OLLAMA_HOST** | The environment variable controlling Ollama's bind address; default `127.0.0.1` is correct, `0.0.0.0` without auth is a known attack surface | | **The ~5 concurrent user ceiling** | Ollama's measured production limit, beyond which latency explodes; the reason Ollama is a dev tool, not a production server | --- ## Lab Exercise See `07-lab-spec.md`. The "Serve Locally" lab: take your quantized model from FT19, serve it via (a) Ollama for dev/single-user and (b) vLLM for simulated multi-user load, hit both with the same eval suite, and report latency under concurrency. The lab that makes the Ollama-vs-vLLM decision *felt*, not theoretical. --- ## References 1. **vLLM documentation** — *vLLM: a high-throughput and memory-efficient LLM serving engine*. The production standard; PagedAttention, continuous batching, tensor parallelism, OpenAI-compatible API. 2. **Ollama privacy policy** — *"We don't see your prompts or data when you run locally."* Accurate for prompt data; does not cover plaintext local history or the `OLLAMA_HOST` exposure risk. 3. **llama.cpp** — *ggml-org/llama.cpp*. The substrate that produces GGUF and ships `llama-server`; the air-gap champion; maximum hardware breadth. 4. **Red Hat Developer** — *Ollama vs. vLLM: a deep dive into performance benchmarking* (2025). The systematic benchmark showing vLLM's throughput and latency advantage under concurrent load. 5. **Apple WWDC25 Session 298** — *Run language models locally on-device with MLX*. Apple's official endorsement of local serving on Apple Silicon; the platform-vendor signal that MLX is a supported path. 6. **Oligo Security** — *More Models, More ProbLLMs: New Vulnerabilities in Ollama* (2025). The six CVEs (including CVE-2025-1975, CVE-2025-51471, CVE-2025-63389) disclosed in Ollama; patched in current releases. 7. **Towards AI** — *I Tested Ollama vs vLLM vs llama.cpp: the 'easiest' one collapses at 5 concurrent users.* The measured concurrency ceiling that defines Ollama's production limit. 8. **Module FT19 — Quantization Formats** — The prerequisite. You arrive at FT20 with a quantized GGUF / AWQ / MLX artifact ready to serve. 9. **Module FT21 — HIPAA and BAA Elimination** and **FT22 — Government and Air-Gap** — The Pillar 7 modules that build directly on this module's no-telemetry and air-gap recipes.