Course: Course 3 — LLM Fine-Tuning Masterclass
Module: FT20 — Serving Stacks
Duration: ~40 minutes (spoken at ~140 wpm)
Format: Verbatim transcript with [SLIDE N] cues. Read aloud or use as speaker notes.
[SLIDE 1 — Title]
Welcome to module FT20, Serving Stacks. This is pillar six, the deploy pillar, and the second half of the export layer. You arrive here from FT19 with a quantized model in your hand — a GGUF, an AWQ, an MLX artifact. The question this module answers is: what do you run it with, and what are the rules when the deployment is sensitive?
There is a recurring mistake this module exists to prevent. Teams pick Ollama because it has the best developer experience, they never benchmark under load, and then they ship it to production and it falls over at five concurrent users. By the end of this module you will know exactly when each runtime is the right answer and when it is the wrong one — and you will know the no-telemetry rules that make a runtime acceptable in a hospital subnet or a classified air-gap.
[SLIDE 2 — The three variables]
Every serving decision collapses to three variables, asked in this order.
First, audience. One developer on a laptop? A small team behind a VPN? Hundreds of concurrent end users hitting an HTTP endpoint? The audience determines whether you need a runtime — one process, one user — or a server — concurrent, queued, observable.
Second, hardware. NVIDIA CUDA? Apple Silicon? A mix? CPU-only edge boxes? ROCm? Hardware determines which runtimes are even options. And on Apple Silicon it determines whether you use a CUDA-targeted stack or the native MLX path.
Third, telemetry posture. Open network? Regulated subnet? True air-gap? This is the variable most engineers underweight, and it is the one this module spends the most time on, because pillar seven builds directly on it.
Hold those three in your head and the rest of the module is just routing.
[SLIDE 3 — The decision matrix]
Here is the routing function. There is no best runtime — there is the runtime your three variables point at.
NVIDIA production, open subnet, many concurrent users — that is vLLM. It is the production standard on CUDA.
Air-gap or regulated subnet, any hardware — that is llama.cpp server. Single binary, no network dependencies, maximum hardware breadth.
Dev or single user, any hardware — that is Ollama. Best DX in the field, but it collapses under concurrency, which we will get to.
Mac fleet, Apple Silicon — that is mlx-lm. Native to the unified memory, the Mac-native path.
The rest of the field — TGI, localai, SGLang, TensorRT-LLM — fills in around these four. TGI is HuggingFace's production server with the best out-of-box observability. SGLang uses RadixAttention for prefix caching and wins on agentic workloads with shared system prompts. TensorRT-LLM is NVIDIA's own squeeze-the-last-millisecond runtime for flagship silicon. Read the matrix as a routing function, not a ranking.
[SLIDE 4 — vLLM, the production standard]
Let's talk about why vLLM is the production standard on NVIDIA. Three engineering choices.
First, PagedAttention. The key-value cache is stored in paged blocks — like an operating system's virtual memory — not a contiguous tensor. A naive contiguous cache pre-allocates a max-length block per sequence; because sequences vary in length, sixty to eighty percent of that allocation is wasted to fragmentation. PagedAttention eliminates the fragmentation, and the reclaimed memory holds many more concurrent sequences.
Second, continuous batching. Naive batching waits for the longest sequence in a batch to finish before starting the next batch. Continuous batching inserts new requests into running batches at every token step, so the GPUs stay saturated and tail latency stays bounded. The throughput difference is often five to ten times under realistic load.
Third, multi-GPU tensor parallelism. A model too big for one GPU is split across several — each holds a slice of every layer, synchronizing at each attention block. It is a flag, not a recompile.
And the API is OpenAI-compatible. Slash v one slash chat slash completions. Any client written for OpenAI works against a local vLLM server by changing one base URL. That is why it slots into existing agent stacks without code changes.
The honest limitation: vLLM's Apple Silicon and CPU support is limited. It is built around CUDA. If your fleet is Macs, you want MLX.
[SLIDE 5 — The telemetry posture]
Now the variable most engineers underweight. Telemetry posture.
Here is the rule of thumb. llama.cpp and vLLM genuinely never need to phone home. Ollama needs the network only for ollama pull; once models are local, block the network and bind OLLAMA_HOST to one twenty seven dot zero dot zero dot one. For a true air-gap, pre-load the models, then sever the network.
That is the whole rule. Let me explain why each runtime lands where it does.
llama.cpp is a single statically-linked binary. There is no network code path for telemetry. Nothing to disable, no flag to flip, no call-home server to reach. The binary reads a local file and serves. You can prove this with a packet capture. It is the air-gap champion by construction.
vLLM is slightly more nuanced but lands in the same place. It is OpenTelemetry-native, which means it emits traces and metrics — but it emits to whatever endpoint you configure via the standard OTel environment variables. Point them at your own collector running on loopback, or leave them unset and the traces are no-ops. There is no vendor phone-home, no license check. The egress story is entirely yours.
Ollama needs the network for exactly one thing in normal operation: ollama pull, which fetches models. Once a model is local, Ollama no longer needs any network to serve it. The recipe for a restricted-network deploy is: pre-load on a connected machine, copy the blob store, bind loopback, sever the network. Ollama will serve from local blobs indefinitely.
[SLIDE 6 — The air-gap recipe]
For a true air-gap — no network egress at all, ever — the recipe is the same regardless of runtime.
One, pre-load models on a staging machine that does have network. Pull from HuggingFace, from the Ollama registry, wherever your artifacts live.
Two, transfer via approved media — approved USB, one-way data diode, whatever your security office permits. Often a sneakernet hop.
Three, validate checksums on the air-gapped side. Models are large; transfers can corrupt. SHA two fifty-six the file on staging, re-hash on the isolated side, compare.
Four, serve with llama.cpp server — preferred, single binary, no runtime dependencies — or a pre-loaded vLLM or Ollama with egress firewalled.
Five, patch on a schedule via the same media. Air-gap does not mean frozen forever. It means updates come through the approved channel, not the live internet. This is the bridge to pillar seven. FT twenty-one and FT twenty-two both build on this recipe.
[SLIDE 7 — Ollama, dev not production]
Now the runtime that causes the most production incidents: Ollama.
For a single developer iterating on a laptop, Ollama is unmatched. Ollama run, and you are chatting. Its privacy policy is direct: we don't see your prompts or data when you run locally. That is true and it matters.
But we don't see your data is not the whole privacy story, and engineers who stop reading there get burned. Three caveats.
One, chat history is stored locally, in plaintext. Ollama run conversations and Modelfile system prompts persist on disk. On a shared laptop or a fleet-managed Mac, that is a data-at-rest concern.
Two, misconfigured OLLAMA_HOST exposes the API. The default bind is loopback, which is correct. But setting it to zero dot zero dot zero dot zero to make it work from another machine exposes an unauthenticated API to the network. There have been multiple incident reports of Ollama instances scraped from the public internet this way.
Three, Oligo found six vulnerabilities. In twenty twenty-five, Oligo Security disclosed six CVEs in Ollama, including a denial of service, a cross-domain token exposure, and an authentication bypass. These are patched in current releases — if you keep Ollama updated. An air-gapped Ollama that never gets patches is an air-gapped Ollama carrying known CVEs.
And then the production ceiling. Ollama collapses at roughly five concurrent users. This is not an opinion; it is a measured, reproducible finding. Response times go from about three seconds at one user to over a minute at five, because Ollama serializes requests and duplicates context per request — no PagedAttention, no real continuous batching. vLLM on the same hardware holds the workload at stable latency. Ollama is a dev tool, not a production server. The moment you have real concurrency, you move to vLLM or TGI. This is the single most expensive serving mistake teams make.
[SLIDE 8 — The Ollama ceiling]
Let me make the ceiling concrete, because the shape is the lesson.
At one concurrent user, Ollama and vLLM are both fine — Ollama maybe three seconds, vLLM maybe one. At three users, Ollama is around eight seconds, vLLM around one. At five users — the ceiling — Ollama is thirty to sixty seconds or more, vLLM around two. At ten users, Ollama is unusable, vLLM around three.
Notice the shape. Ollama's latency explodes somewhere around five concurrent users. vLLM's stays bounded. That shape is the deployment decision. If your traffic is one user, Ollama's DX wins. If your traffic is anything more, vLLM wins, and it is not close.
The root cause is structural, not a bug. Ollama serializes requests and duplicates context per request. vLLM has PagedAttention, which packs sequences via a paged KV cache, and continuous batching, which inserts requests at every token step. You cannot tune your way out of the Ollama ceiling — you switch runtimes. The lab for this module exists to make you feel this ceiling happen on your own machine.
[SLIDE 9 — Apple Silicon and MLX]
If your fleet is Macs — common in healthcare workstations, defense laptops, design studios — the runtime choice is not vLLM. It is MLX.
Apple Silicon has unified memory: the GPU and CPU share the same physical RAM. A model that loads into VRAM on a Mac is really loading into this unified pool. But CUDA-targeted runtimes cannot exploit this — they expect discrete NVIDIA GPUs. Running them on a Mac means falling back to CPU or experimental Metal paths that give up most of the optimizations.
MLX is Apple's answer. An array framework, designed by Apple's machine learning research team, targeting the unified memory and Metal backend natively. mlx-lm is the model-serving layer on top. It is to Apple Silicon what vLLM is to NVIDIA — built for the hardware, not ported to it.
Apple officially embraced local serving with MLX at WWDC twenty twenty-five, session two ninety-eight. The platform vendor's message was explicit: local, on-device, private-by-construction serving is a supported path on macOS, not a hack.
The quantization recommendation lines up with FT nineteen: four-bit is the sweet spot. Roughly seventy-five percent size reduction versus FP sixteen, with minimal measurable quality loss for the model classes you deploy locally. You rarely quantize yourself — you pull pre-quantized models from the mlx-community org on HuggingFace. Performance is the draw: roughly one hundred forty-three tokens per second on a Qwen three VL four B class model on current M-series hardware.
[SLIDE 10 — Federated local serving]
Here is the pattern that makes MLX the right answer for Mac fleets in regulated domains.
For a hospital with five hundred Mac workstations, you do not stand up one central MLX or vLLM server and share it. You run MLX on each workstation. The architecture is federated local serving, not centralized serving.
And this is actually a privacy advantage — there is no central server holding everyone's prompts. The privacy property is structural, not a policy layered on top. There is nothing to breach, nothing to exfiltrate, nothing to subpoena. Each device holds only its own user's data.
The trade-off is operational. Federated local serving is a different model than the vLLM-on-a-GPU-box pattern. Teams used to centralized serving need to adjust. You push model updates to the fleet rather than restarting one central server. You monitor per-device rather than centrally. But for the privacy constraint — patient data cannot leave the device — it is the only architecture that makes the constraint structurally true rather than a policy that can fail.
[SLIDE 11 — Production deployment patterns]
Now the patterns that separate a working dev setup from a service you can page on.
One, bind loopback for local-only. One twenty seven dot zero dot zero dot one, always, by default. The model serves to processes on the same machine. Exposing the raw API to the network is something you do deliberately, with auth, never by accident.
Two, OpenAI-compatible API for app integration. vLLM, llama.cpp, Ollama, TGI, MLX, SGLang all speak the OpenAI chat-completions dialect. Standardize on it. Your application calls slash v one slash chat slash completions with a configurable base URL; you swap that one value between local Ollama in dev and a vLLM cluster in prod with no code change. This is the single highest-leverage integration decision in the serving layer.
Three, concurrency limits and queueing for multi-user. Once you have more than a handful of concurrent users, you need explicit limits: max in-flight requests, a queue with bounded wait, graceful four twenty-nines when the queue fills. vLLM and TGI have these built in; Ollama does not, which is why it collapses. In front of any runtime, put a reverse proxy — nginx, envoy — that enforces per-client rate limits and timeouts.
Four, observability wired to your own collector. OTel for vLLM, Prometheus for TGI, whatever your org standardizes on. The point is: the metrics come to you, you do not phone home to a vendor dashboard. Latency percentiles, tokens per second, queue depth, error rate. If you cannot see the latency tail, you cannot tune the concurrency limit, and you will set it wrong.
[SLIDE 12 — Anti-patterns and the synthesis]
Three anti-patterns to leave with.
First, using Ollama for production multi-user. The cardinal serving mistake. Ollama collapses at five concurrent users. It is a dev tool. The moment you have real concurrency, move to vLLM or TGI.
Second, exposing OLLAMA_HOST to zero dot zero dot zero dot zero without auth. The default loopback bind is correct. Setting it broadly exposes an unauthenticated API, and the Oligo CVEs make raw exposure actively dangerous. Never bind Ollama broadly without an auth layer in front.
Third, not benchmarking under realistic load. Ollama is fast on my laptop with one user is not a production signal. You must benchmark under your realistic concurrent load before declaring any runtime production-ready. vLLM's continuous-batching advantage only shows up under load; Ollama's collapse only shows up under load. Single-request latency tells you nothing about either.
This module is the bridge to pillar seven. The no-telemetry rule of thumb, the air-gap recipe, the federated local serving pattern — these are the technical substrate that HIPAA in FT twenty-one and government air-gap in FT twenty-two build on. If you cannot execute the air-gap recipe cleanly, you are not ready for a regulated deploy.
[SLIDE 13 — What you can now do]
You can now place each major runtime on the decision matrix and justify a pick for a given target. You can state the no-telemetry rule of thumb and the air-gap recipe. You can predict when Ollama collapses and route that workload to vLLM. And you can configure a production deploy — loopback bind, OpenAI-compatible API, concurrency limits, observability to your own collector.
Next, module FT twenty-one: HIPAA and BAA elimination. We take the air-gap recipe from this module and apply it to the specific compliance regime that gates most healthcare deploys. Let's go.
End of module FT20. Duration: approximately forty minutes at one-hundred-forty words per minute.
# Teaching Script — Module FT20: Serving Stacks **Course**: Course 3 — LLM Fine-Tuning Masterclass **Module**: FT20 — Serving Stacks **Duration**: ~40 minutes (spoken at ~140 wpm) **Format**: Verbatim transcript with `[SLIDE N]` cues. Read aloud or use as speaker notes. --- [SLIDE 1 — Title] Welcome to module FT20, Serving Stacks. This is pillar six, the deploy pillar, and the second half of the export layer. You arrive here from FT19 with a quantized model in your hand — a GGUF, an AWQ, an MLX artifact. The question this module answers is: what do you run it with, and what are the rules when the deployment is sensitive? There is a recurring mistake this module exists to prevent. Teams pick Ollama because it has the best developer experience, they never benchmark under load, and then they ship it to production and it falls over at five concurrent users. By the end of this module you will know exactly when each runtime is the right answer and when it is the wrong one — and you will know the no-telemetry rules that make a runtime acceptable in a hospital subnet or a classified air-gap. [SLIDE 2 — The three variables] Every serving decision collapses to three variables, asked in this order. First, audience. One developer on a laptop? A small team behind a VPN? Hundreds of concurrent end users hitting an HTTP endpoint? The audience determines whether you need a runtime — one process, one user — or a server — concurrent, queued, observable. Second, hardware. NVIDIA CUDA? Apple Silicon? A mix? CPU-only edge boxes? ROCm? Hardware determines which runtimes are even options. And on Apple Silicon it determines whether you use a CUDA-targeted stack or the native MLX path. Third, telemetry posture. Open network? Regulated subnet? True air-gap? This is the variable most engineers underweight, and it is the one this module spends the most time on, because pillar seven builds directly on it. Hold those three in your head and the rest of the module is just routing. [SLIDE 3 — The decision matrix] Here is the routing function. There is no best runtime — there is the runtime your three variables point at. NVIDIA production, open subnet, many concurrent users — that is vLLM. It is the production standard on CUDA. Air-gap or regulated subnet, any hardware — that is llama.cpp server. Single binary, no network dependencies, maximum hardware breadth. Dev or single user, any hardware — that is Ollama. Best DX in the field, but it collapses under concurrency, which we will get to. Mac fleet, Apple Silicon — that is mlx-lm. Native to the unified memory, the Mac-native path. The rest of the field — TGI, localai, SGLang, TensorRT-LLM — fills in around these four. TGI is HuggingFace's production server with the best out-of-box observability. SGLang uses RadixAttention for prefix caching and wins on agentic workloads with shared system prompts. TensorRT-LLM is NVIDIA's own squeeze-the-last-millisecond runtime for flagship silicon. Read the matrix as a routing function, not a ranking. [SLIDE 4 — vLLM, the production standard] Let's talk about why vLLM is the production standard on NVIDIA. Three engineering choices. First, PagedAttention. The key-value cache is stored in paged blocks — like an operating system's virtual memory — not a contiguous tensor. A naive contiguous cache pre-allocates a max-length block per sequence; because sequences vary in length, sixty to eighty percent of that allocation is wasted to fragmentation. PagedAttention eliminates the fragmentation, and the reclaimed memory holds many more concurrent sequences. Second, continuous batching. Naive batching waits for the longest sequence in a batch to finish before starting the next batch. Continuous batching inserts new requests into running batches at every token step, so the GPUs stay saturated and tail latency stays bounded. The throughput difference is often five to ten times under realistic load. Third, multi-GPU tensor parallelism. A model too big for one GPU is split across several — each holds a slice of every layer, synchronizing at each attention block. It is a flag, not a recompile. And the API is OpenAI-compatible. Slash v one slash chat slash completions. Any client written for OpenAI works against a local vLLM server by changing one base URL. That is why it slots into existing agent stacks without code changes. The honest limitation: vLLM's Apple Silicon and CPU support is limited. It is built around CUDA. If your fleet is Macs, you want MLX. [SLIDE 5 — The telemetry posture] Now the variable most engineers underweight. Telemetry posture. Here is the rule of thumb. llama.cpp and vLLM genuinely never need to phone home. Ollama needs the network only for ollama pull; once models are local, block the network and bind OLLAMA_HOST to one twenty seven dot zero dot zero dot one. For a true air-gap, pre-load the models, then sever the network. That is the whole rule. Let me explain why each runtime lands where it does. llama.cpp is a single statically-linked binary. There is no network code path for telemetry. Nothing to disable, no flag to flip, no call-home server to reach. The binary reads a local file and serves. You can prove this with a packet capture. It is the air-gap champion by construction. vLLM is slightly more nuanced but lands in the same place. It is OpenTelemetry-native, which means it emits traces and metrics — but it emits to whatever endpoint you configure via the standard OTel environment variables. Point them at your own collector running on loopback, or leave them unset and the traces are no-ops. There is no vendor phone-home, no license check. The egress story is entirely yours. Ollama needs the network for exactly one thing in normal operation: ollama pull, which fetches models. Once a model is local, Ollama no longer needs any network to serve it. The recipe for a restricted-network deploy is: pre-load on a connected machine, copy the blob store, bind loopback, sever the network. Ollama will serve from local blobs indefinitely. [SLIDE 6 — The air-gap recipe] For a true air-gap — no network egress at all, ever — the recipe is the same regardless of runtime. One, pre-load models on a staging machine that does have network. Pull from HuggingFace, from the Ollama registry, wherever your artifacts live. Two, transfer via approved media — approved USB, one-way data diode, whatever your security office permits. Often a sneakernet hop. Three, validate checksums on the air-gapped side. Models are large; transfers can corrupt. SHA two fifty-six the file on staging, re-hash on the isolated side, compare. Four, serve with llama.cpp server — preferred, single binary, no runtime dependencies — or a pre-loaded vLLM or Ollama with egress firewalled. Five, patch on a schedule via the same media. Air-gap does not mean frozen forever. It means updates come through the approved channel, not the live internet. This is the bridge to pillar seven. FT twenty-one and FT twenty-two both build on this recipe. [SLIDE 7 — Ollama, dev not production] Now the runtime that causes the most production incidents: Ollama. For a single developer iterating on a laptop, Ollama is unmatched. Ollama run, and you are chatting. Its privacy policy is direct: we don't see your prompts or data when you run locally. That is true and it matters. But we don't see your data is not the whole privacy story, and engineers who stop reading there get burned. Three caveats. One, chat history is stored locally, in plaintext. Ollama run conversations and Modelfile system prompts persist on disk. On a shared laptop or a fleet-managed Mac, that is a data-at-rest concern. Two, misconfigured OLLAMA_HOST exposes the API. The default bind is loopback, which is correct. But setting it to zero dot zero dot zero dot zero to make it work from another machine exposes an unauthenticated API to the network. There have been multiple incident reports of Ollama instances scraped from the public internet this way. Three, Oligo found six vulnerabilities. In twenty twenty-five, Oligo Security disclosed six CVEs in Ollama, including a denial of service, a cross-domain token exposure, and an authentication bypass. These are patched in current releases — if you keep Ollama updated. An air-gapped Ollama that never gets patches is an air-gapped Ollama carrying known CVEs. And then the production ceiling. Ollama collapses at roughly five concurrent users. This is not an opinion; it is a measured, reproducible finding. Response times go from about three seconds at one user to over a minute at five, because Ollama serializes requests and duplicates context per request — no PagedAttention, no real continuous batching. vLLM on the same hardware holds the workload at stable latency. Ollama is a dev tool, not a production server. The moment you have real concurrency, you move to vLLM or TGI. This is the single most expensive serving mistake teams make. [SLIDE 8 — The Ollama ceiling] Let me make the ceiling concrete, because the shape is the lesson. At one concurrent user, Ollama and vLLM are both fine — Ollama maybe three seconds, vLLM maybe one. At three users, Ollama is around eight seconds, vLLM around one. At five users — the ceiling — Ollama is thirty to sixty seconds or more, vLLM around two. At ten users, Ollama is unusable, vLLM around three. Notice the shape. Ollama's latency explodes somewhere around five concurrent users. vLLM's stays bounded. That shape is the deployment decision. If your traffic is one user, Ollama's DX wins. If your traffic is anything more, vLLM wins, and it is not close. The root cause is structural, not a bug. Ollama serializes requests and duplicates context per request. vLLM has PagedAttention, which packs sequences via a paged KV cache, and continuous batching, which inserts requests at every token step. You cannot tune your way out of the Ollama ceiling — you switch runtimes. The lab for this module exists to make you feel this ceiling happen on your own machine. [SLIDE 9 — Apple Silicon and MLX] If your fleet is Macs — common in healthcare workstations, defense laptops, design studios — the runtime choice is not vLLM. It is MLX. Apple Silicon has unified memory: the GPU and CPU share the same physical RAM. A model that loads into VRAM on a Mac is really loading into this unified pool. But CUDA-targeted runtimes cannot exploit this — they expect discrete NVIDIA GPUs. Running them on a Mac means falling back to CPU or experimental Metal paths that give up most of the optimizations. MLX is Apple's answer. An array framework, designed by Apple's machine learning research team, targeting the unified memory and Metal backend natively. mlx-lm is the model-serving layer on top. It is to Apple Silicon what vLLM is to NVIDIA — built for the hardware, not ported to it. Apple officially embraced local serving with MLX at WWDC twenty twenty-five, session two ninety-eight. The platform vendor's message was explicit: local, on-device, private-by-construction serving is a supported path on macOS, not a hack. The quantization recommendation lines up with FT nineteen: four-bit is the sweet spot. Roughly seventy-five percent size reduction versus FP sixteen, with minimal measurable quality loss for the model classes you deploy locally. You rarely quantize yourself — you pull pre-quantized models from the mlx-community org on HuggingFace. Performance is the draw: roughly one hundred forty-three tokens per second on a Qwen three VL four B class model on current M-series hardware. [SLIDE 10 — Federated local serving] Here is the pattern that makes MLX the right answer for Mac fleets in regulated domains. For a hospital with five hundred Mac workstations, you do not stand up one central MLX or vLLM server and share it. You run MLX on each workstation. The architecture is federated local serving, not centralized serving. And this is actually a privacy advantage — there is no central server holding everyone's prompts. The privacy property is structural, not a policy layered on top. There is nothing to breach, nothing to exfiltrate, nothing to subpoena. Each device holds only its own user's data. The trade-off is operational. Federated local serving is a different model than the vLLM-on-a-GPU-box pattern. Teams used to centralized serving need to adjust. You push model updates to the fleet rather than restarting one central server. You monitor per-device rather than centrally. But for the privacy constraint — patient data cannot leave the device — it is the only architecture that makes the constraint structurally true rather than a policy that can fail. [SLIDE 11 — Production deployment patterns] Now the patterns that separate a working dev setup from a service you can page on. One, bind loopback for local-only. One twenty seven dot zero dot zero dot one, always, by default. The model serves to processes on the same machine. Exposing the raw API to the network is something you do deliberately, with auth, never by accident. Two, OpenAI-compatible API for app integration. vLLM, llama.cpp, Ollama, TGI, MLX, SGLang all speak the OpenAI chat-completions dialect. Standardize on it. Your application calls slash v one slash chat slash completions with a configurable base URL; you swap that one value between local Ollama in dev and a vLLM cluster in prod with no code change. This is the single highest-leverage integration decision in the serving layer. Three, concurrency limits and queueing for multi-user. Once you have more than a handful of concurrent users, you need explicit limits: max in-flight requests, a queue with bounded wait, graceful four twenty-nines when the queue fills. vLLM and TGI have these built in; Ollama does not, which is why it collapses. In front of any runtime, put a reverse proxy — nginx, envoy — that enforces per-client rate limits and timeouts. Four, observability wired to your own collector. OTel for vLLM, Prometheus for TGI, whatever your org standardizes on. The point is: the metrics come to you, you do not phone home to a vendor dashboard. Latency percentiles, tokens per second, queue depth, error rate. If you cannot see the latency tail, you cannot tune the concurrency limit, and you will set it wrong. [SLIDE 12 — Anti-patterns and the synthesis] Three anti-patterns to leave with. First, using Ollama for production multi-user. The cardinal serving mistake. Ollama collapses at five concurrent users. It is a dev tool. The moment you have real concurrency, move to vLLM or TGI. Second, exposing OLLAMA_HOST to zero dot zero dot zero dot zero without auth. The default loopback bind is correct. Setting it broadly exposes an unauthenticated API, and the Oligo CVEs make raw exposure actively dangerous. Never bind Ollama broadly without an auth layer in front. Third, not benchmarking under realistic load. Ollama is fast on my laptop with one user is not a production signal. You must benchmark under your realistic concurrent load before declaring any runtime production-ready. vLLM's continuous-batching advantage only shows up under load; Ollama's collapse only shows up under load. Single-request latency tells you nothing about either. This module is the bridge to pillar seven. The no-telemetry rule of thumb, the air-gap recipe, the federated local serving pattern — these are the technical substrate that HIPAA in FT twenty-one and government air-gap in FT twenty-two build on. If you cannot execute the air-gap recipe cleanly, you are not ready for a regulated deploy. [SLIDE 13 — What you can now do] You can now place each major runtime on the decision matrix and justify a pick for a given target. You can state the no-telemetry rule of thumb and the air-gap recipe. You can predict when Ollama collapses and route that workload to vLLM. And you can configure a production deploy — loopback bind, OpenAI-compatible API, concurrency limits, observability to your own collector. Next, module FT twenty-one: HIPAA and BAA elimination. We take the air-gap recipe from this module and apply it to the specific compliance regime that gates most healthcare deploys. Let's go. --- *End of module FT20. Duration: approximately forty minutes at one-hundred-forty words per minute.*