Lab Specification — Module FT20: Serving Stacks

# Clean venv
python3.11 -m venv ft20-env && source ft20-env/bin/activate

# Serving + load-testing stack
pip install -q vllm "openai>=1.0" httpx
# (Ollama is installed as a system binary, not a pip package — see Phase 1)

# For the load test later
pip install -q "numpy" "rich"

import torch
print(f"PyTorch: {torch.__version__}")
print(f"CUDA available: {torch.cuda.is_available()}")   # expect True on NVIDIA
print(f"MPS available: {torch.backends.mps.is_available()}")  # Apple Silicon

# Pull a model (uses the network for THIS step only)
ollama pull qwen2.5:1.5b

# CRITICAL: bind loopback. Never 0.0.0.0 without auth.
# The default is 127.0.0.1; verify:
echo $OLLAMA_HOST    # should be empty (default 127.0.0.1:11434) or explicitly 127.0.0.1:11434

# Confirm it is listening loopback only:
curl -s http://127.0.0.1:11434/api/tags | head -c 200

# ft20_smoke_ollama.py
from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:11434/v1", api_key="ollama")  # api_key is required by the client but ignored by Ollama

def gen(prompt, max_tokens=100):
    r = client.chat.completions.create(
        model="qwen2.5:1.5b",
        messages=[{"role": "user", "content": prompt}],
        max_tokens=max_tokens,
        temperature=0.0,   # deterministic for eval equivalence
    )
    return r.choices[0].message.content

print(gen("In one sentence, what is PagedAttention?"))

# ft20_eval_suite.py
EVAL_PROMPTS = [
    "In one sentence, what is PagedAttention?",
    "List the three variables in the serving-stack decision matrix.",
    "What does Ollama's official privacy policy say about local prompts?",
    "Name two reasons llama.cpp server is the air-gap champion.",
    "At roughly how many concurrent users does Ollama collapse, and why?",
]

def run_suite(gen_fn, label):
    """gen_fn: str -> str. Returns dict of prompt -> response."""
    print(f"\n=== {label} ===")
    results = {}
    for i, p in enumerate(EVAL_PROMPTS):
        resp = gen_fn(p)
        results[p] = resp
        print(f"[{i+1}] {p}\n    -> {resp[:160]}\n")
    return results

# vLLM serves an OpenAI-compatible API on 127.0.0.1:8000 by default
python -m vllm.entrypoints.openai.api_server \
  --model Qwen/Qwen2.5-1.5B-Instruct \
  --host 127.0.0.1 \
  --port 8000 \
  --max-model-len 2048

# ft20_smoke_vllm.py
from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="vllm")  # api_key ignored by vLLM by default

def gen(prompt, max_tokens=100):
    r = client.chat.completions.create(
        model="Qwen/Qwen2.5-1.5B-Instruct",
        messages=[{"role": "user", "content": prompt}],
        max_tokens=max_tokens,
        temperature=0.0,
    )
    return r.choices[0].message.content

print(gen("In one sentence, what is PagedAttention?"))

# ft20_compare.py
ollama_results = {...}   # paste / load from Phase 2
vllm_results = run_suite(gen, "vLLM")  # uses Phase 3's gen

print("\n=== OUTPUT EQUIVALENCE CHECK ===")
for p in EVAL_PROMPTS:
    same = ollama_results[p].strip() == vllm_results[p].strip()
    # Note: with temperature=0, the two servers SHOULD produce near-identical
    # outputs for the same base model, but small differences (tokenization,
    # sampling impl) can cause minor divergence. Accept "semantically same".
    print(f"[{'OK' if same else '~'}] {p[:50]}")

# ft20_load_test.py
import time, statistics, concurrent.futures
from openai import OpenAI

def make_client(base_url):
    return OpenAI(base_url=base_url, api_key="x", timeout=120.0)

def one_request(client, model, prompt="In two sentences, explain continuous batching."):
    t0 = time.perf_counter()
    client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": prompt}],
        max_tokens=80,
        temperature=0.0,
    )
    return time.perf_counter() - t0

def load_test(base_url, model, concurrency, n=20):
    """Fire n requests at the given concurrency; return p50, p95 latencies in seconds."""
    client = make_client(base_url)
    latencies = []
    with concurrent.futures.ThreadPoolExecutor(max_workers=concurrency) as pool:
        futs = [pool.submit(one_request, client, model) for _ in range(n)]
        for f in concurrent.futures.as_completed(futs):
            try:
                latencies.append(f.result())
            except Exception as e:
                latencies.append(float("inf"))   # timeout / error counts as very slow
    latencies.sort()
    p50 = statistics.median(latencies)
    p95 = latencies[int(0.95 * len(latencies)) - 1] if len(latencies) > 1 else latencies[0]
    return p50, p95, sum(1 for l in latencies if l == float("inf"))  # errors

# Run both servers, one at a time, at each concurrency level.
CONFIGS = [
    ("Ollama", "http://127.0.0.1:11434/v1", "qwen2.5:1.5b"),
    ("vLLM",   "http://127.0.0.1:8000/v1",  "Qwen/Qwen2.5-1.5B-Instruct"),
]

print(f"{'Server':<8} {'Conc':>4} {'p50(s)':>8} {'p95(s)':>8} {'errors':>7}")
print("-" * 40)
for label, base_url, model in CONFIGS:
    for conc in [1, 3, 5, 10]:
        # IMPORTANT: only one server should be under load at a time.
        # Stop the other (ollama stop / Ctrl-C the vllm server) before each row,
        # or run on a machine with enough VRAM to hold both cold.
        p50, p95, errs = load_test(base_url, model, conc, n=20)
        print(f"{label:<8} {conc:>4} {p50:>8.2f} {p95:>8.2f} {errs:>7}")

Server   Conc    p50(s)    p95(s)  errors
----------------------------------------
Ollama      1      0.8       1.1       0
Ollama      3      2.4       3.8       0
Ollama      5      8.5      18.2       0   <-- THE CEILING
Ollama     10     35.0      60.0+      2
vLLM        1      0.4       0.6       0
vLLM        3      0.5       0.8       0
vLLM        5      0.7       1.2       0
vLLM       10      1.1       2.0       0

# Lab Specification — Module FT20: Serving Stacks

**Course**: Course 3 — LLM Fine-Tuning Masterclass
**Module**: FT20 — Serving Stacks
**Duration**: 45–60 minutes
**Environment**: A consumer NVIDIA GPU (RTX 3090/4090 / 16–24GB) OR Apple Silicon (M-series, 16GB+) OR a Colab T4/A100. You must arrive with a quantized model from FT19 (GGUF for Ollama; the same base in FP16/AWQ for vLLM). Python 3.11+. ~10GB free disk.

---

## Learning objectives

By the end of this lab you will have:

1. **Served your FT19 quantized model two ways** — Ollama (dev/single-user path) and vLLM (production path) — both exposing the same OpenAI-compatible API.
2. **Hit both servers with the same eval suite** (identical prompts, identical generation settings) and confirmed output equivalence within sampling noise.
3. **Run a concurrent load test** (1, 3, 5, 10 simulated users) against *both* servers and recorded p50/p95 latency at each concurrency level.
4. **Felt the Ollama ceiling** — the moment around 5 concurrent users where Ollama's latency explodes while vLLM's stays bounded — and written the deployment decision in your own words.
5. **Captured the serving telemetry posture** of each runtime (what reaches the network, what does not) so you can defend the choice for a regulated subnet.

This lab is the one that makes the Ollama-vs-vLLM decision *felt*, not theoretical. You will watch the ceiling happen on your own machine.

---

## Phase 0 — Environment and model prep (5 min)

You need *one model* in two formats. The simplest path: use a well-known small base that has both a GGUF (for Ollama) and a standard HF checkpoint (for vLLM). This lab uses `Qwen/Qwen2.5-1.5B-Instruct` — small enough to run on anything, including a free Colab T4. Substitute your FT19 artifact if you have one.

```bash
# Clean venv
python3.11 -m venv ft20-env && source ft20-env/bin/activate

# Serving + load-testing stack
pip install -q vllm "openai>=1.0" httpx
# (Ollama is installed as a system binary, not a pip package — see Phase 1)

# For the load test later
pip install -q "numpy" "rich"
```

Verify vLLM sees your accelerator:

```python
import torch
print(f"PyTorch: {torch.__version__}")
print(f"CUDA available: {torch.cuda.is_available()}")   # expect True on NVIDIA
print(f"MPS available: {torch.backends.mps.is_available()}")  # Apple Silicon
```

> **vLLM on Apple Silicon note.** vLLM's CUDA-targeted path is the supported one. On a Mac, the vLLM phase of this lab runs in CPU mode (slow but correct) or you substitute `mlx_lm.server` for the production path — the load-test script is identical because both speak the OpenAI API. If you are Mac-only, do Ollama for Phase 1, then swap vLLM for `python -m mlx_lm.server --model mlx-community/Qwen2.5-1.5B-Instruct-4bit` in Phase 3. The lab's lesson is the same.

---

## Phase 1 — Serve with Ollama (the dev path) (10 min)

Install Ollama per the official instructions (`curl -fsSL https://ollama.com/install.sh | sh` on Linux/macOS). Then pull a small model and confirm the loopback bind:

```bash
# Pull a model (uses the network for THIS step only)
ollama pull qwen2.5:1.5b

# CRITICAL: bind loopback. Never 0.0.0.0 without auth.
# The default is 127.0.0.1; verify:
echo $OLLAMA_HOST    # should be empty (default 127.0.0.1:11434) or explicitly 127.0.0.1:11434

# Confirm it is listening loopback only:
curl -s http://127.0.0.1:11434/api/tags | head -c 200
```

**Record**: the `OLLAMA_HOST` value and the confirmation that the API responds on loopback. This is the telemetry-posture checkpoint for Ollama — the model is now local, the network can be severed, the bind is correct.

Smoke-test a single generation via the OpenAI-compatible endpoint:

```python
# ft20_smoke_ollama.py
from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:11434/v1", api_key="ollama")  # api_key is required by the client but ignored by Ollama

def gen(prompt, max_tokens=100):
    r = client.chat.completions.create(
        model="qwen2.5:1.5b",
        messages=[{"role": "user", "content": prompt}],
        max_tokens=max_tokens,
        temperature=0.0,   # deterministic for eval equivalence
    )
    return r.choices[0].message.content

print(gen("In one sentence, what is PagedAttention?"))
```

**Record**: the response. Save it — you will compare it to vLLM's response to the same prompt in Phase 3.

---

## Phase 2 — The eval suite (5 min)

A tiny, deterministic eval suite (5 prompts). Both servers will answer the identical suite so you can compare outputs.

```python
# ft20_eval_suite.py
EVAL_PROMPTS = [
    "In one sentence, what is PagedAttention?",
    "List the three variables in the serving-stack decision matrix.",
    "What does Ollama's official privacy policy say about local prompts?",
    "Name two reasons llama.cpp server is the air-gap champion.",
    "At roughly how many concurrent users does Ollama collapse, and why?",
]

def run_suite(gen_fn, label):
    """gen_fn: str -> str. Returns dict of prompt -> response."""
    print(f"\n=== {label} ===")
    results = {}
    for i, p in enumerate(EVAL_PROMPTS):
        resp = gen_fn(p)
        results[p] = resp
        print(f"[{i+1}] {p}\n    -> {resp[:160]}\n")
    return results
```

Run the suite against Ollama now (using the `gen` function from Phase 1) and save the dict. These are your **baseline outputs**.

---

## Phase 3 — Serve with vLLM (the production path) and re-run the suite (10 min)

Stop Ollama for a moment to free VRAM (`ollama stop qwen2.5:1.5b`, or just leave it idle — it unloads on its own). Start vLLM:

```bash
# vLLM serves an OpenAI-compatible API on 127.0.0.1:8000 by default
python -m vllm.entrypoints.openai.api_server \
  --model Qwen/Qwen2.5-1.5B-Instruct \
  --host 127.0.0.1 \
  --port 8000 \
  --max-model-len 2048
```

Wait for the line `Uvicorn running on http://127.0.0.1:8000`. Then point the same OpenAI client at it:

```python
# ft20_smoke_vllm.py
from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="vllm")  # api_key ignored by vLLM by default

def gen(prompt, max_tokens=100):
    r = client.chat.completions.create(
        model="Qwen/Qwen2.5-1.5B-Instruct",
        messages=[{"role": "user", "content": prompt}],
        max_tokens=max_tokens,
        temperature=0.0,
    )
    return r.choices[0].message.content

print(gen("In one sentence, what is PagedAttention?"))
```

Now re-run the eval suite from Phase 2 against vLLM, and **diff** the two output dicts:

```python
# ft20_compare.py
ollama_results = {...}   # paste / load from Phase 2
vllm_results = run_suite(gen, "vLLM")  # uses Phase 3's gen

print("\n=== OUTPUT EQUIVALENCE CHECK ===")
for p in EVAL_PROMPTS:
    same = ollama_results[p].strip() == vllm_results[p].strip()
    # Note: with temperature=0, the two servers SHOULD produce near-identical
    # outputs for the same base model, but small differences (tokenization,
    # sampling impl) can cause minor divergence. Accept "semantically same".
    print(f"[{'OK' if same else '~'}] {p[:50]}")
```

**Record**: whether the outputs are equivalent. Expect near-identical (minor divergence is normal and fine — the point is that the two servers serve *the same model*, so the deployment choice is about latency/concurrency/telemetry, not output quality).

---

## Phase 4 — The load test: 1, 3, 5, 10 concurrent users (15 min)

This is the heart of the lab. You will fire the same workload at each server with increasing concurrency and record p50/p95 latency. Restart Ollama's model load (`ollama run qwen2.5:1.5b ""` to warm it), then run:

```python
# ft20_load_test.py
import time, statistics, concurrent.futures
from openai import OpenAI

def make_client(base_url):
    return OpenAI(base_url=base_url, api_key="x", timeout=120.0)

def one_request(client, model, prompt="In two sentences, explain continuous batching."):
    t0 = time.perf_counter()
    client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": prompt}],
        max_tokens=80,
        temperature=0.0,
    )
    return time.perf_counter() - t0

def load_test(base_url, model, concurrency, n=20):
    """Fire n requests at the given concurrency; return p50, p95 latencies in seconds."""
    client = make_client(base_url)
    latencies = []
    with concurrent.futures.ThreadPoolExecutor(max_workers=concurrency) as pool:
        futs = [pool.submit(one_request, client, model) for _ in range(n)]
        for f in concurrent.futures.as_completed(futs):
            try:
                latencies.append(f.result())
            except Exception as e:
                latencies.append(float("inf"))   # timeout / error counts as very slow
    latencies.sort()
    p50 = statistics.median(latencies)
    p95 = latencies[int(0.95 * len(latencies)) - 1] if len(latencies) > 1 else latencies[0]
    return p50, p95, sum(1 for l in latencies if l == float("inf"))  # errors

# Run both servers, one at a time, at each concurrency level.
CONFIGS = [
    ("Ollama", "http://127.0.0.1:11434/v1", "qwen2.5:1.5b"),
    ("vLLM",   "http://127.0.0.1:8000/v1",  "Qwen/Qwen2.5-1.5B-Instruct"),
]

print(f"{'Server':<8} {'Conc':>4} {'p50(s)':>8} {'p95(s)':>8} {'errors':>7}")
print("-" * 40)
for label, base_url, model in CONFIGS:
    for conc in [1, 3, 5, 10]:
        # IMPORTANT: only one server should be under load at a time.
        # Stop the other (ollama stop / Ctrl-C the vllm server) before each row,
        # or run on a machine with enough VRAM to hold both cold.
        p50, p95, errs = load_test(base_url, model, conc, n=20)
        print(f"{label:<8} {conc:>4} {p50:>8.2f} {p95:>8.2f} {errs:>7}")
```

> **Running both at once.** A 1.5B model is small; on a 16GB+ GPU you can usually hold both servers resident at once. If VRAM is tight, stop one before testing the other (the load-test script's `CONFIGS` loop is meant to be run one server at a time — comment out the row you are not currently running). The latency comparison is only fair if the servers are not contending for the same GPU.

**Record**: the full table. You are looking for the inflection. Expect something like:

```
Server   Conc    p50(s)    p95(s)  errors
----------------------------------------
Ollama      1      0.8       1.1       0
Ollama      3      2.4       3.8       0
Ollama      5      8.5      18.2       0   <-- THE CEILING
Ollama     10     35.0      60.0+      2
vLLM        1      0.4       0.6       0
vLLM        3      0.5       0.8       0
vLLM        5      0.7       1.2       0
vLLM       10      1.1       2.0       0
```

Your exact numbers will vary with hardware, but the *shape* is what matters: Ollama's p95 blows up somewhere around 5 concurrent users; vLLM's stays bounded. **That shape is the deployment decision.**

---

## Phase 5 — The decision, in your own words (5 min)

No code. Write 4–6 sentences answering:

1. At what concurrency did Ollama's p95 latency exceed 2x its 1-user latency? At what concurrency did it exceed 10x? Where is *your* ceiling on *your* hardware?
2. At the same concurrency levels, how did vLLM's latency behave? Was there an inflection, and if so, where?
3. For a single-user dev workflow, which server would you use, and why? (Hint: DX matters; latency at N=1 matters less than the install friction.)
4. For a 20-person internal tool with bursty concurrent use, which server would you use, and why? What would you put in front of it?
5. If this deploy had to run on a hospital subnet with no internet egress, what would change about your setup, for *each* runtime?

---

## Deliverables

Submit `ft20-lab-report.md`:

- [ ] Phase 1: the `OLLAMA_HOST` value; confirmation of loopback bind; the Ollama smoke-test response.
- [ ] Phase 2: the eval-suite output dict from Ollama (the baseline).
- [ ] Phase 3: the eval-suite output dict from vLLM; the equivalence check (OK / ~ per prompt).
- [ ] Phase 4: the full load-test table (both servers, all four concurrency levels).
- [ ] Phase 5: your 4–6 sentence deployment decision, including the per-runtime air-gap changes.

---

## Solution key

- **Phase 1**: `OLLAMA_HOST` is empty or `127.0.0.1:11434`. The curl to `127.0.0.1:11434/api/tags` returns JSON listing the pulled model. A correct smoke-test produces a one-sentence explanation of PagedAttention (quality depends on the 1.5B model; expect a plausible-but-shallow answer — the point is the *request worked*, not the answer's depth).
- **Phase 2**: the suite returns 5 responses. Save them verbatim for the Phase 3 diff.
- **Phase 3**: vLLM starts and prints `Uvicorn running on http://127.0.0.1:8000`. The equivalence check shows `OK` or `~` for all 5 prompts. Minor divergence (a different word, a slightly different sentence boundary) is expected and acceptable with temperature=0 — the two servers run the same model weights, so semantic equivalence is the bar, not byte-identity.
- **Phase 4**: the table shows Ollama's p95 climbing steeply between 3 and 5 concurrent users, often exceeding 10x its 1-user p95 by 5–10 users. vLLM's p95 stays within ~3x of its 1-user value across all four levels. On a 1.5B model on a 4090, expect Ollama's 5-user p95 in the 5–20s range and vLLM's 5-user p95 under 2s. If a student sees Ollama holding flat, they are either (a) not actually running requests concurrently — check that `ThreadPoolExecutor` is sized to `concurrency`, (b) running the servers against each other so both are starved, or (c) on hardware so fast the ceiling is past 10 — bump the concurrency array to `[1, 5, 10, 20, 40]`.
- **Phase 5**: a correct answer names (a) Ollama for dev/single-user (best DX, fine at N=1); (b) vLLM (or TGI) for the 20-person internal tool, with a reverse proxy (nginx/envoy) in front enforcing rate limits and a queue; (c) for the air-gap: pre-load the GGUF on a connected machine, transfer via approved media, SHA256-verify, serve with llama.cpp server (preferred for air-gap) or a pre-loaded vLLM/Ollama with egress firewalled and `OLLAMA_HOST=127.0.0.1`. The honest note that Ollama air-gap means accepting its CVE history unless patched on a schedule via the same media is a strong answer.

---

## Stretch goals

1. **Substitute SGLang for vLLM.** Install SGLang (`pip install sglang`), serve the same model with `python -m sglang.launch_server --model-path Qwen/Qwen2.5-1.5B-Instruct --port 8000`, and re-run the load test. SGLang's RadixAttention gives it an edge on workloads with shared prefixes (long shared system prompts) — compare its p95 to vLLM's at 10 concurrent users. If your eval prompts share a long system prompt, you may see SGLang win; if they are all unique, vLLM and SGLang are close.
2. **Add the MLX path (Apple Silicon).** On a Mac, serve `mlx-community/Qwen2.5-1.5B-Instruct-4bit` via `python -m mlx_lm.server --port 8000`, run the eval suite against it, and add an MLX row to your load-test table. Expect single-user latency competitive with vLLM and a ceiling somewhere between Ollama and vLLM. This is the Mac-fleet path from 20.3.
3. **Wire up OTel for vLLM.** Run a local OTel collector (`docker run -p 4317:4317 otel/opentelemetry-collector`), set `OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4317`, restart vLLM, run the load test, and confirm traces arrive at your collector — *not* at any vendor. This is the telemetry-posture proof that vLLM is air-gap-acceptable.
4. **Prove the air-gap recipe.** Pre-load `qwen2.5:1.5b`, copy `~/.ollama/models` to a second machine, set `OLLAMA_HOST=127.0.0.1:11434`, disconnect the network entirely, and confirm Ollama serves. Then do the same with a local GGUF and `llama-server`. This is the bridge to FT21/FT22.