Can I run models myself?
Open weights, the local stack, serving, and the hardware math for what fits on your machines.
The path.
- Open-Weight Models The models you can download, self-host, and fine-tune. As of July 2026. Two things to know: (1) "open weight" ≠ open source.
- Local & Self-Hosting Stack Running models on your own machine or servers.
- Inference, Serving & Optimization TTFT (Time To First Token). Latency until the first token appears. Dominated by the prefill (prompt-processing) phase and prompt length.
- AI Hardware & Accelerators The 2026 picture is that NVIDIA still dominates training and general inference.
Check yourself.
One question per step. Take it cold to find where to start, or after reading to see what stuck. Nobody's grading you.
Best so far:
Question 1 of 4
-
A MoE model computes only a fraction of its parameters per token. What does that save you at inference?
MoE models load all experts into memory but only run the active set, so they're fast but still memory-hungry. VRAM needs follow total parameters, roughly params × bytes-per-param.
Covered in step 1 · Open-Weight Models -
When does self-hosting models lose on cost?
Self-hosting wins at high volume thanks to zero marginal cost, but loses cost-efficiency when traffic is low or spiky, because you pay for the GPU whether or not it's busy.
Covered in step 2 · Local & Self-Hosting Stack -
You're serving an open model on your own GPUs. What's the single biggest throughput win in modern serving?
Continuous (in-flight) batching adds and removes requests every step as sequences finish, keeping the GPU saturated. The chapter calls it the single biggest throughput win in modern serving.
Covered in step 3 · Inference, Serving & Optimization -
How should you compare GPUs when buying or renting for inference?
Cost per token is set by throughput, not sticker price: a pricier GPU that serves 3× the tokens comes out cheaper per token, which is why quantization, batching, and the serving engine move the economics most.
Covered in step 4 · AI Hardware & Accelerators