AI Hardware & Accelerators
The 2026 picture is that NVIDIA still dominates training and general inference. A growing field of custom silicon (TPUs, Trainium, in-house hyperscaler chips) and inference-specialist startups (Groq, Cerebras, and transformer-ASIC entrants like Etched) chip away at specific workloads.
NVIDIA: Blackwell / GB-series
Blackwell is NVIDIA's current-generation datacenter architecture. Key parts (as of July 2026):
- B200 / GB200. The Blackwell workhorses. GB200 NVL72 is a rack-scale system linking 72 Blackwell GPUs and 36 Grace CPUs over NVLink into one liquid-cooled unit that acts as a giant GPU. https://www.nvidia.com/en-us/data-center/gb200-nvl72/
- B300 / Blackwell Ultra. A mid-generation refresh that began shipping in late 2025. Per NVIDIA/vendor materials roughly 288 GB HBM3e per GPU, higher FP4 throughput than B200, ~1,400W, liquid-cooled. GB300 NVL72 is the rack-scale version. Emphasis on low-precision (FP4) inference.
- Vera Rubin (R100). The next architecture, first unveiled at GTC 2025 and detailed further through CES/GTC 2026, scheduled for release in H2 2026 (around Q3), and expected to pressure Blackwell pricing over time.
NVIDIA's real moat is CUDA + NVLink + the whole software stack as much as the silicon.
Google TPU
Ironwood is Google's seventh-generation TPU (aka TPU v7 / "TPU7x"), positioned as inference-first and reaching general availability in 2026. Google cites large per-chip gains over prior generations and superpods of up to 9,216 chips with massive shared HBM and Optical Circuit Switching. TPUs are available on Google Cloud (and power Gemini). The tradeoff is a JAX/XLA-centric software path vs. CUDA. https://cloud.google.com/tpu/docs
AMD
Instinct MI300X/MI325X (HBM-heavy, competitive memory capacity per GPU) established AMD as the credible #2. The MI350 series (CDNA 4) is the current-gen competitor to Blackwell, with the MI400 series slated for 2026 and rack-scale "Helios" systems positioned against NVIDIA's NVL racks. AMD's software stack is ROCm. Support has improved a lot (vLLM, PyTorch) but remains less turnkey than CUDA for edge cases.
AWS Trainium & other hyperscaler silicon
AWS Trainium (Trainium 2 in wide use; Trainium 3 now generally available via Trn3 UltraServers) targets cost-efficient training/inference on AWS via the Neuron SDK. Also in the field: Microsoft Maia, Meta MTIA, and Intel Gaudi, mostly captive to their clouds/workloads. Outside the US ecosystem, Huawei Ascend (910C, and the rack-scale CloudMatrix 384 supernode) is the dominant non-NVIDIA training/inference platform, especially in China, and is benchmarked directly against GB200 NVL72. → hiascend.com
Inference-first chips
| Chip | What it is | Watch out |
|---|---|---|
| Groq | Deterministic LPU built for very low-latency token generation; very high tokens/sec on served models | Not a training chip; consumed as an API/cloud with a curated model menu |
| Cerebras | Wafer-scale engine (a whole wafer as one chip); large on-chip memory removes the bandwidth wall for high inference speeds, and it trains too | Specialized; accessed via Cerebras cloud/appliances |
| Etched (Sohu) | A transformer-specialized ASIC that bakes the architecture into silicon for very high throughput and low cost on transformer inference; emerged from stealth in 2026 with large reported pre-orders | Runs only transformer models (a bet the architecture stays stable), and it's early 🔄 |
| SambaNova | Reconfigurable RDU (SN40L) serving very fast tokens via SambaNova Cloud; a Groq/Cerebras peer | Consumed as a cloud/appliance; curated model menu |
| d-Matrix (Corsair) | In-memory-compute inference accelerator, in full production in 2026, aimed at high-throughput low-cost serving | Early; specialized inference-only silicon 🔄 |
VRAM math & cost-per-token intuition
- Can I even load it? Weights ≈
bytes-per-param × params: FP16 = 2 (7B ≈ 14 GB), INT8 = 1 (≈ 7 GB), INT4 = 0.5 (≈ 3.5 GB). Then add KV cache (grows with batch × context) and overhead. A 70B model in 4-bit is ~35 GB of weights alone, so it needs a 48 GB card (tight) or two. - Cost per token is set by throughput, not sticker price:
$/token ≈ (GPU $/hour) ÷ (tokens/hour). A pricier GPU that serves 3× the tokens can be cheaper per token. This is why quantization, batching, and the right serving engine (Inference, Serving & Optimization) move the economics more than the hardware line item. - Cloud vs own: rent (per-hour or per-token APIs) for spiky/uncertain demand and to skip capex and ops. Buy/reserve when utilization is high and steady (owned/reserved GPUs amortize well below on-demand at ~24/7 use). Most teams should rent until utilization and unit economics clearly justify owning.