A step-by-step walkthrough of building a decoder-only Transformer (the architecture behind ChatGPT) from scratch using PyTorch and PyTorch Lightning. Covers tokenization, word embeddings, positional encoding using sine/cosine functions, masked self-attention with query/key/value matrices, residual connections, and a fully connected output layer. Includes training with Adam optimizer and cross-entropy loss on a minimal toy dataset, demonstrating the full pipeline from untrained random outputs to correct predictions.
•31m watch time