Build an LLM From Scratch in Python & PyTorch
Course Syllabus
4 sections · 16 curated lessonsSection 1: Foundations, Tokenization & Data Pipelines
Autoregressive prediction, character tokenizers, context windows, and positional vector embeddings.
+
Section 1: Foundations, Tokenization & Data Pipelines
Autoregressive prediction, character tokenizers, context windows, and positional vector embeddings.
Section 2: Attention Mechanism & Transformer Core
Scaled dot-product attention, causal triangular masks, multi-head projections, and residual connections.
+
Section 2: Attention Mechanism & Transformer Core
Scaled dot-product attention, causal triangular masks, multi-head projections, and residual connections.
Section 3: The Complete Transformer Architecture & Optimization
Feed-forward networks with GELU, full GPT assembly, cross-entropy loss, and AdamW optimization.
+
Section 3: The Complete Transformer Architecture & Optimization
Feed-forward networks with GELU, full GPT assembly, cross-entropy loss, and AdamW optimization.
Section 4: Training, Generation & Deployment
The production training loop, temperature scaling, top-k sampling, and WebGPU deployment.
+
Section 4: Training, Generation & Deployment
The production training loop, temperature scaling, top-k sampling, and WebGPU deployment.
About this course
Course Overview
Large Language Models like GPT-4, Llama 3, and Claude often seem like impenetrable black boxes. But beneath the massive scale lies an elegant, surprisingly compact mathematical architecture: the decoder-only Transformer. When you strip away billions of parameters and hyper-scale distributed clusters, a complete, functioning language model can be written in roughly 250 lines of clean PyTorch code and trained on a personal laptop.
Build an LLM From Scratch in Python & PyTorch takes you on a step-by-step engineering journey from raw Unicode strings to an autoregressive model generating coherent text. Inspired by Andrej Karpathy's nanoGPT and Angelos Perivolaropoulos's educational LLM implementations, this course teaches you every single tensor transformation, gradient update, and sampling algorithm by building it yourself.
What You Will Build
- Custom Tokenizer Pipeline: Construct character-level tokenizers with bidirectional index mapping, vocabulary extraction, and batching matrices.
- Positional Vector Spaces: Combine continuous token embeddings with learnable positional encodings to give sequences spatial geometry.
- Multi-Head Causal Attention: Implement scaled dot-product attention, query/key/value projections, and lower-triangular causal masks that prevent future peeking.
- The Transformer Block: Wire pre-layer normalization, Gaussian Error Linear Units (GELU), residual skip connections, and multi-layer perceptrons.
- The Full GPT Model: Assemble stacked transformer blocks, weight-tied language model projection heads, and cross-entropy loss computation.
- Production Optimization: Implement the AdamW optimizer with decoupled weight decay, gradient norm clipping, and cosine learning rate schedules with linear warmup.
- Interactive Generation: Code autoregressive inference loops with temperature scaling, top-k filtering, and client-side browser deployment.
Course Structure
Section 1: Foundations, Tokenization & Data Pipelines
Autoregressive framing, next-token prediction, character tokenizers, context windows, and positional vector embeddings.
Section 2: Attention Mechanism & Transformer Core
Self-attention intuition, scaled dot-product math, causal triangular masking, multi-head parallel subspaces, and residual highways.
Section 3: The Complete Transformer Architecture & Optimization
Feed-forward networks, GELU activations, full GPT assembly, cross-entropy loss, AdamW, and cosine learning rate warmup schedules.
Section 4: Training, Generation & Deployment
End-to-end training loops, validation perplexity monitoring, temperature scaling, top-k sampling, and in-browser WebGPU inference.
Student Starter Pack & Code Downloads
Complete Offline Starter Bundle (.zip)
Includes the 1.1MB Tiny Shakespeare dataset (input.txt), clean PyTorch model (model.py), training loop (train.py), interactive generation script (generate.py), and quickstart guide.
Prerequisites
Basic familiarity with Python syntax (classes, functions, list comprehensions) and fundamental matrix math concepts (vectors, dot products). No prior deep learning experience or expensive cloud GPUs required: everything runs locally on your computer or directly in the browser.