Phase 10

LLMs From Scratch

Build, train, and understand large language models.

01Tokenizers: BPE, WordPiece, SentencePieceBuild~90 minutesQuiz 02Building a Tokenizer from ScratchBuild~90 minutesQuiz 03Data Pipelines for Pre-TrainingBuild~90 minutesQuiz 04Pre-Training a Mini GPT (124M Parameters)Build~120 minutesQuiz 05Scaling: Distributed Training, FSDP, DeepSpeedBuild~120 minutesQuiz 06Instruction Tuning (SFT)Build~90 minutesQuiz 07RLHF: Reward Model + PPOBuild~90 minutesQuiz 08DPO: Direct Preference OptimizationBuild~90 minutesQuiz 09Constitutional AI and Self-ImprovementBuild~45 minutes 10Evaluation: Benchmarks, Evals, LM HarnessBuild~90 minutesQuiz 11Quantization: Making Models FitBuild~120 minutesQuiz 12Inference OptimizationBuild~120 minutesQuiz 13Building a Complete LLM PipelineBuild~120 minutes 14Open Models: Architecture WalkthroughsLearn~45 minutes 15Speculative Decoding and EAGLE-3Build~75 minutes 16Differential Attention (V2)Build~60 minutes 17Native Sparse Attention (DeepSeek NSA)Build~60 minutes 18Multi-Token Prediction (MTP)Build~60 minutes 19DualPipe ParallelismLearn~60 minutes 20DeepSeek-V3 Architecture WalkthroughLearn~75 minutes 21Jamba — Hybrid SSM-TransformerLearn~60 minutes 22Async and Hogwild! InferenceBuild~60 minutes 25Speculative Decoding and EAGLEBuild~75 minutes 34Gradient Checkpointing and Activation RecomputationBuild~70 minutes 35Training Resource Budgeting: FLOPs, Memory, and Wall-ClockBuild~65 minutesQuiz 36Web Corpus Cleaning: Filtering, Deduplication, and MixingBuild~70 minutesQuiz

“Stay hungry, stay foolish.”

— Steve Jobs