RESOURCES
Self-study resources for AI foundations
The roadmap, books and papers behind my self-study syllabus, from linear algebra to research methods. Each book lists the courses it serves and the chapters actually read.
01 — Roadmap
Roadmap
Ten courses, in order. Courses with a link are published on /syllabus; the rest are not public yet.
Course 02
Linear Algebra
Dot products, matrices as transformations, projections, rank, eigenvectors, SVD.
10 lessonsRead on /syllabus →
Course 03
Calculus & Optimization
Derivatives, the chain rule, gradients, exp and log, gradient descent.
6 lessonsRead on /syllabus →
Course 04
Probability
Distributions, Bayes, likelihood and MLE, the CLT, cross-entropy and KL divergence.
6 lessonsNot published yet
Course 05
Statistical Inference
Confidence intervals, p-values and their limits, power, effect size, pseudo-replication, bootstrap, TOST.
7 lessonsNot published yet
Course 06
Machine Learning
Loss, overfitting and bias–variance, regularization, train/validation/test, cross-validation, metrics, PCA.
8 lessonsNot published yet
Course 07
Deep Learning
MLPs, nonlinearity, backpropagation, initialization, Adam, layer norm, dropout, PyTorch.
7 lessonsNot published yet
Course 08
NLP & Transformers
BPE, embeddings, attention, the Transformer, positional encoding, masks, KV cache and MLA.
8 lessonsNot published yet
Course 09
Modern Training
Self-supervision, pretraining, scaling laws, LoRA, SFT, RL basics, RLHF and DPO.
7 lessonsNot published yet
Course 10
Systems Around the Model
BM25 and TF-IDF, nDCG and MRR, in-context learning, RAG, memory, graph memory and PPR, agents, evaluation.
7 lessonsNot published yet
Course 11
Research Skills
Critical paper reading, ablation, falsifiability, construct validity, null models, pre-registration.
10 lessonsNot published yet
02 — Bookshelf
Bookshelf
Free books link to the authors' own editions; paid books are marked as such.
Core texts
Marc Peter Deisenroth, A. Aldo Faisal, Cheng Soon Ong
Ch. 2–4 for linear algebra (Ch. 4 has the SVD), Ch. 5 and 7 for calculus and optimization, Ch. 6 for probability, Part II for the derivations behind classic ML.
Courses 2, 3, 4, 6Linear Algebra and Its Applications (4th ed.)
PaidGilbert Strang
The geometry: subspaces, projections and least squares, eigenvalues, Markov matrices, the SVD. Almost nothing on optimization or statistics, which is where MML takes over.
Course 2Joseph K. Blitzstein, Jessica Hwang
Main text for probability, taught alongside the Harvard Stat 110 lectures. Ch. 10 covers the CLT, which MML barely mentions.
Course 4Stat 110 lectures ↗Geoff Cumming, Robert Calin-Jageman
Backbone of the inference course: estimation, confidence intervals and effect sizes as the goal, with the p-value as a side note.
Course 5Gareth James, Daniela Witten, Trevor Hastie, Robert Tibshirani, Jonathan Taylor
Backbone of the machine learning course. Ch. 13 (multiple testing: Bonferroni, Holm, Benjamini–Hochberg) is used in the inference course.
Courses 5, 6Andrej Karpathy
Parts 1–6 for deep learning, starting with an autograd engine written from zero. Part 7 builds GPT from scratch; part 8 builds the BPE tokenizer.
Courses 7, 8Simon J. D. Prince
Main reference for deep learning (Ch. 3–9, 11, 20), plus bias–variance (Ch. 8), graph neural networks (Ch. 13) and reinforcement learning (Ch. 19).
Courses 6, 7, 9, 10Daniel Jurafsky, James H. Martin
Tokens, n-grams and transformers for NLP; post-training and alignment (Ch. 10); retrieval and RAG (Ch. 11). Chapter numbers follow the January 2026 draft.
Courses 8, 9, 10Build a Large Language Model (From Scratch)
Free code · paid bookSebastian Raschka
The slower PyTorch path after Karpathy: from raw text to attention to a full GPT, then pretraining and fine-tuning. The code alone is enough to follow.
Courses 8, 9Book (Manning) ↗Christopher D. Manning, Prabhakar Raghavan, Hinrich Schütze
Inverted index (Ch. 1–2), TF-IDF (Ch. 6), evaluation with P@k, MAP and nDCG (Ch. 8), BM25 (Ch. 11). Ch. 18 (LSI is the SVD on text) and Ch. 21 (PageRank) link back to linear algebra.
Course 10
Reference
Stephen Boyd, Lieven Vandenberghe
The applied side: QR, pseudo-inverses, least squares for regression and classification. No eigenvalues or SVD.
Courses 2, 6Deep Learning
FreeIan Goodfellow, Yoshua Bengio, Aaron Courville
For looking up precise definitions, after checking Understanding Deep Learning.
Course 7Richard S. Sutton, Andrew G. Barto
Only Ch. 3 (MDPs) and Ch. 13 (policy gradient), as groundwork for RLHF.
Course 9Mykel J. Kochenderfer, Tim A. Wheeler, Kyle H. Wray
Selected sections: d-separation, policy gradient and PPO, behavioral cloning, belief states.
Courses 9, 10, 11Moritz Hardt, Benjamin Recht
Ch. 8 on what happens to a benchmark when many models are tuned against the same test set.
Courses 5, 6Eric Lehman, F. Thomson Leighton, Albert R. Meyer
Optional: reading proofs (Ch. 1), graphs, and probability on countable sample spaces.
Courses 2, 5, 11Yao Ma, Jiliang Tang
Graphs as matrices and what GNNs learn from neighbours: background for graph Laplacians and graph memory.
Courses 2, 10Yufei Zhao
Ch. 3 only (expander mixing, Cheeger, Alon–Boppana), after the graph Laplacian lesson.
Course 2Jean Gallier, Jocelyn Quaintance
Full proofs for power methods, graph Laplacians, the spectral theorem and the SVD when a course text skips them.
Course 2
03 — Papers
Papers
Papers read inside a course, grouped by the course that uses them.
Course 08 · NLP & Transformers
Attention Is All You Need
Vaswani et al. · 2017 · NeurIPS
Read with the attention lesson: recurrence was dropped for throughput, and the 1/√dₖ scaling rests on a footnote that was never ablated.
Course 09 · Modern Training
Scaling Laws for Neural Language Models
Kaplan et al. · 2020 · arXiv
Loss follows smooth power laws in parameters, data and compute. Its advice on how to spend compute was later revised by Chinchilla.
Training Compute-Optimal Large Language Models
Hoffmann et al. · 2022 · NeurIPS
Chinchilla: model size and training tokens should grow together. Read with Kaplan; the pair is also a case study in the research skills course.
LoRA: Low-Rank Adaptation of Large Language Models
Hu et al. · 2021 · ICLR 2022
Freeze the weights and learn a low-rank update: the low-rank idea from linear algebra, applied to fine-tuning.
Training Language Models to Follow Instructions with Human Feedback
Ouyang et al. · 2022 · NeurIPS
InstructGPT: supervised fine-tuning, a reward model and RL, the pipeline that turns a next-token predictor into an assistant.
Course 11 · Research Skills
Mathematics in the Age of AI
Terence Tao · 2026 · arXiv
An essay based on his public lecture at ICM 2026, read in the research skills course.
Reducing the Costs to Design, Train, and Collect Data for Neural Networks with Combinatorial Optimization
Hieu Pham · 2021 · PhD thesis, CMU (CMU-LTI-21-003)
Read for its structure: one idea in Ch. 2, applied to four problems in Ch. 3–6. A model of how a thesis holds together.
04 — Latest notes
Latest notes
Notes distilled from the courses above, written in English.