MSDM 5054: Statistical Machine Learning
Fall 2026

Course Information

[ Reference ]
• To view .ipynb files below, you may try [ Jupyter NBViewer]
• Linear Regression Python Notebook [ MAFS6010_Regression.ipynb ]
• Linear Classification Python Notebook [ MAFS6010_Classification.ipynb ]

[Reference]:
• To view .ipynb files below, you may try [ Jupyter NBViewer]
• Python Notebook for Model Selection (Subset, Ridge, Lasso, and Principal Component Regression) [ Selection.ipynb ]


• Kaggle: Home Credit Default Risk [ link ]
• Kaggle: M5 Forecasting - Accuracy, Estimate the unit sales of Walmart retail goods. [ link ]

[Reference]:
• To view .ipynb files below, you may try [ Jupyter NBViewer]
• Python Notebook for ISLR Chapter 7 Lab [Ch7_Lab.ipynb ]

[ Seminar ]
• Title: A Statistical View on Implicit Regularization: Gradient Descent Dominates Ridge [ slides ][ link ]
• Speaker: Dr. Jingfeng WU, UC Berkeley
• Abstract: A key puzzle in deep learning is how simple gradient methods find generalizable solutions without explicit regularization. This talk discusses the implicit regularization of gradient descent (GD) through the lens of statistical dominance. Using linear regression as a clean proxy, we present three surprising findings. First, GD dominates ridge regression: with comparable regularization, the excess risk of GD is always within a constant factor of ridge, but ridge can be polynomially worse even when tuned optimally. Second, GD is incomparable with online stochastic gradient descent (SGD). While it is known that for certain problems GD can be polynomially better than SGD, the reverse is also true: we construct problems, inspired by benign overfitting theory, where optimally stopped GD is polynomially worse. Finally, GD dominates SGD for a significant subclass of problems -- those with fast and continuously decaying covariance spectra -- which includes all problems satisfying the standard capacity condition. This is joint work with Peter Bartlett, Sham Kakade, Jason Lee, and Bin Yu.
• Bio: Jingfeng Wu is an assistant professor of Statistics at UC Berkeley. His research focuses on deep learning theory, optimization, and statistical learning. He earned his Ph.D. in Computer Science from Johns Hopkins University in 2023. Prior to that, he received a B.S. in Mathematics (2016) and an M.S. in Applied Mathematics (2019), both from Peking University. In 2023, he was recognized as a Rising Star in Data Science by the University of Chicago and UC San Diego.

[Reference]:
• Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, Nathan Srebro. The Implicit Bias of Gradient Descent on Separable Data. [ arXiv:1710.10345 ]. ICLR 2018. Gradient descent on logistic regression leads to max margin.
• Matus Telgarsky. Margins, Shrinkage, and Boosting. [ arXiv:1303.4172 ]. ICML 2013. An older paper on gradient descent on exponential/logistic loss leads to max margin.
• Yuan Yao, Lorenzo Rosasco and Andrea Caponnetto. On Early Stopping in Gradient Descent Learning. Constructive Approximation, 2007, 26 (2): 289-315. [ link ]
• Jingfeng Wu, Peter L. Bartlett, Jason D. Lee, Sham M. Kakade, and Bin Yu. Risk Comparisons in Linear Regression: Implicit Regularization Dominates Explicit Regularization. [ arXiv:2509.17251 ]

[Reference]:
• To view .ipynb files below, you may try [ Jupyter NBViewer]
• Python Notebook for Decision Trees, Bagging, Random Forests and Boosting [ tree.ipynb ]

[Reference]:
• To view .ipynb files below, you may try [ Jupyter NBViewer]
• Python Notebook for Support Vector Machines [ svm.ipynb ]