MATH 5470: Statistical Machine Learning

statlearning-notebooks, by Sujit Pal, Python implementations of the R labs for the StatLearning: Statistical Learning online course from Stanford taught by Profs Trevor Hastie and Rob Tibshirani.

Homework and Projects:

TBA (To Be Announced)

Schedule

Date	Topic	Instructor	Scriber
07/02/2022, Mon	Lecture 01: A Historic Overview. [ slides (pdf) ]	Y.Y.
09/02/2022, Wed	Lecture 02: An Introduction to Supervised Learning [ slides ]	Y.Y.
14/02/2022, Mon	Lecture 03: Supervised Learning, linear regression and classification [ slides ]	Y.Y.
16/02/2022, Wed	Lecture 04: Linear regression and classification [ slides ] [ Reference ] To view .ipynb files below, you may try [ Jupyter NBViewer] Linear Regression Python Notebook [ Regression.ipynb ] Linear Classification Python Notebook [ Classification.ipynb ]	Y.Y.
21/02/2022, Mon	Lecture 05: Model Assessment and Selection [ slides ] [Reference]: To view .ipynb files below, you may try [ Jupyter NBViewer] Python Notebook for Model Selection (Subset, Ridge, Lasso, and Principal Component Regression) [ Selection.ipynb ]	Y.Y.
23/02/2022, Wed	Lecture 06: Model Assessment and Selection: Subset, Ridge, Lasso, and PCR [ slides ] [Reference]: To view .ipynb files below, you may try [ Jupyter NBViewer] Python Notebook for Model Selection (Subset, Ridge, Lasso, and Principal Component Regression) [ Selection.ipynb ]	Y.Y.
28/02/2022, Mon	Lecture 07: Moving beyond Linearity [ slides ] [Reference]: To view .ipynb files below, you may try [ Jupyter NBViewer] Python Notebook for ISLR Chapter 7 Lab [Ch7_Lab.ipynb ]	Y.Y.
02/03/2022, Wed	Lecture 08: Moving beyond Linearity [ slides ] [Reference]: To view .ipynb files below, you may try [ Jupyter NBViewer] Python Notebook for ISLR Chapter 7 Lab [Ch7_Lab.ipynb ]	Y.Y.
07/03/2022, Mon	Lecture 09: Decision Tree, Bagging, Random Forests and Boosting [ YY's slides ] [Reference]: To view .ipynb files below, you may try [ Jupyter NBViewer] Python Notebook for Decision Trees, Bagging, Random Forests and Boosting [ tree.ipynb ]	Y.Y.
09/03/2022, Wed	Lecture 10: Decision Tree, Bagging, Random Forests and Boosting [ YY's slides ] [Reference]: To view .ipynb files below, you may try [ Jupyter NBViewer] Python Notebook for Decision Trees, Bagging, Random Forests and Boosting [ tree.ipynb ]	Y.Y.
14/03/2022, Mon	Lecture 11: Support Vector Machines [ YY's slides ] [Reference]: To view .ipynb files below, you may try [ Jupyter NBViewer] Python Notebook for Support Vector Machines [ svm.ipynb ]	Y.Y.
16/03/2022, Wed	Lecture 12: Support Vector Machines [ YY's slides ] [Reference]: To view .ipynb files below, you may try [ Jupyter NBViewer] Python Notebook for Support Vector Machines [ svm.ipynb ] Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, Nathan Srebro. The Implicit Bias of Gradient Descent on Separable Data. [ arXiv:1710.10345 ]. ICLR 2018. Gradient descent on logistic regression leads to max margin. Matus Telgarsky. Margins, Shrinkage, and Boosting. [ arXiv:1303.4172 ]. ICML 2013. An older paper on gradient descent on exponential/logistic loss leads to max margin.	Y.Y.
21/03/2022, Mon	Lecture 13: An Introduction to Convolutional Neural Networks [ YY's slides ] [Reference]: To view .ipynb files below, you may try [ Jupyter NBViewer] LeNet5 for MNIST dataset in Pytorch Notebook [ LeNet5_mnist.ipynb ] LeNet5 for Cifar10 dataset in Pytorch Notebook [ LeNet5_cifar10.ipynb ] AlexNet for Cifar10 dataset in Pytorch Notebook [ AlexNet_cifar10.ipynb ] ResNet for Cifar10 dataset in Pytorch Notebook [ ResNet_cifar10.ipynb ]	Y.Y.
23/03/2022, Wed	Lecture 14: Topics on Convolutional Neural Networks [ YY's slides ] and Final Project Assignment [ project.pdf ] [ Topics on CNNs ]: Visualizing CNNs [ YY's slides ] To view .ipynb files below, you may try [ Jupyter NBViewer] Fine-tuning (transfer learning) of ResNet in Pytorch Notebook [ finetuning_resnet.ipynb ] Visualization of VGG16 in Pytorch Notebook [ vgg16-visualization.ipynb ] Class activation heatmap of VGG16 in Pytorch Notebook [ vgg16-heatmap.ipynb ] Neural Style of HKUST at Starry Night in Pytorch Notebook [ neural_style_starry-hkust.ipynb ] Adversarial examples of LeNet5 with MNIST [ LeNet5_mnist_fgsm.ipynb ] [Reading Material]: Shihao Gu, Bryan Kelly and Dacheng Xiu "Empirical Asset Pricing via Machine Learning", Review of Financial Studies, Vol. 33, Issue 5, (2020), 2223-2273. Winner of the 2018 Swiss Finance Institute Outstanding Paper Award. [ link ] Jingwen Jiang, Bryan Kelly and Dacheng Xiu "(Re-)Imag(in)ing Price Trends", Chicago Booth Report, Aug 2021 [ link ] [ Reference ]: Kaggle: Home Credit Default Risk [ link ] Kaggle: M5 Forecasting - Accuracy, Estimate the unit sales of Walmart retail goods. [ link ] Kaggle: M5 Forecasting - Uncertainty, Estimate the uncertainty distribution of Walmart unit sales. [ link ] Kaggle: Ubiquant Market Prediction - Make predictions against future market data. [ link ] Kaggle: G-Research Crypto Forecasting. [ link ] Type-II diabetes and Alzheimer’s disease. [ slides (pdf) ] [ slides (pptx) ]	Y.Y.
28/03/2022, Mon	Lecture 15: An Introduction to Recurrent Neural Networks (RNN) and Long Short Term Memory (LSTM) [ YY's slides ] [Reference]: To view .ipynb files below, you may try [ Jupyter NBViewer] Character-level RNN, LSTM and GRU for Name Classification [ char_rnn_classification_tutorial.ipynb ] Generating Shakespeare's Sonnet: RNN, LSTM, Bidirectional LSTM, and Momentum-LSTM [ rnn_lstm_BiLSTM_mlstm_shakespeare.ipynb ] [ rnn_lstm_biLSTM_shakespeare.ipynb ] [ shakespeare.txt ] RNN for generating Shakespeare's Sonnet [ rnn.ipynb ] [ shakespeare.txt ]	Y.Y.
30/03/2022, Wed	Lecture 16: An Introduction to Recurrent Neural Networks (RNN) and Long Short Term Memory (LSTM) [ YY's slides ] [Reference]: To view .ipynb files below, you may try [ Jupyter NBViewer] Character-level RNN, LSTM and GRU for Name Classification [ char_rnn_classification_tutorial.ipynb ] RNN for generating Shakespeare's Sonnet [ rnn.ipynb ] [ shakespeare.txt ] LSTM for generating Shakespeare's Sonnet [ rnn_lstm_shakespeare.ipynb ] Generating Shakespeare's Sonnet: RNN, LSTM, Bidirectional LSTM [ rnn_lstm_BiLSTM_shakespeare.ipynb ] [ shakespeare.txt ] Bidirectional RNN for MNIST in pytorch [ bidirection_lstm_mnist.ipynb ] [ Youtube ] Jorgen Schmidhuber: [ Deep Learning: Our Miraculous Year 1990-1991 ] [ Critique of Honda Prize for Dr. Hinton ]	Y.Y.
04/04/2022, Mon	Lecture 17: Attention, Transformer and BERT [ slides ] [Reference]: To view .ipynb files below, you may try [ Jupyter NBViewer] Pytorch Twitter Sentiment Analysis: RNN, LSTM, BiLSTM, Multihead Self-Attention. [ RNN (ipynb) ] [ LSTM (ipynb) ] [ BiLSTM (ipynb) ] [ BiLSTM with Multihead Attention (ipynb) ] [ BERT embedding with BiLSTM (ipynb) ] [ Sentiment140 dataset ] Pytorch Sentiment Analysis with IMDB data: RNN, (bi)-LSTM, CNN, Transformer, BERT, etc. [ GitHub ] Illustrated Transformer by Jay Alammar: [ link ] The Annotated Transformer Tutorial by Sasha Rush: [ link ] BERT generation of Shakespeare's sonnet: [ BERT_shakespeare_gen.ipynb ] BERT next sentence generation of Shakespeare's sonnet and Chinese poems: [ BERT_shakespeare_nextsen.ipynb ] Chinese BERT (Whole-Word-Masking): [ link ] [ Presentation ] Final Project Proposal: Detect the disrupted brain connectivity in type-II diabetes patients [ slides ] Speaker: WEI, Yue and XIE, Weiyan	Y.Y.
06/04/2022, Wed	Lecture 18: Attention, Transformer and BERT [ slides ] [Reference]: To view .ipynb files below, you may try [ Jupyter NBViewer] Pytorch Twitter Sentiment Analysis: RNN, LSTM, BiLSTM, Multihead Self-Attention. [ RNN (ipynb) ] [ LSTM (ipynb) ] [ BiLSTM (ipynb) ] [ BiLSTM with Multihead Attention (ipynb) ] [ BERT embedding with BiLSTM (ipynb) ] [ Sentiment140 dataset ] Pytorch Sentiment Analysis with IMDB data: RNN, (bi)-LSTM, CNN, Transformer, BERT, etc. [ GitHub ] Illustrated Transformer by Jay Alammar: [ link ] The Annotated Transformer Tutorial by Sasha Rush: [ link ] BERT generation of Shakespeare's sonnet: [ BERT_shakespeare_gen.ipynb ] BERT next sentence generation of Shakespeare's sonnet and Chinese poems: [ BERT_shakespeare_nextsen.ipynb ] Chinese BERT (Whole-Word-Masking): [ link ]	Y.Y.
11/04/2022, Mon	Lecture 19: An Introduction to Unsupervised Learning: PCA, AutoEncoder, VAE, and GANs [ slides ] [Reference]: To view .ipynb files below, you may try [ Jupyter NBViewer] DCGAN for MNIST Tutorial in Pytorch Notebook [ dcgan_mnist_tutorial.ipynb ]	Y.Y.
20/04/2022, Wed	Lecture 20: Robust Statistics and Generative Adversarial Networks [ slides ] [Reference]: To view .ipynb files below, you may try [ Jupyter NBViewer] Robust_GAN.ipynb : Jupyter Notebook for demonstration Robust-GAN-Center : robust center (mean) estimate via GANs Robust-GAN-Scatter : robust scatter (covariance) estimate via GANs GAO, Chao, Jiyi LIU, Yuan YAO, and Weizhi ZHU. Robust Estimation and Generative Adversarial Nets. ICLR 2019. [ arXiv:1810.02030 ] [ GitHub ] [ GAO, Chao's Simons Talk ] GAO, Chao, Yuan YAO, and Weizhi ZHU. Generative Adversarial Nets for Robust Scatter Estimation: A Proper Scoring Rule Perspective. Journal of Machine Learning Research, 21(160):1-48, 2020. [ arXiv:1903.01944 ] [ GitHub ]	Y.Y.
25/04/2022, Mon	Lecture 21: An Introduction to Self-supervised Learning [ slides ] [ Reference ]: To view .ipynb files below, you may try [ Jupyter NBViewer] A tutorial on MoCo: [ moco_tutorial.ipynb ]	Y.Y.
27/04/2022, Wed	Seminar: Conformal Prediction Speaker: Prof. Emmanuel Candès, Stanford University Abstract: Recent progress in machine learning provides us with many potentially effective tools to learn from datasets of ever-increasing sizes and make useful predictions. How do we know that these tools can be trusted in critical and highly-sensitive domains? If a learning algorithm predicts the GPA of a prospective college applicant, what guarantees do we have concerning the accuracy of this prediction? How do we know that it is not biased against certain groups of applicants? To address questions of this kind, this talk reviews a wonderful field of research known under the name of conformal inference/prediction, pioneered by Vladimir Vovk and his colleagues 20 years ago. After reviewing some of the basic ideas underlying distribution-free predictive inference, we shall survey recent progress in the field touching upon several issues: (1) efficiency: how can we provide tighter predictions?, (2) data-reuse: what do we do when data is scarce? (3) algorithmic fairness: how do we make sure that learned models apply to individuals in an equitable manner?, and (4) causal inference: can we predict the counterfactual response to a treatment given that the patient was not treated? This is the keynote talk at the Bernoulli-IMS One World Symposium on 27 Aug 2020. [ link ] [ Reference ] Alex Gammerman, Volodya Vovk, Vladimir Vapnik. Learning by Transduction. In Proceedings of the Fourteenth Conference on Uncertainty in Artificial Intelligence (UAI1998). 1998. [ arXiv:1301.7375 ] Glenn Shafer, Vladimir Vovk, A Tutorial on Conformal Prediction. Journal of Machine Learning Research 9 (2008) 371-421. [ link ] Jing Lei, Max G’Sell, Alessandro Rinaldo, Ryan J. Tibshirani, and Larry Wasserman. Distribution-Free Predictive Inference for Regression. Journal of the American Statistical Association, 2018, 113(523):1094-1111. [ link ][ arXiv:1604.04173 ] Y. Romano, E. Patterson, E. J. Candès. Conformalized quantile regression. Advances in neural information processing systems 32 (NIPS), 2019. [ arXiv:1905.03222 ] R. F. Barber, E. J. Candès, A. Ramdas, R. J. Tibshirani. Predictive inference with the jackknife+. Ann. Statist.. 2021. [ arXiv:1905.02928 ] Y. Romano, M. Sesia, E. J. Candès. Classification with valid and adaptive coverage. Advances in neural information processing systems 33 (neurips 2020). 2020. [ arXiv:2006.02544 ] I. Gibbs, E. Candès. Adaptive conformal inference under distribution shift. Advances in Neural Information Processing Systems 34 (NeurIPS 2021). 2021. [ arXiv:2106.00170 ] L. Lei, E. J. Candès. Conformal inference of counterfactuals and individual treatment effects. Journal of the Royal Statistical Society Series B. 2021. [ arXiv:2006.06138 ] R. F. Barber, E. J. Candès, A. Ramdas, R. J. Tibshirani. Conformal prediction beyond exchangeability. 2022. [ arXiv:2202.13415 ]	Y.Y.
04/05/2022, Wed	Final Project Presentations. [ Groups ]: 1. CHAN Wing Chun. Empirical Asset Pricing via Machine Learning. [ report (pptx) ] [ slides (pptx) ] [ source (zip) ] [ presentation (youtube) ] 2. CHU Zhuang, YANG Zilan. (Re-)Imag(in)ing Price Trends. [ report (pptx) ] [ slides(pdf) ] [ source (GitHub) ] [ presentation (youtube) ] 3. Alvin Lo, Ning Liu, Tony Tan, Ziqing Guo. (Re-)Imag(in)ing Price Trends. [ report (pdf) ] [ slides (pdf) ] [ source (github) ] 4. DI Yining, DING Hongxing, XU Meng. Traffic Network Imputation and Congestion Recognition via Attentive Graph Neural Processes. [ slides (pptx) ] [ presentation (hkust) ]	Y.Y.
11/05/2022, Wed	Final Project Presentations. [ Groups ] LI Jiabao, ZHU Zhihan. Comparison of models for Ubiquant Market Prediction. WEI Xinyi, KUANG Liangyawei. Home Credit Default Risk. WEI Yue, XIE Weiyan. Detect the Disrupted Brain Connectivity in Type-II Diabetes Patients. CAO, Zhefeng and CHU, Mengyuan. Empirical Asset Pricing via Machine Learning. [ Final Report Collection ] Description of Final Project: [ pdf ] GitHub Repository for reports of Final Project [ GitHub ] [ Paper Replication: Empirical Asset Pricing via Machine Learning ] 1. CHAN Wing Chun. [ report (pptx) ] [ slides (pptx) ] [ source (zip) ] [ presentation (youtube) ] 2. ZENG Jialin, JIN Tianyu, HU Tiankai. [ report (pdf) ] [ slides (pdf) ] [ source (ipynb) ] [ presentation (zoom) ] 3. SHANG Zhenhang, SUN Lei, QUAN Xueyang. [ report (pdf) ] [ slides(pdf) ] [ source (GitHub) ] [ presentation (youtube) ] 4. SHI, Runhao. [ report (pdf) ] [ slides (pdf) ] [ source (github) ] [ presentation (bilibili) ] 5. XIONG Wei, LIU Chen, WANG Zhe, JI Wen. [ report (pdf) ] [ source (github) ] [ presentation (bilibili) ] 6. ZHANG, Kaixi. [ report (pdf) ] [ source (R) ] [ presentation (bilibili) ] 7. CAO, Zhefeng and CHU, Mengyuan. [ report (docx) ] [ slides (pdf) ] [ source (github) ] [ Paper Replication: (Re-)Imag(in)ing Price Trends ] 1. YU Wentao and CAO Ruoxiao. [ report (pdf) ] [ slides (pptx) ] [ source (hkust) ] [ presentation (zoom) ] 2. HUANG Bo, CHEN Mingyang. [ report (pdf) ] [ Huang's 'slides (pptx) ] [ Chen's 'slides (pptx) ] [ source I (github) ] [ source II (github) ] [ presentation (youtube) ] 3. CHU Zhuang, YANG Zilan. [ report (pptx) ] [ slides(pdf) ] [ source (GitHub) ] [ presentation (youtube) ] 4. Hu, Mingyun; Ma, Wanteng; Zhang, Jiaxin. [ report (pdf) ] [ slides (pdf) ] [ source (github) ] [ presentation (youtube) ] 5. Alvin Lo, Ning Liu, Tony Tan, Ziqing Guo. [ report (pdf) ] [ slides (pdf) ] [ source (github) ] 6. WONG Wing Kin, WAN Ho Yin. [ report (pdf) ] [ source (R) ] [ presentation (youtube) ] [ Kaggle ] 1. Yuxuan Qin, Wenxue Li, Yubo Wang, Decang Sun. M5 Forecasting - Accuracy. [ report (pptx) ] [ source (github) ] [ presentation (bilibili) ] 2. HUI Chun, LIU Shiyi. Home Credit Default Risk. [ report (pdf) ] [ slides (pptx) "] [ source (zip) ] [ presentation (bilibili) ] 3. Haoran Li, Jiaxin Bai, Qi Hu, Ying Su. Home Credit Default Risk. [ report (pdf) ] [ slides (pdf) "] [ source (github) ] [ presentation (youtube) ] 4. WEI Xinyi, KUANG Liangyawei. Home Credit Default Risk. [ report (pdf) ] [ source (ipynb) ] [ presentation (youtube) ] 5. WU Hao, ZHU Yiming, TANG Jihong, WANG Xian. Home Credit Default Risk. [ report (pdf) ] [ slides (pptx) "] [ source (github) ] [ presentation (youtube) ] 6. YE, Peng. Home Credit Default Risk. [ report (pdf) ] [ slides (pptx) "] [ presentation (hkust) ] 7. LI Jiabao, ZHU Zhihan. Comparison of models for Ubiquant Market Prediction. [ report (pdf) ] [ slides (pdf) ] [ source (github) ] [ presentation (youtube) ] 8. LIN Zizhen, YAO Duanyi. Ubiquant Market Prediction Project Report. [ report (pdf) ] [ slides (pptx) ] [ source (ipynb) ] [ presentation (youtube) ] [ Self Proposals ] 1. WEI Yue, XIE Weiyan. Detect the Disrupted Brain Connectivity in Type-II Diabetes Patients. [ report (pdf) ] [ slides (pptx) ] [ source (github) ] 2. Yue Cui, Qichen Tan, Jing Zhao. Open Target Debiasing. [ slides (pptx) ] [ presentation (youtube) ] 3. DI Yining, DING Hongxing, XU Meng. Traffic Network Imputation and Congestion Recognition via Attentive Graph Neural Processes. [ slides (pptx) ] [ presentation (hkust) ] [Reading Material]: Shihao Gu, Bryan Kelly and Dacheng Xiu "Empirical Asset Pricing via Machine Learning", Review of Financial Studies, Vol. 33, Issue 5, (2020), 2223-2273. Winner of the 2018 Swiss Finance Institute Outstanding Paper Award. [ link ] Jingwen Jiang, Bryan Kelly and Dacheng Xiu "(Re-)Imag(in)ing Price Trends", Chicago Booth Report, Aug 2021 [ link ] [ Reference ]: Kaggle: Home Credit Default Risk [ link ] Kaggle: M5 Forecasting - Accuracy, Estimate the unit sales of Walmart retail goods. [ link ] Kaggle: M5 Forecasting - Uncertainty, Estimate the uncertainty distribution of Walmart unit sales. [ link ] Kaggle: Ubiquant Market Prediction - Make predictions against future market data. [ link ] Kaggle: G-Research Crypto Forecasting. [ link ] Type-II diabetes and Alzheimer’s disease. [ slides (pdf) ] [ slides (pptx) ]	Y.Y.

by YAO, Yuan.