ECE 595ML: Machine Learning I

By Stanley H. Chan

Electrical and Computer Engineering, Purdue University, West Lafayette, IN

Category

Courses

Published on

Abstract

Spring 2020 - This course is in production

Course Website: https://engineering.purdue.edu/ChanGroup/ECE595/index.html

Course Outline:

  • Part 1: Mathematical Background
    • Linear Regression and Optimization
  • Part 2: Classification
    • Methods to train linear classifiers
    • Feature analysis, Geometry, Bayesian decision rule, logistic regression, perceptron algorithm, support vector machine
  • Part 3: Handling Uncertainty
    • Imperfect data: noisy label, unbalanced data, missing data, knowledge transfer
    • Robustness study: adversarial attack and defense
  • Part 4: Learning Theory
    • Evaluation of a classifier.
    • Feasibility of learning, VC dimension, bias-variance, validation

Bio

Stanley H. Chan Stanley H. Chan is currently an assistant professor in the School of Electrical and Computer Engineering and the Department of Statistics at Purdue University.

Dr. Chan received the Ph.D. degree in Electrical Engineering and the M.A. degree in Mathematics from the University of California at San Diego, in 2011 and 2009, respectively, and the B.Eng. degree (with first class honor) in Electrical Engineering from the University of Hong Kong in 2007. Prior to joining Purdue, he was a postdoctoral research fellow at Harvard John A. Paulson School of Engineering and Applied Sciences from 2012 to 2014. His research interests include signal and image processing, applied statistics, and large-scale numerical optimization.

References

Textbook and References

  • Duda, Hart and Stork, Pattern Classification, Wiley-Interscience; 2nd edition, 2000.
  • Bishop, Pattern Recognition and Machine Learning, Springer, 2006.
  • Abu-Mostafa, Magdon-Ismail and Lin, Learning from Data, AMLBook, 2012.
  • Hastie, Tibshirani and Friedman, Elements of Statistical Learning, Springer, 2nd edition, 2009.

Pre-requesite Background References

  • Linear Algebra: Gilbert Strang, Linear Algebra and Its Applications, 5th Edition
  • Optimization: Stephen Boyd and Lieven Vandenberghe, Convex Optimization, Cambridge 2004.
  • Probability: Dimitri Bertsekas, Introduction to Probability, Athena Scientific, 2008, 2nd Edition.

Cite this work

Researchers should cite this work as follows:

  • Stanley H. Chan (2020), "ECE 595ML: Machine Learning I," https://nanohub.org/resources/32203.

    BibTex | EndNote

Location

WTHR 200, Purdue University, West Lafayette, IN

Tags

Lecture Number/Topic Online Lecture Video Lecture Notes Supplemental Material Suggested Exercises
ECE 595ML: Introduction YouTube
ECE 595ML: Course Overview View HTML
View Notes (pdf)
ECE 595ML Lecture 1.1: Linear Regression View HTML
View Notes (pdf) YouTube
ECE 595ML Lecture 1.2: Linear Regression - Geometry View HTML
View Notes (pdf)
ECE 595ML Lecture 2.1: Regularized Linear Regression View HTML
View Notes (pdf) YouTube
ECE 595ML Lecture 2.2: Regularized Linear Regression - LASSO Regression View HTML
View Notes (pdf)
ECE 595ML Lecture 3.1: Linear Regression with Kernels - Kernel Method View HTML
View Notes (pdf) YouTube
ECE 595ML Lecture 3.2: Linear Regression with Kernels - Examples View HTML
View Notes (pdf)
ECE 595ML Lecture 4.1: Introduction to Optimization - Unconstrained Optimization View HTML
View Notes (pdf) YouTube
ECE 595ML Lecture 4.2: Introduction to Optimization - Convexity View HTML
View Notes (pdf)
ECE 595ML Lecture 4.3: Introduction to Optimization - Constrained Optimization View HTML
View Notes (pdf)
ECE 595ML Lecture 5.1: Gradient Descent View HTML
View Notes (pdf) YouTube
ECE 595ML Lecture 5.2: Gradient Descent - Stochastic Gradient Descent View HTML
View Notes (pdf)
ECE 595ML Lecture 6.1: Linear Separatability - Notations View HTML
View Notes (pdf) YouTube
ECE 595ML Lecture 6.2: Linear Separatability - Geometry of Discriminant Function View HTML
View Notes (pdf)
ECE 595ML Lecture 6.3: Linear Separatability View HTML
View Notes (pdf)
ECE 595ML Lecture 7.1: Feature Analysis via PCA View HTML
View Notes (pdf) YouTube
ECE 595ML Lecture 7.2: Feature Analysis via PCA - Kernal PCA View HTML
View Notes (pdf)
ECE 595ML Lecture 8.1: Hand-Crafted and Deep Features - Convolution View HTML
View Notes (pdf) YouTube
ECE 595ML Lecture 8.2: Hand-Crafted and Deep Features - SIFT and HOG View HTML
View Notes (pdf)
ECE 595ML Lecture 8.3: Hand-Crafted and Deep Features - Deep Features View HTML
View Notes (pdf)
ECE 595ML Lecture 9.1: Bayesian Decision - Review of High-Dimensional Gaussian View HTML
View Notes (pdf) YouTube
ECE 595ML Lecture 9.2: Bayesian Decision - Basic Principle View HTML
View Notes (pdf)
ECE 595ML Lecture 9.3: Bayesian Decision - The Three Cases View HTML
View Notes (pdf)
ECE 595ML Lecture 10: Minimum Probability of Error Rule YouTube
ECE 595ML Lecture 11.1: Maximum-Likelihood Estimation - Basic Principles View HTML
View Notes (pdf) YouTube
ECE 595ML Lecture 11.2: Maximum-Likelihood Estimation - Examples View HTML
View Notes (pdf)
ECE 595ML Lecture 12.1: Bayesian Parameter Estimation - Basic Principles View HTML
View Notes (pdf) YouTube
ECE 595ML Lecture 12.2: Bayesian Parameter Estimation - Choosing Priors View HTML
View Notes (pdf)
ECE 595ML Lecture 13.1: Connecting Bayesian with Linear Regression - Linear Regression Review View HTML
View Notes (pdf) YouTube
ECE 595ML Lecture 13.2: Connecting Bayesian with Linear Regression View HTML
View Notes (pdf)
ECE 595ML Lecture 14.1: Logistic Regression - From Linear to Logistic View HTML
View Notes (pdf) YouTube
ECE 595ML Lecture 14.2: Logistic Regression - Interpreting Logistic View HTML
View Notes (pdf)
ECE 595ML Lecture 14.3: Logistic Regression - Convexity View HTML
View Notes (pdf)
ECE 595ML Lecture 15.1: Logistic Regression - Gradient Descent View HTML
View Notes (pdf) YouTube
ECE 595ML Lecture 15.2: Logistic Regression - Connection with Bayes View HTML
View Notes (pdf)
ECE 595ML Lecture 15.3: Logistic Regression - Comparison with Linear Regression View HTML
View Notes (pdf)
ECE 595ML Lecture 16.1: Preceptron - From Logistic to Preceptron View HTML
View Notes (pdf) YouTube
ECE 595ML Lecture 16.2: Preceptron - Properties of Preceptron Loss View HTML
View Notes (pdf)
ECE 595ML Lecture 17.1: Perceptron - Perceptron Algorithm View HTML
View Notes (pdf) YouTube
ECE 595ML Lecture 17.2: Perceptron - Optimality View HTML
View Notes (pdf)
ECE 595ML Lecture 17.3: Perceptron - Convergence View HTML
View Notes (pdf)
ECE 595ML Lecture 18.1: Multi-Layer Perceptron View HTML
View Notes (pdf) YouTube
ECE 595ML Lecture 18.2: Multi-Layer Perceptron - Back Propagation View HTML
View Notes (pdf)
ECE 595ML Lecture 19.1: Support Vector Machine - Concept of Margin View HTML
View Notes (pdf) YouTube
ECE 595ML Lecture 19.2: Support Vector Machine - SVM View HTML
View Notes (pdf)
ECE 595ML Lecture 20.1: Support Vector Machine - Lagrange Duality View HTML
View Notes (pdf) YouTube
ECE 595ML Lecture 20.2: Support Vector Machine - Dual SVM View HTML
View Notes (pdf)
ECE 595ML Lecture 21.1: Support Vector Machine - Soft SVM View HTML
View Notes (pdf) YouTube
ECE 595ML Lecture 21.2: Support Vector Machine - Kernel Trick View HTML
View Notes (pdf)
ECE 595ML Lecture 22.1: Is Learning Feasible? - What Constitutes a Learning Problem? View HTML
View Notes (pdf) YouTube
ECE 595ML Lecture 22.2: Is Learning Feasible? View HTML
View Notes (pdf)
ECE 595ML Lecture 22.3: Is Learning Feasible? - Testing vs. Training View HTML
View Notes (pdf)
ECE 595ML Lecture 23.1: Probability Inequality - Basic Inequalities View HTML
View Notes (pdf) YouTube
ECE 595ML Lecture 23.2: Probability Inequality - Advanced Inequalities View HTML
View Notes (pdf)
ECE 595ML Lecture 23.3: COVID-19/Probability Inequality - Hoeffding Inequality View HTML
View Notes (pdf) YouTube
ECE 595ML Lecture 24.1: Probability Approximate Correct - Two Ingredients of Generalization View HTML
View Notes (pdf) YouTube
ECE 595ML Lecture 24.2: Probability Approximate Correct - PAC Framework View HTML
View Notes (pdf) YouTube
ECE 595ML Lecture 25.1: Generalization Bound - M Hypothesis View HTML
View Notes (pdf) YouTube
ECE 595ML Lecture 25.2: Generalization Bound View HTML
View Notes (pdf) YouTube
ECE 595ML Lecture 25.3: Generalization Bound - Handling M Hypothesis View HTML
View Notes (pdf)
ECE 595ML Lecture 26.1: Growth Function - Overcoming the M Factor View HTML
View Notes (pdf) YouTube
ECE 595ML Lecture 26.2: Growth Function - Examples of mH(N) View HTML
View Notes (pdf) YouTube
ECE 595ML Lecture 27.1: VC Dimension - From Dichotomy to Shattering View HTML
View Notes (pdf) YouTube
ECE 595ML Lecture 27.2: VC Dimension - Example of VC Dimension View HTML
View Notes (pdf) YouTube
ECE 595ML Lecture 28.1: Sample and Model Complexity - Generalization Bound using VC Dimension View HTML
View Notes (pdf) YouTube
ECE 595ML Lecture 28.2: Sample and Model Complexity View HTML
View Notes (pdf) YouTube
ECE 595ML Lecture 29.1: Bias and Variance - From VC Analysis to Bias-Variance View HTML
View Notes (pdf) YouTube
ECE 595ML Lecture 29.2: Bias and Variance - Example View HTML
View Notes (pdf) YouTube
ECE 595ML Lecture 30.1: Overfit - Source of Overfit View HTML
View Notes (pdf) YouTube
ECE 595ML Lecture 30.2: Overfit - Analyzing Overfit View HTML
View Notes (pdf) YouTube
ECE 595ML Lecture 31.1: Regularization - Motivation for Regularization View HTML
View Notes (pdf) YouTube
ECE 595ML Lecture 31.2: Regularization - Two Regularization Techniques View HTML
View Notes (pdf) YouTube
ECE 595ML Lecture 31.3: Regularization - Choosing a Regularization View HTML
View Flash
View Notes (pdf) YouTube
ECE 595ML Lecture 32.1: Validation View HTML
View Notes (pdf) YouTube
ECE 595ML Lecture 32.2: Validation - Model Selection View HTML
View Notes (pdf) YouTube
ECE 595ML Lecture 32.3: Validation - Validation in Regularization View HTML
View Notes (pdf)
ECE 595ML Lecture 33.1: Adversarial Attack - An Overview View HTML
View Notes (pdf)
ECE 595ML Lecture 33.2: Adversarial Attack - Basic Terminologies View HTML
View Notes (pdf)