# .

Lecture notes for the Statistical Machine Learning course taught  at the Department of Information Technology, University of Uppsala (Sweden.) Updated in March 2019. Authors: Andreas Lindholm, Niklas Wahlström, Fredrik Lindsten, and Thomas B. Schön. Source: page 61 in these lecture notes

Available as a PDF, here (original) or here (mirror).

Content

1 Introduction 7

1.1 What is machine learning all about?
1.2 Regression and classification
1.3 Overview of these lecture notes

2 The regression problem and linear regression 11

2.1 The regression problem
2.2 The linear regression model

• Describe relationships — classical statistics
• Predicting future outputs — machine learning

2.3 Learning the model from training data

• Maximum likelihood
• Least squares and the normal equations

2.4 Nonlinear transformations of the inputs – creating more features
2.5 Qualitative input variables
2.6 Regularization

• Ridge regression
• LASSO
• General cost function regularization

2.A Derivation of the normal equations

• A calculus approach
• A linear algebra approach

3 The classification problem and three parametric classifiers 25

3.1 The classification problem
3.2 Logistic regression

• Learning the logistic regression model from training data
• Decision boundaries for logistic regression
• Logistic regression for more than two classes

3.3 Linear and quadratic discriminant analysis (LDA & QDA)

• Using Gaussian approximations in Bayes’ theorem
• Using LDA and QDA in practice

3.4 Bayes’ classifier — a theoretical justification for turning p(y | x) into yb

• Bayes’ classifier
• Optimality of Bayes’ classifier
• Bayes’ classifier in practice: useless, but a source of inspiration
• Is it always good to predict according to Bayes’ classifier?

3.5 More on classification and classifiers

• Regularization
• Evaluating binary classifiers

4 Non-parametric methods for regression and classification: k-NN and trees 43

4.1 k-NN

• Decision boundaries for k-NN
• Choosing k
• Normalization

4.2 Trees

• Basics
• Training a classification tree
• Other splitting criteria
• Regression trees

5 How well does a method perform? 53

5.1 Expected new data error Enew: performance in production
5.2 Estimating Enew

• Etrain 6≈ Enew: We cannot estimate Enew from training data
• Etest ≈ Enew: We can estimate Enew from test data
• Cross-validation: Eval ≈ Enew without setting aside test data

5.3 Understanding Enew

• Enew = Etrain+ generalization error
• Enew = bias2 + variance + irreducible error

6 Ensemble methods 67

6.1 Bagging

• Variance reduction by averaging
• The bootstrap

6.2 Random forests
6.3 Boosting

• The conceptual idea
• Binary classification, margins, and exponential loss
• Boosting vs. bagging: base models and ensemble size
• Robust loss functions and gradient boosting

6.A Classification loss functions

7 Neural networks and deep learning 83

7.1 Neural networks for regression

• Generalized linear regression
• Two-layer neural network
• Matrix notation
• Deep neural network
• Learning the network from data

7.2 Neural networks for classification

• Learning classification networks from data

7.3 Convolutional neural networks

• Data representation of an image
• The convolutional layer
• Condensing information with strides
• Multiple channels
• Full CNN architecture

7.4 Training a neural network

• Initialization
• Learning rate
• Dropout

A Probability theory 101

A.1 Random variables

• Marginalization
• Conditioning

A.2 Approximating an integral with a sum

B Unconstrained numerical optimization 105

B.1 A general iterative solution
B.2 Commonly used search directions

• Steepest descent direction
• Newton direction
• Quasi-Newton
{"ft_lang_label":"__label__en","ft_lang_prob":0.7415424,"math_prob":0.7708317,"size":2267,"snap":"2021-21-2021-25","text_gpt3_token_len":579,"char_repetition_ratio":0.12417145,"word_repetition_ratio":0.0,"special_character_ratio":0.24922806,"punctuation_ratio":0.13151927,"nsfw_num_words":0,"has_unicode_error":false,"math_prob_llama3":0.996125,"pos_list":[0,1,2,3,4],"im_url_duplicate_count":[null,null,null,3,null],"WARC_HEADER":"{\"WARC-Type\":\"response\",\"WARC-Date\":\"2021-05-11T10:45:53Z\",\"WARC-Record-ID\":\"<urn:uuid:78423735-2923-4a0b-aaf0-132bf1130bd4>\",\"Content-Length\":\"87291\",\"Content-Type\":\"application/http; msgtype=response\",\"WARC-Warcinfo-ID\":\"<urn:uuid:abd19544-8a52-4fa9-b794-7404aa797a32>\",\"WARC-Concurrent-To\":\"<urn:uuid:c2ca655c-20fc-4278-b26e-13db31e69920>\",\"WARC-IP-Address\":\"172.67.74.34\",\"WARC-Target-URI\":\"https://www.datasciencecentral.com/profiles/blogs/free-book-lecture-notes-on-machine-learning\",\"WARC-Payload-Digest\":\"sha1:JR6NCRHCEF4HNS73KMP44V7X47QHCBW5\",\"WARC-Block-Digest\":\"sha1:ZGLL5FWLZS7TD3YEF75HX2H3FX6SZQQD\",\"WARC-Identified-Payload-Type\":\"text/html\",\"warc_filename\":\"/cc_download/warc_2021/CC-MAIN-2021-21/CC-MAIN-2021-21_segments_1620243991982.8_warc_CC-MAIN-20210511092245-20210511122245-00427.warc.gz\"}"}