Machine learning 1000-135UM

(I) Classical machine learning

0. Basic concepts, tasks, methods.
• Concepts: data – sample from population, loss function, risk/error, prediction, parameters, hyperparameters, learning/training/fitting/estimation, learning error, generalization error.
• Tasks: unsupervised learning: feature dimension reduction (PCA) and data dimension reduction (clustering); supervised learning: prediction = regression or classification.
• Learning by means of (penalized) empirical risk minimization.

1. Unsupervised learning.
• Variable dimension reduction: Principal Component Analysis (PCA) via SVD.
• Clustering using the k-means algorithm as an example. Soft k-means for a mixture of normal distributions.

2. Linear prediction. Fitting a logistic model using Iteratively Reweighted Least Squares.

3. Penalization of linear prediction.
• Variable selection, or L0 penalization.
• Ridge Regression, or L2 penalization. Penalty tuning via SVD.
• Lasso, or L1 penalization. Model fitting using Coordinate Descent. Penalty tuning using cross-validation.

4. Enhancement of linear prediction using kernel transformations of explanatory variables: Kernel Regularized Least Squares and Support Vector Machines (SVM). Fitting SVM using Gradient Descent (GD) and quadratic optimization with constraints.

5. Tree based prediction.
• Location, dispersion and dependency measures based on probability densities.
• Greedy tree generation and pruning using L0 penalization.
• Random Forests.
• Enhancement of linear and tree based prediction using Gradient Boosting.

6. Enhancement of linear and tree based prediction via Gradient Boosting method.

(II) Introduction to deep neural networks.

7. Basic concepts: network, architecture, layer, backbone, data batch, learning epoch.

8. Multi-layer perceptron (MLP) – fitting using the Stochastic Gradient Descent algorithm with an automatic differentiation (back-propagation).

9. MLP for data representation. An autoencoder network (nonlinear equivalent of PCA) with application to classification of handwritten digits.

10. Convolutional Neural Networks (ConvNets) with application to image classification. Convolution with local or global kernels. Computation of convolutions (discrete, cyclic, 1-2 dim) with global kernel using the Fourier transform (convolution theorem and FFT).

11. Transformers with application to sequence processing (text, speech, time series). Embeddings of words and natural numbers in R^512 and an averaging transformation called attention.