Objectives

The objective of this course is to introduce linear and non-linear regression (logistic regression and generalized linear models). Regression plays a key role in many problems and it is absolutely essential for a datascientist to understand the theory and the practice of regression analysis. It is also an important vehicle to address the statistical challenges in statistical learning : model selection, penalisation, resampling (bootstrap, cross-validation) robustness, detection of outliers, and also methods to detect deviations from an assumed model. The course will also serve as a motivation to sharpen the understanding of statistical techniques, covering both estimation and tests.

Syllabus

1.    Introduction to statistical learning
Regression: Learning objectives and applications
Linear models : interpretation, examples
Least-Square estimators properties (bias, variance)
Case study: univariate and multivariate regression
Multivariate Linear Regression: Parametric casee
Construction of least-square estimators
2.    Parametric true model
Distribution of least-squares estimatorsAsymptotic properties
Gaussian case (distribution of the parameters, confidence regions)
Confidence intervals and tests
Classical regression diagnostic (leverage points)
Case studyAlgo: understanding  multiple linear regression with R (lm summary, detecting outliers, understanding classical regression diagnosis)
Multivariate Linear Regression: Non parametric casemodel validation

3.    Residual analysis (homoscedasticity, non-linear dependence)
    Outlier detection (leverage effects, influence, introduction to robsut statistics)
Functional modelintroduction to non-parametric regression : from parameters to functions
Multiple models for a single problemFunction classes, model selection
Variable choice / Basis / Spline
Bias / Variance (Approximation error / Estimation Error)
Case study : Spline regression

4.    Model Selection and Resampling
Approximation Error / Estimation Error
Learning Error / Generalization Error
Resampling based method: jacknife, bootstrap, and Cross Validation
Case study: model selection with CV

5.    Model Selection and Unbiased Risk Estimation
Unbiased Risk Estimation
AIC/BIC Penalization and Exhaustive Exploration
Forward / Backward and Stochastic Exploration
Multiple tests
Case study : model selection with exploration
Categorical variable
Binary variable : logisitic regression framework
Logistic loss
Algorithm : gradient descent algorithm, Iteratively Reweighted Least-Squares
Generalized linear models

6.    Model Selection and Penalization
Restricted Model and Penalization
Ridge and Lasso
Numerical algorithm: Gradient Descent and Coordinate Descent
Case study: Coordinate Descent and Lasso

7.    Logistic Regression
Classification and Binary output
Maximum Likelihood Approach
Penalization
Numerical Algorithm: Gradient Descent, Stocahstic Gradient Descent
The exponential family (definitions, examples, log-partition function)
Generalized linear models basics (ML/MAP estimators)
Probit regresion : latent variable interpretation, multinomial probit models
Case study: Logistic and model selection

8.    Generalized Linear Models
The exponential family (definitions, examples, log-partition function)
Generalized linear models basics (ML/MAP estimators)
Probit regresion : latent variable interpretation, multinomial probit models
Classical variable selection methods (forward, backward)
Multiple tests (Bonferonni, false discovery)
Algo: Cross Validation in R
Case study: multiclass