1. Supervised learning

• 1.1. Generalized Linear Models
  • 1.1.1. Ordinary Least Squares
    • 1.1.1.1. Ordinary Least Squares Complexity
  • 1.1.2. Ridge Regression
    • 1.1.2.1. Ridge Complexity
    • 1.1.2.2. Setting the regularization parameter: generalized Cross-Validation
  • 1.1.3. Lasso
    • 1.1.3.1. Setting regularization parameter
      • 1.1.3.1.1. Using cross-validation
      • 1.1.3.1.2. Information-criteria based model selection
  • 1.1.4. Multi-task Lasso
  • 1.1.5. Elastic Net
  • 1.1.6. Multi-task Elastic Net
  • 1.1.7. Least Angle Regression
  • 1.1.8. LARS Lasso
    • 1.1.8.1. Mathematical formulation
  • 1.1.9. Orthogonal Matching Pursuit (OMP)
  • 1.1.10. Bayesian Regression
    • 1.1.10.1. Bayesian Ridge Regression
    • 1.1.10.2. Automatic Relevance Determination - ARD
  • 1.1.11. Logistic regression
  • 1.1.12. Stochastic Gradient Descent - SGD
  • 1.1.13. Perceptron
  • 1.1.14. Passive Aggressive Algorithms
  • 1.1.15. Robustness regression: outliers and modeling errors
    • 1.1.15.1. Different scenario and useful concepts
    • 1.1.15.2. RANSAC: RANdom SAmple Consensus
      • 1.1.15.2.1. Details of the algorithm
    • 1.1.15.3. Theil-Sen estimator: generalized-median-based estimator
      • 1.1.15.3.1. Theoretical considerations
  • 1.1.16. Polynomial regression: extending linear models with basis functions
• 1.2. Linear and Quadratic Discriminant Analysis
  • 1.2.1. Dimensionality reduction using Linear Discriminant Analysis
  • 1.2.2. Mathematical formulation of the LDA and QDA classifiers
  • 1.2.3. Mathematical formulation of LDA dimensionality reduction
  • 1.2.4. Shrinkage
  • 1.2.5. Estimation algorithms
• 1.3. Kernel ridge regression
• 1.4. Support Vector Machines
  • 1.4.1. Classification
    • 1.4.1.1. Multi-class classification
    • 1.4.1.2. Scores and probabilities
    • 1.4.1.3. Unbalanced problems
  • 1.4.2. Regression
  • 1.4.3. Density estimation, novelty detection
  • 1.4.4. Complexity
  • 1.4.5. Tips on Practical Use
  • 1.4.6. Kernel functions
    • 1.4.6.1. Custom Kernels
      • 1.4.6.1.1. Using Python functions as kernels
      • 1.4.6.1.2. Using the Gram matrix
      • 1.4.6.1.3. Parameters of the RBF Kernel
  • 1.4.7. Mathematical formulation
    • 1.4.7.1. SVC
    • 1.4.7.2. NuSVC
    • 1.4.7.3. SVR
  • 1.4.8. Implementation details
• 1.5. Stochastic Gradient Descent
  • 1.5.1. Classification
  • 1.5.2. Regression
  • 1.5.3. Stochastic Gradient Descent for sparse data
  • 1.5.4. Complexity
  • 1.5.5. Tips on Practical Use
  • 1.5.6. Mathematical formulation
    • 1.5.6.1. SGD
  • 1.5.7. Implementation details
• 1.6. Nearest Neighbors
  • 1.6.1. Unsupervised Nearest Neighbors
    • 1.6.1.1. Finding the Nearest Neighbors
    • 1.6.1.2. KDTree and BallTree Classes
  • 1.6.2. Nearest Neighbors Classification
  • 1.6.3. Nearest Neighbors Regression
  • 1.6.4. Nearest Neighbor Algorithms
    • 1.6.4.1. Brute Force
    • 1.6.4.2. K-D Tree
    • 1.6.4.3. Ball Tree
    • 1.6.4.4. Choice of Nearest Neighbors Algorithm
    • 1.6.4.5. Effect of leaf_size
  • 1.6.5. Nearest Centroid Classifier
    • 1.6.5.1. Nearest Shrunken Centroid
  • 1.6.6. Approximate Nearest Neighbors
    • 1.6.6.1. Locality Sensitive Hashing Forest
    • 1.6.6.2. Mathematical description of Locality Sensitive Hashing
• 1.7. Gaussian Processes
  • 1.7.1. Gaussian Process Regression (GPR)
  • 1.7.2. GPR examples
    • 1.7.2.1. GPR with noise-level estimation
    • 1.7.2.2. Comparison of GPR and Kernel Ridge Regression
    • 1.7.2.3. GPR on Mauna Loa CO2 data
  • 1.7.3. Gaussian Process Classification (GPC)
  • 1.7.4. GPC examples
    • 1.7.4.1. Probabilistic predictions with GPC
    • 1.7.4.2. Illustration of GPC on the XOR dataset
    • 1.7.4.3. Gaussian process classification (GPC) on iris dataset
  • 1.7.5. Kernels for Gaussian Processes
    • 1.7.5.1. Gaussian Process Kernel API
    • 1.7.5.2. Basic kernels
    • 1.7.5.3. Kernel operators
    • 1.7.5.4. Radial-basis function (RBF) kernel
    • 1.7.5.5. Matérn kernel
    • 1.7.5.6. Rational quadratic kernel
    • 1.7.5.7. Exp-Sine-Squared kernel
    • 1.7.5.8. Dot-Product kernel
    • 1.7.5.9. References
  • 1.7.6. Legacy Gaussian Processes
    • 1.7.6.1. An introductory regression example
    • 1.7.6.2. Fitting Noisy Data
    • 1.7.6.3. Mathematical formulation
      • 1.7.6.3.1. The initial assumption
      • 1.7.6.3.2. The best linear unbiased prediction (BLUP)
      • 1.7.6.3.3. The empirical best linear unbiased predictor (EBLUP)
    • 1.7.6.4. Correlation Models
    • 1.7.6.5. Regression Models
    • 1.7.6.6. Implementation details
• 1.8. Cross decomposition
• 1.9. Naive Bayes
  • 1.9.1. Gaussian Naive Bayes
  • 1.9.2. Multinomial Naive Bayes
  • 1.9.3. Bernoulli Naive Bayes
  • 1.9.4. Out-of-core naive Bayes model fitting
• 1.10. Decision Trees
  • 1.10.1. Classification
  • 1.10.2. Regression
  • 1.10.3. Multi-output problems
  • 1.10.4. Complexity
  • 1.10.5. Tips on practical use
  • 1.10.6. Tree algorithms: ID3, C4.5, C5.0 and CART
  • 1.10.7. Mathematical formulation
    • 1.10.7.1. Classification criteria
    • 1.10.7.2. Regression criteria
• 1.11. Ensemble methods
  • 1.11.1. Bagging meta-estimator
  • 1.11.2. Forests of randomized trees
    • 1.11.2.1. Random Forests
    • 1.11.2.2. Extremely Randomized Trees
    • 1.11.2.3. Parameters
    • 1.11.2.4. Parallelization
    • 1.11.2.5. Feature importance evaluation
    • 1.11.2.6. Totally Random Trees Embedding
  • 1.11.3. AdaBoost
    • 1.11.3.1. Usage
  • 1.11.4. Gradient Tree Boosting
    • 1.11.4.1. Classification
    • 1.11.4.2. Regression
    • 1.11.4.3. Fitting additional weak-learners
    • 1.11.4.4. Controlling the tree size
    • 1.11.4.5. Mathematical formulation
      • 1.11.4.5.1. Loss Functions
    • 1.11.4.6. Regularization
      • 1.11.4.6.1. Shrinkage
      • 1.11.4.6.2. Subsampling
    • 1.11.4.7. Interpretation
      • 1.11.4.7.1. Feature importance
      • 1.11.4.7.2. Partial dependence
  • 1.11.5. VotingClassifier
    • 1.11.5.1. Majority Class Labels (Majority/Hard Voting)
      • 1.11.5.1.1. Usage
    • 1.11.5.2. Weighted Average Probabilities (Soft Voting)
    • 1.11.5.3. Using the VotingClassifier with GridSearch
      • 1.11.5.3.1. Usage
• 1.12. Multiclass and multilabel algorithms
  • 1.12.1. Multilabel classification format
  • 1.12.2. One-Vs-The-Rest
    • 1.12.2.1. Multiclass learning
    • 1.12.2.2. Multilabel learning
  • 1.12.3. One-Vs-One
    • 1.12.3.1. Multiclass learning
  • 1.12.4. Error-Correcting Output-Codes
    • 1.12.4.1. Multiclass learning
• 1.13. Feature selection
  • 1.13.1. Removing features with low variance
  • 1.13.2. Univariate feature selection
  • 1.13.3. Recursive feature elimination
  • 1.13.4. Feature selection using SelectFromModel
    • 1.13.4.1. L1-based feature selection
    • 1.13.4.2. Randomized sparse models
    • 1.13.4.3. Tree-based feature selection
  • 1.13.5. Feature selection as part of a pipeline
• 1.14. Semi-Supervised
  • 1.14.1. Label Propagation
• 1.15. Isotonic regression
• 1.16. Probability calibration
• 1.17. Neural network models (supervised)
  • 1.17.1. Multi-layer Perceptron
  • 1.17.2. Classification
  • 1.17.3. Regression
  • 1.17.4. Algorithms
  • 1.17.5. Complexity
  • 1.17.6. Mathematical formulation
  • 1.17.7. Tips on Practical Use
  • 1.17.8. More control with warm_start