Deep Learning

Notes from:

• Understanding Deep Learning

Some of these notes are also included in Machine Learning.

Foundations

• Tensor
• Einstein Notation

Neural Networks

• Supervised Learning
  • Linear Regression
• Discriminative vs. Generative Models

Shallow Neural Networks

• Shallow Neural Network
• Universal Approximation Theorem
• Multivariate Inputs and Outputs

Deep Neural Networks

• Composing Shallow Networks
• Deep Neural Network
• Shallow vs. Deep Networks

Loss Functions

• Loss Function
• Conditional Probabilistic Perspective of Learning
• Maximum Likelihood Criterion
• Log-Likelihood Criterion
• Loss Function Recipe
  • Univariate Regression
    • Heteroscedastic Regression
  • Binary Classification
  • Multi-class Classification
• Cross-Entropy Loss

Model Fitting/Training

• Model Optimization
• Gradient Descent
  • Gradient Descent for Linear Regression
  • Gradient Descent for Non-convex Gabor Model
• Stochastic Gradient Descent
• Momentum (ML)
  • Nesterov Accelerated Momentum
• Adam
• Line Search

Gradients and Initialization

• Backpropagation Intuition
• Backpropagation Algorithm
• Backpropagation Scalar Example
• Backpropagation 3-Layer Example
• Parameter Initialization

Model Performance

• Sources of Test Error
  • MNIST 1D Test Error Example
• Mathematical Formulation of Test Error
• Reducing Model Error
• Double Descent
• Inductive Bias
• Curse of Dimensionality
• Hyperparameter Search
• Cross-Validation
• Model Capacity

Regularization

• Regularization
• Explicit Regularization

Learning Theory

• Neural Tangent Kernel

Practical

• Einops
• PyTorch Tensor Creation

Exercises

• UDL Chapter 2 Problems
• UDL Chapter 3 Problems
• UDL Chapter 4 Problems
• UDL Chapter 5 Problems
• UDL Chapter 6 Problems
• UDL Chapter 7 Problems
• UDL Chapter 8 Problems

Projects

• Counterfactual Probing