Deep Learning

Notes from:

• Understanding Deep Learning

Some of these notes are also included in Machine Learning.

Foundations

• Tensor
• Einstein Notation

Neural Networks

• Supervised Learning
  • Linear Regression
• Discriminative vs. Generative Models

Shallow Neural Networks

• Shallow Neural Network
• Universal Approximation Theorem
• Multivariate Inputs and Outputs

Deep Neural Networks

• Composing Shallow Networks
• Deep Neural Network
• Shallow vs. Deep Networks

Loss Functions

• Loss Function
• Conditional Probabilistic Perspective of Learning
• Maximum Likelihood Criterion
• Log-Likelihood Criterion
• Loss Function Recipe
  • Univariate Regression
    • Heteroscedastic Regression
  • Binary Classification
  • Multi-class Classification
• Cross-Entropy Loss

Model Fitting/Training

• Model Optimization
• Gradient Descent
  • Gradient Descent for Linear Regression
  • Gradient Descent for Non-convex Gabor Model
• Stochastic Gradient Descent
• Momentum (ML)
  • Nesterov Accelerated Momentum
• Adam
• Line Search

Gradients and Initialization

• Backpropagation Intuition
• Backpropagation Toy Example
• Backpropagation Algorithm
• Backpropagation

Learning Theory

• Neural Tangent Kernel

Practical

• Einops
• PyTorch Tensor Creation
• Einsum

Exercises

• UDL Chapter 2 Problems
• UDL Chapter 3 Problems
• UDL Chapter 4 Problems
• UDL Chapter 5 Problems
• UDL Chapter 6 Problems
• UDL Chapter 7 Problems