Machine Learning

Relevant Areas

• Statistics
• Linear Algebra
• Calculus III

Fundamentals

Regression

• Regression
• Ordinary Least Squares
• Ridge Regression

Neural Networks

• Neural Networks
  • Artificial Neuron
  • Neural Network Layer
  • Multi-layer Neural Network
• Shallow Neural Network
• Deep Neural Network
• Multivariate Inputs and Outputs for Neural Networks
• Universal Approximation Theorem
• Backpropagation
• Neural Network Training
• Neural Network Weight Initialization

Classification

• Linear Classifier
  • Perceptron
    • Averaged Perceptron
    • Perceptron Convergence Theorem
• Margin
• Logistic Regression
  • Gradient Descent for Logistic Regression
• Support Vector Machine

Features

• Feature Representation
  • Feature Engineering
  • Polynomial Basis

Activation Functions

• Activation Function
  • Sigmoid
  • Rectified Linear Unit
  • Step function
  • Hyperbolic Tangent
  • Softmax
• Output Activation Functions

Loss Functions

• Machine Learning as Optimization
• Loss Function
• Negative log-likelihood
  • Negative log-likelihood multi-class
• Hinge Loss

Regularization

• Regularization
• L1 Regularization
• L2 Regularization
• Early stopping
• Weight Decay
• Weight Perturbation
• Dropout
• Batch Normalization

Gradient Descent

• Gradient Descent
  • 1D Gradient Descent
  • Multiple Dimension Gradient Descent
• Stochastic Gradient Descent
• Batch Gradient Descent

Optimizing Neural Network Parameters

• Adaptive step-size
  • Running Averages
  • Momentum (ML)
  • Adadelta
  • Adam

Convolutional Neural Networks

• Convolutional Neural Networks
• Convolution Filter
• 2D Convolution Operation
• Image Filter Bank
• Convolution Layer
• Tensors
• Max Pooling
• CNN Architecture

Recurrent Networks

• Recurrent Neural Networks
• LSTM

Reinforcement Learning

• RL Conceptual Framework
• State Machine
  • State Transition Diagram
  • Linear Time-invariant Systems
• Markov Decision Process
  • Policy Evaluation
  • Finite-horizon MDP Solutions
  • Infinite-horizon MDP Solutions

Computer Vision

• Radar Perception

3D View Synthesis

• NeRF

• EmerNeRF

• Instant NGP

• FeatureNeRF

• Polynomial Neural Fields

• Gaussian Splatting

• WATONeRF

Self-Supervised Learning

• SimCLR
• MoCo

Tools & Techniques

• Distributed Training
• ML Compilers

Other

• World Models

Problems

• MIT 6.036x Problems
• SYDE 572
  • SYDE 572 A1
  • SYDE 572 A2
  • SYDE 572 A3
  • SYDE 572 A4
  • SYDE 572 A5
• Understanding Deep Learning
  • UDL Chapter 2 Problems
  • UDL Chapter 3 Problems

Resources

• MIT 6.036x
• Understanding Deep Learning
• Deep Learning - Foundations and Concepts