Statistics

Notes from:

• Chapters 2 and 3 of Deep Learning - Foundations and Concepts
• Chapter 6 of The Algorithm Design Manual

Probability

Probability Basics

• Basic Probability
• Sum and Product Rules of Probability
• Bayes’ Theorem
• Prior and Posterior
• Independent Variables
• Law of Total Probability
• Addition Law of Probability

Probability Densities

• Probability Density Function
  • Multivariate Probability Density
• Cumulative Distribution Function
• Expected Value
  • Expectation Algebra
• Variance
• Covariance
• Standard Deviation

Transformation of Densities

• Transformation of Densities
• Transformation of Multivariate Distributions

Distribution Basics

• Gaussian Distribution
• Likelihood Function
• Maximum Likelihood Estimation
  • Gaussian Maximum Likehood Estimation
  • Bessel’s Correction
• Linear Regression as MLE
• Confidence Interval

Information Theory

• Information Theory
• Measure of Information
• Kullback-Leibler Divergence
  • Convex Function
  • Jensen’s Inequality
  • Density Estimation with KL Divergence
• Mutual Information

Entropy

• Information Entropy
  • Statistical Entropy
• Differential Entropy
• Maximum Entropy
• Kullback-Leibler Divergence (Relative entropy)
• Conditional Entropy

Bayesian Probability

• Bayesian Probability
• Bayesian Machine Learning

Standard Distributions