Essentials of Mathematical Methods

About

This repo contains my online evolving book Essentials of Mathematical Methods: Foundations, Principles, and Algorithms. This book surveys fundamental results in major branches of applied mathematics, with emphasized applications in stochastic system modeling, statistical learning, and optimal decision-making.
The book takes an approach that resembles many math and CS theory textbooks. Each section starts with basic concepts, motivations and definitions, then important properties in Lemma, Theorem, and Corollary (with proof), and then computational procedures summarized in Methods and Algorithms. I believe this approach can help distinguish different concepts and state important results in a clean and organized manner.
I am hoping this book can be beneficial to you, either you are preparing for interviews or you are doing machine learning applications and research.

中文简介

This book is free. You can support me by purchasing it from LeanPub. If you have any questions or suggestions, you can create a pull request or send me an email at [email protected]

Middle/Long term plan

The long term goal is to build a comprehensive, interactive electronic book that is handy to students, teachers, researchers, and practitioners who are passionate about mathematical modeling, artificial intelligence and machine learning.

Currently I am working on several mid-term items:

• Add more diagrams, figures, and examples to demonstrate the intuition and application.
• Organize implementations for core algorithms.
• Create a new Part "Deep Learning Methods" (now it is a lengthy chapter under the Part Statistical Learning Methods), devoted to deep learning fundamentals and its applications in computer vision, time series, recommender, and NLP.

Long term detailed action items:

• Create an interactive electronic book that can host dynamic visuals and short video tuturials for key parts.

Downloads

Whole book

Front Matter
All-in-One

Selective topics

Linear Algebra and Matrix Analysis
Mathematical Optimization
Probability and Statistical Estimation
Stochastic Process
Markov Chain and Random Walk
Linear Regression Analysis
Statistical Learning
Neural Network and Deep Learning
(Deep) Reinforcement Learning

Table of Contents

I Mathematical Foundations

• Sets, Sequences and Series
• Metric Space
• Advanced Calculus
• Linear Algebra and Matrix Analysis
• Basic Functional Analysis

II Mathematical Optimization Methods

• Unconstrained Nonlinear Optimization
• Constrained Nonlinear Optimization
• Linear Optimization
• Convex Analysis and Convex Optimization
• Basic Game Theory

III Classical Statistical Methods

• Probability Theory
• Statistical Distributions
• Statistical Estimation Theory
• Multivariate Statistical Methods
• Linear Regression Analysis
• Monte Carlo Methods

IV Dynamics Modeling Methods

• Models and estimation in linear systems
• Stochastic Process
• Stochastic Calculus
• Markov Chain and Random Walk
• Time Series Analysis

V Statistical Learning Methods

• Supervised Learning Principles and Methods
• Linear Models for Regression
• Linear Models for Classification
• Generative Models
• K Nearest Neighbors
• Tree Methods
• Ensemble and Boosting Methods
• Unsupervised Statistical Learning
• Neural Network and Deep Learning

VI Optimal Control and Reinforcement Learning Methods

• Classical Optimal Control Theory
• Reinforcement Learning

Appendix: Supplemental Mathematical Facts

License statement

You are free to redistribute the material in any medium or format under the following terms:

-Attribution: You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.
-NonCommercial: You may not use the material for commercial purposes.

*The licensor cannot revoke these freedoms as long as you follow the license terms. This licence is created via creative commons (https://creativecommons.org). If you have any questions regarding the license, please contact the author.

Demonstration

Linear algebra

SVD (Singular Value Decomposition) is one of the most important results in linear algebra. It is the cornerstone of many important methods in applied math, statistics, and machine learning. This books summarizes the properties of SVD with the following theorem and diagram. The proof is concise with all the supporting theorems and lemma included in the book.

The following diagram shows the shape of resulting matrices and captures the relationship between complete SVD and compact SVD.

A common mistake on the relationship between U and V is discussed as wells.

Applied functional analysis

Contraction mapping and its closely related fixed point theorem are essential results in applied functional analysis. We start with a diagram to help understand contraction mapping.

We then state its definition and a simple example.

We also discuss how to identify a contraction mapping in practice.

Finally, with the understanding of contraction mapping, we introduce the fixed point theorem.

We give a detailed and concise proof.

Statistics

In multivariate Gaussian statistics, the affine transformation theorem is used to prove a number of important properties of Gaussian random variables (such as addition, condition, etc.). This book first gives a proof of this theorem based on moment generating functions.

Then the book gives the application of this theorem to the sum of multivariate Gaussian random variables. It is worth mentioning that the author emphasizes in the footnote that these result only hold when the joint normality conditions holds.

Machine learning

Fisher linear discriminant analysis

We start with Fisher linear discriminant analysis by contrasting with PCA.

We use diagrams to build the intuition of deriving Fisher linear discriminant.

The intuition of deriving Fisher linear discriminant can be summarized by following definition.

We summarize the solution to the optimization problem in the following Lemma.

Unified linear models

SVM, logistic regression and Perceptron learning are commonly used linear classification model in machine learning. The three models can be unified under the same mathematical optimization framework, with each method corresponds to a different loss function. This book elaborates and shows how to transform these three models into this unified framework.

SVD for collaborative filtering

One important application of SVD in machine learning area is collaborative filtering for recommenders.

Reinforcement learning

Value iteration is one of the cornerstone theorems in reinforcement learning. However, many textbooks and online resources simply skip the proof. This book puts together a concise proof through contraction mapping and fixed point theorem. The contraction mapping and fixed point theorem, which are important tools in applied math, are introduced in detail in Part I of this book.

Then the book gives an algorithm based on the value iteration theorem.