vinsis / Math And Ml Notes

Links to some important research papers or links. I plan to add notes as I go through each topic one by one.

✔ 1. Information theory based (unsupervised) learning

• [x] Invariant Information Clustering
  • Notes
• [x] Mutual Information Neural Estimation
  • Notes
• [x] Deep Infomax
  • Notes
• [x] Learning Representations by Maximizing Mutual Information Across Views
  • Notes
• [x] How Google decoupled MI maximization and representation learning: On Mutual Information Maximization for Representation Learning
  • Notes

✔ 2. Disentangled representations

• [x] Quick overview by Google
  • Notes
• [x] β-VAE, pdf
  • Notes
• [x] Understanding disentangling in β-VAE
  • Notes

Decided not to delve deeper into this topic. It is not mature yet.
* [Disentangling Disentanglement in Variational Autoencoders](https://arxiv.org/abs/1812.02833)
* [Isolating Sources of Disentanglement in Variational Autoencoders](https://arxiv.org/abs/1802.04942)
* [InfoGAN-CR: Disentangling Generative Adversarial Networks with Contrastive Regularizers](https://arxiv.org/abs/1906.06034)
* [Disentangling by Factorising, pdf](https://www.cs.toronto.edu/~amnih/papers/disentangling_nips_ws.pdf)

✔ 3. Contrastive Coding

• [x] Representation Learning with Contrastive Predictive Coding
• [x] Data-Efficient Image Recognition with Contrastive Predictive Coding
  • Combined notes
• [x] Contrastive Multiview Coding
  • Notes
• [x] Momentum Contrast for Unsupervised Visual Representation Learning
  • Notes
• Google dispelling a lot of misconceptions about disentangled representations: Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations
  • I have become sort of disillusioned with disentangled representations. I feel they are not at a mature stage yet. So I have decided not to spend time on this paper.

✔ 4. Automatic differentiation

• [x] Automatic differentiation in machine learning: a survey
  • Annotated pdf
• [x] Automatic Reverse-Mode Differentiation: Lecture Notes
  • Annotated pdf
• [x] Reverse mode automatic differentiation

5. NNs and ODEs

• [ ] Neural Ordinary Differential Equations
  • [ ] Adjoint tutorial
• [ ] Augmented Neural ODEs
• [ ] Invertible ResNets
• [ ] Universal Differential Equations for Scientific Machine Learning

6. Probabilistic Programming

• [x] Probabilistic models of cognition
  • Notes
• [ ] The Design and Implementation of Probabilistic Programming Languages
• [ ] Composition in Probabilistic Language Understanding

7. Miscellaneous

Memorization in neural networks

• Blog post by BAIR
• Identity Crisis: Paper

Online Learning

• A Modern Introduction to Online Learning

Graph Neural Networks

• [ ] A Gentle Introduction to Deep Learning for Graphs, pdf

Normalizing Flows

• Detailed hands-on introduction
• Normalizing Flows for Probabilistic Modeling and Inference
• PyTorch implementations of density estimation algorithms

Transformers

• [x] Attention is all you need
  • [x] A nice visual introduction
  • [x] Annonated paper
  • Python notebook with illustrations and working code
• Reformer: The Efficient Transformer

Others

• Zero-shot knowledge transfer
• SpecNet
• Deep Learning & Symbolic Mathematics
• Learning to Predict Layout-to-image Conditional Convolutions for Semantic Image Synthesis
• Deep Equilibrium Models

8. Theory of neural networks

Lottery tickets

• Lottery ticket hypothesis
• Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask
• Rigging the Lottery: Making All Tickets Winners

Others

• What's Hidden in a Randomly Weighted Neural Network?
• Topological properties of the set of functions generated by neural networks of fixed size
• YOUR CLASSIFIER IS SECRETLY AN ENERGY BASED MODEL AND YOU SHOULD TREAT IT LIKE ONE
• Neural Persistence: A Complexity Measure for Deep Neural Networks Using Algebraic Topology

9. Advanced Variational Inference

• Amortized Population Gibbs Samplers with Neural Sufficient Statistics
• Evaluating Combinatorial Generalization in Variational Autoencoders

10. Causal Inference

• Elements of Causal Inference, pdf
• Causal Inference: What If, pdf
• Theoretical Impediments to Machine Learning With Seven Sparks from the Causal Revolution