Deep Learning with Tensorflow 2.0

Getting Started • About • Table of Contents • Donate • Acknowledgment • FAQ •

Made by Mukesh Mithrakumar • 🌌 https://mukeshmithrakumar.com

This is the GitHub version of the Deep Learning with Tensorflow 2.0 by Mukesh Mithrakumar. Feel free to watch for updates, you can also follow me to get notified when I make a new post.

📋 Getting Started

• Read the book in its entirety online at https://www.adhiraiyan.org/DeepLearningWithTensorflow.html

• Run the code using the Jupyter notebooks available in this repository's notebooks directory.

• Launch executable versions of these notebooks using Google Colab:

• Launch a live notebook server with these notebooks using binder:

About

This Book is a practical guide to Deep Learning with Tensorflow 2.0. We will be using the Deep Learning Book by Ian Goodfellow as our guide. Ian Goodfellows' Deep Learning Book is an excellent, comprehensive textbook on deep learning that I found so far but this book can be challenging because this is a highly theoretical book written as an academic text and the best way to learn these concepts would be by practicing it, working on problems and solving programming examples which resulted in me writing Deep Learning with Tensorflow 2.0 as a practical guide with explanations for complex concepts, summaries for others and practical examples and exercises in Tensorflow 2.0 to help anyone with limited mathematics, machine learning and programming background to get started.

Read more about the book in Introduction.

Finally I would like to ask for your help, this Book is for you, and I would love to hear from you, if you need more explanations, have doubts on certain sections, many others will feel the same so please feel free to reach out to me via:

with your questions, comments or even if you just want to say Hi.

Table of Contents

^{▴ Back to top}

0. Index

1. Introduction

01.00 Preface 01.01 Introduction 01.02 Who should read this book 01.03 A Short History of Deep Learning

2. Linear Algebra

02.01 Scalars, Vectors, Matrices and Tensors 02.02 Multiplying Matrices and Vectors 02.03 Identity and Inverse Matrices 02.04 Linear Dependence and Span 02.05 Norms 02.06 Special Kinds of Matrices and Vectors 02.07 Eigendecomposition 02.08 Singular Value Decomposition 02.09 The Moore-Penrose Pseudoinverse 02.10 The Trace Operator 02.11 The Determinant 02.12 Example: Principal Components Analysis

3. Probability and Information Theory

03.01 Why Probability? 03.02 Random Variables 03.03 Probability Distributions 03.04 Marginal Probability 03.05 Conditional Probability 03.06 The Chain Rule of Conditional Probabilities 03.07 Independence and Conditional Independence 03.08 Expectation, Variance and Covariance 03.09 Common Probability Distributions 03.10 Useful Properties of Common Functions 03.11 Bayes' Rule 03.12 Technical Details of Continuous Variables 03.13 Information Theory 03.14 Structured Probabilistic Models

4. Numerical Computation

04.01 Overflow and Underflow 04.02 Poor Conditioning 04.03 Gradient-Based Optimization 04.04 Constrained Optimization 04.05 Example: Linear Least Squares

5. Machine Learning Basics

05.01 Learning Algorithms 05.02 Capacity, Overfitting and Underfitting 05.03 Hyperparameters and Validation Sets 05.04 Estimators, Bias and Variance 05.05 Maximum Likelihood Estimation 05.06 Bayesian Statistics 05.07 Supervised Learning Algorithms 05.08 Unsupervised Learning Algorithms 05.09 Stochastic Gradient Descent 05.10 Building a Machine Learning Algorithm 05.11 Challenges Motivating Deep Learning

6. Deep Feedforward Networks

06.01 Example: Learning XOR 06.02 Gradient-Based Learning 06.03 Hidden Units 06.04 Architecture Design 06.05 Back-Propagation and Other Differentiation Algorithms 06.06 Historical Notes

7. Regularization for Deep Learning

07.01 Parameter Norm Penalties 07.02 Norm Penalties as Constrained Optimization 07.03 Regularization and Under-Constrained Problems 07.04 Dataset Augmentation 07.05 Noise Robustness 07.06 Semi-Supervised Learning 07.07 Multitask Learning 07.08 Early Stopping 07.09 Parameter Tying and Parameter Sharing 07.10 Sparse Representations 07.11 Bagging and Other Ensemble Methods 07.12 Dropout 07.13 Adversarial Training 07.14 Tangent Distance, Tangent Prop and Manifold Tangent Classifier

8. Optimization for Training Deep Models

08.01 How Learning Differs from Pure Optimization 08.02 Challenges in Neural Network Optimization 08.03 Basic Algorithms 08.04 Parameter Initialization Strategies 08.05 Algorithms with Adaptive Learning Rates 08.06 Approximate Second-Order Methods 08.07 Optimization Strategies and Meta-Algorithms

9. Convolutional Networks

09.01 The Convolution Operation 09.02 Motivation 09.03 Pooling 09.04 Convolution and Pooling as an Infinitely Strong Prior 09.05 Variants of the Basic Convolution Function 09.06 Structured Outputs 09.07 Data Types 09.08 Efficient Convolution Algorithms 09.09 Random or Unsupervised Features 09.10 The Neuroscientific Basis for Convolutional Networks 09.11 Convolutional Networks and the History of Deep Learning

10. Sequence Modeling: Recurrent and Recursive Nets

10.01 Unfolding Computational Graphs 10.02 Recurrent Neural Networks 10.03 Bidirectional RNNs 10.04 Encoder-Decoder Sequence-to-Sequence Architectures 10.05 Deep Recurrent Networks 10.06 Recursive Neural Networks 10.07 The Challenge of Long-Term Dependencies 10.08 Echo State Networks 10.09 Leaky Units and Other Strategies for Multiple Time Scales 10.10 The Long Short-Term Memory and Other Gated RNNs 10.11 Optimization for Long-Term Dependencies 10.12 Explicit Memory

11. Practical Methodology

11.01 Performance Metrics 11.02 Default Baseline Models 11.03 Determining Whether to Gather More Data 11.04 Selecting Hyperparameters 11.05 Debugging Strategies 11.06 Example: Multi-Digit Number Recognition

12. Applications

12.01 Large-Scale Deep Learning 12.02 Computer Vision 12.03 Speech Recognition 12.04 Natural Language Processing 12.05 Other Applications

13. Linear Factor Models

13.01 Probabilistic PCA and Factor Analysis 13.02 Independent Component Analysis 13.03 Slow Feature Analysis 13.04 Sparse Coding 13.05 Manifold Interpretation of PCA

14. Autoencoders

14.01 Undercomplete Autoencoders 14.02 Regularized Autoencoders 14.03 Representational Power, Layer Size and Depth 14.04 Stochastic Encoders and Decoders 14.05 Denoising Autoencoders 14.06 Learning Manifolds with Autoencoders 14.07 Contractive Autoencoders 14.08 Predictive Sparse Decomposition 14.09 Applications of Autoencoders

15. Representation Learning

15.01 Greedy Layer-Wise Unsupervised Pretraining 15.02 Transfer Learning and Domain Adaptation 15.03 Semi-Supervised Disentangling of Causal Factors 15.04 Distributed Representation 15.05 Exponential Gains from Depth 15.06 Providing Clues to Discover Underlying Causes

16. Structured Probabilistic Models for Deep Learning

16.01 The Challenge of Unstructured Modeling 16.02 Using Graphs to Describe Model Structure 16.03 Sampling from Graphical Models 16.04 Advantages of Structured Modeling 16.05 Learning about Dependencies 16.06 Inference and Approximate Inference 16.07 The Deep Learning Approach to Structured Probabilistic Models

17. Monte Carlo Methods

17.01 Sampling and Monte Carlo Methods 17.02 Importance Sampling 17.03 Markov Chain Monte Carlo Methods 17.04 Gibbs Sampling 17.05 The Challenge of Mixing between Separated Modes

18. Confronting the Partition Function

18.01 The Log-Likelihood Gradient 18.02 Stochastic Maximum Likelihood and Contrastive Divergence 18.03 Pseudolikelihood 18.04 Score Matching and Ratio Matching 18.05 Denoising Score Matching 18.06 Noise-Contrastive Estimation 18.07 Estimating the Partition Function

19. Approximate Inference

19.01 Inference as Optimization 19.02 Expectation Maximization 19.03 MAP Inference and Sparse Coding 19.04 Variational Inference and Learning 19.05 Learned Approximate Inference

20. Deep Generative Models

20.01 Boltzmann Machines 20.02 Restricted Boltzmann Machines 20.03 Deep Belief Networks 20.04 Deep Boltzmann Machines 20.05 Boltzmann Machines for Real-Valued Data 20.06 Convolutional Boltzmann Machines 20.07 Boltzmann Machines for Structured or Sequential Outputs 20.08 Other Boltzmann Machines 20.09 Back-Propagation through Random Operations 20.10 Directed Generative Nets 20.11 Drawing Samples from Autoencoders 20.12 Generative Stochastic Networks 20.13 Other Generation Schemes 20.14 Evaluating Generative Models 20.15 Conclusion

Acknowledgment

^{▴ Back to top}

To cite the Deep Learning Book by GoodFellow, please use this bibtex entry:

@book{Goodfellow-et-al-2016,
    title={Deep Learning},
    author={Ian Goodfellow and Yoshua Bengio and Aaron Courville},
    publisher={MIT Press},
    note={\url{http://www.deeplearningbook.org}},
    year={2016}
}

To cite the Deep Learning with Tensorflow 2.0 Book by Mukesh Mithrakumar, please use this bibtex entry:

@book{MukeshMithrakumar-2019,
    title={Deep Learning with Tensorflow 2.0},
    author={Mukesh Mithrakumar},
    note={\url{https://github.com/adhiraiyan/DeepLearningWithTF2.0}},
    year={2019}
}

💬 FAQ

^{▴ Back to top}