MyGrad's Documentation

Introducing mygrad

mygrad is a simple, NumPy-centric autograd library. An autograd library enables you to automatically compute derivatives of mathematical functions. This library is designed to serve both as a tool for prototyping/testing and as an education tool for learning about gradient-based machine learning; it is easy to install, has a readable and easily customizable code base, and provides a sleek interface that mimics NumPy. Furthermore, it leverages NumPy's vectorization to achieve good performance despite the library's simplicity.

This is not meant to be a competitor to libraries like PyTorch (which mygrad most closely resembles) or TensorFlow. Rather, it is meant to serve as a useful tool for students who are learning about training neural networks using back propagation.

Installing mygrad (this project)

To install MyGrad, you can pip-install it:

pip install mygrad

or clone this repository and navigate to the MyGrad directory, then run:

python setup.py install

MyGrad requires numpy. It is highly recommended that you utilize a release of numpy built with MKL for access to optimized math routines.

A Simple Application

Let's use mygrad to compute the derivative of , which is .

mygrad.Tensor behaves nearly identically to NumPy's ndarray, in addition to having the machinery needed to compute the analytic derivatives of functions. Suppose we want to compute this derivative at x = 3. We can create a 0-dimensional tensor (a scalar) for x and compute f(x):

>>> import mygrad as mg
>>> x = mg.Tensor(3.0)
>>> f = x ** 2
>>> f
Tensor(9.0)

Invoking f.backward() instructs mygrad to trace through the computational graph that produced f and compute the derivatives of f with respect to all of its independent variables. Thus, executing f.backward() will compute and will store the value in x.grad:

>>> f.backward()  # triggers computation of `df/dx`
>>> x.grad  # df/dx = 2x = 6.0
array(6.0)

This is the absolute tip of the iceberg. mygrad can compute derivatives of multivariable composite functions of tensor-valued variables!

Some Bells and Whistles

mygrad supports all of NumPy's essential features, including:

• N-dimensional tensors that can be reshaped and have their axes transposed
• vectorization
• broadcasting
• basic and advanced indexing (including all varieties of mixed indexing schemes) for both getting and setting items.
• fully-fledged support for einsum (including broadcasting and traces, which are not supported by PyTorch, TensorFlow, or HIPS-autograd)

mygrad.Tensor plays nicely with NumPy-arrays, which behave as constants when they are used in computational graphs:

>>> import numpy as np
>>> x = mg.Tensor([2.0, 2.0, 2.0])
>>> y = np.array([1.0, 2.0, 3.0])
>>> f = x ** y  # (2 ** 1, 2 ** 2, 2 ** 3)
>>> f.backward()
>>> x.grad
array([ 1.,  4., 12.])

mygrad.nnet supplies essential functions for machine learning, including:

• N-dimensional convolutions (with striding, dilation, and padding)
• N-dimensional pooling
• A gated recurrent unit for sequence-learning (with input-level dropout and variational hidden-hidden dropout)

It leverages a nice sliding window view function, which produces convolution-style windowed views of arrays/tensors without making copies of them, to intuitively (and quite efficiently) perform the neural network-style convolutions and pooling.

Advanced Example

The following is an example of using mygrad to compute the hinge loss of classification scores and to "backpropagate" through (compute the gradient of) this loss. This example demonstrates some of mygrad's ability to perform backpropagation through broadcasted operations, basic indexing, advanced indexing, and in-place assignments.

>>> from mygrad import Tensor
>>> import numpy as np
>>> class_scores = Tensor(10 * np.random.rand(100, 10))         # 100 samples, 10 possible classes for each
>>> class_labels = np.random.randint(low=0, high=10, size=100)  # correct label for each datum
>>> class_labels = (range(len(class_labels)), class_labels)
>>> correct_class_scores = class_scores[class_labels]

>>> Lij = class_scores - correct_class_scores[:, np.newaxis] + 1.  # 100x10 margins
>>> Lij[Lij <= 0] = 0      # scores within the hinge incur no loss
>>> Lij[class_labels] = 0  # the score corresponding to the correct label incurs no loss

>>> loss = Lij.sum() / class_scores.shape[0]  # compute mean hinge loss
>>> loss.backward()    # compute gradient of loss w.r.t all dependent tensors
>>> class_scores.grad  # d(loss)/d(class_scores)
array([[ 0.  ,  0.01,  0.  , -0.04,  0.  ,  0.  ,  0.01,  0.  ,  0.01, 0.01], ...])

Computational Graph Visualization

mygrad uses Graphviz and a Python interface for Graphviz to render the computational graphs built using tensors. These graphs can be rendered in Jupyter notebooks, allowing for quick checks of graph structure, or can be saved to file for later reference.

The dependencies can be installed with:

conda install graphviz
conda install python-graphviz