pytest-notebook

A pytest plugin for regression testing and regenerating Jupyter Notebooks.

Purpose

The purpose of the plugin is to ensure that changes to the python environment (e.g. updating packages), have not affected the outputs of the notebook. If the notebook has changed, this plugin can also regenerate the notebooks, saving the new outputs.

Features

• Recognise, collect, execute (optionally output) then diff input vs. output Jupyter Notebooks.

• Provides clear and colorized diffs of the notebooks (using nbdime)

• Regenerate failing notebooks.

• Integration with coverage and pytest_cov.

• A well defined API allows notebook regression tests to be run:

  1. Using the pytest test collection architecture.
  2. As a pytest fixtures (nb_regression.check(filename)).
  3. Using the pytest_notebook python package.

• All stages are highly configurable via:

  1. The pytest command-line interface.
  2. The pytest configuration file.
  3. The notebook and cell level metadata.

• Post-processor plugin entry-points, allow for customisable modifications of the notebook, including source code formatting with black

Installation

To install from Conda (recommended):

>> conda install -c conda-forge pytest-notebook

To install via pip from PyPI:

>> pip install pytest-notebook

To install the development version:

>> git clone https://github.com/chrisjsewell/pytest-notebook .
>> cd pytest-notebook
>> pip install --upgrade pip
>> pip install -e .
>> # pip install -e .[code_style,testing,docs] # install extras for more features

Usage

See the documentation at: http://pytest-notebook.readthedocs.io/

If you want to test some sample notebooks, add the --nb-test-files flag:

>> git clone https://github.com/chrisjsewell/pytest-notebook
>> cd pytest-notebook/samples
>> pip install pytest ipykernel pytest-notebook
>> pytest --nb-test-files *.ipynb

You should see success.ipynb passed the test and fail.ipynb failed the test.

You can regenerate the outputs of fail.ipynb with the --nb-force-regen flag:

>> pytest --nb-test-files --nb-force-regen *.ipynb

Contributing

Contributions are very welcome.

The following will discover and run all unit test:

>> pip install -e .[testing]
>> pytest -v

Coding Style Requirements

The code style is tested using flake8, with the configuration set in .flake8, and code should be formatted with black.

Installing with pytest-notebook[code_style] makes the pre-commit package available, which will ensure these tests are passed by reformatting the code and testing for lint errors before submitting a commit. It can be setup by:

>> cd pytest-notebook
>> pre-commit install

Optionally you can run black and flake8 separately:

>> black .
>> flake8 .

Editors like VS Code also have automatic code reformat utilities, which can adhere to this standard.

License

Distributed under the terms of the BSD-3 license, pytest-notebook is free and open source software.

Issues

If you encounter any problems, please file an issue along with a detailed description.

Acknowledgements

• nbdime
• nbval