CortexNet

This repo contains the PyTorch implementation of CortexNet.
Check the project website for further information.

Project structure

The project consists of the following folders and files:

• data/: contains Bash scripts and a Python class definition inherent video data loading;
• image-pretraining/: hosts the code for pre-training TempoNet's discriminative branch;
• model/: stores several network architectures, including PredNet, an additive feedback Model01, and a modulatory feedback Model02 (CortexNet);
• notebook/: collection of Jupyter Notebooks for data exploration and results visualisation;
• utils/: scripts for
  • (current or former) training error plotting,
  • experiments diff,
  • multi-node synchronisation,
  • generative predictions visualisation,
  • network architecture graphing;
• [email protected]: link to the location where experimental results will be saved within 3-digit folders;
• new_experiment.sh*: creates a new experiment folder, updates [email protected], prints a memo about last used settings;
• [email protected]: symbolic link pointing to a new results sub-directory created by new_experiment.sh;
• main.py: training script for CortexNet in MatchNet or TempoNet configuration;

Dependencies

• scikit-video: accessing images / videos

pip install sk-video

• tqdm: progress bar

conda config --add channels conda-forge
conda update --all
conda install tqdm

IDE

This project has been realised with PyCharm by JetBrains and the Vim editor. Grip has been also fundamental for crafting decent documtation locally.

Initialise environment

Once you've determined where you'd like to save your experimental results — let's call this directory <my saving location> — run the following commands from the project's root directory:

ln -s <my saving location> results  # replace <my saving location>
mkdir results/000 && touch results/000/train.log  # init. placeholder
ln -s results/000 last  # create pointer to the most recent result

Setup new experiment

Ready to run your first experiment? Type the following:

./new_experiment.sh

GPU selection

Let's say your machine has N GPUs. You can choose to use any of these, by specifying the index n = 0, ..., N-1. Therefore, type CUDA_VISIBLE_DEVICES=n just before python ... in the following sections.

Train MatchNet

• Download e-VDS35 (e.g. e-VDS35-May17.tar) from here.
• Use data/resize_and_split.sh to prepare your (video) data for training. It resizes videos present in folders of folders (i.e. directory of classes) and may split them into training and validation set. May also skip short videos and trim longer ones. Check data/README.md for more details.
• Run the main.py script to start training. Use -h to print the command line interface (CLI) arguments help.

python -u main.py --mode MatchNet <CLI arguments> | tee last/train.log

Train TempoNet

• Download e-VDS35 (e.g. e-VDS35-May17.tar) from here.
• Pre-train the forward branch (see image-pretraining/) on an image data set (e.g. 33-image-set.tar from here);
• Use data/resize_and_sample.sh to prepare your (video) data for training. It resizes videos present in folders of folders (i.e. directory of classes) and samples them. Videos are then distributed across training and validation set. May also skip short videos and trim longer ones. Check data/README.md for more details.
• Run the main.py script to start training. Use -h to print the CLI arguments help.

python -u main.py --mode TempoNet --pre-trained <path> <CLI args> | tee last/train.log

GPU selection

To run on a specific GPU, say n, type CUDA_VISIBLE_DEVICES=n just before python ....