Algorithm Whiteboard Resources
This is where we share notebooks and projects used in our youtube channel.
Video 1: DIET Architecture - How it Works
This video explains the parts of the DIET architecture. It does not discuss any code.
Video 2: DIET Architecture - Design Decisions
This video explains the parts of the DIET architecture. It does not discuss any code.
Video 3: DIET Architecture - Benchmarks
In this video we make changes to a configuration file. The configuration files, the streamlit application as well as an instructions manual can be found in the diet folder.
Video 4: Word Embeddings - Letter Embeddings
In this video we demonstrate how to train letter embeddings in order to gain intuition on what word embeddings are.
The kaggle dataset that we use in this video can be found here.
We've added the two notebooks in this repo in the letter-embeddings folder. But you can also run them yourself in google colab. The notebooks are mostly identical but the v1 notebook only uses one token to predict the next one while v2 uses two tokens to predict the next one.
Notebook with one token input:
Notebook with two token input:
Video 5: Word Embeddings - CBOW & SkipGram
This video explains two algorithms but it does not discuss any code.
Video 6: Word Embeddings - GloVe
This video discusses GloVe but also offers code to train a variant of your own. The keras model can be found in the glove folder.
The glove.py file contains just the keras algorithm while the notebook
contains the full code. You can also go online to colab
and play with the full notebook from there.
The full notebook:
Video 7: Word Embeddings - WhatLies
This video discusses a small visualisation package we've open sourced. The documentation for it can be found here.
The notebook that we made in this video can be found in the whatlies folder.
Video 8: Attention - Self Attention
This video discusses the idea behind attention (you may notice some similarities with a convolution) but it does not discuss any code.
Video 9: Attention: Keys, Values, Queries
This video discusses how you can add more context to the self attention mechanism by introducing layers. This video does not discuss any code though.
Video 10: Attention: Multi Head Attention
This video explains how you can increase the potential of attention by introducing multiple layers of keys, queries and values. The video does not discuss any code though.
Video 11: Attention: Transformers
Given the lessons from the previous videos, this video wraps everything together by combining everything into a transformer block. There is no code for this video.
Video 12: StarSpace
This video discusses the StarSpace algorithm. The video serves as an introduction to the TED policy. This video contains no code.
Video 13: TED Policy
This video only discusses the theory behind the TED algorithm. The next video will show how TED more on a practical level. This video contains no code.
Video 14: TED Policy in Practice
This video makes use of a rasa project that can be found here. By tuning the history hyperparameter we see how the chatbot is able
to deal with context switches over a long period in the dialogue.
Video 15: Response Selection
This video explains how a response selection model might make your model more accurate in a FAQ/Chitchat scenario. There is no code for this video.
Video 16: Response Selection
This video explains how a response selection model is implemented internally. There is no code for this video.
Video 17: CountVectors
This video explains why CountVectors are still the unsung hero of natural language processing. There is no code attachment for this video.
Video 18: Subword Embeddings
This video tries to combine the ideas from word embeddings with the idea of countvectors. To reproduce, check out whatlies.
Video 19: Subword Implementation
This video explains how you might implement subword embeddings from a neural network design perspective. There is no code for this video.
Video 20: BytePair Embeddings
This video explains how BytePair embeddings work. If you want to use these embeddings in Rasa please check out rasa-nlu-examples.
Video 21: Levenshtein Vectors
This video explains how count vector mights be turned from sparse into dense layers. While doing this, we also learn that these vectors also encode levensthein distance.
Video 22: Bias in Word Embeddings
This video explains how you might measure gender bias in word embeddings. It's part of a larger series and the code for it can be found in the bias folder of this repository.
Video 23: De-Biasing Projections
There's a lot of research on how we might remove bias from word-embeddings. In this video we'll discuss one such technique. For the code, check the bias folder of this repository.
Video 24: Remain Careful with Debiasing
In this video we explain why de-biasing techniques have limits. For the code, check the bias folder of this repository.
Video 25: Why Debiasing is Hard
In this video we explain why de-biasing techniques have limits. For the code, check the bias folder of this repository.
Video 26: Word Analogies
In this video we explain why "word analogies" don't really work by merely applying arithmetic on word-vectors. For the code, check the analogies folder of this repository.
Video 27: Toxic Language
In this video we explain why detecting toxic language is harder than it might seem. Code for the video can be found in the toxic folder in this repository.
Video 28: Lexical Ambiguity
In this video we explain why detecting, in general, NLP models fall short. Models don't really understand language, they merely model it.
Video 29: Fallback Detection
It's important to understand the limits of our models. They can sometimes tell us when they're uncertain about a prediction and this information should not be ignored.
Video 30: Language Detection
What might an assistant do if it sees a text from a language that it isn't trained on? It might make assumptions because it's unlike anything it has seen before and a standard fallback mechanism might not be able to pick it up.
For the code, check the language folder of this repository.
Video 31: Incremental Training
Sometimes we don't need to completely retrain our algorithms. At times we can just finetune on new data. In this video we explain how that might be done with DIET.
Video 31: Bulk Labelling UI
This video demonstrates a new feature in our bulk labelling demo. The code can be found here.
Video 32: Language Agnostic BERT (LaBSE)
In this episode, I'll discuss how you might tweak the standard BERT model to accommodate multiple languages at the same time. We'll also demonstrate a pre-trained model that you can use right away! If you're interested in the paper, you can find it here.
Video 33: Iterate on Data
Instead of debugging a model, it might be much more effective to consider debugging your data. In this video, we'll discuss some techniques that you can start with while also demonstrating some new features in Rasa X.
Video 34: Meaningful Benchmarks
It's easy to get distracted when you go down the rabbit hole of performance statistics. But! Not every impressive benchmark is meaningful and it's important to make the distinction. In this video, we're going to explore one benchmark to demonstrate what we mean by this.
The code for this can be found in the intent-benchmark folder.
Video 35: Model Confidence
If we're going to apply a fallback, we better make sure that we have a good measure for confidence. In this video we explain an update that we've made to DIET that makes the confidence measure a more representative number.
Video 36: FlashText Entity Extraction
If we're going to apply a fallback, we better make sure that we have a good measure for confidence. In this video we explain an update that we've made to DIET that makes the confidence measure a more representative number.