PyTorch Transformers Tutorials

Introduction

The field of NLP was revolutionized in the year 2018 by introduction of BERT and his Transformer friends(RoBerta, XLM etc.).

These novel transformer based neural network architectures and new ways to training a neural network on natural language data introduced transfer learning to NLP problems. Transfer learning had been giving out state of the art results in the Computer Vision domain for a few years now and introduction of transformer models for NLP brought about the same paradigm change in NLP.

Companies like Google and Facebook trained their neural networks on large swathes of Natural Language Data to grasp the intricacies of language thereby generating a Language model. Finally these models were fine tuned to specific domain dataset to achieve state of the art results for a specific problem statement. They also published these trained models to open source community. The community members were now able to fine tune these models to their specific use cases.

Hugging Face made it easier for community to access and fine tune these models using their Python Package: Transformers.

Motivation

Despite these amazing technological advancements applying these solutions to business problems is still a challenge given the niche knowledge required to understand and apply these method on specific problem statements. Hence, In the following tutorials i will be demonstrating how a user can leverage technologies along with some other python tools to fine tune these Language models to specific type of tasks.

Before i proceed i will like to mention the following groups for the fantastic work they are doing and sharing which have made these notebooks and tutorials possible:

Please review these amazing sources of information and subscribe to their channels/sources.

• Hugging Face Team
• Abhishek Thakur for his amazing Youtube videos

The problem statements that i will be working with are:

Notebook	Github Link	Colab Link	Kaggle Kernel
Text Classification: Multi-Class	Github		Kaggle
Text Classification: Multi-Label	Github		Kaggle
Sentiment Classification with Experiment Tracking in WandB!	Github
Named Entity Recognition: with TPU processing!	Github		Kaggle
Question Answering
Summary Writing: with Experiment Tracking in WandB!	Github		Kaggle

Directory Structure

1. data: This folder contains all the toy data used for fine tuning.
2. utils: This folder will contain any miscellaneous script used to prepare for the fine tuning.
3. models: Folder to save all the artifacts post fine tuning.

Further Watching/Reading

I will try to cover the practical and implementation aspects of fine tuning of these language models on various NLP tasks. You can improve your knowledge on this topic by reading/watching the following resources.

• Watching

  • Introduction in Simple terms
  • Transfer Learning in NLP
  • BERT Research Series from ChrisMcCormickAI

• Reading

  • Transformers Documentation
  • Pytorch Documentation
  • Google AI Blog