All Projects β†’ dredwardhyde β†’ gpt-neo-fine-tuning-example

dredwardhyde / gpt-neo-fine-tuning-example

Licence: Apache-2.0 license
Fine-Tune EleutherAI GPT-Neo And GPT-J-6B To Generate Netflix Movie Descriptions Using Hugginface And DeepSpeed

Programming Languages

python
139335 projects - #7 most used programming language

Projects that are alternatives of or similar to gpt-neo-fine-tuning-example

gpt-j
A GPT-J API to use with python3 to generate text, blogs, code, and more
Stars: ✭ 101 (-35.67%)
Mutual labels:  gpt-3, gpt-neo-text-generation, gpt-j
gpt-j-api
API for the GPT-J language model 🦜. Including a FastAPI backend and a streamlit frontend
Stars: ✭ 248 (+57.96%)
Mutual labels:  text-generation, gpt-3, gpt-j
finetune-gpt2xl
Guide: Finetune GPT2-XL (1.5 Billion Parameters) and finetune GPT-NEO (2.7 B) on a single GPU with Huggingface Transformers using DeepSpeed
Stars: ✭ 353 (+124.84%)
Mutual labels:  deepspeed, gpt-neo, gpt-neo-fine-tuning
awesome-text-style-transfer
A list of resources about Text Style Transfer
Stars: ✭ 43 (-72.61%)
Mutual labels:  text-generation
pistoBot
Create an AI that chats like you
Stars: ✭ 121 (-22.93%)
Mutual labels:  text-generation
Basic-UI-for-GPT-J-6B-with-low-vram
A repository to run gpt-j-6b on low vram machines (4.2 gb minimum vram for 2000 token context, 3.5 gb for 1000 token context). Model loading takes 12gb free ram.
Stars: ✭ 90 (-42.68%)
Mutual labels:  gpt-neo
Gumbel-CRF
Implementation of NeurIPS 20 paper: Latent Template Induction with Gumbel-CRFs
Stars: ✭ 51 (-67.52%)
Mutual labels:  text-generation
transformer-drg-style-transfer
This repository have scripts and Jupyter-notebooks to perform all the different steps involved in Transforming Delete, Retrieve, Generate Approach for Controlled Text Style Transfer
Stars: ✭ 97 (-38.22%)
Mutual labels:  text-generation
Keras-Generating-Sentences-from-a-Continuous-Space
Text Variational Autoencoder inspired by the paper 'Generating Sentences from a Continuous Space' Bowman et al. https://arxiv.org/abs/1511.06349
Stars: ✭ 32 (-79.62%)
Mutual labels:  text-generation
gap-text2sql
GAP-text2SQL: Learning Contextual Representations for Semantic Parsing with Generation-Augmented Pre-Training
Stars: ✭ 83 (-47.13%)
Mutual labels:  text-generation
Skin Lesions Classification DCNNs
Transfer Learning with DCNNs (DenseNet, Inception V3, Inception-ResNet V2, VGG16) for skin lesions classification
Stars: ✭ 47 (-70.06%)
Mutual labels:  fine-tuning
porn-description-generator
Generates new porn descriptions based on an edited dataset of xhamster video descriptions uploaded between 2007-2016.
Stars: ✭ 40 (-74.52%)
Mutual labels:  text-generation
caffe-char-rnn
Multi-layer Recurrent Neural Networks (with LSTM) for character-level language models in Caffe
Stars: ✭ 25 (-84.08%)
Mutual labels:  text-generation
machine learning course
Artificial intelligence/machine learning course at UCF in Spring 2020 (Fall 2019 and Spring 2019)
Stars: ✭ 47 (-70.06%)
Mutual labels:  fine-tuning
backprop
Backprop makes it simple to use, finetune, and deploy state-of-the-art ML models.
Stars: ✭ 229 (+45.86%)
Mutual labels:  fine-tuning
hangul ipsum
ν•œκΈ€ λ²„μ „μ˜ lorem ipsum 생성기
Stars: ✭ 17 (-89.17%)
Mutual labels:  text-generation
text-generator
Golang text generator for generate SEO texts
Stars: ✭ 18 (-88.54%)
Mutual labels:  text-generation
Neural-Scam-Artist
Web Scraping, Document Deduplication & GPT-2 Fine-tuning with a newly created scam dataset.
Stars: ✭ 18 (-88.54%)
Mutual labels:  fine-tuning
Vehicle-Detection
Vehicle Detection Using Deep Learning and YOLO Algorithm
Stars: ✭ 96 (-38.85%)
Mutual labels:  fine-tuning
kaldi ag training
Docker image and scripts for training finetuned or completely personal Kaldi speech models. Particularly for use with kaldi-active-grammar.
Stars: ✭ 14 (-91.08%)
Mutual labels:  fine-tuning

GPT-Neo-2.7B & GPT-J-6B Fine-Tuning Examples Using HuggingFace & DeepSpeed

medium Python3.8.6 PyTorch1.8.1

Installation

cd venv/bin
./pip install -r ../../requirements.txt 
./pip install deepspeed==0.5.9

GPT-Neo

Example with GPT-Neo-1.3B without DeepSpeed
Training and testing log with GPT-Neo-1.3B
Example with GPT-Neo-2.7B with DeepSpeed
DeepSpeed configuration with GPT-Neo-2.7B
Training and testing log with GPT-Neo-2.7B

GPU VRAM load during GPT-Neo-2.7B training

RAM load during GPT-Neo-2.7B training

Results

GPT-J-6B

Example with GPT-J-6B with DeepSpeed
DeepSpeed configuration with GPT-J-6B
Training and testing log with GPT-J-6B

GPU VRAM load during GPT-J-6B training

RAM load during GPT-J-6B training

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].