GitPlanet

Cheap and reliable Node.js hosting starts at $3/month, and $1/month static HTML hosting

Created with love in Canada, visit hostnodejs.com today

Feel like to post an Ad? Learn Details
All Projects → Oneflow-Inc → libai

Oneflow-Inc / libai

Licence: Apache-2.0 license
LiBai(李白): A Toolbox for Large-Scale Distributed Parallel Training

Programming Languages

python
139335 projects - #7 most used programming language
C++
36643 projects - #6 most used programming language

Labels

nlpdeep-learningtransformerlarge-scaledata-parallelismmodel-parallelismdistributed-trainingself-supervised-learningoneflowpipeline-parallelismvision-transformer

Projects that are alternatives of or similar to libai

YOLOS
You Only Look at One Sequence (NeurIPS 2021)
Stars: ✭ 612 (+115.49%)
Mutual labels:  transformer, vision-transformer
image-classification
A collection of SOTA Image Classification Models in PyTorch
Stars: ✭ 70 (-75.35%)
Mutual labels:  transformer, vision-transformer
pytorch-gpt-x
Implementation of autoregressive language model using improved Transformer and DeepSpeed pipeline parallelism.
Stars: ✭ 21 (-92.61%)
Mutual labels:  transformer, pipeline-parallelism
transformer-ls
Official PyTorch Implementation of Long-Short Transformer (NeurIPS 2021).
Stars: ✭ 201 (-29.23%)
Mutual labels:  transformer, vision-transformer
SwinIR
SwinIR: Image Restoration Using Swin Transformer (official repository)
Stars: ✭ 1,260 (+343.66%)
Mutual labels:  transformer, vision-transformer
semantic-segmentation
SOTA Semantic Segmentation Models in PyTorch
Stars: ✭ 464 (+63.38%)
Mutual labels:  transformer, vision-transformer
towhee
Towhee is a framework that is dedicated to making neural data processing pipelines simple and fast.
Stars: ✭ 821 (+189.08%)
Mutual labels:  transformer, vision-transformer
BossNAS
(ICCV 2021) BossNAS: Exploring Hybrid CNN-transformers with Block-wisely Self-supervised Neural Architecture Search
Stars: ✭ 125 (-55.99%)
Mutual labels:  transformer, self-supervised-learning
SIGIR2021 Conure
One Person, One Model, One World: Learning Continual User Representation without Forgetting
Stars: ✭ 23 (-91.9%)
Mutual labels:  transformer, self-supervised-learning
LaTeX-OCR
pix2tex: Using a ViT to convert images of equations into LaTeX code.
Stars: ✭ 1,566 (+451.41%)
Mutual labels:  transformer, vision-transformer
PLSC
Paddle Large Scale Classification Tools，supports ArcFace, CosFace, PartialFC, Data Parallel + Model Parallel. Model includes ResNet, ViT, DeiT, FaceViT.
Stars: ✭ 113 (-60.21%)
Mutual labels:  large-scale, distributed-training
keras-vision-transformer
The Tensorflow, Keras implementation of Swin-Transformer and Swin-UNET
Stars: ✭ 91 (-67.96%)
Mutual labels:  transformer, vision-transformer
Deepspeed
DeepSpeed is a deep learning optimization library that makes distributed training easy, efficient, and effective.
Stars: ✭ 6,024 (+2021.13%)
Mutual labels:  data-parallelism, model-parallelism
TransMorph Transformer for Medical Image Registration
TransMorph: Transformer for Unsupervised Medical Image Registration (PyTorch)
Stars: ✭ 130 (-54.23%)
Mutual labels:  transformer, vision-transformer
PASSL
PASSL包含 SimCLR，MoCo v1/v2，BYOL，CLIP，PixPro，BEiT，MAE等图像自监督算法以及 Vision Transformer，DEiT，Swin Transformer，CvT，T2T-ViT，MLP-Mixer，XCiT，ConvNeXt，PVTv2 等基础视觉算法
Stars: ✭ 134 (-52.82%)
Mutual labels:  self-supervised-learning, vision-transformer
visualization
a collection of visualization function
Stars: ✭ 189 (-33.45%)
Mutual labels:  transformer, vision-transformer
Ghostnet
CV backbones including GhostNet, TinyNet and TNT, developed by Huawei Noah's Ark Lab.
Stars: ✭ 1,744 (+514.08%)
Mutual labels:  transformer, vision-transformer
VT-UNet
[MICCAI2022] This is an official PyTorch implementation for A Robust Volumetric Transformer for Accurate 3D Tumor Segmentation
Stars: ✭ 151 (-46.83%)
Mutual labels:  transformer, vision-transformer
bytekit
Java 字节操作的工具库(不是字节码的工具库)
Stars: ✭ 40 (-85.92%)
Mutual labels:  transformer
ru-dalle
Generate images from texts. In Russian
Stars: ✭ 1,606 (+465.49%)
Mutual labels:  transformer
View All Similar Projects ➔

LiBai

docs GitHub GitHub release PRs Welcome Python Checks Docs Release Status

Introduction

English | 简体中文

LiBai is a large-scale open-source model training toolbox based on OneFlow. The main branch works with OneFlow 0.7.0.

Highlights

  • Support a collection of parallel training components

    LiBai provides multiple parallelisms such as Data Parallelism, Tensor Parallelism, and Pipeline Parallelism. It's also extensible for other new parallelisms.

  • Varied training techniques

    LiBai provides many out-of-the-box training techniques such as Distributed Training, Mixed Precision Training, Activation Checkpointing, Recomputation, Gradient Accumulation, and Zero Redundancy Optimizer(ZeRO).

  • Support for both CV and NLP tasks

    LiBai has predifined data process for both CV and NLP datasets such as CIFAR, ImageNet, and BERT Dataset.

  • Easy to use

    LiBai's components are designed to be modular for easier usage as follows:

    • LazyConfig system for more flexible syntax and no predefined structures
    • Friendly trainer and engine
    • Used as a library to support building research projects on it. See projects/ for some projects that are built based on LiBai

  • High Efficiency

Installation

See Installation instructions.

Getting Started

See Quick Run for the basic usage of LiBai.

Documentation

See LiBai's documentation for full API documentation and tutorials.

ChangeLog

Beta 0.2.0 was released in 07/07/2022, the general changes in 0.2.0 version are as follows:

Features:

  • Support evaluation enabled and set eval_iter
  • Support customized sampler in config.py
  • Support rdma for pipeline-model-parallel
  • Support multi fused kernel
    • fused_scale_mask_softmax_dropout
    • fused_scale_tril_softmax_mask_scale
    • fused_self_attention in branch libai_bench
  • User Experience Optimization
  • Optimization for training throughput, see benchmark for more details

Supported Models:

  • Support 3D parallel Roberta model
  • Support 2D parallel (data parallel + tensor model parallel) SimCSE model
  • Support Data parallel MAE model
  • Support Data parallel MOCOV3 model

See changelog for details and release history.

Contributing

We appreciate all contributions to improve LiBai. See CONTRIBUTING for the contributing guideline.

License

This project is released under the Apache 2.0 license.

Citation

If you find this project useful for your research, consider cite:

@misc{of2021libai,
  author =       {Xingyu Liao and Peng Cheng and Tianhe Ren and Depeng Liang and
                  Kai Dang and Yi Wang and Xiaoyu Xu},
  title =        {LiBai},
  howpublished = {\url{https://github.com/Oneflow-Inc/libai}},
  year =         {2021}
}

Join the WeChat group

LiBai_Wechat_QRcode

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].