GitPlanet

Cheap and reliable Node.js hosting starts at $3/month, and $1/month static HTML hosting

Created with love in Canada, visit hostnodejs.com today

Feel like to post an Ad? Learn Details
All Projects → alibaba → HybridBackend

alibaba / HybridBackend

Licence: Apache-2.0 License
Efficient training of deep recommenders on cloud.

Programming Languages

python
139335 projects - #7 most used programming language
C++
36643 projects - #6 most used programming language
shell
77523 projects
Makefile
30231 projects

Labels

deep-learninggpuparquetrecommender-systemhybrid-parallelism

Projects that are alternatives of or similar to HybridBackend

glip-lib
An OpenGL Image Processing Library (in C++/GLSL).
Stars: ✭ 14 (-53.33%)
Mutual labels:  gpu
meepo
异构存储数据迁移
Stars: ✭ 29 (-3.33%)
Mutual labels:  parquet
briefmatch
BriefMatch real-time GPU optical flow
Stars: ✭ 36 (+20%)
Mutual labels:  gpu
BifurcationKit.jl
A Julia package to perform Bifurcation Analysis
Stars: ✭ 185 (+516.67%)
Mutual labels:  gpu
experiments
Code examples for my blog posts
Stars: ✭ 21 (-30%)
Mutual labels:  parquet
peakperf
Achieve peak performance on x86 CPUs and NVIDIA GPUs
Stars: ✭ 33 (+10%)
Mutual labels:  gpu
FGPU
No description or website provided.
Stars: ✭ 30 (+0%)
Mutual labels:  gpu
monolish
monolish: MONOlithic LInear equation Solvers for Highly-parallel architecture
Stars: ✭ 166 (+453.33%)
Mutual labels:  gpu
ELM-pytorch
Extreme Learning Machine implemented in Pytorch
Stars: ✭ 68 (+126.67%)
Mutual labels:  gpu
lambdacube-quake3
Quake 3 map viewer in Haskell using LambdaCube 3D
Stars: ✭ 66 (+120%)
Mutual labels:  gpu
Recommender-Systems-with-Collaborative-Filtering-and-Deep-Learning-Techniques
Implemented User Based and Item based Recommendation System along with state of the art Deep Learning Techniques
Stars: ✭ 41 (+36.67%)
Mutual labels:  recommender-system
MARank
Multi-order Attentive Ranking Model for Sequential Recommendation
Stars: ✭ 25 (-16.67%)
Mutual labels:  recommender-system
gpubootcamp
This repository consists for gpu bootcamp material for HPC and AI
Stars: ✭ 227 (+656.67%)
Mutual labels:  gpu
RenderScriptOps
🚀 TensorFlow running with RenderScript on Android GPU
Stars: ✭ 15 (-50%)
Mutual labels:  gpu
bifrost
A stream processing framework for high-throughput applications.
Stars: ✭ 48 (+60%)
Mutual labels:  gpu
session4rec
GRu4Rec in TensorFlow
Stars: ✭ 14 (-53.33%)
Mutual labels:  recommender-system
centurion
Kotlin Bigdata Toolkit
Stars: ✭ 320 (+966.67%)
Mutual labels:  parquet
recommendation-api
Rank and serve recommendations
Stars: ✭ 14 (-53.33%)
Mutual labels:  recommender-system
RecSysDatasets
This is a repository of public data sources for Recommender Systems (RS).
Stars: ✭ 272 (+806.67%)
Mutual labels:  recommender-system
fun-rec
推荐系统入门教程，在线阅读地址：https://datawhalechina.github.io/fun-rec/
Stars: ✭ 1,367 (+4456.67%)
Mutual labels:  recommender-system
View All Similar Projects ➔

HybridBackend

License PRs Welcome cibuild: cpu Documentation Status

Introduction

HybridBackend is a training framework for deep recommenders which bridges the gap between evolving cloud infrastructure and complex training process. See documentation for more information.

bridging

Requirements

For Linux/macOS installation:

  • Ubuntu 18.04 or later (64-bit)
  • Python 3.6 or later
  • Pip 19.0 or later
  • TensorFlow 1.15 or TensorFlow 1.14
  • For GPU support, CUDA SDK 11.3 or later is required
  • Docker is the future

For docker-phobes using macOS:

  • macOS 11.0 or later (x86 64-bit)
  • Python 3.7 or later
  • Pip 19.0 or later
  • Tebnsorflow 1.15 or TensorFlow 1.14
  • Other libraries installed by brew

Install

For TensorFlow 1.15 CPU version:

pip install hybridbackend-cpu

For TensorFlow 1.14 CPU version:

pip install hybridbackend-cpu-legacy

For GPU support:

PAI DLC docker images are prefered to use.

Build from source on Linux/macOS w/ docker

  • Fetch source from git and sync submodules.
git submodule sync
git submodule update --init
  • Step into developer docker.
# Environment variable `DOCKER_RUN_IMAGE` can be used to replace
# developer docker.
cibuild/run
  • Configure.
# Only build CPU releated functions.
export HYBRIDBACKEND_WITH_CUDA=OFF
# For TensorFlow 1.14, zero-copy is not supported.
export HYBRIDBACKEND_WITH_ARROW_ZEROCOPY=OFF
# Use below command to verify C++ ABI of installed TensorFlow.
python -c 'import tensorflow as tf; print(tf.sysconfig.get_compile_flags())'
# Must be consistent with installed TensorFlow.
export HYBRIDBACKEND_USE_CXX11_ABI=0
  • Build.
cibuild/run make -j8

Build from source on Linux w/o docker

  • Fetch source from git and sync submodules.
git submodule sync
git submodule update --init
  • Install TensorFlow and other requirements.

See Dockerfiles.

  • Configure.
# Only build CPU releated functions.
export HYBRIDBACKEND_WITH_CUDA=OFF
# For TensorFlow 1.15, zero-copy is supported.
export HYBRIDBACKEND_WITH_ARROW_ZEROCOPY=ON
# Use below command to verify C++ ABI of installed TensorFlow.
python -c 'import tensorflow as tf; print(tf.sysconfig.get_compile_flags())'
# Must be consistent with installed TensorFlow.
export HYBRIDBACKEND_USE_CXX11_ABI=0
  • Build.
make -j8

Build from source on macOS w/o docker

  • Fetch source from git and sync submodules.
git submodule sync
git submodule update --init
  • Install TensorFlow and other requirements.
brew install wget [email protected] [email protected] utf8proc zstd snappy re2 thrift zlib
brew uninstall grpc abseil || true
export PATH="/usr/local/opt/[email protected]/bin:$PATH"
export LDFLAGS="-L/usr/local/opt/[email protected]/lib"
export PKG_CONFIG_PATH="/usr/local/opt/[email protected]/lib/pkgconfig"

pip3.7 install -i https://mirrors.aliyun.com/pypi/simple/ \
    tensorflow==1.14 \
    "pybind11[global]"
  • Configure.
# Only build CPU releated functions.
export HYBRIDBACKEND_WITH_CUDA=OFF
# For TensorFlow 1.14, zero-copy is not supported.
export HYBRIDBACKEND_WITH_ARROW_ZEROCOPY=OFF
# Use below command to verify C++ ABI of installed TensorFlow.
python -c 'import tensorflow as tf; print(tf.sysconfig.get_compile_flags())'
# Must be consistent with installed TensorFlow.
export HYBRIDBACKEND_USE_CXX11_ABI=0

# Set path of thridparty libraries.
export PYTHON=python3.7
export PYTHON_HOME=/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/Current
export PYTHON_IMPL=python3.7
export PYTHON_IMPL_FLAG=m
export SSL_HOME=/usr/local/opt/[email protected]
export RE2_HOME=/usr/local/opt/re2
export THRIFT_HOME=/usr/local/opt/thrift
export UTF8PROC_HOME=/usr/local/opt/utf8proc
export SNAPPY_HOME=/usr/local/opt/snappy
export ZSTD_HOME=/usr/local/opt/zstd
export ZLIB_HOME=/usr/local/opt/zlib
  • Build.
make -j8

Contributing

We appreciate all contributions to improve HybridBackend. Please see Contributing Guide for more details.

Community

If you are intrested in adoption of HybridBackend in your organization, you can add your organization name to our list of adopters by submitting a pull request. We will discuss new feature requirements with you in advance.

Further more, if you would like to share your experiences with others, you are welcome to contact us in DingTalk:

community

License

HybridBackend is licensed under the Apache 2.0 License.

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].