GitPlanet

Cheap and reliable Node.js hosting starts at $3/month, and $1/month static HTML hosting

Created with love in Canada, visit hostnodejs.com today

Feel like to post an Ad? Learn Details
All Projects → ildoonet → remote-dataloader

ildoonet / remote-dataloader

Licence: MIT license
PyTorch DataLoader processed in multiple remote computation machines for heavy data processings

Programming Languages

python
139335 projects - #7 most used programming language

Labels

distributed-systemspytorchpreprocessingdata-loadermultinode

Projects that are alternatives of or similar to remote-dataloader

text-normalizer
Normalize text string
Stars: ✭ 12 (-80.33%)
Mutual labels:  preprocessing
podium
Podium: a framework agnostic Python NLP library for data loading and preprocessing
Stars: ✭ 55 (-9.84%)
Mutual labels:  preprocessing
torcharrow
High performance model preprocessing library on PyTorch
Stars: ✭ 566 (+827.87%)
Mutual labels:  preprocessing
skippa
SciKIt-learn Pipeline in PAndas
Stars: ✭ 33 (-45.9%)
Mutual labels:  preprocessing
postcss-each
PostCSS plugin to iterate through values
Stars: ✭ 93 (+52.46%)
Mutual labels:  preprocessing
HyperGBM
A full pipeline AutoML tool for tabular data
Stars: ✭ 172 (+181.97%)
Mutual labels:  preprocessing
Preprocessing-Method-for-STEMI-Detection
Official source code of "Preprocessing Method for Performance Enhancement in CNN-based STEMI Detection from 12-lead ECG"
Stars: ✭ 12 (-80.33%)
Mutual labels:  preprocessing
AutoTS
Automated Time Series Forecasting
Stars: ✭ 665 (+990.16%)
Mutual labels:  preprocessing
preprocess-conll05
Scripts for preprocessing the CoNLL-2005 SRL dataset.
Stars: ✭ 17 (-72.13%)
Mutual labels:  preprocessing
redux-autoloader
A higher order component for declarative data loading in React and Redux.
Stars: ✭ 56 (-8.2%)
Mutual labels:  data-loader
pywedge
Makes Interactive Chart Widget, Cleans raw data, Runs baseline models, Interactive hyperparameter tuning & tracking
Stars: ✭ 49 (-19.67%)
Mutual labels:  preprocessing
BrainPrep
Preprocessing pipeline on Brain MR Images through FSL and ANTs, including registration, skull-stripping, bias field correction, enhancement and segmentation.
Stars: ✭ 107 (+75.41%)
Mutual labels:  preprocessing
Igel
a delightful machine learning tool that allows you to train, test, and use models without writing code
Stars: ✭ 2,956 (+4745.9%)
Mutual labels:  preprocessing
tweets-preprocessor
Repo containing the Twitter preprocessor module, developed by the AUTH OSWinds team
Stars: ✭ 26 (-57.38%)
Mutual labels:  preprocessing
graphql-dependency
Cross service dependencies for GraphQL API with underlying @imqueue services
Stars: ✭ 17 (-72.13%)
Mutual labels:  data-loader
veridical-flow
Making it easier to build stable, trustworthy data-science pipelines.
Stars: ✭ 28 (-54.1%)
Mutual labels:  preprocessing
PyTOUGH
A Python library for automating TOUGH2 simulations of subsurface fluid and heat flow
Stars: ✭ 64 (+4.92%)
Mutual labels:  preprocessing
3D Ground Segmentation
A ground segmentation algorithm for 3D point clouds based on the work described in “Fast segmentation of 3D point clouds: a paradigm on LIDAR data for Autonomous Vehicle Applications”, D. Zermas, I. Izzat and N. Papanikolopoulos, 2017. Distinguish between road and non-road points. Road surface extraction. Plane fit ground filter
Stars: ✭ 55 (-9.84%)
Mutual labels:  preprocessing
Benzina
Benzina is an image-loader package that greatly accelerates image loading onto GPUs using their built-in hardware codecs.
Stars: ✭ 36 (-40.98%)
Mutual labels:  data-loader
Multi-Node-TimescaleDB
The multi-node setup of TimescaleDB 🐯🐯🐯 🐘 🐯🐯🐯
Stars: ✭ 42 (-31.15%)
Mutual labels:  multinode
View All Similar Projects ➔

remote-dataloader

DataLoader processed in multiple remote computation machines for heavy data processing.

Architecture

Usage

RemoteDataLoader

total_trainset = torchvision.datasets.CIFAR10(root='./data', train=True, download=True, transform=transform_train)
loader = RemoteDataLoader(total_trainset, batch_size=32, timeout=5)

Example.py

example.py contains a simple example to process cifar10 images using remote nodes.

$ python example.py     # run server(dataloader)
$ python remote_dataloader/worker.py --server {master_ip}:1958      # run multiple workers
$ python remote_dataloader/worker.py --server {master_ip}:1958
$ ...
Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].