All Projects → HenrikBengtsson → Future.apply

HenrikBengtsson / Future.apply

🚀 R package: future.apply - Apply Function to Elements in Parallel using Futures

Programming Languages

r
7636 projects

Projects that are alternatives of or similar to Future.apply

Future
🚀 R package: future: Unified Parallel and Distributed Processing in R for Everyone
Stars: ✭ 735 (+362.26%)
Mutual labels:  asynchronous, parallel-computing, parallel-processing, parallelization, programming, distributed-computing, hpc
future.callr
🚀 R package future.callr: A Future API for Parallel Processing using 'callr'
Stars: ✭ 52 (-67.3%)
Mutual labels:  programming, asynchronous, parallel-computing, parallelization, parallel-processing
ParallelUtilities.jl
Fast and easy parallel mapreduce on HPC clusters
Stars: ✭ 28 (-82.39%)
Mutual labels:  hpc, parallel, parallel-computing, distributed-computing
future.batchtools
🚀 R package future.batchtools: A Future API for Parallel and Distributed Processing using batchtools
Stars: ✭ 77 (-51.57%)
Mutual labels:  package, hpc, parallel, distributed-computing
Index
Metarhia educational program index 📖
Stars: ✭ 2,045 (+1186.16%)
Mutual labels:  asynchronous, parallel, programming
parallel
PARALLEL: Stata module for parallel computing
Stars: ✭ 97 (-38.99%)
Mutual labels:  hpc, parallel, parallelization
t8code
Parallel algorithms and data structures for tree-based AMR with arbitrary element shapes.
Stars: ✭ 37 (-76.73%)
Mutual labels:  hpc, parallel, parallel-computing
hp2p
Heavy Peer To Peer: a MPI based benchmark for network diagnostic
Stars: ✭ 17 (-89.31%)
Mutual labels:  hpc, parallel, parallel-computing
java-multithread
Códigos feitos para o curso de Multithreading com Java, no canal RinaldoDev do YouTube.
Stars: ✭ 24 (-84.91%)
Mutual labels:  parallel, parallel-computing, parallel-processing
Core
parallel finite element unstructured meshes
Stars: ✭ 124 (-22.01%)
Mutual labels:  parallel, parallel-computing, hpc
cruise
User space POSIX-like file system in main memory
Stars: ✭ 27 (-83.02%)
Mutual labels:  hpc, parallel, parallel-computing
Easylambda
distributed dataflows with functional list operations for data processing with C++14
Stars: ✭ 475 (+198.74%)
Mutual labels:  parallel, distributed-computing, hpc
Hamsters.js
100% Vanilla Javascript Multithreading & Parallel Execution Library
Stars: ✭ 517 (+225.16%)
Mutual labels:  parallel, parallel-computing, parallel-processing
Pwrake
Parallel Workflow extension for Rake, runs on multicores, clusters, clouds.
Stars: ✭ 57 (-64.15%)
Mutual labels:  parallel, parallel-computing, distributed-computing
Floops.jl
Fast sequential, threaded, and distributed for-loops for Julia—fold for humans™
Stars: ✭ 96 (-39.62%)
Mutual labels:  parallel, distributed-computing
Parallel
This project now lives on in a rewrite at https://gitlab.redox-os.org/redox-os/parallel
Stars: ✭ 1,181 (+642.77%)
Mutual labels:  parallel, parallel-computing
Charm
The Charm++ parallel programming system. Visit https://charmplusplus.org/ for more information.
Stars: ✭ 96 (-39.62%)
Mutual labels:  parallel-computing, hpc
Parenchyma
An extensible HPC framework for CUDA, OpenCL and native CPU.
Stars: ✭ 71 (-55.35%)
Mutual labels:  parallel-computing, hpc
Parapet
A purely functional library to build distributed and event-driven systems
Stars: ✭ 106 (-33.33%)
Mutual labels:  parallel-computing, distributed-computing
Onemkl
oneAPI Math Kernel Library (oneMKL) Interfaces
Stars: ✭ 122 (-23.27%)
Mutual labels:  parallel-computing, hpc

future.apply: Apply Function to Elements in Parallel using Futures

Introduction

The purpose of this package is to provide worry-free parallel alternatives to base-R "apply" functions, e.g. apply(), lapply(), and vapply(). The goal is that one should be able to replace any of these in the core with its futurized equivalent and things will just work. For example, instead of doing:

library("datasets")
library("stats")
y <- lapply(mtcars, FUN = mean, trim = 0.10)

one can do:

library("future.apply")
plan(multisession) ## Run in parallel on local computer

library("datasets")
library("stats")
y <- future_lapply(mtcars, FUN = mean, trim = 0.10)

Reproducibility is part of the core design, which means that perfect, parallel random number generation (RNG) is supported regardless of the amount of chunking, type of load balancing, and future backend being used. To enable parallel RNG, use argument future.seed = TRUE.

Role

Where does the future.apply package fit in the software stack? You can think of it as a sibling to foreach, furrr, BiocParallel, plyr, etc. Just as parallel provides parLapply(), foreach provides foreach(), BiocParallel provides bplapply(), and plyr provides llply(), future.apply provides future_lapply(). Below is a table summarizing this idea:

Package Functions Backends
future.apply

Future-versions of common goto *apply() functions available in base R (of the 'base' package):
future_apply(), future_by(), future_eapply(), future_lapply(), future_Map(), future_mapply(), future_.mapply(), future_replicate(), future_sapply(), future_tapply(), and future_vapply().
The following function is yet not implemented:
future_rapply()
All future backends
parallel mclapply(), mcmapply(), clusterMap(), parApply(), parLapply(), parSapply(), ... Built-in and conditional on operating system
foreach foreach(), times() All future backends via doFuture
furrr future_imap(), future_map(), future_pmap(), future_map2(), ... All future backends
BiocParallel Bioconductor's parallel mappers:
bpaggregate(), bpiterate(), bplapply(), and bpvec()
All future backends via doFuture (because it supports foreach) or via BiocParallel.FutureParam (direct BiocParallelParam support; prototype)
plyr **ply(..., .parallel = TRUE) functions:
aaply(), ddply(), dlply(), llply(), ...
All future backends via doFuture (because it uses foreach internally)

Note that, except for the built-in parallel package, none of these higher-level APIs implement their own parallel backends, but they rather enhance existing ones. The foreach framework leverages backends such as doParallel, doMC and doFuture, and the future.apply framework leverages the future ecosystem and therefore backends such as built-in parallel, future.callr, and future.batchtools.

By separating future_lapply() and friends from the future package, it helps clarifying the purpose of the future package, which is to define and provide the core Future API, which higher-level parallel APIs can build on and for which any futurized parallel backends can be plugged into.

Roadmap

  1. Implement future_*apply() versions for all common *apply() functions that exist in base R. This also involves writing a large set of package tests asserting the correctness and the same behavior as the corresponding *apply() functions.

  2. Harmonize all future_*apply() functions with each other, e.g. the future-specific arguments.

  3. Consider additional future_*apply() functions and features that fit in this package but don't necessarily have a corresponding function in base R. Examples of this may be "apply" functions that return futures rather than values, mechanisms for benchmarking, and richer control over load balancing.

The API and identity of the future.apply package will be kept close to the *apply() functions in base R. In other words, it will neither keep growing nor be expanded with new, more powerful apply-like functions beyond those core ones in base R. Such extended functionality should be part of a separate package.

Installation

R package future.apply is available on CRAN and can be installed in R as:

install.packages("future.apply")

Pre-release version

To install the pre-release version that is available in Git branch develop on GitHub, use:

remotes::install_github("HenrikBengtsson/future.apply", ref="develop")

This will install the package from source.

Contributing

To contribute to this package, please see CONTRIBUTING.md.

Software status

Resource CRAN GitHub Actions Travis CI AppVeyor CI
Platforms: Multiple Multiple Linux & macOS Windows
R CMD check CRAN version Build status Build status Build status
Test coverage Coverage Status
Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].