All Projects → joblib → Joblib

joblib / Joblib

Licence: bsd-3-clause
Computing with Python functions.

Programming Languages

python
139335 projects - #7 most used programming language
shell
77523 projects

Projects that are alternatives of or similar to Joblib

think-async
🌿 Exploring cooperative concurrency primitives in Python
Stars: ✭ 178 (-93.21%)
Mutual labels:  multiprocessing, threading
Cachier
Persistent, stale-free, local and cross-machine caching for Python functions.
Stars: ✭ 359 (-86.3%)
Mutual labels:  caching, memoization
distex
Distributed process pool for Python
Stars: ✭ 101 (-96.15%)
Mutual labels:  multiprocessing, parallel-computing
Fooproxy
稳健高效的评分制-针对性- IP代理池 + API服务,可以自己插入采集器进行代理IP的爬取,针对你的爬虫的一个或多个目标网站分别生成有效的IP代理数据库,支持MongoDB 4.0 使用 Python3.7(Scored IP proxy pool ,customise proxy data crawler can be added anytime)
Stars: ✭ 195 (-92.56%)
Mutual labels:  threading, multiprocessing
Tutorials
机器学习相关教程
Stars: ✭ 9,616 (+267.02%)
Mutual labels:  threading, multiprocessing
maki
[beta] persistent memoization of computations, e.g. for repeatable tests and benchmarks
Stars: ✭ 16 (-99.39%)
Mutual labels:  caching, memoization
Kernels
This is a set of simple programs that can be used to explore the features of a parallel platform.
Stars: ✭ 287 (-89.05%)
Mutual labels:  parallel-computing, threading
LruClockCache
A low-latency LRU approximation cache in C++ using CLOCK second-chance algorithm. Multi level cache too. Up to 2.5 billion lookups per second.
Stars: ✭ 35 (-98.66%)
Mutual labels:  caching, memoization
Multitasking
Non-blocking Python methods using decorators
Stars: ✭ 87 (-96.68%)
Mutual labels:  threading, multiprocessing
Schwimmbad
A common interface to processing pools.
Stars: ✭ 82 (-96.87%)
Mutual labels:  parallel-computing, multiprocessing
bkt
bkt is a subprocess caching utility, available as a command line binary and a Rust library.
Stars: ✭ 117 (-95.53%)
Mutual labels:  caching, memoization
React Native Multithreading
🧵 Fast and easy multithreading for React Native using JSI
Stars: ✭ 164 (-93.74%)
Mutual labels:  threading, multiprocessing
python-graceful-shutdown
Example of a Python code that implements graceful shutdown while using asyncio, threading and multiprocessing
Stars: ✭ 109 (-95.84%)
Mutual labels:  multiprocessing, threading
mantichora
A simple interface to Python multiprocessing and threading
Stars: ✭ 13 (-99.5%)
Mutual labels:  multiprocessing, threading
funboost
pip install funboost,python全功能分布式函数调度框架,。支持python所有类型的并发模式和全球一切知名消息队列中间件,python函数加速器,框架包罗万象,一统编程思维,兼容50% python编程业务场景,适用范围广。只需要一行代码即可分布式执行python一切函数。旧名字是function_scheduling_distributed_framework
Stars: ✭ 351 (-86.6%)
Mutual labels:  multiprocessing, threading
Pebble
Multi threading and processing eye-candy.
Stars: ✭ 276 (-89.47%)
Mutual labels:  threading, multiprocessing
Cacheout
A caching library for Python
Stars: ✭ 238 (-90.92%)
Mutual labels:  caching, memoization
atpbar
Progress bars for threading and multiprocessing tasks on terminal and Jupyter Notebook
Stars: ✭ 74 (-97.18%)
Mutual labels:  multiprocessing, threading
Cached
Rust cache structures and easy function memoization
Stars: ✭ 530 (-79.77%)
Mutual labels:  caching, memoization
Frontend Computer Science
A list of Computer Science topics important for a Front-End Developer to learn 📝
Stars: ✭ 113 (-95.69%)
Mutual labels:  caching, memoization

Joblib version Azure CI status Doc status Codecov coverage

The homepage of joblib with user documentation is located on:

https://joblib.readthedocs.io

Getting the latest code

To get the latest code using git, simply type:

git clone git://github.com/joblib/joblib.git

If you don't have git installed, you can download a zip or tarball of the latest code: http://github.com/joblib/joblib/archives/master

Installing

You can use pip to install joblib:

pip install joblib

from any directory or:

python setup.py install

from the source directory.

Dependencies

  • Joblib has no mandatory dependencies besides Python (supported versions are 3.6+).
  • Joblib has an optional dependency on Numpy (at least version 1.6.1) for array manipulation.
  • Joblib includes its own vendored copy of loky for process management.
  • Joblib can efficiently dump and load numpy arrays but does not require numpy to be installed.
  • Joblib has an optional dependency on python-lz4 as a faster alternative to zlib and gzip for compressed serialization.
  • Joblib has an optional dependency on psutil to mitigate memory leaks in parallel worker processes.
  • Some examples require external dependencies such as pandas. See the instructions in the Building the docs section for details.

Workflow to contribute

To contribute to joblib, first create an account on github. Once this is done, fork the joblib repository to have your own repository, clone it using 'git clone' on the computers where you want to work. Make your changes in your clone, push them to your github account, test them on several computers, and when you are happy with them, send a pull request to the main repository.

Running the test suite

To run the test suite, you need the pytest (version >= 3) and coverage modules. Run the test suite using:

pytest joblib

from the root of the project.

Building the docs

To build the docs you need to have sphinx (>=1.4) and some dependencies installed:

pip install -U -r .readthedocs-requirements.txt

The docs can then be built with the following command:

make doc

The html docs are located in the doc/_build/html directory.

Making a source tarball

To create a source tarball, eg for packaging or distributing, run the following command:

python setup.py sdist

The tarball will be created in the dist directory. This command will compile the docs, and the resulting tarball can be installed with no extra dependencies than the Python standard library. You will need setuptool and sphinx.

Making a release and uploading it to PyPI

This command is only run by project manager, to make a release, and upload in to PyPI:

python setup.py sdist bdist_wheel
twine upload dist/*

Note that the documentation should automatically get updated at each git push. If that is not the case, try building th doc locally and resolve any doc build error (in particular when running the examples).

Updating the changelog

Changes are listed in the CHANGES.rst file. They must be manually updated but, the following git command may be used to generate the lines:

git log --abbrev-commit --date=short --no-merges --sparse
Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].