All Projects → mendrugory → piton

mendrugory / piton

Licence: MIT License
Run your Python algorithms in parallel and avoid the GIL

Programming Languages

elixir
2628 projects
python
139335 projects - #7 most used programming language

Labels

Projects that are alternatives of or similar to piton

reactphp-parallel
ReactPHP bindings around ext-parallel
Stars: ✭ 15 (-62.5%)
Mutual labels:  parallel
swarmci
Swarm CI - Docker Swarm-based CI system or enhancement to existing systems.
Stars: ✭ 48 (+20%)
Mutual labels:  parallel
multipart-download
Speed up download of a single file with multiple HTTP GET connections running in parallel
Stars: ✭ 29 (-27.5%)
Mutual labels:  parallel
hatrack
Fast, multi-reader, multi-writer, lockless data structures for parallel programming
Stars: ✭ 55 (+37.5%)
Mutual labels:  parallel
cruise
User space POSIX-like file system in main memory
Stars: ✭ 27 (-32.5%)
Mutual labels:  parallel
jobflow
runs stuff in parallel (like GNU parallel, but much faster and memory-efficient)
Stars: ✭ 67 (+67.5%)
Mutual labels:  parallel
FoldsCUDA.jl
Data-parallelism on CUDA using Transducers.jl and for loops (FLoops.jl)
Stars: ✭ 48 (+20%)
Mutual labels:  parallel
asynckit
Minimal async jobs utility library, with streams support
Stars: ✭ 21 (-47.5%)
Mutual labels:  parallel
VieCut
VieCut 1.00 - Shared-memory Minimum Cuts
Stars: ✭ 34 (-15%)
Mutual labels:  parallel
split-tests
Split test files in Jest and Cypress into parallel CI jobs
Stars: ✭ 22 (-45%)
Mutual labels:  parallel
web-scraping-engine
A simple web scraping engine supporting concurrent and anonymous scraping
Stars: ✭ 27 (-32.5%)
Mutual labels:  parallel
laravel-parallel
Laravel parallel
Stars: ✭ 41 (+2.5%)
Mutual labels:  parallel
ezpq
an easy parallel queueing system
Stars: ✭ 42 (+5%)
Mutual labels:  parallel
go-worker-thread-pool
A visual working example of a Thread Pool pattern, based on a known blog article.
Stars: ✭ 24 (-40%)
Mutual labels:  parallel
pblat
parallelized blat with multi-threads support
Stars: ✭ 34 (-15%)
Mutual labels:  parallel
zmq
ZeroMQ based distributed patterns
Stars: ✭ 27 (-32.5%)
Mutual labels:  parallel
hack parallel
The core parallel and shared memory library used by Hack, Flow, and Pyre
Stars: ✭ 39 (-2.5%)
Mutual labels:  parallel
k-means
Code accompanying my blog post on k-means in Python, C++ and CUDA
Stars: ✭ 56 (+40%)
Mutual labels:  parallel
future.batchtools
🚀 R package future.batchtools: A Future API for Parallel and Distributed Processing using batchtools
Stars: ✭ 77 (+92.5%)
Mutual labels:  parallel
tbslas
A parallel, fast solver for the scalar advection-diffusion and the incompressible Navier-Stokes equations based on semi-Lagrangian/Volume-Integral method.
Stars: ✭ 21 (-47.5%)
Mutual labels:  parallel

Piton

hex.pm hexdocs.pm Build Status

Piton is a library which will help you to run your Python code.

You can implement your own Piton.Port and run your python code but I highly recommend to use Piton.Pool, a pool which will allow to run Python code in parallel, a way of avoiding the GIL, and it will protect you from python exceptions.

Installation

Add piton to your list of dependencies in mix.exs:

def deps do
  [{:piton, "~> 0.4.0"}]
end

How to use it

Define your own port

  • The Easiest one
defmodule MySimplePort do
  use Piton.Port
end
  • A port with some wrapper functions which will help you to call the python function: YOUR_MODULE.execute(pid, python_module, python_function, list_of_arguments)
defmodule MyCustomPort do
  use Piton.Port
  def start_link(), do: MyCustomPort.start_link([path: Path.expand("python_folder"), python: "python"], [name: __MODULE__])
  def fun(n), do: MyCustomPort.execute(__MODULE__, :functions, :fun, [n])
end
  • A port prepared to be run by Piton.Pool They have to have a function start() and it has not to be linked.
defmodule MyPoolPort do
  use Piton.Port
  def start(), do: MyPoolPort.start([path: Path.expand("python_folder"), python: "python"], [])
  def fun(pid, n), do: MyPoolPort.execute(pid, :functions, :fun, [n])
end

Run a Pool

Pay attention to the number of Pythons you want to run in parallel. It does not exist an optimal number, maybe it is the number of cores, maybe half or maybe double. Test it with your application.

{:ok, pool} = Piton.Pool.start_link([module: MyPoolPort, pool_number: pool_number], [])

Call a Port (No pool)

iex> MyCustomPort.execute(pid_of_the_port, python_module, python_function, list_of_arguments_of_python_function)

Call a Pool

iex> Piton.Pool.execute(pid_of_the_pool, elixir_function, list_of_arguments_of_elixir_function)

Tasks

Some Mix.Tasks have been included in order to facilitate the integration of a python project

  • Mix.Tasks.Piton.Venv: It creates a Python Virtual Environment.
  • Mix.Tasks.Piton.Pip: It upgrades the Python pip.
  • Mix.Tasks.Piton.Requirements: It gets the dependencies of the Python project.

Test

Run the tests.

mix test 

Name

Pitón is only Python in Spanish 😜 🐍

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].