daskos / Daskos
Licence: apache-2.0
Apache Mesos backend for Dask scheduling library
Stars: ✭ 28
Programming Languages
python
139335 projects - #7 most used programming language
Projects that are alternatives of or similar to Daskos
mesos-framework
A wrapper around the Mesos HTTP APIs for Schedulers and Executors. Write your Mesos framework in pure JavaScript!
Stars: ✭ 61 (+117.86%)
Mutual labels: scheduler, mesos
scheduler
Maintenance fork of Apache Aurora's Scheduler
Stars: ✭ 21 (-25%)
Mutual labels: scheduler, mesos
mentos
Fresh Python Mesos Scheduler and Executor driver
Stars: ✭ 18 (-35.71%)
Mutual labels: scheduler, mesos
Cook
Fair job scheduler on Kubernetes and Mesos for batch workloads and Spark
Stars: ✭ 314 (+1021.43%)
Mutual labels: scheduler, mesos
Crono
A time-based background job scheduler daemon (just like Cron) for Rails
Stars: ✭ 637 (+2175%)
Mutual labels: scheduler
Singularity
Scheduler (HTTP API and webapp) for running Mesos tasks—long running processes, one-off tasks, and scheduled jobs. #hubspot-open-source
Stars: ✭ 793 (+2732.14%)
Mutual labels: mesos
Gocron
Easy and fluent Go cron scheduling. This is a fork from https://github.com/jasonlvhit/gocron
Stars: ✭ 605 (+2060.71%)
Mutual labels: scheduler
Opencue
A render management system you can deploy for visual effects and animation productions.
Stars: ✭ 578 (+1964.29%)
Mutual labels: scheduler
Airflow Maintenance Dags
A series of DAGs/Workflows to help maintain the operation of Airflow
Stars: ✭ 914 (+3164.29%)
Mutual labels: dag
Bree
🚥 The best job scheduler for Node.js and JavaScript with cron, dates, ms, later, and human-friendly support. Works in Node v10+ and browsers, uses workers to spawn sandboxed processes, and supports async/await, retries, throttling, concurrency, and graceful shutdown. Simple, fast, and lightweight. Made for @ForwardEmail and @ladjs.
Stars: ✭ 933 (+3232.14%)
Mutual labels: scheduler
Blockchain
Compilation of useful documents and scientific papers about Blockchain & cryptocurrencies.
Stars: ✭ 751 (+2582.14%)
Mutual labels: dag
Swiftcoroutine
Swift coroutines for iOS, macOS and Linux.
Stars: ✭ 690 (+2364.29%)
Mutual labels: scheduler
Cortex M Rtic
Real-Time Interrupt-driven Concurrency (RTIC) framework for ARM Cortex-M microcontrollers
Stars: ✭ 623 (+2125%)
Mutual labels: scheduler
Linkerdcosdockerfile
Linker Dcos DockerFile&DockerCompose yml file
Stars: ✭ 8 (-71.43%)
Mutual labels: mesos
Go Health
Library for enabling asynchronous health checks in your service
Stars: ✭ 588 (+2000%)
Mutual labels: mesos
Smart Scheduler Android
A utility library for scheduling periodic and non-periodic jobs efficiently.
Stars: ✭ 930 (+3221.43%)
Mutual labels: scheduler
Toil
A scalable, efficient, cross-platform (Linux/macOS) and easy-to-use workflow engine in pure Python.
Stars: ✭ 733 (+2517.86%)
Mutual labels: mesos
Apache Mesos backend for Dask scheduling library. Run distributed python dask workflows on your Mesos cluster.
Notable Features
- distributively run tasks in docker container
- specify resource requirements per task
- bin packing for optimized resource utilization
Installation
Prerequisits: satyr, dask, toolz. All of them should be installed w/ the following commands:
pip install dask.mesos
or use lensa/dask.mesos Docker image
Configuration:
MESOS_MASTER=zk://127.0.0.1:2181/mesos
ZOOKEEPER_HOST=127.0.0.1:2181
Example
from __future__ import absolute_import, division, print_function
from daskos import mesos, MesosExecutor
@mesos(cpus=0.1, mem=64)
def add(x, y):
"""Run add on mesos with specified resources"""
return x + y
@mesos(cpus=0.3, mem=128, image='lensa/daskos')
def mul(x, y):
"""Run mul on mesos in specified docker image"""
return x * y
with MesosExecutor(name='dask') as executor:
"""This context handles Mesos scheduler's lifecycle"""
a, b = 23, 89
alot = add(a, b)
gigalot = mul(alot, add(10, 2))
gigalot.compute(get=executor.get) # (a + b) * (10 + 2)
executor.compute([alot, gigalot]) # list of futures
Configuring daskos Tasks
You can configure your mesos tasks in your decorator, currently the following options are available:
- cpus: The amount of cpus to use for the task.
- mem: Memory in MB to reserver for the task.
- disk: The amount of disk to use for the task.
- image: A docker image name. If not set, mesos containerizer will be used.
Both mem and cpus are defaults to some low values set in satyr so you're encouraged to override them here. More options like constraints, other resources are on the way.
Note that the project description data, including the texts, logos, images, and/or trademarks,
for each open source project belongs to its rightful owner.
If you wish to add or remove any projects, please contact us at [email protected].