GitPlanet

Cheap and reliable Node.js hosting starts at $3/month, and $1/month static HTML hosting

Created with love in Canada, visit hostnodejs.com today

Feel like to post an Ad? Learn Details
All Projects → getindata → kedro-airflow-k8s

getindata / kedro-airflow-k8s

Licence: Apache-2.0 license
Kedro Plugin to support running pipelines on Kubernetes using Airflow.

Programming Languages

python
139335 projects - #7 most used programming language
Jinja
831 projects

Labels

airflowk8smachinelearningkuberentesmlopskedrokedro-pluginkedro-airflow

Projects that are alternatives of or similar to kedro-airflow-k8s

kedro-airflow
Kedro-Airflow makes it easy to deploy Kedro projects to Airflow.
Stars: ✭ 121 (+450%)
Mutual labels:  airflow, kedro, kedro-plugin
Kedro
A Python framework for creating reproducible, maintainable and modular data science code.
Stars: ✭ 4,764 (+21554.55%)
Mutual labels:  mlops, kedro
kedro
A Python framework for creating reproducible, maintainable and modular data science code.
Stars: ✭ 6,068 (+27481.82%)
Mutual labels:  mlops, kedro
Clearml
ClearML - Auto-Magical CI/CD to streamline your ML workflow. Experiment Manager, MLOps and Data-Management
Stars: ✭ 2,868 (+12936.36%)
Mutual labels:  machinelearning, mlops
awesome-open-mlops
The Fuzzy Labs guide to the universe of open source MLOps
Stars: ✭ 304 (+1281.82%)
Mutual labels:  machinelearning, mlops
k3ai
A lightweight tool to get an AI Infrastructure Stack up in minutes not days. K3ai will take care of setup K8s for You, deploy the AI tool of your choice and even run your code on it.
Stars: ✭ 105 (+377.27%)
Mutual labels:  airflow, mlops
airflow-dbt-python
A collection of Airflow operators, hooks, and utilities to elevate dbt to a first-class citizen of Airflow.
Stars: ✭ 111 (+404.55%)
Mutual labels:  airflow
mindsdb native
Machine Learning in one line of code
Stars: ✭ 34 (+54.55%)
Mutual labels:  machinelearning
Paperboy
A web frontend for scheduling Jupyter notebook reports
Stars: ✭ 221 (+904.55%)
Mutual labels:  airflow
Airflow Scheduler Failover Controller
A process that runs in unison with Apache Airflow to control the Scheduler process to ensure High Availability
Stars: ✭ 204 (+827.27%)
Mutual labels:  airflow
AirflowETL
Blog post on ETL pipelines with Airflow
Stars: ✭ 20 (-9.09%)
Mutual labels:  airflow
GIMLeT
GIMLeT – Gestural Interaction Machine Learning Toolkit
Stars: ✭ 33 (+50%)
Mutual labels:  machinelearning
hawking
A Natural Language Date Time Parser that Extract date and time from text with context and parse to the required format
Stars: ✭ 168 (+663.64%)
Mutual labels:  machinelearning
LiuAlgoTrader
Framework for algorithmic trading
Stars: ✭ 514 (+2236.36%)
Mutual labels:  machinelearning
mlzero to hero
No description or website provided.
Stars: ✭ 40 (+81.82%)
Mutual labels:  machinelearning
Example Airflow Dags
Example DAGs using hooks and operators from Airflow Plugins
Stars: ✭ 243 (+1004.55%)
Mutual labels:  airflow
pipeline
PipelineAI Kubeflow Distribution
Stars: ✭ 4,154 (+18781.82%)
Mutual labels:  airflow
Awesome Apache Airflow
Curated list of resources about Apache Airflow
Stars: ✭ 2,755 (+12422.73%)
Mutual labels:  airflow
popmon
Monitor the stability of a Pandas or Spark dataframe ⚙︎
Stars: ✭ 434 (+1872.73%)
Mutual labels:  mlops
Vector-Map-Generation-from-Aerial-Imagery-using-Deep-Learning-GeoSpatial-UNET
We propose a simple yet efficient technique to leverage semantic segmentation model to extract and separate individual buildings in densely compacted areas using medium resolution satellite/UAV orthoimages. We adopted standard UNET architecture, additionally added batch normalization layer after every convolution, to label every pixel in the ima…
Stars: ✭ 74 (+236.36%)
Mutual labels:  machinelearning
View All Similar Projects ➔

Kedro Airflow K8S Plugin

Python Version License SemVer PyPI version Downloads

Maintainability Test Coverage Documentation Status

About

The main purpose of this plugin is to enable running kedro pipeline with Airflow on Kubernetes Cluster. In difference to kedro-airflow this plugin does not require additional libraries installed in airflow runtime, it uses K8S infrastructure instead. It supports translation from Kedro pipeline DSL to airflow (python API) and generation of airflow DAGs.

The plugin can be used together with kedro-docker to simplify preparation of docker image for pipeline execution.

Documentation

For detailed documentation refer to https://kedro-airflow-k8s.readthedocs.io/

Usage guide

Usage: kedro airflow-k8s [OPTIONS] COMMAND [ARGS]...
 
Options:
  -e, --env TEXT  Environment to use.
  -p, --pipeline TEXT  Pipeline name to pick.
  -h, --help      Show this message and exit.

Commands:
  compile          Create an Airflow DAG for a project
  init             Initializes configuration for the plugin
  list-pipelines   List pipelines generated by this plugin
  run-once         Uploads pipeline to Airflow and runs once
  schedule         Uploads pipeline to Airflow with given schedule
  ui               Open Apache Airflow UI in new browser tab
  upload-pipeline  Uploads pipeline to Airflow DAG location
Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].