teamclairvoyant / Airflow Maintenance Dags
Licence: apache-2.0
A series of DAGs/Workflows to help maintain the operation of Airflow
Stars: ✭ 914
Programming Languages
python
139335 projects - #7 most used programming language
Projects that are alternatives of or similar to Airflow Maintenance Dags
Argo Workflows
Workflow engine for Kubernetes
Stars: ✭ 10,024 (+996.72%)
Mutual labels: dag, airflow, workflow
Xene
A distributed workflow runner focusing on performance and simplicity.
Stars: ✭ 56 (-93.87%)
Mutual labels: dag, airflow, workflow
Aws Ecs Airflow
Run Airflow in AWS ECS(Elastic Container Service) using Fargate tasks
Stars: ✭ 107 (-88.29%)
Mutual labels: dag, airflow
Docker Registry Pruner
Tool to apply retention logic to docker images in a Docker Registry
Stars: ✭ 122 (-86.65%)
Mutual labels: cleanup, maintenance
Cyclone
Powerful workflow engine and end-to-end pipeline solutions implemented with native Kubernetes resources. https://cyclone.dev
Stars: ✭ 978 (+7%)
Mutual labels: dag, workflow
Cwl Airflow
Python package to extend Airflow functionality with CWL1.1 support
Stars: ✭ 130 (-85.78%)
Mutual labels: dag, workflow
Example Airflow Dags
Example DAGs using hooks and operators from Airflow Plugins
Stars: ✭ 243 (-73.41%)
Mutual labels: dag, airflow
beekeeper
Service for automatically managing and cleaning up unreferenced data
Stars: ✭ 43 (-95.3%)
Mutual labels: maintenance, cleanup
aircal
Visualize Airflow's schedule by exporting future DAG runs as events to Google Calendar.
Stars: ✭ 66 (-92.78%)
Mutual labels: airflow, dag
bitnami-docker-airflow-scheduler
Bitnami Docker Image for Apache Airflow Scheduler
Stars: ✭ 19 (-97.92%)
Mutual labels: workflow, airflow
Bitnami Docker Airflow
Bitnami Docker Image for Apache Airflow
Stars: ✭ 89 (-90.26%)
Mutual labels: airflow, workflow
Dataspherestudio
DataSphereStudio is a one stop data application development& management portal, covering scenarios including data exchange, desensitization/cleansing, analysis/mining, quality measurement, visualization, and task scheduling.
Stars: ✭ 1,195 (+30.74%)
Mutual labels: airflow, workflow
Airflow
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
Stars: ✭ 24,101 (+2536.87%)
Mutual labels: airflow, workflow
aircan
💨🥫 A Data Factory system for running data processing pipelines built on AirFlow and tailored to CKAN. Includes evolution of DataPusher and Xloader for loading data to DataStore.
Stars: ✭ 24 (-97.37%)
Mutual labels: airflow, dag
Cloudneo
CWL implementation of CloudNeo: A cloud pipeline for identifying patient-specific tumor neoantigens
Stars: ✭ 13 (-98.58%)
Mutual labels: workflow
Actions Toolkit
🛠 A toolkit for building GitHub Actions in Node.js
Stars: ✭ 848 (-7.22%)
Mutual labels: workflow
airflow-maintenance-dags
A series of DAGs/Workflows to help maintain the operation of Airflow
DAGs/Workflows
- backup-configs
- A maintenance workflow that you can deploy into Airflow to periodically take backups of various Airflow configurations and files.
- clear-missing-dags
- A maintenance workflow that you can deploy into Airflow to periodically clean out entries in the DAG table of which there is no longer a corresponding Python File for it. This ensures that the DAG table doesn't have needless items in it and that the Airflow Web Server displays only those available DAGs.
- db-cleanup
- A maintenance workflow that you can deploy into Airflow to periodically clean out the DagRun, TaskInstance, Log, XCom, Job DB and SlaMiss entries to avoid having too much data in your Airflow MetaStore.
- kill-halted-tasks
- A maintenance workflow that you can deploy into Airflow to periodically kill off tasks that are running in the background that don't correspond to a running task in the DB.
- This is useful because when you kill off a DAG Run or Task through the Airflow Web Server, the task still runs in the background on one of the executors until the task is complete.
- log-cleanup
- A maintenance workflow that you can deploy into Airflow to periodically clean out the task logs to avoid those getting too big.
- delete-broken-dags
- A maintenance workflow that you can deploy into Airflow to periodically delete DAG files and clean out entries in the ImportError table for DAGs which Airflow cannot parse or import properly. This ensures that the ImportError table is cleaned every day.
Note that the project description data, including the texts, logos, images, and/or trademarks,
for each open source project belongs to its rightful owner.
If you wish to add or remove any projects, please contact us at [email protected].