2055 Open source projects that are alternatives of or similar to Example Airflow Dags

Run Airflow in AWS ECS(Elastic Container Service) using Fargate tasks

ETLy is an add-on dashboard service on top of Apache Airflow.

ETL (extract, transform and load) tools for ingesting Polygon blockchain data to Google BigQuery and Pub/Sub

💨🥫 A Data Factory system for running data processing pipelines built on AirFlow and tailored to CKAN. Includes evolution of DataPusher and Xloader for loading data to DataStore.

Visualize Airflow's schedule by exporting future DAG runs as events to Google Calendar.

Rest API for a food delivery application - Built with Express, Postgres, Redis, MongoDB and Nodemailer

豆瓣电影top250、斗鱼爬取json数据以及爬取美女图片、淘宝、有缘、CrawlSpider爬取红娘网相亲人的部分基本信息以及红娘网分布式爬取和存储redis、爬虫小demo、Selenium、爬取多点、django开发接口、爬取有缘网信息、模拟知乎登录、模拟github登录、模拟图虫网登录、爬取多点商城整站数据、爬取微信公众号历史文章、爬取微信群或者微信好友分享的文章、itchat监听指定微信公众号分享的文章

Sync data between persistence engines, like ETL only not stodgy

A series of DAGs/Workflows to help maintain the operation of Airflow

华炎魔方低代码/无代码平台。内核采用了元数据、微服务、微前端、K8S等最新的技术架构。Steedos Low-Code / No-Code Platform in nodejs and mongodb.

Workflow engine for Kubernetes

a go daemon that syncs MongoDB to Elasticsearch in realtime

A luigi powered analytics / warehouse stack

A light-weight, modular, message representation and mail delivery framework for Python.

ACK is an E(T)L tool specialized in API data ingestion. It is accessible through a Command-Line Interface. The application allows you to easily extract, stream and load data (with minimum transformations), from the API source to the destination of your choice.

Example of an ETL Pipeline using Airflow

A scalable general purpose micro-framework for defining dataflows. You can use it to create dataframes, numpy matrices, python objects, ML models, etc.

Blog post on ETL pipelines with Airflow

Udacity Data Engineering Nano Degree (DEND)

Generate beautiful documentation for your data pipelines in markdown format

Astro allows rapid and clean development of {Extract, Load, Transform} workflows using Python and SQL, powered by Apache Airflow.

MongoDB dockerized backup agent. Runs schedule backups with retention, S3 & SFTP upload, notifications, instrumentation with Prometheus and more.

DataSphereStudio is a one stop data application development& management portal, covering scenarios including data exchange, desensitization/cleansing, analysis/mining, quality measurement, visualization, and task scheduling.

Example end to end data engineering project.

Mass processing data with a complete ETL for .net developers

A distributed workflow runner focusing on performance and simplicity.

File storage abstraction for Yii2

A MongoDB to Elasticsearch connector

Next.js HOC to integrate Google Analytics on every page change

Dockerized Selenium and Python with support for Chrome, Firefox and PhantomJS

本仓库收集整理爬虫相关资源，开发语言以Java为主

Data access layer for PostgreSQL, CockroachDB, MySQL, SQLite and MongoDB with ORM-like features.

Salesforce integration for Django's ORM using the SF REST API.

MongoDB Plugin for Java

R client for the Elasticsearch HTTP API

招聘网数据爬虫

A comprehensive collection of test examples created with JUnit 5

微服务 spring dubbo项目：dubbo rpc；druid数据源连接池；mybatis配置集成，多数据源；jmx监控MBean；定时任务；aop；ftp；测试；Metrics监控；参数验证；跨域处理；shiro权限控制；consul服务注册，发现；redis分布式锁；SPI服务机制；cat监控；netty服务代理；websocket；disconf；mongodb集成；rest；docker；fescar

The official MongoDB driver for Swift

Salesforce Lightning Design System components built with React

🚀Complete Node.js API built using 👉Typescript | Jest | MongoDB | Express

recheck for web apps – change comparison tool with local Golden Masters, Git-like ignore syntax and "Unbreakable Selenium" tests.

A monorepo that holds all of Foursquare's opensource projects

Web-interface + rest API for classification and regression (https://jeff1evesque.github.io/machine-learning.docs)

Nestjs Starter using Mongodb and Passportjs

Python AI assistant 🧠

`scrape_linkedin` is a python package that allows you to scrape personal LinkedIn profiles & company pages - turning the data into structured json.

Python Client and Toolkit for DataFrames, Big Data, Machine Learning and ETL in Elasticsearch

Easy, powerful and ultrafast MongoDB ODM for Laravel.

Course demos and handout material for Talk Python's MongoDB Quickstart course

MongoDB Enterprise Kubernetes Operator

Full Reactive Stack with Spring Boot (WebFlux), MongoDB and Angular

Lightweight MongoDB migration tool for Java

🚀 TypeScript Express Starter

Mongo2Go - MongoDB for integration tests (.NET Core)

web spider built by puppeteer, support task-queue and task-scheduling by decorators，support nedb / mongodb, support data visualization; 基于puppeteer的web爬虫框架，提供灵活的任务队列管理调度方案，提供便捷的数据保存方案（nedb/mongodb），提供数据可视化和用户交互的实现方案

pac4j security library for Spring Security: OAuth, CAS, SAML, OpenID Connect, LDAP, JWT...

A web frontend for scheduling Jupyter notebook reports

VMime Mail Library