1414 Open source projects that are alternatives of or similar to Dataspherestudio

Workflow engine for Kubernetes

同步Hive数据仓库数据到Elasticsearch的小工具

Easy parsing of Apache HTTPD and NGINX access logs with Java, Hadoop, Hive, Pig, Flink, Beam, Storm, Drill, ...

All code samples, scripts and more in-depth examples for The Apache Ignite Book. Include Apache Ignite 2.6 or above

Prosto is a data processing toolkit radically changing how data is processed by heavily relying on functions and operations with functions - an alternative to map-reduce and join-groupby

Example projects for 'BigInsights for Apache Hadoop' on IBM Bluemix

ETL (extract, transform and load) tools for ingesting Polygon blockchain data to Google BigQuery and Pub/Sub

Manage Apache Atlas and Ranger configuration for your Hadoop environment.

ACID Data Source for Apache Spark based on Hive ACID

IBIS is a workflow creation-engine that abstracts the Hadoop internals of ingesting RDBMS data.

A SQL-like language for performing ETL transformations.

A Data Engineering & Machine Learning Knowledge Hub

Simplified ETL process in Hadoop using Apache Spark. Has complete ETL pipeline for datalake. SparkSession extensions, DataFrame validation, Column extensions, SQL functions, and DataFrame transformations

When Apache Pulsar meets Apache Spark

Run Airflow in AWS ECS(Elastic Container Service) using Fargate tasks

Mapreduce project by Hadoop, Nutch, AWS EMR, Pig, Tez, Hive

最好的大数据项目。《Titan数据运营系统》，本项目是一个全栈闭环系统，我们有用作数据可视化的web系统，然后用flume-kafaka-flume进行日志的读取，在hive设计数仓，编写spark代码进行数仓表之间的转化以及ads层表到mysql的迁移，使用azkaban进行定时任务的调度，使用技术：Java/Scala语言，Hadoop、Spark、Hive、Kafka、Flume、Azkaban、SpringBoot，Bootstrap， Echart等；

Astro allows rapid and clean development of {Extract, Load, Transform} workflows using Python and SQL, powered by Apache Airflow.

와이빅타 엔지니어링팀의 자료를 정리해두는 곳입니다.

Identifies and collects data on cc-licensed content across web crawl data and public apis.

Data Lineage Tracking And Visualization Solution

Ytk-learn is a distributed machine learning library which implements most of popular machine learning algorithms(GBDT, GBRT, Mixture Logistic Regression, Gradient Boosting Soft Tree, Factorization Machines, Field-aware Factorization Machines, Logistic Regression, Softmax).

Kyuubi is a unified multi-tenant JDBC interface for large-scale data processing and analytics, built on top of Apache Spark

简单的可视化湖州通话数据 假设数据量很大，没法用浏览器直接绘制热力图，把绘制热力图这一步骤放到线下计算分析。使用Apache Spark并行计算数据之后，再使用Apache Spark绘制热力图，然后用leafletjs加载OpenStreetMap图层和热力图图层，以达到良好的交互效果。现在使用Apache Spark实现绘制，可能是Apache Spark不擅长这方面的计算或者是我没有设计好算法，并行计算的速度比不上单机计算。Apache Spark绘制热力图和计算代码在这 https://github.com/yuanzhaokang/ParallelizeHeatmap.git .

Bitnami Docker Image for Apache Airflow Scheduler

Minikube for big data with Scala and Spark

Example of an ETL Pipeline using Airflow

Bitnami Docker Image for Apache Airflow

Iceberg is a table format for large, slow-moving tabular data

Docker Apache Airflow

Create clusters of VMs on the cloud and configure them with Ansible.

A simplified, lightweight ETL Framework based on Apache Spark

An orchestration platform for the development, production, and observation of data assets.

Building Large-Scale AI Applications for Distributed Big Data

Apache Hive

Official repository of Trino, the distributed SQL query engine for big data, formerly known as PrestoSQL (https://trino.io)

80+ DevOps & Data CLI Tools - AWS, GCP, GCF Python Cloud Function, Log Anonymizer, Spark, Hadoop, HBase, Hive, Impala, Linux, Docker, Spark Data Converters & Validators (Avro/Parquet/JSON/CSV/INI/XML/YAML), Travis CI, AWS CloudFormation, Elasticsearch, Solr etc.

一个大数据架构师应该掌握的技能

Hive UDFs for funnel analysis

Moonbox is a DVtaaS (Data Virtualization as a Service) Platform

A Scala feature transformation library for data science and machine learning

Web UI for Trino, Presto, Hive, Elasticsearch, SparkSQL

Code for Agile Data Science 2.0, O'Reilly 2017, Second Edition

Data science Python notebooks: Deep learning (TensorFlow, Theano, Caffe, Keras), scikit-learn, Kaggle, big data (Spark, Hadoop MapReduce, HDFS), matplotlib, pandas, NumPy, SciPy, Python essentials, AWS, and various command lines.

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

编程电子书，电子书，编程书籍，包括C，C#，Docker，Elasticsearch，Git，Hadoop，HeadFirst，Java，Javascript，jvm，Kafka，Linux，Maven，MongoDB，MyBatis，MySQL，Netty，Nginx，Python，RabbitMQ，Redis，Scala，Solr，Spark，Spring，SpringBoot，SpringCloud，TCPIP，Tomcat，Zookeeper，人工智能，大数据类，并发编程，数据库类，数据挖掘，新面试题，架构设计，算法系列，计算机类，设计模式，软件测试，重构优化，等更多分类

Yet Another UserAgent Analyzer

Waimak is an open-source framework that makes it easier to create complex data flows in Apache Spark.

Cloudflow enables users to quickly develop, orchestrate, and operate distributed streaming applications on Kubernetes.

Generic Data Ingestion & Dispersal Library for Hadoop

💎🔥大数据学习笔记

Real Time Analytics and Data Pipelines based on Spark Streaming

Example project implementing best practices for PySpark ETL jobs and applications.

电商用户行为分析大数据平台

H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.

An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.

A facebook for data

Kylo is a data lake management software platform and framework for enabling scalable enterprise-class data lakes on big data technologies such as Teradata, Apache Spark and/or Hadoop. Kylo is licensed under Apache 2.0. Contributed by Teradata Inc.

50+ DockerHub public images for Docker & Kubernetes - Hadoop, Kafka, ZooKeeper, HBase, Cassandra, Solr, SolrCloud, Presto, Apache Drill, Nifi, Spark, Consul, Riak, TeamCity and DevOps tools built on the major Linux distros: Alpine, CentOS, Debian, Fedora, Ubuntu

Web-based notebook that enables data-driven, interactive data analytics and collaborative documents with SQL, Scala and more.