397 Open source projects that are alternatives of or similar to learning-hadoop-and-spark

Mapreduce project by Hadoop, Nutch, AWS EMR, Pig, Tez, Hive

gomrjob - a Go Framework for Hadoop Map Reduce Jobs

Data science Python notebooks: Deep learning (TensorFlow, Theano, Caffe, Keras), scikit-learn, Kaggle, big data (Spark, Hadoop MapReduce, HDFS), matplotlib, pandas, NumPy, SciPy, Python essentials, AWS, and various command lines.

Data pipeline project

大数据生态圈学习

RDMA accelerated, high-performance, scalable and efficient ShuffleManager plugin for Apache Spark

Scalable RNA-seq analysis

A collection of tutorials on Hadoop, MapReduce, Spark, Docker

The Archives Unleashed Toolkit is an open-source toolkit for analyzing web archives.

A light-weight distributed stream computing framework for Golang

Simplified ETL process in Hadoop using Apache Spark. Has complete ETL pipeline for datalake. SparkSession extensions, DataFrame validation, Column extensions, SQL functions, and DataFrame transformations

A high-performance, scalable and efficient ShuffleManager plugin for Apache Spark, utilizing UCX communication layer

.NET for Apache® Spark™ makes Apache Spark™ easily accessible to .NET developers.

Behemoth is an open source platform for large scale document analysis based on Apache Hadoop.

简单的可视化湖州通话数据 假设数据量很大，没法用浏览器直接绘制热力图，把绘制热力图这一步骤放到线下计算分析。使用Apache Spark并行计算数据之后，再使用Apache Spark绘制热力图，然后用leafletjs加载OpenStreetMap图层和热力图图层，以达到良好的交互效果。现在使用Apache Spark实现绘制，可能是Apache Spark不擅长这方面的计算或者是我没有设计好算法，并行计算的速度比不上单机计算。Apache Spark绘制热力图和计算代码在这 https://github.com/yuanzhaokang/ParallelizeHeatmap.git .

Basin is a visual programming editor for building Spark and PySpark pipelines. Easily build, debug, and deploy complex ETL pipelines from your browser

Example MapReduce jobs in Java, Hive, Pig, and Hadoop Streaming that work on Avro data.

大数据入门指南 ⭐

Map Reduce Implementation of Connected Component on Apache Spark

A complete example of a big data application using : Kubernetes (kops/aws), Apache Spark SQL/Streaming/MLib, Apache Flink, Scala, Python, Apache Kafka, Apache Hbase, Apache Parquet, Apache Avro, Apache Storm, Twitter Api, MongoDB, NodeJS, Angular, GraphQL

💎🔥大数据学习笔记

🎯 🌟[大数据面试题]分享自己在网络上收集的大数据相关的面试题以及自己的答案总结.目前包含Hadoop/Hive/Spark/Flink/Hbase/Kafka/Zookeeper框架的面试题知识总结

Asakusa Framework

个人学习知识库涉及到数据仓库建模、实时计算、大数据、Java、算法等。

Distributed Deep Learning, with a focus on distributed training, using Keras and Apache Spark.

C# and F# language binding and extensions to Apache Spark

Griffon Data Science Virtual Machine

Cascading is a feature rich API for defining and executing complex and fault tolerant data processing flows locally or on a cluster. See https://github.com/Cascading/cascading for the release repository.

MapReduce, Spark, Java, and Scala for Data Algorithms Book

网站点击流离线日志分析

Apache-Spark based Data Flow(ETL) Framework which supports multiple read, write destinations of different types and also support multiple categories of transformation rules.

Fundamentals of Spark with Python (using PySpark), code examples

大数据学习笔记，学习路线，技术案例整理。

Astronomy Broker based on Apache Spark

Docker image for main Apache Hadoop components (Yarn/Hdfs)

Pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).

Repository for integration with Apache Kafka

Scalable data processing pipelines in JavaScript

The most popular open source electronic health records and medical practice management solution.

Simple Java Framework,designed for easily develop Spring based java program.Support Bigdata And metadata management.A common elasticsearch comm query tool and so on.

Free and open source Electronic Health Record

Open Source Phasor Data Concentrator

Real-world Spark pipelines examples

Distributed Java micro-services using ZooKeeper

This plugin provides native AWS instrumentation for monitoring and metrics collection, including: health and metrics for various AWS services, such as EC2, RDS, ELB, and more, as well as handlers for EC2, SES, and SNS.

Open Source Healthcare ERP / Management System

Management and automation platform for Stateful Distributed Systems

Django REST framework-based Digital Patient Registration and EMR backend

PDD Graph : Bridging MIMIC-III and Linked Data Cloud

Big Data Management and Analysis Final Project

Install hadoop cluster with ansible

Sample code for AWS data service and ML courses on LinkedIn Learning

SBT plugin for Apache Spark on AWS EMR

Word analysis, by domain, on the Common Crawl data set for the purpose of finding industry trends

An example Terraform project that will configure a Secure and Customizable Spark Cluster on Amazon EMR.

curated list of awesome tools and libraries for specific domains

Node.js WebHDFS REST API client

TonY is a framework to natively run deep learning frameworks on Apache Hadoop.

Routines and data structures for using isarn-sketches idiomatically in Apache Spark

Aplicación web para ANDES