687 Open source projects that are alternatives of or similar to Iceberg

一个大数据架构师应该掌握的技能

Data science Python notebooks: Deep learning (TensorFlow, Theano, Caffe, Keras), scikit-learn, Kaggle, big data (Spark, Hadoop MapReduce, HDFS), matplotlib, pandas, NumPy, SciPy, Python essentials, AWS, and various command lines.

A big data platform for analyzing web access logs

The Archives Unleashed Toolkit is an open-source toolkit for analyzing web archives.

Code examples for my blog posts

🚢 Docker image for Apache Spark

简单的可视化湖州通话数据 假设数据量很大，没法用浏览器直接绘制热力图，把绘制热力图这一步骤放到线下计算分析。使用Apache Spark并行计算数据之后，再使用Apache Spark绘制热力图，然后用leafletjs加载OpenStreetMap图层和热力图图层，以达到良好的交互效果。现在使用Apache Spark实现绘制，可能是Apache Spark不擅长这方面的计算或者是我没有设计好算法，并行计算的速度比不上单机计算。Apache Spark绘制热力图和计算代码在这 https://github.com/yuanzhaokang/ParallelizeHeatmap.git .

Fast Data Cluster (Apache Cassandra, Kafka, Spark, Flink, YARN and HDFS with Vagrant and VirtualBox)

Open-source distribute workflow schedule tools, also support streaming task.

Fundamentals of Spark with Python (using PySpark), code examples

🎼 Docker compose files for various kafka stacks

Optimized Analytics Package for Spark* Platform

Generic Data Ingestion & Dispersal Library for Hadoop

Apache Drill is a distributed MPP query layer for self describing data

Avro SerDe for Apache Spark structured APIs.

Data Preview 🈸 extension for importing 📤 viewing 🔎 slicing 🔪 dicing 🎲 charting 📊 & exporting 📥 large JSON array/config, YAML, Apache Arrow, Avro, Parquet & Excel data files

WASP is a framework to build complex real time big data applications. It relies on a kind of Kappa/Lambda architecture mainly leveraging Kafka and Spark. If you need to ingest huge amount of heterogeneous data and analyze them through complex pipelines, this is the framework for you.

Read and write Parquet in Scala. Use Scala classes as schema. No need to start a cluster.

A pandas-like deferred expression system, with first-class SQL support

Apache Parquet implementation in Rust

Kylo is a data lake management software platform and framework for enabling scalable enterprise-class data lakes on big data technologies such as Teradata, Apache Spark and/or Hadoop. Kylo is licensed under Apache 2.0. Contributed by Teradata Inc.

Code Library for My Blog

low-level helpers for Apache Spark libraries and tests

ETL Framework for .NET / c# (Parser / Writer for CSV, Flat, Xml, JSON, Key-Value, Parquet, Yaml, Avro formatted files)

Enabling queries on compressed data.

Fair job scheduler on Kubernetes and Mesos for batch workloads and Spark

Code snippets for solving common big data problems in various platforms. Inspired by Rosetta Code

Spark structured streaming examples with using of version 3.0.0

Spark-Crawler: Apache Nutch-like crawler that runs on Apache Spark.

ClickHouse Native Protocol JDBC implementation

Profile Photo Camera support for Laravel Spark

Gradle plugin to interact with Confluent Schema-Registry.

Example source code accompanying O'Reilly's "Hadoop: The Definitive Guide" by Tom White

Template for Spark Projects

A SQL-like language for performing ETL transformations.

TensorFlowOnSpark brings TensorFlow programs to Apache Spark clusters.

Web tool for Avro Schema Registry |

酷玩 Spark: Spark 源代码解析、Spark 类库等

本项目收藏这些年来看过或者听过的一些不错的书籍，在整理文件时看见这些，发现删掉有点可惜，放着又太浪费空间，本着分享的原则，就把它们共享出来，一方面给需要的读者提供这些书籍，另一方面也是一种像知识库的积累吧

One click deploy docker-compose with Kafka, Spark Streaming, Zeppelin UI and Monitoring (Grafana + Kafka Manager)

This is the github repo for Learning Spark: Lightning-Fast Data Analytics [2nd Edition]

Showcases how to build a small Event-sourced application using Spring Boot, Spring Kafka, Apache Avro and Apache Kafka

Spark Streaming+Flume+Kafka+HBase+Hadoop+Zookeeper实现实时日志分析统计；SpringBoot+Echarts实现数据可视化展示

Simple framework agnostic UI router for SPAs

spark structured streaming via HTTP communication

Kylo integration with PDND (previously DAF).

An open-source storage layer that brings scalable, ACID transactions to Apache Spark™ and big data workloads.

dllib is a distributed deep learning library running on Apache Spark

Make Your Company Data Driven. Connect to any data source, easily visualize, dashboard and share your data.

Apache Hive

UC Berkeley team's submission for RecSys Challenge 2018

phData Pulse application log aggregation and monitoring

尚硅谷大数据Spark-2019版最新 Spark 学习

Zeek Analysis Tools (ZAT): Processing and analysis of Zeek network data with Pandas, scikit-learn, Kafka and Spark

Developing Spark External Data Sources using the V2 API

Docker build project to setup a lightweight hadoop cluster containing hadoop, pig, zookeeper, hbase, phoenix, storm, kafka, kafka manager

Qubole Sparklens tool for performance tuning Apache Spark

A curated list of awesome resources related to the Ada and SPARK programming language

Prosto is a data processing toolkit radically changing how data is processed by heavily relying on functions and operations with functions - an alternative to map-reduce and join-groupby