Bigdata PlaygroundA complete example of a big data application using : Kubernetes (kops/aws), Apache Spark SQL/Streaming/MLib, Apache Flink, Scala, Python, Apache Kafka, Apache Hbase, Apache Parquet, Apache Avro, Apache Storm, Twitter Api, MongoDB, NodeJS, Angular, GraphQL
Stars: ✭ 177 (-35.87%)
ParquetviewerSimple windows desktop application for viewing & querying Apache Parquet files
Stars: ✭ 145 (-47.46%)
SparkrdmaRDMA accelerated, high-performance, scalable and efficient ShuffleManager plugin for Apache Spark
Stars: ✭ 215 (-22.1%)
Amazon S3 Find And ForgetAmazon S3 Find and Forget is a solution to handle data erasure requests from data lakes stored on Amazon S3, for example, pursuant to the European General Data Protection Regulation (GDPR)
Stars: ✭ 115 (-58.33%)
MorpheusMorpheus brings the leading graph query language, Cypher, onto the leading distributed processing platform, Spark.
Stars: ✭ 303 (+9.78%)
Griffon VmGriffon Data Science Virtual Machine
Stars: ✭ 128 (-53.62%)
MmlsparkSimple and Distributed Machine Learning
Stars: ✭ 2,899 (+950.36%)
Spark.NET for Apache® Spark™ makes Apache Spark™ easily accessible to .NET developers.
Stars: ✭ 1,721 (+523.55%)
HydrographA visual ETL development and debugging tool for big data
Stars: ✭ 144 (-47.83%)
Data AcceleratorData Accelerator for Apache Spark simplifies onboarding to Streaming of Big Data. It offers a rich, easy to use experience to help with creation, editing and management of Spark jobs on Azure HDInsights or Databricks while enabling the full power of the Spark engine.
Stars: ✭ 247 (-10.51%)
mmtf-sparkMethods for the parallel and distributed analysis and mining of the Protein Data Bank using MMTF and Apache Spark.
Stars: ✭ 20 (-92.75%)
MistServerless proxy for Spark cluster
Stars: ✭ 309 (+11.96%)
GafferA large-scale entity and relation database supporting aggregation of properties
Stars: ✭ 1,642 (+494.93%)
Parquet MrApache Parquet
Stars: ✭ 1,278 (+363.04%)
datalake-etl-pipelineSimplified ETL process in Hadoop using Apache Spark. Has complete ETL pipeline for datalake. SparkSession extensions, DataFrame validation, Column extensions, SQL functions, and DataFrame transformations
Stars: ✭ 39 (-85.87%)
sparkucxA high-performance, scalable and efficient ShuffleManager plugin for Apache Spark, utilizing UCX communication layer
Stars: ✭ 32 (-88.41%)
mmtf-workshop-2018Structural Bioinformatics Training Workshop & Hackathon 2018
Stars: ✭ 50 (-81.88%)
Eel SdkBig Data Toolkit for the JVM
Stars: ✭ 140 (-49.28%)
Awkward 0.xManipulate arrays of complex data structures as easily as Numpy.
Stars: ✭ 216 (-21.74%)
Spark With PythonFundamentals of Spark with Python (using PySpark), code examples
Stars: ✭ 150 (-45.65%)
spark-recordsBulletproof Apache Spark jobs with fast root cause analysis of failures.
Stars: ✭ 67 (-75.72%)
DaFlowApache-Spark based Data Flow(ETL) Framework which supports multiple read, write destinations of different types and also support multiple categories of transformation rules.
Stars: ✭ 24 (-91.3%)
gan deeplearning4jAutomatic feature engineering using Generative Adversarial Networks using Deeplearning4j and Apache Spark.
Stars: ✭ 19 (-93.12%)
leaflet heatmap简单的可视化湖州通话数据 假设数据量很大,没法用浏览器直接绘制热力图,把绘制热力图这一步骤放到线下计算分析。使用Apache Spark并行计算数据之后,再使用Apache Spark绘制热力图,然后用leafletjs加载OpenStreetMap图层和热力图图层,以达到良好的交互效果。现在使用Apache Spark实现绘制,可能是Apache Spark不擅长这方面的计算或者是我没有设计好算法,并行计算的速度比不上单机计算。Apache Spark绘制热力图和计算代码在这 https://github.com/yuanzhaokang/ParallelizeHeatmap.git .
Stars: ✭ 13 (-95.29%)
pyspark-cheatsheetPySpark Cheat Sheet - example code to help you learn PySpark and develop apps faster
Stars: ✭ 115 (-58.33%)
DrillApache Drill is a distributed MPP query layer for self describing data
Stars: ✭ 1,619 (+486.59%)
awesome-toolscurated list of awesome tools and libraries for specific domains
Stars: ✭ 31 (-88.77%)
SynapseMLSimple and Distributed Machine Learning
Stars: ✭ 3,355 (+1115.58%)
autThe Archives Unleashed Toolkit is an open-source toolkit for analyzing web archives.
Stars: ✭ 111 (-59.78%)
MusicSharpMusicSharp is a lightweight, cross-platform TUI music player written in C#.
Stars: ✭ 41 (-85.14%)
ibmpairsopen source tools for interaction with IBM PAIRS:
Stars: ✭ 23 (-91.67%)
lensMirror of Apache Lens
Stars: ✭ 57 (-79.35%)
loggingGeneric file logger for .NET Core (FileLoggerProvider) with minimal dependencies
Stars: ✭ 109 (-60.51%)
spark-acidACID Data Source for Apache Spark based on Hive ACID
Stars: ✭ 91 (-67.03%)
HybridBackendEfficient training of deep recommenders on cloud.
Stars: ✭ 30 (-89.13%)
DatahubThe Metadata Platform for the Modern Data Stack
Stars: ✭ 4,232 (+1433.33%)
UnitynugetProvides a service to install NuGet packages into a Unity project via the Unity Package Manager
Stars: ✭ 257 (-6.88%)
WebGPU.NETThis repository contains low-level bindings for WebGPU used in WaveEngine.
Stars: ✭ 35 (-87.32%)
CleanArchitectureASP.NET Core 6 Web API Clean Architecture Solution Template
Stars: ✭ 312 (+13.04%)
Equinox.NET Event Sourcing library with CosmosDB, EventStoreDB, SqlStreamStore and integration test backends. Focused at stream level; see https://github.com/jet/propulsion for cross-stream projections/subscriptions/reactions
Stars: ✭ 260 (-5.8%)
bigstatsrR package for statistical tools with big matrices stored on disk.
Stars: ✭ 139 (-49.64%)
vxqueryMirror of Apache VXQuery
Stars: ✭ 19 (-93.12%)
connected-componentMap Reduce Implementation of Connected Component on Apache Spark
Stars: ✭ 68 (-75.36%)
HAL-9000Automatically setup a productive development environment with Ansible on macOS
Stars: ✭ 72 (-73.91%)
alluxio-pyAlluxio Python client - Access Any Data Source with Python
Stars: ✭ 18 (-93.48%)