Bigdata PlaygroundA complete example of a big data application using : Kubernetes (kops/aws), Apache Spark SQL/Streaming/MLib, Apache Flink, Scala, Python, Apache Kafka, Apache Hbase, Apache Parquet, Apache Avro, Apache Storm, Twitter Api, MongoDB, NodeJS, Angular, GraphQL
Stars: ✭ 177 (+22.07%)
gan deeplearning4jAutomatic feature engineering using Generative Adversarial Networks using Deeplearning4j and Apache Spark.
Stars: ✭ 19 (-86.9%)
DrillApache Drill is a distributed MPP query layer for self describing data
Stars: ✭ 1,619 (+1016.55%)
HydrographA visual ETL development and debugging tool for big data
Stars: ✭ 144 (-0.69%)
MmlsparkSimple and Distributed Machine Learning
Stars: ✭ 2,899 (+1899.31%)
mmtf-sparkMethods for the parallel and distributed analysis and mining of the Protein Data Bank using MMTF and Apache Spark.
Stars: ✭ 20 (-86.21%)
mmtf-workshop-2018Structural Bioinformatics Training Workshop & Hackathon 2018
Stars: ✭ 50 (-65.52%)
Eel SdkBig Data Toolkit for the JVM
Stars: ✭ 140 (-3.45%)
datalake-etl-pipelineSimplified ETL process in Hadoop using Apache Spark. Has complete ETL pipeline for datalake. SparkSession extensions, DataFrame validation, Column extensions, SQL functions, and DataFrame transformations
Stars: ✭ 39 (-73.1%)
SynapseMLSimple and Distributed Machine Learning
Stars: ✭ 3,355 (+2213.79%)
pyspark-cheatsheetPySpark Cheat Sheet - example code to help you learn PySpark and develop apps faster
Stars: ✭ 115 (-20.69%)
sparkucxA high-performance, scalable and efficient ShuffleManager plugin for Apache Spark, utilizing UCX communication layer
Stars: ✭ 32 (-77.93%)
Griffon VmGriffon Data Science Virtual Machine
Stars: ✭ 128 (-11.72%)
Awkward 0.xManipulate arrays of complex data structures as easily as Numpy.
Stars: ✭ 216 (+48.97%)
Spark With PythonFundamentals of Spark with Python (using PySpark), code examples
Stars: ✭ 150 (+3.45%)
Data AcceleratorData Accelerator for Apache Spark simplifies onboarding to Streaming of Big Data. It offers a rich, easy to use experience to help with creation, editing and management of Spark jobs on Azure HDInsights or Databricks while enabling the full power of the Spark engine.
Stars: ✭ 247 (+70.34%)
Amazon S3 Find And ForgetAmazon S3 Find and Forget is a solution to handle data erasure requests from data lakes stored on Amazon S3, for example, pursuant to the European General Data Protection Regulation (GDPR)
Stars: ✭ 115 (-20.69%)
awesome-toolscurated list of awesome tools and libraries for specific domains
Stars: ✭ 31 (-78.62%)
DaFlowApache-Spark based Data Flow(ETL) Framework which supports multiple read, write destinations of different types and also support multiple categories of transformation rules.
Stars: ✭ 24 (-83.45%)
MorpheusMorpheus brings the leading graph query language, Cypher, onto the leading distributed processing platform, Spark.
Stars: ✭ 303 (+108.97%)
MistServerless proxy for Spark cluster
Stars: ✭ 309 (+113.1%)
GafferA large-scale entity and relation database supporting aggregation of properties
Stars: ✭ 1,642 (+1032.41%)
spark-recordsBulletproof Apache Spark jobs with fast root cause analysis of failures.
Stars: ✭ 67 (-53.79%)
autThe Archives Unleashed Toolkit is an open-source toolkit for analyzing web archives.
Stars: ✭ 111 (-23.45%)
SparkrdmaRDMA accelerated, high-performance, scalable and efficient ShuffleManager plugin for Apache Spark
Stars: ✭ 215 (+48.28%)
leaflet heatmap简单的可视化湖州通话数据 假设数据量很大,没法用浏览器直接绘制热力图,把绘制热力图这一步骤放到线下计算分析。使用Apache Spark并行计算数据之后,再使用Apache Spark绘制热力图,然后用leafletjs加载OpenStreetMap图层和热力图图层,以达到良好的交互效果。现在使用Apache Spark实现绘制,可能是Apache Spark不擅长这方面的计算或者是我没有设计好算法,并行计算的速度比不上单机计算。Apache Spark绘制热力图和计算代码在这 https://github.com/yuanzhaokang/ParallelizeHeatmap.git .
Stars: ✭ 13 (-91.03%)
Parquet MrApache Parquet
Stars: ✭ 1,278 (+781.38%)
AzuredatalakeSamples and Docs for Azure Data Lake Store and Analytics
Stars: ✭ 128 (-11.72%)
Sparkling GraphSparklingGraph provides easy to use set of features that will give you ability to proces large scala graphs using Spark and GraphX.
Stars: ✭ 139 (-4.14%)
FeastFeature Store for Machine Learning
Stars: ✭ 2,576 (+1676.55%)
Scalable Data ScienceScalable Data Science, course sets in big data Using Apache Spark over databricks and their mathematical, statistical and computational foundations using SageMath.
Stars: ✭ 142 (-2.07%)
PoseidonA search engine which can hold 100 trillion lines of log data.
Stars: ✭ 1,793 (+1136.55%)
RichdemHigh-performance Terrain and Hydrology Analysis
Stars: ✭ 127 (-12.41%)
Parquet4sRead and write Parquet in Scala. Use Scala classes as schema. No need to start a cluster.
Stars: ✭ 125 (-13.79%)
Mobydq🐳 Tool to automate data quality checks on data pipelines
Stars: ✭ 123 (-15.17%)
YaxlibYet Another XML Serialization Library for the .NET Framework and .NET Core
Stars: ✭ 124 (-14.48%)
Parquet RsApache Parquet implementation in Rust
Stars: ✭ 144 (-0.69%)
AcceleratorThe Accelerator is a tool for fast and reproducible processing of large amounts of data.
Stars: ✭ 137 (-5.52%)
Report自动化配置报表平台。演示地址http://58.87.112.247/report 账号 visitor密码123456
Stars: ✭ 123 (-15.17%)
PrestasharpCSharp .Net client library for the PrestaShop API via web service
Stars: ✭ 134 (-7.59%)
SlimmessagebusLightweight message bus interface for .NET (pub/sub and request-response) with transport plugins for popular message brokers.
Stars: ✭ 120 (-17.24%)
SigmfThe Signal Metadata Format Specification
Stars: ✭ 120 (-17.24%)
NsubstituteA friendly substitute for .NET mocking libraries.
Stars: ✭ 1,646 (+1035.17%)
Hdfs ShellHDFS Shell is a HDFS manipulation tool to work with functions integrated in Hadoop DFS
Stars: ✭ 117 (-19.31%)
NsoupNSoup is a .NET port of the jsoup (http://jsoup.org) HTML parser and sanitizer originally written in Java
Stars: ✭ 145 (+0%)
FleckC# Websocket Implementation
Stars: ✭ 1,869 (+1188.97%)
Borderless WindowA minimal borderless window with the Windows API
Stars: ✭ 131 (-9.66%)