ljcan / Sparkstreaming
Spark Streaming+Flume+Kafka+HBase+Hadoop+Zookeeper实现实时日志分析统计；SpringBoot+Echarts实现数据可视化展示
Stars: ✭ 349
Labels
Projects that are alternatives of or similar to Sparkstreaming
Spark Notebook
Interactive and Reactive Data Science using Scala and Spark.
Stars: ✭ 3,081 (+782.81%)
Mutual labels: spark
Learningsparkv2
This is the github repo for Learning Spark: Lightning-Fast Data Analytics [2nd Edition]
Stars: ✭ 307 (-12.03%)
Mutual labels: spark
Ytk Learn
Ytk-learn is a distributed machine learning library which implements most of popular machine learning algorithms(GBDT, GBRT, Mixture Logistic Regression, Gradient Boosting Soft Tree, Factorization Machines, Field-aware Factorization Machines, Logistic Regression, Softmax).
Stars: ✭ 337 (-3.44%)
Mutual labels: spark
Elasticluster
Create clusters of VMs on the cloud and configure them with Ansible.
Stars: ✭ 298 (-14.61%)
Mutual labels: spark
Delta
An open-source storage layer that brings scalable, ACID transactions to Apache Spark™ and big data workloads.
Stars: ✭ 3,903 (+1018.34%)
Mutual labels: spark
Clickhouse Native Jdbc
ClickHouse Native Protocol JDBC implementation
Stars: ✭ 310 (-11.17%)
Mutual labels: spark
Cloudflow
Cloudflow enables users to quickly develop, orchestrate, and operate distributed streaming applications on Kubernetes.
Stars: ✭ 278 (-20.34%)
Mutual labels: spark
Sparklens
Qubole Sparklens tool for performance tuning Apache Spark
Stars: ✭ 345 (-1.15%)
Mutual labels: spark
Wirbelsturm
Wirbelsturm is a Vagrant and Puppet based tool to perform 1-click local and remote deployments, with a focus on big data tech like Kafka.
Stars: ✭ 332 (-4.87%)
Mutual labels: spark
Awesome Ada
A curated list of awesome resources related to the Ada and SPARK programming language
Stars: ✭ 299 (-14.33%)
Mutual labels: spark
Cook
Fair job scheduler on Kubernetes and Mesos for batch workloads and Spark
Stars: ✭ 314 (-10.03%)
Mutual labels: spark
Spark Hbase Connector
Connect Spark to HBase for reading and writing data with ease
Stars: ✭ 299 (-14.33%)
Mutual labels: spark
Iql
An ad hoc query service based on the spark sql engine.(基于spark sql引擎的即席查询服务)
Stars: ✭ 341 (-2.29%)
Mutual labels: spark
Spark Druid Olap
Sparkline BI Accelerator provides fast ad-hoc query capability over Logical Cubes. This has been folded into our SNAP Platform(http://bit.ly/2oBJSpP) an Integrated BI platform on Apache Spark.
Stars: ✭ 282 (-19.2%)
Mutual labels: spark
Scalnet
A Scala wrapper for Deeplearning4j, inspired by Keras. Scala + DL + Spark + GPUs
Stars: ✭ 342 (-2.01%)
Mutual labels: spark
Sparklint
A tool for monitoring and tuning Spark jobs for efficiency.
Stars: ✭ 316 (-9.46%)
Mutual labels: spark
Sparing Streaming实时日志分析
- 前言：使用scala和java混编完成，其中也涉及到python脚本来自动生成日志，linux crontab调度工具来定时执行脚本生成实时日志。生成的数据主要是模拟某学习网站学习视频课程的访问量（其中模拟的日志中URL以"/class"开头的表示实战课程，然后通过流水线Flume+Kafka+SparkStreaming进行实时日志的收集，HBase来存储数据）*
注意事项（使用的软件工具及环境配置）
- hadoop-2.6.0-cdh5.7.0
- hbase-1.2.0-cdh5.7.0
- zookeeper-3.4.5-cdh5.7.0
- spark-2.2.0-bin-2.6.0-cdh5.7.0
- apache-flume-1.6.0-cdh5.7.0-bin
- kafka_2.11-0.9.0.0
- apache-maven-3.3.9
- jdk1.8.0_181
- scala-2.11.8
需求分析
- 统计该网站实战课程的访问量。
- 统计该网站实战课程从不同搜索引擎引流过来的访问量，通过结果可为该网站的课程广告投资的方向做出更准确的决策。
数据清洗
- 使用Spark Streaming剔除掉不符合要求的数据。
- 过滤URL以"/class"开头的数据清洗出实战课程相关数据。
数据分析
使用Spark Streaming进行实时日志分析然后写入HBase数据库。
Spark Streaming Web可视化展示
- 使用SpringBoot+Echarts开发可视化展示页面 *
可视化部分结果展示
需求一结果展示
需求二结果展示
为什么使用HBase来存储数据？
在数据量大的情况下存储数据，因为是实时存储数据，在向数据库中写入数据的时候，需要按天来累加访问量，如果使用其他的RDBMS，需要进行多步API操作，先按插入数据，然后按条件查询数据，最后更新数据，这样在一定会使应用程序变慢很多，也消耗系统资源。而如果使用HBase利用它的前缀查询，只需一步API就可以存储相应的数据。
Note that the project description data, including the texts, logos, images, and/or trademarks,
for each open source project belongs to its rightful owner.
If you wish to add or remove any projects, please contact us at [email protected].