xinghalo / Teddy
Spark Streaming监控平台,支持任务部署与告警、自启动
Stars: ✭ 120
Programming Languages
java
68154 projects - #9 most used programming language
Projects that are alternatives of or similar to Teddy
fastdata-cluster
Fast Data Cluster (Apache Cassandra, Kafka, Spark, Flink, YARN and HDFS with Vagrant and VirtualBox)
Stars: ✭ 20 (-83.33%)
Mutual labels: spark, yarn
Sparta
Real Time Analytics and Data Pipelines based on Spark Streaming
Stars: ✭ 513 (+327.5%)
Mutual labels: spark, streaming
kafka-spark-streaming-zeppelin-docker
One click deploy docker-compose with Kafka, Spark Streaming, Zeppelin UI and Monitoring (Grafana + Kafka Manager)
Stars: ✭ 82 (-31.67%)
Mutual labels: streaming, spark
Azure Event Hubs Spark
Enabling Continuous Data Processing with Apache Spark and Azure Event Hubs
Stars: ✭ 140 (+16.67%)
Mutual labels: spark, streaming
Cuesheet
A framework for writing Spark 2.x applications in a pretty way
Stars: ✭ 86 (-28.33%)
Mutual labels: spark, yarn
Data Accelerator
Data Accelerator for Apache Spark simplifies onboarding to Streaming of Big Data. It offers a rich, easy to use experience to help with creation, editing and management of Spark jobs on Azure HDInsights or Databricks while enabling the full power of the Spark engine.
Stars: ✭ 247 (+105.83%)
Mutual labels: spark, streaming
Enterprise gateway
A lightweight, multi-tenant, scalable and secure gateway that enables Jupyter Notebooks to share resources across distributed clusters such as Apache Spark, Kubernetes and others.
Stars: ✭ 412 (+243.33%)
Mutual labels: spark, yarn
Azure Event Hubs
☁️ Cloud-scale telemetry ingestion from any stream of data with Azure Event Hubs
Stars: ✭ 233 (+94.17%)
Mutual labels: spark, streaming
Spark Twitter Stream Example
"Sentiment analysis" on a live Twitter feed with Apache Spark and Apache Bahir
Stars: ✭ 73 (-39.17%)
Mutual labels: spark, streaming
Bigdata Interview
🎯 🌟[大数据面试题]分享自己在网络上收集的大数据相关的面试题以及自己的答案总结.目前包含Hadoop/Hive/Spark/Flink/Hbase/Kafka/Zookeeper框架的面试题知识总结
Stars: ✭ 857 (+614.17%)
Mutual labels: spark, yarn
Spark
.NET for Apache® Spark™ makes Apache Spark™ easily accessible to .NET developers.
Stars: ✭ 1,721 (+1334.17%)
Mutual labels: spark, streaming
Kyuubi
Kyuubi is a unified multi-tenant JDBC interface for large-scale data processing and analytics, built on top of Apache Spark
Stars: ✭ 363 (+202.5%)
Mutual labels: spark, yarn
Mobius
C# and F# language binding and extensions to Apache Spark
Stars: ✭ 929 (+674.17%)
Mutual labels: spark, streaming
Flink Learning
flink learning blog. http://www.54tianzhisheng.cn/ 含 Flink 入门、概念、原理、实战、性能调优、源码解析等内容。涉及 Flink Connector、Metrics、Library、DataStream API、Table API & SQL 等内容的学习案例,还有 Flink 落地应用的大型项目案例(PVUV、日志存储、百亿数据实时去重、监控告警)分享。欢迎大家支持我的专栏《大数据实时计算引擎 Flink 实战与性能优化》
Stars: ✭ 11,378 (+9381.67%)
Mutual labels: spark, streaming
Spark Mllib Twitter Sentiment Analysis
🌟 ✨ Analyze and visualize Twitter Sentiment on a world map using Spark MLlib
Stars: ✭ 113 (-5.83%)
Mutual labels: spark
Cube.js
📊 Cube — Open-Source Analytics API for Building Data Apps
Stars: ✭ 11,983 (+9885.83%)
Mutual labels: spark
Teddy 泰迪
项目背景 公司的spark集群是CDH,暂时不能良好的支持spark2.2的streaming,所以考虑自己写一个监控程序。 至于为什么要起这样一个名字,一方面公司的项目大多以狗狗的名字命名,另外,自己还养了一只小泰迪(名字叫做 三月),希望自己也能用心的对待这个项目。
主要功能
- Streaming任务部署
- 任务监控与告警
- 任务自启动
- (new)任务资源自定义配置
- (new)Spark Streaming代码零侵入
效果展示
使用说明
前提:
- 部署web服务的机器上需要有spark(否则无法提交任务)
- 部署web服务的机器上需要能使用app命令操作yarn(否则无法手动停止任务)
1 主要需要注意的参数
###########
# 环境配置 #
###########
# 本地spark的安装目录
spark.home=/var/lib/hadoop-hdfs/app/spark
# 本地资源的上传目录
lib.home=/home/xinghailong/monitor/lib/
# spark任务重定向日志文件,如果不嫌弃,可以直接重定向到日志文件
log.file=/home/xinghailong/monitor2/teddy/logs/teddy.log
# yarn的连接地址,用于Http方式查询spark任务的状态
yarn.cluster=hnode1:8088,hnode2:8088
# 邮件配置
mail.host=smtp.mxhichina.com
[email protected]
mail.passwd=123
###########
# 性能配置 #
###########
# 告警时间配置,秒单位,默认一分钟
# 注意:如果告警时间很短,小心邮件爆炸!
alert.interval=60
# 状态刷新时间配置,秒单位,默认5秒钟
state.refresh.interval=5
# 自动重启间隔时间,3分钟
# 注意:如果自定重启时间很短,可能会导致任务的重复启动
auto.restart.interval=180
# 尝试重启的次数
auto.restart.retries=3
2 启动teddy
下载代码后执行mvn install,即可编译打包项目工程。如果不想编译,也可以直接下载下面的版本:
待补充
编译后,在target目录下,可以找到对应的teddy-release压缩包。 目前仅支持Linux系统,修改相应的环境配置,启动即可。
链接
Note that the project description data, including the texts, logos, images, and/or trademarks,
for each open source project belongs to its rightful owner.
If you wish to add or remove any projects, please contact us at [email protected].