All Projects → DTStack → Jlogstash

DTStack / Jlogstash

Licence: apache-2.0
java 版本的logstash

Programming Languages

java
68154 projects - #9 most used programming language

Projects that are alternatives of or similar to Jlogstash

Json Logging Python
Python logging library to emit JSON log that can be easily indexed and searchable by logging infrastructure such as ELK, EFK, AWS Cloudwatch, GCP Stackdriver
Stars: ✭ 143 (-42.8%)
Mutual labels:  logstash
Dockerfile
some personally made dockerfile
Stars: ✭ 2,021 (+708.4%)
Mutual labels:  logstash
Kafka Influxdb
High performance Kafka consumer for InfluxDB. Supports collectd message formats.
Stars: ✭ 206 (-17.6%)
Mutual labels:  logstash
Logrus mate
tool for logrus, let it easy to use
Stars: ✭ 148 (-40.8%)
Mutual labels:  logstash
Elk Docker
Elasticsearch, Logstash, Kibana (ELK) Docker image
Stars: ✭ 1,973 (+689.2%)
Mutual labels:  logstash
Microservices Sample
Sample project to create an application using microservices architecture
Stars: ✭ 167 (-33.2%)
Mutual labels:  logstash
Quicklogger
Library for logging on files, console, memory, email, rest, eventlog, syslog, slack, telegram, redis, logstash, elasticsearch, influxdb, graylog, Sentry, Twilio, ide debug messages and throw events for Delphi/Firemonkey/freepascal/.NET (Windows/Linux/OSX/IOS/Android).
Stars: ✭ 137 (-45.2%)
Mutual labels:  logstash
Logstash Output Jdbc
JDBC output for Logstash
Stars: ✭ 230 (-8%)
Mutual labels:  logstash
Sweetie Data
This repo contains logstash of various honeypots
Stars: ✭ 163 (-34.8%)
Mutual labels:  logstash
Docker Elastic
Deploy Elastic stack in a Docker Swarm cluster. Ship application logs and metrics using beats & GELF plugin to Elasticsearch
Stars: ✭ 202 (-19.2%)
Mutual labels:  logstash
Elk Stack
ELK Stack ... based on Elastic Stack 5.x
Stars: ✭ 148 (-40.8%)
Mutual labels:  logstash
Logrus Logstash Hook
ℹ️ Logstash hook for logrus
Stars: ✭ 150 (-40%)
Mutual labels:  logstash
Docker Elastic Stack
ELK Stack Dockerfile
Stars: ✭ 175 (-30%)
Mutual labels:  logstash
Filebeat Kubernetes
Filebeat container, alternative to fluentd used to ship kubernetes cluster and pod logs
Stars: ✭ 147 (-41.2%)
Mutual labels:  logstash
Microservice Scaffold
基于Spring Cloud(Greenwich.SR2)搭建的微服务脚手架(适用于在线系统),已集成注册中心(Nacos Config)、配置中心(Nacos Discovery)、认证授权(Oauth 2 + JWT)、日志处理(ELK + Kafka)、限流熔断(AliBaba Sentinel)、应用指标监控(Prometheus + Grafana)、调用链监控(Pinpoint)、以及Spring Boot Admin。
Stars: ✭ 211 (-15.6%)
Mutual labels:  logstash
Logstash Filter Verifier
Stars: ✭ 143 (-42.8%)
Mutual labels:  logstash
Synesis lite suricata
Suricata IDS/IPS log analytics using the Elastic Stack.
Stars: ✭ 167 (-33.2%)
Mutual labels:  logstash
Chipmunk
log analysis tool
Stars: ✭ 247 (-1.2%)
Mutual labels:  logstash
Go Stash
go-stash is a high performance, free and open source server-side data processing pipeline that ingests data from Kafka, processes it, and then sends it to ElasticSearch.
Stars: ✭ 214 (-14.4%)
Mutual labels:  logstash
Elastiflow
Network flow analytics (Netflow, sFlow and IPFIX) with the Elastic Stack
Stars: ✭ 2,322 (+828.8%)
Mutual labels:  logstash

注释:

  jlogstash前期的有部分代码引用了hangout项目里的代码,这里也感谢hangout的作者。

说明:

用java版本重写logstash目的是提升性能,跟ruby logstash 对比 请看 https://github.com/DTStack/jlogstash-performance-testing

不同的业务,性能会有所不同,dtstack 后台日志解析 java 版本是ruby版本的5倍,在单台4g 4cores 虚拟机上。

jlogstash 的参数配置和使用看wiki介绍,现在的插件跟ruby版本相比还太少,希望更多的人参与开发。

Inputs详情:

https://github.com/DTStack/jlogstash/tree/master/pipeline/inputs/README.md

Filters详情:

https://github.com/DTStack/jlogstash/tree/master/pipeline/filters/README.md

Outputs详情:

https://github.com/DTStack/jlogstash/tree/master/pipeline/outputs/README.md

性能采集:

目前支持将jlogstash运行时的性能数据采集到promethues.

详情:https://github.com/DTStack/jlogstash/tree/master/core/metrics.md

Jar放置目录(编译的jar必须要有版本号 ):

jlogstash 核心代码放在jlogstash/lib/下

插件的代码分别的放到jlogstash/plugin 下的filter,input,output目录下

启动命令:

sh jlogstash.sh -f /home/admin/ysq.yaml vv

启动参数:

-name:任务名称

-f:配置文件 yaml格式路径(必填)

-l:日志文件路径

-i:input queue size coefficient 默认 200f/1024

-w:filter work number 默认是根据的机器cpu核数+2

-o:output work number 默认是根据的机器cpu核数

-c:output queue size coefficient 默认 500f/1024

-dev: 开发模式,直接在pom.xml引用包即可。

v: error级别

vv: warn级别

vvv:info级别

vvvv:debug级别

vvvvv:trace级别

插件开发:

 1.现在各自的plugin 的包 都会有各自的classloder去加载,parent classloder是AppClassLoder,所以各自的plugin的代码即使引用了相同的第三的jar版   本不一样也不会导致版本冲突  

2.各个插件的代码不能相互引用,如果有公共代码需要打入到各自的jar包中

3.所需依赖到maven中心库 搜索 jlogstash(http://search.maven.org/https://oss.sonatype.org)

4.插件开发样列 https://github.com/DTStack/jlogstash/tree/master/src/test/java/com/dtstack/jlogstash

5.每一个plugin打的包名的前缀要跟插件的类名一致,不区分大小写,不然会报类找不到,列如:input.kafka-1.0.0.jar 或    kafka-1.0.0.jar

招聘:

1.大数据平台开发工程师,想了解岗位详细信息可以添加本人微信号ysqwhiletrue,注明招聘,如有意者发送简历至[email protected]

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].