ahujaraman / Live_log_analyzer_spark
Spark Application for analysis of Apache Access logs and detect anamolies! Along with Medium Article.
Stars: ✭ 14
Programming Languages
python
139335 projects - #7 most used programming language
Projects that are alternatives of or similar to Live log analyzer spark
Spark With Python
Fundamentals of Spark with Python (using PySpark), code examples
Stars: ✭ 150 (+971.43%)
Mutual labels: spark, analytics, apache-spark, pyspark
Azure Cosmosdb Spark
Apache Spark Connector for Azure Cosmos DB
Stars: ✭ 165 (+1078.57%)
Mutual labels: spark, apache-spark, pyspark
Sparkle
Haskell on Apache Spark.
Stars: ✭ 419 (+2892.86%)
Mutual labels: spark, analytics, apache-spark
Agile data code 2
Code for Agile Data Science 2.0, O'Reilly 2017, Second Edition
Stars: ✭ 413 (+2850%)
Mutual labels: spark, analytics, apache-spark
Spark
.NET for Apache® Spark™ makes Apache Spark™ easily accessible to .NET developers.
Stars: ✭ 1,721 (+12192.86%)
Mutual labels: spark, analytics, apache-spark
Mmlspark
Simple and Distributed Machine Learning
Stars: ✭ 2,899 (+20607.14%)
Mutual labels: spark, pyspark, apache-spark
aut
The Archives Unleashed Toolkit is an open-source toolkit for analyzing web archives.
Stars: ✭ 111 (+692.86%)
Mutual labels: spark, apache-spark, pyspark
Spark Structured Streaming Book
The Internals of Spark Structured Streaming
Stars: ✭ 371 (+2550%)
Mutual labels: spark, apache-spark
Redash
Make Your Company Data Driven. Connect to any data source, easily visualize, dashboard and share your data.
Stars: ✭ 20,147 (+143807.14%)
Mutual labels: spark, analytics
Devops Python Tools
80+ DevOps & Data CLI Tools - AWS, GCP, GCF Python Cloud Function, Log Anonymizer, Spark, Hadoop, HBase, Hive, Impala, Linux, Docker, Spark Data Converters & Validators (Avro/Parquet/JSON/CSV/INI/XML/YAML), Travis CI, AWS CloudFormation, Elasticsearch, Solr etc.
Stars: ✭ 406 (+2800%)
Mutual labels: spark, pyspark
Pyspark Boilerplate
A boilerplate for writing PySpark Jobs
Stars: ✭ 318 (+2171.43%)
Mutual labels: apache-spark, pyspark
Sparta
Real Time Analytics and Data Pipelines based on Spark Streaming
Stars: ✭ 513 (+3564.29%)
Mutual labels: spark, analytics
Sparkling Titanic
Training models with Apache Spark, PySpark for Titanic Kaggle competition
Stars: ✭ 12 (-14.29%)
Mutual labels: spark, pyspark
Sparkmeasure
This is the development repository of SparkMeasure, a tool for performance troubleshooting of Apache Spark workloads. It simplifies the collection and analysis of Spark task metrics data.
Stars: ✭ 368 (+2528.57%)
Mutual labels: spark, apache-spark
Kyuubi
Kyuubi is a unified multi-tenant JDBC interface for large-scale data processing and analytics, built on top of Apache Spark
Stars: ✭ 363 (+2492.86%)
Mutual labels: spark, analytics
Wirbelsturm
Wirbelsturm is a Vagrant and Puppet based tool to perform 1-click local and remote deployments, with a focus on big data tech like Kafka.
Stars: ✭ 332 (+2271.43%)
Mutual labels: spark, apache-spark
Kafka Storm Starter
Code examples that show to integrate Apache Kafka 0.8+ with Apache Storm 0.9+ and Apache Spark Streaming 1.1+, while using Apache Avro as the data serialization format.
Stars: ✭ 728 (+5100%)
Mutual labels: spark, apache-spark
Scriptis
Scriptis is for interactive data analysis with script development(SQL, Pyspark, HiveQL), task submission(Spark, Hive), UDF, function, resource management and intelligent diagnosis.
Stars: ✭ 696 (+4871.43%)
Mutual labels: spark, pyspark
live_log_analyzer_spark
Spark Application for analysis of Apache Access logs and detect anamolies!
Link to My Medium Article in Towards Data Science Publication : Article
The Application has features to extract information from unstrucutred text logs, and analyze the trends the trends based on Response codes, Traffic, Frequent Visitors, Top End Points, Content Flow etc to detect the anamolies.
Frequent End Points
Response Code Analysis
Traffic Flow Weekly Analysis
Traffic Flow Particular Day
The Above plot, depicts the anamolies of spikes on certain times, which can be crucial starting point to detect the anamolies.
Note that the project description data, including the texts, logos, images, and/or trademarks,
for each open source project belongs to its rightful owner.
If you wish to add or remove any projects, please contact us at [email protected].