GitPlanet

Cheap and reliable Node.js hosting starts at $3/month, and $1/month static HTML hosting

Created with love in Canada, visit hostnodejs.com today

Feel like to post an Ad? Learn Details
All Projects → tyjhart → flowanalyzer

tyjhart / flowanalyzer

Licence: BSD-3-Clause license
Manito Networks Flow Analyzer

Programming Languages

python
139335 projects - #7 most used programming language
shell
77523 projects

Labels

elasticsearchkibanasflownetflowipfixnetflow-v9netflow-v5

Projects that are alternatives of or similar to flowanalyzer

Elastiflow
Network flow analytics (Netflow, sFlow and IPFIX) with the Elastic Stack
Stars: ✭ 2,322 (+5428.57%)
Mutual labels:  kibana, sflow, netflow, ipfix
goflow2
High performance sFlow/IPFIX/NetFlow Collector
Stars: ✭ 125 (+197.62%)
Mutual labels:  sflow, netflow, ipfix
Ntopng
Web-based Traffic and Security Network Traffic Monitoring
Stars: ✭ 4,313 (+10169.05%)
Mutual labels:  sflow, netflow, ipfix
Fastnetmon
FastNetMon - very fast DDoS sensor with sFlow/Netflow/IPFIX/SPAN support
Stars: ✭ 2,860 (+6709.52%)
Mutual labels:  sflow, netflow, ipfix
Dynamite Nsm
DynamiteNSM is a free Network Security Monitor developed by Dynamite Analytics to enable network visibility and advanced cyber threat detection
Stars: ✭ 92 (+119.05%)
Mutual labels:  kibana, netflow
docker grafana statsd elk
Docker repo for a general purpose graphing and logging container - includes graphite+carbon, grafana, statsd, elasticsearch, kibana, nginx, logstash indexer (currently using redis as an intermediary)
Stars: ✭ 19 (-54.76%)
Mutual labels:  kibana
ncedc-earthquakes
The complete set of earthquake data with the Elastic Stack demo.
Stars: ✭ 22 (-47.62%)
Mutual labels:  kibana
osint-combiner
Combining OSINT sources in Elastic Stack
Stars: ✭ 77 (+83.33%)
Mutual labels:  kibana
EnterpriseApplicationLog
Enterprise Application Log with RabbitMQ, LogStash, ElasticSearch and Kibana
Stars: ✭ 88 (+109.52%)
Mutual labels:  kibana
elk-dashboard-v5-docker
My production setup for the latest version of ELK stack running in a compose, displaying a basic -but powerfull- security and performance dashboard.
Stars: ✭ 25 (-40.48%)
Mutual labels:  kibana
generator-mitosis
A micro-service infrastructure generator based on Yeoman/Chatbot, Kubernetes/Docker Swarm, Traefik, Ansible, Jenkins, Spark, Hadoop, Kafka, etc.
Stars: ✭ 78 (+85.71%)
Mutual labels:  kibana
kibana-plugin-metric-percent
Plugin for Kibana Version 4.6 or 5.x.x or 6.x.x
Stars: ✭ 31 (-26.19%)
Mutual labels:  kibana
search-guard-module-kibana-multitenancy
[OUTDATED] Provide multitenancy and other features for kibana
Stars: ✭ 15 (-64.29%)
Mutual labels:  kibana
Location-based-Restaurants-Recommendation-System
Big Data Management and Analysis Final Project
Stars: ✭ 44 (+4.76%)
Mutual labels:  kibana
rubban
Kibana Automatic Index Pattern Discovery and Other Elastic Stack Curating Tasks
Stars: ✭ 49 (+16.67%)
Mutual labels:  kibana
vagrant-vault-consul-docker-monitoring
No description or website provided.
Stars: ✭ 20 (-52.38%)
Mutual labels:  kibana
kibana notification center
This plugin helps you to use kibana's notifications more usefully.
Stars: ✭ 33 (-21.43%)
Mutual labels:  kibana
kbn circles vis
Kibana 4.4.1 D3 Circles Packing Visualization
Stars: ✭ 30 (-28.57%)
Mutual labels:  kibana
elk-upgrade
Elastic Stack Upgrade with Ansible
Stars: ✭ 28 (-33.33%)
Mutual labels:  kibana
jmx-monitoring-stacks
No description or website provided.
Stars: ✭ 170 (+304.76%)
Mutual labels:  kibana
View All Similar Projects ➔

Manito Networks Flow Analyzer

The Flow Analyzer is a Netflow, IPFIX, and sFlow collector and parser, available under the BSD 3-Clause License, that stores flows in Elasticsearch and visualizes them in Kibana. It is designed to run on Ubuntu Server, either as a single installation or as part of an Elasticsearch cluster.

Visualizations and Dashboards are provided to support network flow analysis right out of the box.

See the License section below for licensing details.

  1. Project Goals
  2. Features
  3. Requirements
  4. Installation
  5. Ports and Protocols
  6. Data Access
  7. Limitations
  8. Debugging
  9. Contributing
  10. License
  11. Attributions

Project Goals

Our goal is to provide superior Netflow and IPFIX collection, visualization, and analysis. We do that by creating:

  • Efficient, accessible, and sustainable software
  • Scalable solutions that can evolve as you grow
  • Superior documentation from architecture through installation, configuration, tuning, and troubleshooting

One other goal is to make Elasticsearch and Kibana easy to implement and accessible to those who haven't used it before. The learning curve for distributed search systems and dashboarding software can be steap, but we think that everyone should be able to realize the benefits of meaningful, beautiful data visualization.

Features

The Flow Analyzer has the flow collection, tagging, and categorizing capabilities to satisfy enterprise and service provider requirements.

Quick Installation

Go from zero to visualized flow data in less than one hour. See the installation documentation for easy to follow step-by-step instructions.

Broad Protocol Support

The Manito Networks Flow Analyzer supports the following flow data protocols:

  • Netflow v5 (Cisco)
  • Netflow v9 (Cisco)
  • IPFIX (IEEE, aka Netflow v10)
  • sFlow (InMon Corporation)
  • Traffic Flow (Mikrotik, Netflow-equivalent)
  • Netstream (Huawei Technologies, Netflow-equivalent)

If you're not familiar with Netflow or IPFIX take a look at the Introduction to Netflow and IPFIX. For information on sFlow and supported sFlow structures see the Flow Analyzer sFlow document. Our software collects Netflow, IPFIX, and sFlow data and then parses and tags it. The parsed and tagged data is stored in Elasticsearch for you to query and graph in Kibana.

Industry-Standard Fields

The Flow Analyzer supports all Netflow v5 fields, non-proprietary Netflow v9 fields, IPFIX fields specified in the RFC, and almost all sFlow structures defined by InMon Corporation's enterprise ID. See the Fields document for a description of Netflow (v5, v9) and IPFIX fields. The sFlow document includes descriptions of supported Flow and Counter structures.

Kibana Visualizations and Dashboards are included so you can leverage supported fields and structures right away.

Some limitations exist, mostly around proprietary or undocumented fields in Netflow and proprietary structures in sFlow. See the Limitations section for details. Efforts are made to skip over unsupported or proprietary elements and continue parsing data uninterrupted.

Service and Protocol Tagging

Our custom Netflow, IPFIX, and sFlow collectors ingest, parse, and tag flow data. We record not only the basic protocol and port numbers but we also take it a step further and correlate the following:

  • IANA protocol numbers to names (eg protocol 1 to "ICMP", 6 to "TCP", 89 to "OSPF")
  • IANA-registered port numbers to services (eg port 80 to "HTTP", 53 to "DNS")
  • Services to categories (eg HTTP, HTTPS, Alt-HTTP to "Web")

This tagging functionality is running by default and happens transparently in the background. For more information on tagging functionality see the Tagging documentation.

DNS Reverse Lookups

A reverse lookup against observed IPs is done if DNS lookups are enabled. Resolved domains are cached for 30 minutes to reduce the impact on DNS servers. Popular domains like facebook.com and cnn.com are categorized with content tags like "Social Media" and "News" to provide insight into website browsing on the network.

MAC Address Correlation

Correlation of MAC address OUI's to top manufacturers is done to help graph traffic sources in hetergenous environments.

Note: This feature is in beta, and the list of OUI's to be built is quite extensive.

Requirements

See the Requirements documentation for information on RAM, CPU, storage, and operating system requirements.

Installation

Clone the Git repo, run the installation script, and import the pre-built Kibana objects. See the installation documentation for more information.

Ports and Protocols

All services listen for UDP flow packets on the following default ports:

Service Protocol Default Port Purpose
Netflow v5 UDP 2055 Basic flow monitoring
Netflow v9 UDP 9995 Intermediate flow monitoring
IPFIX UDP 4739 Advanced flow monitoring
sFlow UDP 6343 Advanced flow and performance monitoring

Data Access

Access your flow data however you want - Kibana dashboards, Elasticsearch JSON-formatted queries, or curl HTTP requests. Access to Kibana can be restricted using Squid via a reverse proxy and the directions for setting that up are included.

See the installation documentation for more information.

Limitations

The following protocols, vendor features, or vendor-proprietary functions are NOT supported by the Flow Analyzer project.

Vendor Features and Protocols

These technologies may use a supported protocol for transport but there are proprietary fields, codes, or structures in use. Some protocols may require parsing that is undocumented or proprietary to the vendor.

Protocol Limitations

  • Netflow v5 does not support IPv6.

Debugging

If you run into any issues during or after installation check out the Debugging page for helpful commands and debugging options.

Contributing

We encourage people who use the Flow Analyzer to contribute to the project if they find a bug or documentation issue, or want to see a feature added. See the Contributing page for more information about contributing code to the project. Interactions between participants should be within the bounds of the Code of Conduct.

License

Copyright (c) 2017, Manito Networks, LLC All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

  1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.

  2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

  3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Attributions

Trademark, copyright, and license disclosures below:

  • "Elasticsearch" and "Kibana" are registered trademarks of Elasticsearch BV.
  • "Elasticsearch" and "Kibana" are distributed under the Apache 2 license by Elasticsearch BV.
  • "Ubuntu" is a registered trademark of Canonical Ltd.
  • "sFlow" is a registered trademark of InMon Corporation.
  • "Cisco" is a registered trademark of Cisco Systems, Inc.
  • "Mikrotik" is a trademark of Mikrotikls SIA.
  • "Huawei" is a trademark of Huawei Technologies Co., Ltd.
  • "NVIDIA" is a trademark of NVIDIA Corporation.
  • "Broadcom" is a trademark of AVAGO TECHNOLOGIES GENERAL IP (SINGAPORE) PTE. LTD.

---

Copyright (c) 2017, Manito Networks, LLC All rights reserved.

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].