All Projects → synacker → Daggy

synacker / Daggy

Licence: mit
Daggy - Data Aggregation Utility. Open source, free, cross-platform, server-less, useful utility for remote or local data aggregation and streaming

Projects that are alternatives of or similar to Daggy

Joincap
Merge multiple pcap files together, gracefully.
Stars: ✭ 159 (+74.73%)
Mutual labels:  network, network-analysis, pcap
Bruteshark
Network Analysis Tool
Stars: ✭ 1,335 (+1367.03%)
Mutual labels:  network-analysis, pcap, sniffing
Libtins
High-level, multiplatform C++ network packet sniffing and crafting library.
Stars: ✭ 1,609 (+1668.13%)
Mutual labels:  network, pcap, sniffing
Graph sampling
Graph Sampling is a python package containing various approaches which samples the original graph according to different sample sizes.
Stars: ✭ 99 (+8.79%)
Mutual labels:  data-mining, network, network-analysis
Deepgraph
Analyze Data with Pandas-based Networks. Documentation:
Stars: ✭ 232 (+154.95%)
Mutual labels:  data-mining, network, network-analysis
Nfstream
NFStream: a Flexible Network Data Analysis Framework.
Stars: ✭ 622 (+583.52%)
Mutual labels:  data-mining, network-analysis, pcap
Cocoadebug
iOS Debugging Tool 🚀
Stars: ✭ 3,769 (+4041.76%)
Mutual labels:  network, network-analysis, debugging-tool
Scapy
Scapy: the Python-based interactive packet manipulation program & library. Supports Python 2 & Python 3.
Stars: ✭ 6,932 (+7517.58%)
Mutual labels:  network, network-analysis, pcap
Net2pcap
Net2PCAP is a simple network-to-pcap capture file for Linux. Its goal is to be as simple as possible to be used in hostile environments
Stars: ✭ 36 (-60.44%)
Mutual labels:  network, pcap
Graphrole
Automatic feature extraction and node role assignment for transfer learning on graphs (ReFeX & RolX)
Stars: ✭ 38 (-58.24%)
Mutual labels:  network, network-analysis
Jiamimao
加密猫:一个免费、开源的文件加密软件
Stars: ✭ 48 (-47.25%)
Mutual labels:  serverless, serverless-applications
Erewhon Game
Video game about programming your spaceships to destroy other programmed spaceships o/
Stars: ✭ 35 (-61.54%)
Mutual labels:  network, cross-platform
Xiringuito
SSH-based "VPN for poors"
Stars: ✭ 969 (+964.84%)
Mutual labels:  network, ssh
Moveit
🚀 NLW #4 | React+ TypeScript + NextJS + StyledComponents + Firebase + MongoDb +Axios
Stars: ✭ 39 (-57.14%)
Mutual labels:  serverless, serverless-applications
Sandpolis
Experimental remote monitoring and management
Stars: ✭ 30 (-67.03%)
Mutual labels:  cross-platform, administration
Cheatsheat Tcpdump
cheatsheat-tcpdump
Stars: ✭ 54 (-40.66%)
Mutual labels:  network, sniffing
Sshkey Audit
Automate SSH key management
Stars: ✭ 55 (-39.56%)
Mutual labels:  network, ssh
Subdue
The Subdue graph miner discovers highly-compressing patterns in an input graph.
Stars: ✭ 20 (-78.02%)
Mutual labels:  data-mining, network
Serverless Application Model
AWS Serverless Application Model (SAM) is an open-source framework for building serverless applications
Stars: ✭ 8,305 (+9026.37%)
Mutual labels:  serverless, serverless-applications
Ios Sdk
AppSpector is a debugging service for mobile apps
Stars: ✭ 56 (-38.46%)
Mutual labels:  network-analysis, debugging-tool

synacker


Daggy - Data Aggregation Utility

Daggy - Data Aggregation Utility. Server-less remote or local data aggregation and streaming.

Daggy main goals are simplicity and ease-of-use.

Daggy is server-less, cross-platform solution and don't require installation on remote servers. Aggregation and streaming work under SSH transport protocol or via local processes execution.

Daggy can be helpful for developers, QA, DevOps and engineers for debug, analyze and control distributed network systems, for example, based on micro-service architecture.

Getting Started

Getting Daggy

Windows

Download installation package (.exe) or portable (.zip) from last release page and install it

MacOS

brew install https://raw.githubusercontent.com/synacker/daggy_homebrew/master/daggy.rb

Fedora

sudo dnf install daggy daggy-devel

Linux

Download .rpm/.deb or portable .zip package from last release page and install it

All daggy dependencies (Qt5, libssh2, libyaml-cpp) are included into separate dir and does not effect system applications.

Build from source

Environment requirenments

Conan, cmake, git and C++17 compiler.

Build steps

git clone https://github.com/synacker/daggy.git
mkdir build
cd build
conan remote add bincrafters https://api.bintray.com/conan/bincrafters/public-conan
conan install ../daggy --build=missing
conan build ../daggy
cmake install

Check installation

daggy --help
Usage: daggy [options] *.yaml|*.yml|*.json

Options:
  -o, --output <folder>       Set output folder
  -f, --format <json|yaml>    Source format
  -i, --stdin                 Read data aggregation sources from stdin
  -t, --timeout <time in ms>  Auto complete timeout
  -h, --help                  Displays help on commandline options.
  --help-all                  Displays help including Qt specific options.
  -v, --version               Displays version information.

Arguments:
  file                        data aggregation sources file

Getting Started with data aggregation and streaming

Simple Data Aggregation Source

Create simple.yaml

sources:
    localhost:
        type: local
        commands:
            pingYa:
                exec: ping ya.ru
                extension: log

Run daggy

daggy simple.yaml

Check console output

23:07:23:977 | AppStat  | Start aggregation in 01-04-20_23-07-23-977_simple
23:07:23:977 | ProvStat | localhost       | New state: Started
23:07:23:977 | CommStat | localhost       | pingYa          | New state: Starting
23:07:23:977 | CommStat | localhost       | pingYa          | New state: Started

There are all commands from simple.yaml/simple.json are streams in 01-04-20_23-07-23-977_simple with output files

Tailing streams from Simple Data Source

tail -f 01-04-20_23-07-23-977_simple/*
64 bytes from ya.ru (87.250.250.242): icmp_seq=99 ttl=249 time=21.2 ms
64 bytes from ya.ru (87.250.250.242): icmp_seq=100 ttl=249 time=18.8 ms
64 bytes from ya.ru (87.250.250.242): icmp_seq=101 ttl=249 time=23.5 ms
64 bytes from ya.ru (87.250.250.242): icmp_seq=102 ttl=249 time=18.8 ms
64 bytes from ya.ru (87.250.250.242): icmp_seq=103 ttl=249 time=18.8 ms
64 bytes from ya.ru (87.250.250.242): icmp_seq=104 ttl=249 time=17.4 ms
64 bytes from ya.ru (87.250.250.242): icmp_seq=105 ttl=249 time=17.4 ms
64 bytes from ya.ru (87.250.250.242): icmp_seq=106 ttl=249 time=20.1 ms
64 bytes from ya.ru (87.250.250.242): icmp_seq=107 ttl=249 time=25.8 ms
64 bytes from ya.ru (87.250.250.242): icmp_seq=108 ttl=249 time=35.1 ms
64 bytes from ya.ru (87.250.250.242): icmp_seq=109 ttl=249 time=21.1 ms

Stop data aggregation and streaming

Type CTRL+C for stopping data aggregation and streaming. Type CTRL+C twice for hard stop application, without waiting cancelation of child local and remote processes.

23:07:23:977 | AppStat  | Start aggregation in 01-04-20_23-07-23-977_simple
23:07:23:977 | ProvStat | localhost       | New state: Started
23:07:23:977 | CommStat | localhost       | pingYa          | New state: Starting
23:07:23:977 | CommStat | localhost       | pingYa          | New state: Started
^C23:17:56:667 | ProvStat | localhost       | New state: Finishing
23:17:56:668 | CommStat | localhost       | pingYa          | New state: Finished. Exit code: 0
23:17:56:668 | ProvStat | localhost       | New state: Finished
23:17:56:668 | AppStat  | Stop aggregation in 01-04-20_23-07-23-977_simple

Investigate aggregated data

ls -l 01-04-20_23-07-23-977_simple/
-rw-r--r-- 1 muxa muxa 45574 апр  1 23:17 localhost-pingYa.log

Example of Data Aggregation Sources with multiple commands and remote data aggregation and streaming

aliases:  
    - &my_commands
        pingYa:
            exec: ping ya.ru
            extension: log
        pingGoo:
            exec: ping goo.gl
            extension: log
        
    - &ssh_auth
        user: {{env_USER}}
        passphrase: {{env_PASSWORD}}
            
sources:
    localhost:
        type: local
        commands: *my_commands
    remotehost:
        host: 192.168.1.9
        type: ssh2
        parameters: *ssh_auth
        commands: *my_commands
    remotehost2:
        host: 192.168.1.9
        type: ssh2
        parameters: *ssh_auth
        commands: *my_commands
    remotehost3:
        host: 192.168.1.9
        type: ssh2
        parameters: *ssh_auth
        commands: *my_commands
Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].