gosquito

gosquito ("go" + "mosquito") is a pluggable tool for data gathering, data processing and data transmitting to various destinations. Main goal is to replace various in-house automated tasks with a single tool and move those tasks at the edge. See docs and examples for additional info.

graph TD;
    I1(kafka)-->P1(process plugins);
    I2(resty)-->P1;
    I3(rss)-->P1;
    I4(telegram)-->P1;
    I5(twitter)-->P1;
    P1-->O1(kafka);
    P1-->O2(mattermost);
    P1-->O3(resty);
    P1-->O4(slack);
    P1-->O5(smtp);
    P1-->O6(telegram);

Main features:

• Pluggable architecture. Data processing organized as chains of plugins.
• Flow approach. Flow consists of: input plugin (receive), process plugins (filter/transform), output plugin (send).
• Declarative YAML configurations with templates support.
• Consider data as new by configurable signature (data timestamp by default). Force fetching is supported.
• Dependencies between process plugins. Plugin "B" will process data only if plugin "A" derived some data.
• Parallel executions limits.
• Export metrics to Prometheus.

Input plugins:

Process plugins:

Output plugins:

ToDo:

0. prometheus: last message received timestamp.
1. prometheus: extend metrics with invalid flows.
2. core: cron mode.
3. regexp/ml: storage of interests.
4. input: match_ignore_case option for match_signature.
5. docs: add complex examples, docker compose environments.
6. telegram: add careful (api limits) support for download/sending unread files/messages.
7. plugins: new (echo plugin more suitable?) file plugin for saving text to files.
8. core/ml: auto learning interval.
9. process: ocr plugin.
10. process: lang detect plugin.
11. core: file deduplication cache.
12. process: exec plugin.
13. core: top content ratio.
14. core: flow schedule.
15. kafka: warn about connection problems.
16. core: log level for flow.
17. core: flow enable/disable regexp support.