All Projects → HariSekhon → Haproxy Configs

HariSekhon / Haproxy Configs

80+ HAProxy Configs for Hadoop, Big Data, NoSQL, Docker, Elasticsearch, SolrCloud, HBase, MySQL, PostgreSQL, Apache Drill, Hive, Presto, Impala, Hue, ZooKeeper, SSH, RabbitMQ, Redis, Riak, Cloudera, OpenTSDB, InfluxDB, Prometheus, Kibana, Graphite, Rancher etc.

Programming Languages

shell
77523 projects

Projects that are alternatives of or similar to Haproxy Configs

Szt Bigdata
深圳地铁大数据客流分析系统🚇🚄🌟
Stars: ✭ 826 (+679.25%)
Mutual labels:  zookeeper, hadoop, hive, mysql, redis, elasticsearch, hbase
Nagios Plugins
450+ AWS, Hadoop, Cloud, Kafka, Docker, Elasticsearch, RabbitMQ, Redis, HBase, Solr, Cassandra, ZooKeeper, HDFS, Yarn, Hive, Presto, Drill, Impala, Consul, Spark, Jenkins, Travis CI, Git, MySQL, Linux, DNS, Whois, SSL Certs, Yum Security Updates, Kubernetes, Cloudera etc...
Stars: ✭ 1,000 (+843.4%)
Mutual labels:  zookeeper, hadoop, mysql, redis, elasticsearch, cassandra, hbase
Bigdata docker
Big Data Ecosystem Docker
Stars: ✭ 161 (+51.89%)
Mutual labels:  zookeeper, hadoop, hive, presto, mysql, hbase
Pifpaf
Python fixtures and daemon managing tools for functional testing
Stars: ✭ 161 (+51.89%)
Mutual labels:  zookeeper, mysql, redis, postgresql, elasticsearch, influxdb
Spring Boot 2.x Examples
Spring Boot 2.x code examples
Stars: ✭ 104 (-1.89%)
Mutual labels:  presto, mysql, redis, postgresql, elasticsearch, hbase
Stream Reactor
Streaming reference architecture for ETL with Kafka and Kafka-Connect. You can find more on http://lenses.io on how we provide a unified solution to manage your connectors, most advanced SQL engine for Kafka and Kafka Streams, cluster monitoring and alerting, and more.
Stars: ✭ 753 (+610.38%)
Mutual labels:  redis, elasticsearch, influxdb, cassandra, hbase
Firecamp
Serverless Platform for the stateful services
Stars: ✭ 194 (+83.02%)
Mutual labels:  zookeeper, redis, postgresql, elasticsearch, cassandra
Dockerfiles
50+ DockerHub public images for Docker & Kubernetes - Hadoop, Kafka, ZooKeeper, HBase, Cassandra, Solr, SolrCloud, Presto, Apache Drill, Nifi, Spark, Consul, Riak, TeamCity and DevOps tools built on the major Linux distros: Alpine, CentOS, Debian, Fedora, Ubuntu
Stars: ✭ 847 (+699.06%)
Mutual labels:  zookeeper, hadoop, presto, cassandra, hbase
Bigdata
💎🔥大数据学习笔记
Stars: ✭ 488 (+360.38%)
Mutual labels:  zookeeper, hadoop, hive, mysql, hbase
dockerfiles
Multi docker container images for main Big Data Tools. (Hadoop, Spark, Kafka, HBase, Cassandra, Zookeeper, Zeppelin, Drill, Flink, Hive, Hue, Mesos, ... )
Stars: ✭ 29 (-72.64%)
Mutual labels:  cassandra, hive, hadoop, hbase, zookeeper
Springboot Templates
springboot和dubbo、netty的集成,redis mongodb的nosql模板, kafka rocketmq rabbit的MQ模板, solr solrcloud elasticsearch查询引擎
Stars: ✭ 100 (-5.66%)
Mutual labels:  zookeeper, hive, redis, elasticsearch, hbase
Flink Learning
flink learning blog. http://www.54tianzhisheng.cn/ 含 Flink 入门、概念、原理、实战、性能调优、源码解析等内容。涉及 Flink Connector、Metrics、Library、DataStream API、Table API & SQL 等内容的学习案例,还有 Flink 落地应用的大型项目案例(PVUV、日志存储、百亿数据实时去重、监控告警)分享。欢迎大家支持我的专栏《大数据实时计算引擎 Flink 实战与性能优化》
Stars: ✭ 11,378 (+10633.96%)
Mutual labels:  mysql, redis, elasticsearch, influxdb, hbase
Dev Setup
macOS development environment setup: Easy-to-understand instructions with automated setup scripts for developer tools like Vim, Sublime Text, Bash, iTerm, Python data analysis, Spark, Hadoop MapReduce, AWS, Heroku, JavaScript web development, Android development, common data stores, and dev-based OS X defaults.
Stars: ✭ 5,590 (+5173.58%)
Mutual labels:  mysql, redis, postgresql, elasticsearch
Smartsql
SmartSql = MyBatis in C# + .NET Core+ Cache(Memory | Redis) + R/W Splitting + PropertyChangedTrack +Dynamic Repository + InvokeSync + Diagnostics
Stars: ✭ 775 (+631.13%)
Mutual labels:  zookeeper, mysql, redis, postgresql
Books Recommendation
程序员进阶书籍(视频),持续更新(Programmer Books)
Stars: ✭ 558 (+426.42%)
Mutual labels:  zookeeper, mysql, elasticsearch, prometheus
Bigdataguide
大数据学习,从零开始学习大数据,包含大数据学习各阶段学习视频、面试资料
Stars: ✭ 817 (+670.75%)
Mutual labels:  zookeeper, hadoop, hive, hbase
Repository
个人学习知识库涉及到数据仓库建模、实时计算、大数据、Java、算法等。
Stars: ✭ 92 (-13.21%)
Mutual labels:  zookeeper, hadoop, hive, hbase
Javakeeper
✍️ Java 工程师必备架构体系知识总结:涵盖分布式、微服务、RPC等互联网公司常用架构,以及数据存储、缓存、搜索等必备技能
Stars: ✭ 502 (+373.58%)
Mutual labels:  zookeeper, mysql, redis, elasticsearch
Docs4dev
后端开发常用框架文档及中文翻译,包含 Spring 系列文档(Spring, Spring Boot, Spring Cloud, Spring Security, Spring Session),大数据(Apache Hive, HBase, Apache Flume),日志(Log4j2, Logback),Http Server(NGINX,Apache),Python,数据库(OpenTSDB,MySQL,PostgreSQL)等最新官方文档以及对应的中文翻译。
Stars: ✭ 974 (+818.87%)
Mutual labels:  hive, mysql, postgresql, hbase
Hadoop cookbook
Cookbook to install Hadoop 2.0+ using Chef
Stars: ✭ 82 (-22.64%)
Mutual labels:  zookeeper, hadoop, hive, hbase

Advanced HAProxy Configs for Big Data, NoSQL, Web and Infrastructure technologies

Codacy CodeFactor Quality Gate Status Maintainability Rating Reliability Rating Security Rating GitHub stars GitHub forks GitHub Last Commit Lines of Code

Linux Mac Docker StarTrack StarCharts

CI Builds Overview Jenkins Concourse GoCD TeamCity

Travis CI AppVeyor Drone CircleCI Codeship Status for HariSekhon/HAProxy-configs Shippable BuildKite Codefresh Cirrus CI Semaphore Wercker

Azure DevOps Pipeline BitBucket Pipeline GitLab Pipeline AWS CodeBuild GCP Cloud Build

Repo on Azure DevOps Repo on GitHub Repo on GitLab Repo on BitBucket

GitHub Actions Ubuntu Mac Mac 10.15 Ubuntu Ubuntu 14.04 Ubuntu 16.04 Ubuntu 18.04 Ubuntu 20.04 Debian Debian 8 Debian 9 Debian 10 CentOS CentOS 7 CentOS 8 Fedora Alpine Alpine 3

git.io/haproxy-configs

Advanced HAProxy configurations for Multi-Master, Active-Standby (Hadoop, HBase, Presto) and Peer-to-Peer technologies (Elasticsearch, SolrCloud etc).

Designed both for production-grade High Availability and also to make scripting and monitoring easier when connecting to APIs.

Forked as a submodule from the Advanced Nagios Plugins Collection, these configurations contain specialised Health Checks for each system based on experience and code from the Advanced Nagios Plugins Collection and DevOps Python Tools github repos.

Recommended to run with High Availability HAProxy using VRRP to create full production-grade High Availability load balancer solutions.

Hari Sekhon

Cloud & Big Data Contractor, United Kingdom

(ex-Cloudera, former Hortonworks Consultant)

My LinkedIn

(you're welcome to connect with me on LinkedIn)

Features

Each config comes pre-tuned with:

  1. Specialised Health Checks per technology's API
  2. Load Balancing algorithm selected best suited per technology (depending on Active/Passive, Peer-to-Peer etc)
  3. Various buffer sizes, keepalives, connection limits and server retries already tuned (many via shared global config 10-global.cfg)
  4. ACL protections - limits access to private IP addresses - safer by default but easy to disable if you need to serve the internet
  5. Additional logging
  6. Optional Stats Admin UI (excellent for admin, monitoring and debugging - password protected, change the default password from 'test' in 20-stats.cfg if you source that config)

You should use an expert consultant to fine tune to your needs but these should be extremely close to your finished production configurations. In most cases all you need to do is put in your addresses for the backend servers.

Usage

Configurations are split by service in the form of <service>.cfg for mix-and-match convenience and must be combined with 10-global.cfg settings like so:

$ haproxy -f 10-global.cfg -f elasticsearch.cfg

For multiple services just add those service configurations to the command line options:

$ haproxy -f 10-global.cfg -f elasticsearch.cfg -f solrcloud.cfg

Some technologies default to the same port number (eg. Ambari and Presto both use port 8080), so if running both on the same HAProxy host then just modify one of the frontend listening port numbers.

If you want to add a Stats Admin UI then include the 20-stats.cfg configuration (remember to change the default password test):

$ haproxy -f 10-global.cfg -f 20-stats.cfg -f elasticsearch.cfg ...

Shortcut

To shorten the command line you can simply use the run.sh script instead:

./run.sh elasticsearch.cfg

and easily with multiple configs

./run.sh elasticsearch.cfg solrcloud.cfg ...

Backend Server Addresses (set these to your cluster hosts)

In most cases all you'll need to do is add your server addresses in to the backend config and then start HAProxy.

Common backend server addresses have been pre-populated for convenience including:

  • <service> - generic service name matching the proxied technology - could be resolved by DNS to multiple IPs to be balanced across
  • 192.168.99.100 - the common Docker Machine / MiniKube / MiniShift IP address
  • docker - convenient to use DNS or /etc/hosts to point this to your Docker host

These addresses are used in Continuous Integration tests run on this repo from the Advanced Nagios Plugins Collection.

More Configs

See the untested/ directory for a few more including SSL config versions I haven't got round to testing yet but should work.

Tests

Forked as a submodule from the Advanced Nagios Plugins Collection. These HAProxy configs are tested against Docker containers as part of CI tests in that repo.

See Also

  • find_active_server.py from the DevOps Python Tools repo and its related adjacent programs for on-the-fly command line determination of active masters or first responding peers across many of these same technologies

  • DevOps Bash Tools - 550+ DevOps Bash Scripts, Advanced .bashrc, .vimrc, .screenrc, .tmux.conf, .gitconfig, CI configs & Utility Code Library - AWS, GCP, Kubernetes, Docker, Kafka, Hadoop, SQL, BigQuery, Hive, Impala, PostgreSQL, MySQL, LDAP, DockerHub, Jenkins, Spotify API & MP3 tools, Git tricks, GitHub API, GitLab API, BitBucket API, Code & build linting, package management for Linux / Mac / Python / Perl / Ruby / NodeJS / Golang, and lots more random goodies

  • SQL Scripts - 100+ SQL Scripts - PostgreSQL, MySQL, AWS Athena, Google BigQuery

  • Templates - dozens of Code & Config templates - AWS, GCP, Docker, Jenkins, Terraform, Vagrant, Puppet, Python, Bash, Go, Perl, Java, Scala, Groovy, Maven, SBT, Gradle, Make, GitHub Actions Workflows, CircleCI, Jenkinsfile, Makefile, Dockerfile, docker-compose.yml, M4 etc.

  • Kubernetes templates - Kubernetes YAML templates - Best Practices, Tips & Tricks are baked right into the templates for future deployments

  • DevOps Python Tools - 80+ DevOps CLI tools for AWS, Hadoop, HBase, Spark, Log Anonymizer, Ambari Blueprints, AWS CloudFormation, Linux, Docker, Spark Data Converters & Validators (Avro / Parquet / JSON / CSV / INI / XML / YAML), Elasticsearch, Solr, Travis CI, Pig, IPython

  • The Advanced Nagios Plugins Collection - 450+ programs for Nagios monitoring your Hadoop & NoSQL clusters. Covers every Hadoop vendor's management API and every major NoSQL technology (HBase, Cassandra, MongoDB, Elasticsearch, Solr, Riak, Redis etc.) as well as message queues (Kafka, RabbitMQ), continuous integration (Jenkins, Travis CI) and traditional infrastructure (SSL, Whois, DNS, Linux)

  • DevOps Perl Tools - 25+ DevOps CLI tools for Hadoop, HDFS, Hive, Solr/SolrCloud CLI, Log Anonymizer, Nginx stats & HTTP(S) URL watchers for load balanced web farms, Dockerfiles & SQL ReCaser (MySQL, PostgreSQL, AWS Redshift, Snowflake, Apache Drill, Hive, Impala, Cassandra CQL, Microsoft SQL Server, Oracle, Couchbase N1QL, Dockerfiles, Pig Latin, Neo4j, InfluxDB), Ambari FreeIPA Kerberos, Datameer, Linux...

  • Dockerfiles - 50+ DockerHub public images for Docker & Kubernetes - Hadoop, Kafka, ZooKeeper, HBase, Cassandra, Solr, SolrCloud, Presto, Apache Drill, Nifi, Spark, Mesos, Consul, Riak, OpenTSDB, Jython, Advanced Nagios Plugins & DevOps Tools repos on Alpine, CentOS, Debian, Fedora, Ubuntu, Superset, H2O, Serf, Alluxio / Tachyon, FakeS3

Stargazers over time

Stargazers over time

git.io/haproxy-configs

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].