All Projects → splunkenizer → Splunkenizer

splunkenizer / Splunkenizer

Licence: apache-2.0
Ansible framework providing a fast and simple way to spin up complex Splunk environments.

Projects that are alternatives of or similar to Splunkenizer

Vagrant Box Templates
Stars: ✭ 100 (+36.99%)
Mutual labels:  ansible, ansible-playbooks, vagrant, virtualbox
Molecule Ansible Docker Aws
Example project showing how to test Ansible roles with Molecule using Testinfra and a multiscenario approach with Docker, Vagrant & AWS EC2 as infrastructure providers
Stars: ✭ 72 (-1.37%)
Mutual labels:  aws, ansible, vagrant, virtualbox
Packer Ubuntu 1804
This build has been moved - see README.md
Stars: ✭ 101 (+38.36%)
Mutual labels:  ansible, vagrant, virtualbox
Rhcsa8env
This is a RHCSA8 study environment built with Vagrant/Ansible
Stars: ✭ 108 (+47.95%)
Mutual labels:  ansible, vagrant, virtualbox
Packer Centos 7
This build has been moved - see README.md
Stars: ✭ 223 (+205.48%)
Mutual labels:  ansible, vagrant, virtualbox
Beetbox
Pre-provisioned L*MP stack
Stars: ✭ 94 (+28.77%)
Mutual labels:  ansible, vagrant, virtualbox
Drupal Vm
A VM for Drupal development
Stars: ✭ 1,348 (+1746.58%)
Mutual labels:  ansible, vagrant, virtualbox
Packer Templates
Scripts and Templates used for generating Vagrant images
Stars: ✭ 219 (+200%)
Mutual labels:  ansible, vagrant, virtualbox
Packer Centos 6
This build has been moved - see README.md
Stars: ✭ 78 (+6.85%)
Mutual labels:  ansible, vagrant, virtualbox
Packer
Packer helpers and templates for Docker, IIS, SQL Server and Visual Studio on Windows and Ubuntu
Stars: ✭ 242 (+231.51%)
Mutual labels:  aws, vagrant, virtualbox
Packer Build
Packer Automated VM Image and Vagrant Box Builds
Stars: ✭ 199 (+172.6%)
Mutual labels:  aws, vagrant, virtualbox
Kube Ansible
Build a Kubernetes cluster via Ansible playbook. 🔧 🔧 🔧
Stars: ✭ 354 (+384.93%)
Mutual labels:  ansible, vagrant, virtualbox
Devbox Golang
A Vagrant box with Ansible provisioning for setting up a vim-based Go(lang) development environment
Stars: ✭ 84 (+15.07%)
Mutual labels:  ansible, vagrant, virtualbox
Packer Boxes
Jeff Geerling's Packer build configurations for Vagrant boxes.
Stars: ✭ 495 (+578.08%)
Mutual labels:  ansible, vagrant, virtualbox
Packer Ubuntu 1404
DEPRECATED - Packer Example - Ubuntu 14.04 Vagrant Box using Ansible provisioner
Stars: ✭ 81 (+10.96%)
Mutual labels:  ansible, vagrant, virtualbox
Trellis
WordPress LEMP stack with PHP 8.0, Composer, WP-CLI and more
Stars: ✭ 2,295 (+3043.84%)
Mutual labels:  ansible, vagrant, virtualbox
Linchpin
ansible based multicloud orchestrator
Stars: ✭ 107 (+46.58%)
Mutual labels:  aws, ansible, ansible-playbooks
Ansible For Devops
Ansible for DevOps examples.
Stars: ✭ 5,265 (+7112.33%)
Mutual labels:  aws, ansible, vagrant
Ansible Playbook
Ansible playbook to deploy distributed technologies
Stars: ✭ 61 (-16.44%)
Mutual labels:  aws, ansible, ansible-playbooks
Ansible Role Packer rhel
Ansible Role - Packer RHEL/CentOS Configuration for Vagrant VirtualBox
Stars: ✭ 45 (-38.36%)
Mutual labels:  ansible, virtualbox

Splunkenizer

Splunkenizer Overview

License

Ever wanted to build a complex Splunk environment for testing, which looks as close as possible to a production deployment? Need to test a Splunk upgrade? See how Splunk indexer- or search head clustering works? Or just need to verify some configuration changes? This is the right place for you! The aim of this framework is to produce a Splunk environment in a fast and convenient way for testing purposes or maybe also for production use. The created Splunk installation and setup follows best practices using base config apps from Splunk. There are many ways to configure a Splunk environment, in terms of configuration file locations, so this is just another example how to do it.

Table of Contents

Support

Note: This framework is not officially supported by Splunk. I am developing this on best effort in my spare time.

Features

  • Build complex, reproducible Splunk environments in one shot, including all roles available for Splunk Enterprise.
  • Building Cluster Master, Indexer Clusters, Deployer, Search Head Clusters, Deployment Server, Universal Forwarders, Heavy Forwarders, License Master and Monitoring Console. All ready to use.
  • Configuration done according best practices with configuration apps
  • Splunk environment definition stored in one simple yaml file
  • Example configuration files for different setups included
  • Deployment and configuration done with Ansible
  • Controlled by Vagrant
  • Virtualized by Virtualbox or in AWS Cloud. It can be extended to other technologies like VMWare, Docker and such in the future.
  • Tested on MacOSX and Linux as vagrant and virtualbox host

Roadmap

See the upcoming features in the Roadmap

Changelog

Implemented changes are to be found in the Changelog

Installation

The Framework is currently tested on Mac OSX and Linux, but any other Unix, which is supported by Virtualbox, should work too.

Framework Installation (Mac OSX)

  1. Download and install Vagrant.
  2. Install the hostmanager plugin for Vagrant: vagrant plugin install vagrant-hostmanager
  3. Install Ansible, I personally prefer Brew which makes it as easy as brew install ansible. For supported Ansible versions check here
  4. Create a folder called Vagrant and change into it.
  5. Download and extract a Splunkenizer release here or clone from GitHub when using the master branch: git clone https://github.com/splunkenizer/Splunkenizer.git
  6. Create a folder called Software.
  7. Download the tgz. archive for the Splunk Software and put in the Software directory
    1. Splunk Enterprise
    2. Splunk Universal Forwarder
  8. Download Splunk Professional Services Best Practices Base Config Apps and extract them into the Software directory
    1. Configurations Base Apps
    2. Configurations Cluster Apps
  9. If you have a Splunk License file, link it to the name Splunk_Enterprise.lic inside the Software directory.

Your directory structure should now look like this:

./Vagrant/Splunkenizer/...
./Vagrant/Software/Configurations - Base/...
./Vagrant/Software/Configurations - Index Replication/...
./Vagrant/Software/splunk-8.0.3-a6754d8441bf-Linux-x86_64.tgz
./Vagrant/Software/splunkforwarder-8.0.3-a6754d8441bf-Linux-x86_64.tgz
./Vagrant/Software/Splunk_Enterprise.lic

Install Virtualbox support (optional)

  1. Download and install Virtualbox.
  2. Install the Virtualbox plugin for Vagrant: vagrant plugin install vagrant-vbguest

Install and configure AWS support (optional)

  1. Install the aws vagrant plugin: vagrant plugin install vagrant-aws
  2. Download the vagrant dummy box for aws: vagrant box add aws-dummy https://github.com/mitchellh/vagrant-aws/raw/master/dummy.box
  3. Generate AWS ACCESS Keys, described here
  4. Optional, but recommended:
    1. Add AWS_ACCESS_KEY_ID= as environment variable
    2. Add AWS_SECRET_ACCESS_KEY= as environment variable
  5. Create an ssh key pair described here and store the public key on your disk for later reference in the config file
  6. Create an AWS security group and name it for example 'Splunk' and add the following TCP ports

Example Basic AWS Security Group

Type Protocol Port Range Source Description
All TCP TCP 0 - 65535 172.31.0.0/16 Allow all internal traffic
Custom TCP TCP 8000 0.0.0.0/0 Splunk Web Interface
SSH TCP 22 0.0.0.0/0 SSH to all hosts

Building Windows Virtual Machine Template

To build your own windows vagrant image follow Setup Windows Vagrant image

Framework Usage

First start and initialization

Run vagrant the first time to initialize itself and create needed directories. You must execute vagrant always in side the Splunkenizer directory where the Vagrantfile sits, otherwise it will not work correctly. You will see the usage page, when executing vagrant without options.

cd Splunkenizer
vagrant

Copy a configuration file

There is one single configuration file, where all settings for your deployment are defined. Copy one configuration file from the examples to config/splunk_config.yml and adjust the setting to your needs. For a standard setup you should be fine with most of the default settings, but there are a lot of things you can adjust for special cases. See the configuration description file, where all existing values are described.

AWS: See instruction here when deploying into Amazon Cloud. You can start with splunk_config_aws.yml for a simple environment. Copy splunk_idxclusters, splunk_shclusters and splunk_hosts sections from other examples for more complex deployments.

Start the deployment

When building virtual machines (for virtualbox) the first time it will pull an os image from the internet. The box images are cached here: ~/.vagrant.d/boxes.

Create the Virtual Machines

vagrant up

Run Ansible playbooks to deploy and configure the Splunk software

The vagrant up command only creates the virtual machines. To deploy Splunk afterwards, run this command:

ansible-playbook ansible/deploy_site.yml

To run both steps with one command use:

vagrant up; ansible-playbook ansible/deploy_site.yml

Stop hosts

This will gracefully shutdown all the virtual machines.

vagrant halt

Destroy hosts

You can destroy all the virtual machines with one command.

vagrant destroy [-f] [<hostname>]

Rerun provisioning

Ansible playbooks can be run over and over again. If the virtual machine is already built, you can rerun the playbooks on a certain host again. This can be needed if something fails and you fixed the error.

ansible-playbook ansible/deploy_site.yml [--limit <hostname>]

Important Note: Always run vagrant status after updating the splunk_config.yml file, otherwise the changes are not populated to the Ansible inventory.

Login to the hosts

Login to Splunk Browser Interface

To login to one of the hosts just open the index.html file created in the Splunkenizer/config directory. You will find links to every role of your deployment.

Login by SSH

Vagrant deployes an ssh key for the vagrant user to login without a password.

vagrant ssh <hostname>

Environment Users

User vagrant

Vagrant uses a dedicated user to work inside the virtual machines. The user name is vagrant and has sudo rights to switch to root or other users.

User splunk

Splunk Enterprise is installed and run as user splunk. You can switch to this user by sudo su - splunk. For convenience, I have added some command aliases to the user vagrant and user splunk.

alias

Copy files

You can copy files from your host system to the virtual nodes with the vagrant command. You need to install the vagrant plugin vagrant-scp to have this feature available. Check Vagrant Docs on how to do this.

vagrant scp <file> <hostname>:/destdir

Deploying on Amazon Cloud

Splunkenizer can talk to the AWS cloud and create virtual machines with Splunk in the cloud. Vagrant is using the plugin vagrant-aws for that. Follow these steps to setup Splunkenizer for AWS. In the example there is a simple network setup, with only one Security group, covering all ports. More complex network setups should be possible, but make sure the host, where Splunkenizer is running does have ssh access to all instances.

To prepare the configuration file for Amazon deployments

  • Take the AWS example and fill in the values you like in the 'aws' section. You need at least:
    • access_key_id, secret_access_key if not specified as ENV vars.
    • keypair_name
    • ssh_private_key_path
    • security_groups
    • you can use the new 'splunk_download' section in 'splunk_defaults', if you do not want to upload the splunk binaries from your host all the time. This will download them from splunk.com instead.

You can copy splunk_hosts and cluster configs from other example files to the AWS template to create more complex environments. There can be all configuration option used, which are described in the vargant-aws plugin. They can also set individually on the splunk hosts, if needed. Just add a aws: section to the host.

Ansible playbooks only

You can also use the ansible playbooks without vagrant. Like that you can deploy Splunk to an existing set of hosts (virtual or physical). You have to create some config files, which is normally done by vagrant. Vagrant dynamically creates the ansible inventory file with the host and group variables for your configuration. Everything can be found in the inventory directory. The easiest way would be to create the same configuration with vagrant (ex. on your laptop) and copy the created files to your other Ansible environment.

Known issues, limitations

  • ulimit settings not working on Ubuntu 14 (without systemd)
  • Forwarding data from a universal forwarder to a heavy forwarder cannot be configured in the config file. This must be done manually after installation.
  • Virtualbox: Virtual host startup does not respond sometimes, if it fails, recreate the host again.
  • Virtualbox has some issues with clock time skew, when not using virtualbox additions. I added a workaround with forcing time clock sync every 5 minutes. A working internet connection on the Virtualbox host is needed.
  • AWS: Due to security reasons the login page with the admin password information has been disabled and https is enabled with splunk's own self signed certs.
  • AWS: OS images (AMI) do not have ntp configured by default. This will be added in Splunkenizer later.

Supported Ansible Versions

The following Ansible versions are tested and working with Splunkenizer, but any newer version should work as well.

✅ Ansible 2.7.x ✅ Ansible 2.8.x ✅ Ansible 2.9.x ✅ Ansible 2.10.x

Authors

Splunkenizer is created by Marco Stadler - a passionate Splunker.

License

License

See COPYING to see the full text.

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].