All Projects → octodns → Octodns

octodns / Octodns

Licence: mit
Tools for managing DNS across multiple providers

Programming Languages

python
139335 projects - #7 most used programming language

Projects that are alternatives of or similar to Octodns

Dnscontrol
Synchronize your DNS to multiple providers from a simple DSL
Stars: ✭ 2,089 (-0.05%)
Mutual labels:  infrastructure-as-code, dns, workflow
Sooty
The SOC Analysts all-in-one CLI tool to automate and speed up workflow.
Stars: ✭ 867 (-58.52%)
Mutual labels:  dns, workflow
Dart Basic Utils
A dart package for many helper methods fitting common situations
Stars: ✭ 153 (-92.68%)
Mutual labels:  dns
Terraform Aws Devops
Info about many of my Terraform, AWS, and DevOps projects.
Stars: ✭ 159 (-92.39%)
Mutual labels:  infrastructure-as-code
Machine Learning Workflow With Python
This is a comprehensive ML techniques with python: Define the Problem- Specify Inputs & Outputs- Data Collection- Exploratory data analysis -Data Preprocessing- Model Design- Training- Evaluation
Stars: ✭ 157 (-92.49%)
Mutual labels:  workflow
Alfred Reddit
Browse Reddit from Alfred
Stars: ✭ 155 (-92.58%)
Mutual labels:  workflow
Yii2 Workflow
A simple workflow engine for Yii2
Stars: ✭ 157 (-92.49%)
Mutual labels:  workflow
Infrastructure As Code Tutorial
Infrastructure As Code Tutorial. Covers Packer, Terraform, Ansible, Vagrant, Docker, Docker Compose, Kubernetes
Stars: ✭ 1,954 (-6.51%)
Mutual labels:  infrastructure-as-code
Sflive Paris2016 Workflow
Demo app for the new symfony/workflow component
Stars: ✭ 160 (-92.34%)
Mutual labels:  workflow
Batchflow
BatchFlow helps you conveniently work with random or sequential batches of your data and define data processing and machine learning workflows even for datasets that do not fit into memory.
Stars: ✭ 156 (-92.54%)
Mutual labels:  workflow
Expand
DevExpress XAF extension framework. 𝗹𝗶𝗻𝗸𝗲𝗱𝗶𝗻.𝗲𝘅𝗽𝗮𝗻𝗱𝗳𝗿𝗮𝗺𝗲𝘄𝗼𝗿𝗸.𝗰𝗼𝗺, 𝘆𝗼𝘂𝘁𝘂𝗯𝗲.𝗲𝘅𝗽𝗮𝗻𝗱𝗳𝗿𝗮𝗺𝗲𝘄𝗼𝗿𝗸.𝗰𝗼𝗺 and 𝘁𝘄𝗶𝘁𝘁𝗲𝗿 @𝗲𝘅𝗽𝗮𝗻𝗱𝗳𝗿𝗮𝗺𝗲𝘄𝗼𝗿𝗸 and or simply 𝗦𝘁𝗮𝗿/𝘄𝗮𝘁𝗰𝗵 this repository and get notified from 𝗚𝗶𝘁𝗛𝘂𝗯
Stars: ✭ 158 (-92.44%)
Mutual labels:  workflow
Pytest Testinfra
With Testinfra you can write unit tests in Python to test actual state of your servers configured by management tools like Salt, Ansible, Puppet, Chef and so on.
Stars: ✭ 1,987 (-4.93%)
Mutual labels:  infrastructure-as-code
Flares
Flares 🔥 is a CloudFlare DNS backup tool
Stars: ✭ 156 (-92.54%)
Mutual labels:  dns
Salt
Software to automate the management and configuration of any infrastructure or application at scale. Get access to the Salt software package repository here:
Stars: ✭ 12,086 (+478.28%)
Mutual labels:  infrastructure-as-code
Routedns
DNS stub resolver, proxy and router with support for DoT, DoH, DoQ, and DTLS
Stars: ✭ 153 (-92.68%)
Mutual labels:  dns
Nictool
NicTool: a DNS management solution
Stars: ✭ 159 (-92.39%)
Mutual labels:  dns
Django Lb Workflow
Reusable workflow library for Django
Stars: ✭ 153 (-92.68%)
Mutual labels:  workflow
Git Town
Generic, high-level Git workflow support!
Stars: ✭ 1,937 (-7.32%)
Mutual labels:  workflow
Dnsbin
The request.bin of DNS request
Stars: ✭ 157 (-92.49%)
Mutual labels:  dns
Terrible
An Ansible playbook that apply the principle of the Infrastructure as Code on a QEMU/KVM environment.
Stars: ✭ 161 (-92.3%)
Mutual labels:  infrastructure-as-code

OctoDNS Logo

DNS as code - Tools for managing DNS across multiple providers

In the vein of infrastructure as code OctoDNS provides a set of tools & patterns that make it easy to manage your DNS records across multiple providers. The resulting config can live in a repository and be deployed just like the rest of your code, maintaining a clear history and using your existing review & workflow.

The architecture is pluggable and the tooling is flexible to make it applicable to a wide variety of use-cases. Effort has been made to make adding new providers as easy as possible. In the simple case that involves writing of a single class and a couple hundred lines of code, most of which is translating between the provider's schema and OctoDNS's. More on some of the ways we use it and how to go about extending it below and in the /docs directory.

Table of Contents

Getting started

Workspace

Running through the following commands will install the latest release of OctoDNS and set up a place for your config files to live. To determine if provider specific requirements are necessary see the Supported providers table below.

$ mkdir dns
$ cd dns
$ python -m venv env
...
$ source env/bin/activate
$ pip install octodns <provider-specific-requirements>
$ mkdir config

Installing a specific commit SHA

If you'd like to install a version that has not yet been released in a repetable/safe manner you can do the following. In general octoDNS is fairly stable inbetween releases thanks to the plan and apply process, but care should be taken regardless.

$ pip install -e git+https://[email protected]/octodns/octodns.git@<SHA>#egg=octodns

Config

We start by creating a config file to tell OctoDNS about our providers and the zone(s) we want it to manage. Below we're setting up a YamlProvider to source records from our config files and both a Route53Provider and DynProvider to serve as the targets for those records. You can have any number of zones set up and any number of sources of data and targets for records for each. You can also have multiple config files, that make use of separate accounts and each manage a distinct set of zones. A good example of this this might be ./config/staging.yaml & ./config/production.yaml. We'll focus on a config/production.yaml.

---
manager:
  max_workers: 2

providers:
  config:
    class: octodns.provider.yaml.YamlProvider
    directory: ./config
    default_ttl: 3600
    enforce_order: True
  dyn:
    class: octodns.provider.dyn.DynProvider
    customer: 1234
    username: 'username'
    password: env/DYN_PASSWORD
  route53:
    class: octodns.provider.route53.Route53Provider
    access_key_id: env/AWS_ACCESS_KEY_ID
    secret_access_key: env/AWS_SECRET_ACCESS_KEY

zones:
  example.com.:
    sources:
      - config
    targets:
      - dyn
      - route53

  example.net.:
    alias: example.com.

class is a special key that tells OctoDNS what python class should be loaded. Any other keys will be passed as configuration values to that provider. In general any sensitive or frequently rotated values should come from environmental variables. When OctoDNS sees a value that starts with env/ it will look for that value in the process's environment and pass the result along.

Further information can be found in the docstring of each source and provider class.

The max_workers key in the manager section of the config enables threading to parallelize the planning portion of the sync.

In this example, example.net is an alias of zone example.com, which means they share the same sources and targets. They will therefore have identical records.

Now that we have something to tell OctoDNS about our providers & zones we need to tell it about our records. We'll keep it simple for now and just create a single A record at the top-level of the domain.

config/example.com.yaml

---
'':
  ttl: 60
  type: A
  values:
    - 1.2.3.4
    - 1.2.3.5

Further information can be found in Records Documentation.

Noop

We're ready to do a dry-run with our new setup to see what changes it would make. Since we're pretending here we'll act like there are no existing records for example.com. in our accounts on either provider.

$ octodns-sync --config-file=./config/production.yaml
...
********************************************************************************
* example.com.
********************************************************************************
* route53 (Route53Provider)
*   Create <ARecord A 60, example.com., [u'1.2.3.4', '1.2.3.5']>
*   Summary: Creates=1, Updates=0, Deletes=0, Existing Records=0
* dyn (DynProvider)
*   Create <ARecord A 60, example.com., [u'1.2.3.4', '1.2.3.5']>
*   Summary: Creates=1, Updates=0, Deletes=0, Existing Records=0
********************************************************************************
...

There will be other logging information presented on the screen, but successful runs of sync will always end with a summary like the above for any providers & zones with changes. If there are no changes a message saying so will be printed instead. Above we're creating a new zone in both providers so they show the same change, but that doesn't always have to be the case. If to start one of them had a different state you would see the changes OctoDNS intends to make to sync them up.

Making changes

WARNING: OctoDNS assumes ownership of any domain you point it to. When you tell it to act it will do whatever is necessary to try and match up states including deleting any unexpected records. Be careful when playing around with OctoDNS. It's best to experiment with a fake zone or one without any data that matters until you're comfortable with the system.

Now it's time to tell OctoDNS to make things happen. We'll invoke it again with the same options and add a --doit on the end to tell it this time we actually want it to try and make the specified changes.

$ octodns-sync --config-file=./config/production.yaml --doit
...

The output here would be the same as before with a few more log lines at the end as it makes the actual changes. After which the config in Route53 and Dyn should match what's in the yaml file.

Workflow

In the above case we manually ran OctoDNS from the command line. That works and it's better than heading into the provider GUIs and making changes by clicking around, but OctoDNS is designed to be run as part of a deploy process. The implementation details are well beyond the scope of this README, but here is an example of the workflow we use at GitHub. It follows the way GitHub itself is branch deployed.

The first step is to create a PR with your changes.

GitHub user interface of a pull request

Assuming the code tests and config validation statuses are green the next step is to do a noop deploy and verify that the changes OctoDNS plans to make are the ones you expect.

Output of a noop deployment command

After that comes a set of reviews. One from a teammate who should have full context on what you're trying to accomplish and visibility in to the changes you're making to do it. The other is from a member of the team here at GitHub that owns DNS, mostly as a sanity check and to make sure that best practices are being followed. As much of that as possible is baked into octodns-validate.

After the reviews it's time to branch deploy the change.

Output of a deployment command

If that goes smoothly, you again see the expected changes, and verify them with dig and/or octodns-report you're good to hit the merge button. If there are problems you can quickly do a .deploy dns/master to go back to the previous state.

Bootstrapping config files

Very few situations will involve starting with a blank slate which is why there's tooling built in to pull existing data out of providers into a matching config file.

$ octodns-dump --config-file=config/production.yaml --output-dir=tmp/ example.com. route53
2017-03-15T13:33:34  INFO  Manager __init__: config_file=tmp/production.yaml
2017-03-15T13:33:34  INFO  Manager dump: zone=example.com., sources=('route53',)
2017-03-15T13:33:36  INFO  Route53Provider[route53] populate:   found 64 records
2017-03-15T13:33:36  INFO  YamlProvider[dump] plan: desired=example.com.
2017-03-15T13:33:36  INFO  YamlProvider[dump] plan:   Creates=64, Updates=0, Deletes=0, Existing Records=0
2017-03-15T13:33:36  INFO  YamlProvider[dump] apply: making changes

The above command pulled the existing data out of Route53 and placed the results into tmp/example.com.yaml. That file can be inspected and moved into config/ to become the new source. If things are working as designed a subsequent noop sync should show zero changes.

Supported providers

Provider Requirements Record Support Dynamic Notes
AzureProvider azure-identity, azure-mgmt-dns, azure-mgmt-trafficmanager A, AAAA, CAA, CNAME, MX, NS, PTR, SRV, TXT Alpha (A, AAAA, CNAME)
Akamai edgegrid-python A, AAAA, CNAME, MX, NAPTR, NS, PTR, SPF, SRV, SSHFP, TXT No
CloudflareProvider A, AAAA, ALIAS, CAA, CNAME, LOC, MX, NS, PTR, SPF, SRV, TXT No CAA tags restricted
ConstellixProvider A, AAAA, ALIAS (ANAME), CAA, CNAME, MX, NS, PTR, SPF, SRV, TXT Yes CAA tags restricted
DigitalOceanProvider A, AAAA, CAA, CNAME, MX, NS, TXT, SRV No CAA tags restricted
DnsMadeEasyProvider A, AAAA, ALIAS (ANAME), CAA, CNAME, MX, NS, PTR, SPF, SRV, TXT No CAA tags restricted
DnsimpleProvider All No CAA tags restricted
DynProvider dyn All Both
EasyDNSProvider A, AAAA, CAA, CNAME, MX, NAPTR, NS, SRV, TXT No
EtcHostsProvider A, AAAA, ALIAS, CNAME No
EnvVarSource TXT No read-only environment variable injection
GandiProvider A, AAAA, ALIAS, CAA, CNAME, DNAME, MX, NS, PTR, SPF, SRV, SSHFP, TXT No
GCoreProvider A, AAAA, NS, MX, TXT, SRV, CNAME, PTR Dynamic
GoogleCloudProvider google-cloud-dns A, AAAA, CAA, CNAME, MX, NAPTR, NS, PTR, SPF, SRV, TXT No
HetznerProvider A, AAAA, CAA, CNAME, MX, NS, SRV, TXT No
MythicBeastsProvider Mythic Beasts A, AAAA, ALIAS, CNAME, MX, NS, SRV, SSHFP, CAA, TXT No
Ns1Provider ns1-python All Yes
OVH ovh A, AAAA, CAA, CNAME, MX, NAPTR, NS, PTR, SPF, SRV, SSHFP, TXT, DKIM No
PowerDnsProvider All No
Rackspace A, AAAA, ALIAS, CNAME, MX, NS, PTR, SPF, TXT No
Route53 boto3 A, AAAA, CAA, CNAME, MX, NAPTR, NS, PTR, SPF, SRV, TXT Both CNAME health checks don't support a Host header
Selectel A, AAAA, CNAME, MX, NS, SPF, SRV, TXT No
Transip transip A, AAAA, CNAME, MX, NS, SRV, SPF, TXT, SSHFP, CAA No
UltraDns A, AAAA, CAA, CNAME, MX, NS, PTR, SPF, SRV, TXT No
AxfrSource A, AAAA, CAA, CNAME, LOC, MX, NS, PTR, SPF, SRV, TXT No read-only
ZoneFileSource A, AAAA, CAA, CNAME, MX, NS, PTR, SPF, SRV, TXT No read-only
TinyDnsFileSource A, CNAME, MX, NS, PTR No read-only
YamlProvider All Yes config

Notes

  • ALIAS support varies a lot from provider to provider care should be taken to verify that your needs are met in detail.
    • Dyn's UI doesn't allow editing or view of TTL, but the API accepts and stores the value provided, this value does not appear to be used when served
    • Dnsimple's uses the configured TTL when serving things through the ALIAS, there's also a secondary TXT record created alongside the ALIAS that octoDNS ignores
  • octoDNS itself supports non-ASCII character sets, but in testing Cloudflare is the only provider where that is currently functional end-to-end. Others have failures either in the client libraries or API calls

Compatibilty & Compliance

lenient

lenient mostly focuses on the details of Records and standards compliance. When set to true octoDNS will allow allow non-compliant configurations & values where possible. For example CNAME values that don't end with a ., label length restrictions, and invalid geo codes on dynamic records. When in lenient mode octoDNS will log validation problems at WARNING and try and continue with the configuration or source data as it exists. See Lenience for more information on the concept and how it can be configured.

strict_supports (Work In Progress)

strict_supports is a Provider level parameter that comes into play when a provider has been asked to create a record that it is unable to support. The simplest case of this would be record type, e.g. SSHFP not being supported by AzureProvider. If such a record is passed to an AzureProvider as a target the provider will take action based on the strict_supports. When true it will throw an exception saying that it's unable to create the record, when set to false it will log at WARNING with information about what it's unable to do and how it is attempting to working around it. Other examples of things that cannot be supported would be dynamic records on a provider that only supports simple or the lack of support for specific geos in a provider, e.g. Route53Provider does not support NA-CA-*.

It is worth noting that these errors will happen during the plan phase of things so that problems will be visible without having to make changes.

This concept is currently a work in progress and only partially implemented. While work is on-going strict_supports will default to false. Once the work is considered complete & ready the default will change to true as it's a much safer and less surprising default as what you configure is what you'll get unless an error is throw telling you why it cannot be done. You will then have the choice to explicitly request that things continue with work-arounds with strict_supports set to false. In the meantime it is encouraged that you manually configure the parameter to true` in your provider configs.

Configuring strict_supports

The strict_supports parameter is available on all providers and can be configured in YAML as follows:

providers:
  someprovider:
    class: whatever.TheProvider
    ...
    strict_supports: true

Custom Sources and Providers

You can check out the source and provider directory to see what's currently supported. Sources act as a source of record information. AxfrSource and TinyDnsFileSource are currently the only OSS sources, though we have several others internally that are specific to our environment. These include something to pull host data from gPanel and a similar provider that sources information about our network gear to create both A & PTR records for their interfaces. Things that might make good OSS sources might include an ElbSource that pulls information about AWS Elastic Load Balancers and dynamically creates CNAMEs for them, or Ec2Source that pulls instance information so that records can be created for hosts similar to how our GPanelProvider works.

Most of the things included in OctoDNS are providers, the obvious difference being that they can serve as both sources and targets of data. We'd really like to see this list grow over time so if you use an unsupported provider then PRs are welcome. The existing providers should serve as reasonable examples. Those that have no GeoDNS support are relatively straightforward. Unfortunately most of the APIs involved to do GeoDNS style traffic management are complex and somewhat inconsistent so adding support for that function would be nice, but is optional and best done in a separate pass.

The class key in the providers config section can be used to point to arbitrary classes in the python path so internal or 3rd party providers can easily be included with no coordination beyond getting them into PYTHONPATH, most likely installed into the virtualenv with OctoDNS.

For examples of building third-party sources and providers, see Related Projects & Resources.

Other Uses

Syncing between providers

While the primary use-case is to sync a set of yaml config files up to one or more DNS providers, OctoDNS has been built in such a way that you can easily source and target things arbitrarily. As a quick example the config below would sync githubtest.net. from Route53 to Dyn.

---
providers:
  route53:
    class: octodns.provider.route53.Route53Provider
    access_key_id: env/AWS_ACCESS_KEY_ID
    secret_access_key: env/AWS_SECRET_ACCESS_KEY
  dyn:
    class: octodns.provider.dyn.DynProvider
    customer: env/DYN_CUSTOMER
    username: env/DYN_USERNAME
    password: env/DYN_PASSWORD

zones:

  githubtest.net.:
    sources:
      - route53
    targets:
      - dyn

Dynamic sources

Internally we use custom sources to create records based on dynamic data that changes frequently without direct human intervention. An example of that might look something like the following. For hosts this mechanism is janitorial, run periodically, making sure the correct records exist as long as the host is alive and ensuring they are removed after the host is destroyed. The host provisioning and destruction processes do the actual work to create and destroy the records.

---
providers:
  gpanel-site:
    class: github.octodns.source.gpanel.GPanelProvider
    host: 'gpanel.site.github.foo'
    token: env/GPANEL_SITE_TOKEN
  powerdns-site:
    class: octodns.provider.powerdns.PowerDnsProvider
    host: 'internal-dns.site.github.foo'
    api_key: env/POWERDNS_SITE_API_KEY

zones:

  hosts.site.github.foo.:
    sources:
      - gpanel-site
    targets:
      - powerdns-site

Contributing

Please see our contributing document if you would like to participate!

Getting help

If you have a problem or suggestion, please open an issue in this repository, and we will do our best to help. Please note that this project adheres to the Contributor Covenant Code of Conduct.

Related Projects & Resources

If you know of any other resources, please do let us know!

License

OctoDNS is licensed under the MIT license.

The MIT license grant is not for GitHub's trademarks, which include the logo designs. GitHub reserves all trademark and copyright rights in and to all GitHub trademarks. GitHub's logos include, for instance, the stylized designs that include "logo" in the file title in the following folder: https://github.com/octodns/octodns/tree/master/docs/logos/

GitHub® and its stylized versions and the Invertocat mark are GitHub's Trademarks or registered Trademarks. When using GitHub's logos, be sure to follow the GitHub logo guidelines.

Authors

OctoDNS was designed and authored by Ross McFarland and Joe Williams. See https://github.com/octodns/octodns/graphs/contributors for a complete list of people who've contributed.

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].