grammarly / Omniconf

• Omniconf

  I [[CHANGELOG.md][https://img.shields.io/badge/-changelog-blue.svg]] I [[https://circleci.com/gh/grammarly/omniconf][https://circleci.com/gh/grammarly/omniconf/tree/master.png]] I [[https://clojars.org/com.grammarly/omniconf][16.4k downloads]] I

  Command-line arguments. Environment variables. Configuration files. Java properties. Almost every program requires some configuration which is usually spread around multiple sources. Keeping track of all of them, mapping ones to others, making sure they are present and correct, passing them around is a laborious and thankless task.

  Configuration libraries, which [[https://github.com/weavejester/environ][there]] [[https://github.com/juxt/aero][are]] [[https://github.com/tolitius/cprop][plenty]] [[https://github.com/reborg/fluorine][of]], promise to solve the configuration problem, and they do. However, they usually provide only the mapping of configuration sources to Clojure data, leaving out the verification part. Omniconf's value proposition, among other features, is the requirement to declare the expected configuration upfront, and the ability to validate the configuration early and display helpful messages if the application is misconfigured.

  In terms of configuration sources, Omniconf supports:

  • Environment variables
  • CLI arguments
  • Java properties
  • [[#providing-configuration-as-files][EDN files]]
  • [[https://github.com/grammarly/omniconf#fetching-configuration-from-aws-systems-manager-ssm][AWS SSM (+dynamic reconfiguration)]]

** Rationale

Omniconf is developed with the following principles in mind:

1. Explicit over implicit. Most configuration libraries allow to grab a configuration value (e.g. from ENV) at any point of time from any place in the code. This makes it impossible to list all configuration options that the program uses without reading the entire source. Omniconf requires you to declare all possible configuration options upfront, at the beginning of the program execution.
2. All configuration sources must be unified. It shouldn't matter where the option is set from --- environment variables, CLI arguments, or config files. It is uniformly initialized, verified, and accessed as regular Clojure data.
3. Maximum verification. We don't want to see NumberFormatException stacktraces in the middle of your program run because of a typo in the environment variable. The whole configuration should be checked early and automatically before the rest of the code is executed. If there are any problems with it, a helpful message should be presented to the user.

** Installation

Add this line to your list of dependencies:

[[https://clojars.org/com.grammarly/omniconf][https://clojars.org/com.grammarly/omniconf/latest-version.svg]]

** Usage

1. You start by defining a set of supported options. =cfg/define= takes a map of options to their different parameters. The following small example shows the syntax:

   #+BEGIN_SRC clojure (require '[omniconf.core :as cfg]) (cfg/define {:hostname {:description "where service is deployed" :type :string :required true} :port {:description "HTTP port" :type :number :default 8080}}) #+END_SRC

 The full list of supported parameters is described [[https://github.com/grammarly/omniconf#configuration-scheme-syntax][here]].

2. Populate the configuration from available sources:

   #+BEGIN_SRC clojure (cfg/populate-from-cmd args) ;; args is a command-line arguments list (when-let [conf (cfg/get :conf)] (cfg/populate-from-file conf)) (cfg/populate-from-properties) (cfg/populate-from-env) #+END_SRC

   The order in which to tap the sources is up to you. Perhaps you want to make environment variables overwrite command-line args, or give the highest priority to the config file. In the above example we get the path to the configuration file as =--conf= CMD argument. For more information, see [[https://github.com/grammarly/omniconf#providing-configuration-as-files][this]].

3. Call =verify=. It marks the boundary in your system, after which the whole configuration is guaranteed to be complete and correct.

   #+BEGIN_SRC clojure (cfg/verify) #+END_SRC

   If there is something wrong with the configuration, =verify= will throw a proper exception. If called not in the REPL environment, the exception will be stripped of its stacktrace, so that you only see the exact error.

   If everything is alright, =verify= will pretty-print the whole configuration map into the standard output. It is convenient because it gives you one final chance to look at your config values and make sure they are good. =:silent true= can be passed to =verify= to prevent it from printing the map.

4. Use =get= to extract arbitrary value from the configuration.

   #+BEGIN_SRC clojure (cfg/get :hostname) #+END_SRC

   For nested values you can pass an address of the value, either as a vector, or like varargs:

   #+BEGIN_SRC clojure (cfg/get :database :ip) (cfg/get [:database :ip]) #+END_SRC

   =set= allows you to change a value. It is definitely not recommended to be used in production code, but may be convenient during development:

   #+BEGIN_SRC clojure (cfg/set :database :port 3306) (cfg/set [:database :port] 3306) #+END_SRC

** Examples

Sample programs that use Omniconf: [[./example-lein][example-lein]] and [[./example-boot][example-boot]]. There is not much difference in using Omniconf with these build tools, but Boot requires a little hack to achieve parity with Leiningen.

** Configuration scheme syntax

Configuration scheme is a map of option names to maps of their parameters. Option name is a keyword that denotes how the option is retrieved inside the program, and how it maps to configuration sources. Naming rules are the following:

For command-line arguments:

: :some-option => --some-option

For environment variables:

: :some-option => SOME_OPTION

For Java properties:

: :some-option => some-option (java -Dsome-option=... if set from command line)

Each option can have the following parameters:

• =:description= --- string that describes this option. This description will be used to generate a help message for the program.

• =:type= --- currently the following types are supported: =:string=, =:keyword=, =:number=, =:boolean=, =:edn=, =:file=, =:directory=. Setting a type automatically defines how to parse a value for this option from a string, and also verifies that the resulting value has the correct Clojure type.

  Boolean types have special treatment. When setting them from the command line, one can omit the value completely.

  : (cfg/define {:foo {:type :boolean}, :bar {:type :boolean}}) : ... : $ my-app --foo --bar # Confmap is {:foo true, :baz true}

  A string parser for booleans treats strings "0" and "false" as =false=, anything else as =true=.

• =:parser= --- a single-arg function that converts a string value (given in command-line option or environment variable) into a Clojure value. This option can be used instead of =:type= if you need a custom option type.

• =:default= --- the option will be initialized with this value. The default value must be specified as a Clojure datatype, not as a string yet to be parsed.

  The value for =:default= can be a nullary function used to generate the actual default value. This function will be invoked during the verification phase or on first direct access to the value, whichever happens first; thus, default functions will have access to other config values provided by the user. Note that you must invoke =(cfg/enable-functions-as-defaults)= first for this feature to work. Example:

  #+BEGIN_SRC clojure (cfg/enable-functions-as-defaults) (cfg/define {:host {:type :string} :port {:type :number} :connstring {:type :string :default #(str (cfg/get :host) ":" (cfg/get :port))}}) (cfg/populate-from-map {:host "localhost", :port 8888}) (cfg/get :connstring) ;; => "localhost:8888" #+END_SRC

  Even if a config option has a functional default, its value can be explicitly set from any configuration source to a normal value, and in that case the default function won't be invoked.

  Make sure that you don't try to =cfg/get= an option with a function default value before the values that function depends on are populated.

• =:required= --- if true, the value for this option must be provided, otherwise =verify= will fail. The value of this parameter can also be a nullary function: if the function returns true then the option value must be provided. It is convenient if the necessity of an option depends on the values of some other options. Example:

  #+BEGIN_SRC clojure (cfg/define {:storage {:one-of [:file :s3]} :s3-bucket {:required #(= (cfg/get :storage) :s3)}}) #+END_SRC

• =:one-of= --- a sequence of values that an option is allowed to take. If the value isn't present in the =:one-of= list, =verify= will fail. =:one-of= automatically implies =:required true= unless you add =nil= as a permitted value.

• =:verifier= --- a function of =[option-name value]= that should throw an exception if the value is not correct. Verifier is only executed if the value is not nil, so it doesn't imply =:required true=. Predefined verifiers:

  • =cfg/verify-file-exists=
  • =cfg/verify-directory-non-empty= --- checks if the value is a directory, and if it is non-empty.

• =:delayed-transform= --- a function of option value that will be called not immediately, but the first time when the option is accessed in the code. Transform will be applied only once, and after that the option will store the transformed value. Usefulness of this feature is yet in question. You can mimic it by using a custom parser that wraps the value in a =delay=, the only difference that you will also have to dereference it manually every time.

• =:nested= --- a map that has the same structure as the top-level configuration scheme. Nested options have the same rights as top-level ones: they can have parsers, verifiers, defaults, etc. Example:

  #+BEGIN_SRC clojure (cfg/define {:statsd {:nested {:host {:type :string :required true :description "IP address of the StatsD server"} :port {:type :number :default 8125}}}}) #+END_SRC

  CLI and ENV arguments have special transformation rules for nested options --- dot as a separator for CLI arguments and Java properties, and double underscore for ENV.

  : [:statsd :host] => --statsd.host (cmdline args) : [:statsd :host] => -Dstatsd.host (properties) : [:statsd :host] => STATSD__HOST (env variables)

  In the program you can use =cfg/get= to fetch a concrete value, or a whole map at any level:

  #+BEGIN_SRC clojure (cfg/get :statsd :port) ;=> 8125 (cfg/get :statsd) ;=> {:host "127.0.0.1", :port 8125} #+END_SRC

• =:secret= --- if true, the value of this option won't be printed out by =cfg/verify=. You will see == instead. Useful for passwords, API keys, and such.

** Providing configuration as files

Omniconf can use EDN files as a configuration source. A file must contain a map of options to their values, which will be merged into the config when =populate-from-file= is called. The values should already have the format the option requires (number, keyword); but you can also use strings so that parser will be called on them.

You can hardcode the name of the file where to look for configuration (e.g. =config.edn= in the current directory). It is somewhat trickier to tell the name of the file dynamically. One of the solutions is to expect the configuration file to be provided in one of the command-line arguments. So you have to =populate-from-cmd= first, and then to populate from config file if it has been provided. However, this way the configuration file will have the priority over CLI arguments which is not always desirable. As a workaround, you can call =populate-from-cmd= again, but only if your CLI args are idempotent (i.e. they don't contain =^:concat=, see below).

Optionally, =data-readers= may be provided when reading from a file which contain library specific reader macros. For example:

#+BEGIN_SRC clojure (cfg/populate-from-cmd args) ;; args is a command-line arguments list (when-let [conf (cfg/get :conf)] (binding [cfg/data-readers {'ig/ref ig/ref 'ig/refset ig/refset}] ; e.g. reader macros used by integrant (cfg/populate-from-file conf))) #+END_SRC

** Fetching configuration from AWS Systems Manager (SSM)

Since version 0.3, Omniconf supports [[https://aws.amazon.com/systems-manager/][Amazon SSM]], particularly its [[https://aws.amazon.com/systems-manager/features/][Parameter Store]], as a configuration source. SSM works well as a storage for secrets --- passwords, tokens, and other sensitive things that you don't want to check into the source control.

To use SSM backend, you'll need to add an extra dependency:

[[https://clojars.org/com.grammarly/omniconf.ssm][https://clojars.org/com.grammarly/omniconf.ssm/latest-version.svg]]

The function =omniconf.core/populate-from-ssm= will be available now. It takes =path= as an argument which will be treated as root path to nested SSM parameters. For example:

#+BEGIN_SRC clojure (cfg/define {:db {:nested {:password {:type :string :secret true}}}})

(cfg/populate-from-ssm "/prod/myapp/") #+END_SRC

This will fetch =/prod/myapp/db/password= parameter from SSM and save it as =[:db :password]= in Omniconf.

You can also specify explicit mapping between SSM and Omniconf like this:

#+BEGIN_SRC clojure (cfg/define {:db {:nested {:password {:type :string :secret true}}} :github-token {:type :string :secret true :ssm-name "/myteam/github/oauth-token"}})

(cfg/populate-from-ssm "/prod/myapp/") #+END_SRC

Parameters with an absolute =:ssm-name= parameter will ignore the =path= argument and will fetch the value directly by name. In case you still want to use =path= for some keys but the layout in SSM differs from one in Omniconf, you can use =./= as a prefix to signify that it is relative to the path:

#+BEGIN_SRC clojure (cfg/define {:db {:nested {:password {:type :string :secret true :ssm-name "./db-pass"}}}})

(cfg/populate-from-ssm "/prod/myapp/") #+END_SRC

This will set =[:db :password]= parameter from =/prod/myapp/db-pass=.

*** Dynamic reconfiguration from SSM

Unlike environment variables and command-line arguments, SSM Parameter Store
values can change independently as your program is running. You might want
to use this, so that you can change some configuration without restarting
the program. There are plenty of usecases for this, like switching the
upstream hostname on the fly, or gradually changing the rate of requests to
an experimental server you are testing.

To tap into this functionality, use =populate-from-ssm-continually= instead
of =populate-from-ssm=. It accepts the same =path= argument, and an extra
one --- interval in seconds between polling SSM. Polling is used because SSM
doesn't expose an event-based API for this; but it's not too bad since you'd
probably set the interval to 5-10 seconds, so the overhead of polling is not
too big. Also, Omniconf would report setting only the values that actually
have changed.

#+BEGIN_SRC clojure

;; Poll values under /prod/myapp/ prefix (and all absolute :ssm-name values too) every 10 seconds. (cfg/populate-from-ssm-continually "/prod/myapp/" 10) #+END_SRC

Note that for now, the verification step is not re-run after fetching
updated values from SSM, so it is possible to break =:verifier= invariants
with this.

** Tips, tricks, and FAQ

*** Are there any drawbacks? What's the catch?

There are a few. First of all, Omniconf is much more complex and intertwined
than, say, Environ. This might put off some developers, although we suspect
they are re-implementing half of Omniconf functionality on top of Environ
anyway (like we did before).

Omniconf configuration map is a global mutable singleton. It adds a bit of
convenience that you don't have to drag the config map around, or require it
in every namespace. However, there might be usecases where this approach
does not fit.

Omniconf is an application-level tool. You most likely don't want to make
your library depend on it, forcing the library users to configure through
Omniconf too.

*** Why are there no convenient Leiningen plugins/Boot tasks for Omniconf?

In the end we distribute and deploy our applications as uberjars. As a
standalone JAR our program doesn't have access to Leiningen or Boot. Hence,
it is better not to offload anything to plugins to avoid spawning
differences between development and production time.

*** CLI help command

=:help= option gets a special treatment in Omniconf. It can have
=:help-name= and =:help-description= parameters that will be used when
printing the help message. If =populate-from-cmd= encounters =--help= on
the arguments list, it prints the help message and quits.

*** Useful functions and macros

=with-options= works as =let= for configuration values, i.e. it takes a binding
list of symbols that should have the same names as options' keyword names.
Only top-level options are supported, destructuring of nested values is not
possible right now.

#+BEGIN_SRC clojure

(cfg/with-options [username password] ;; Binds (cfg/get :username) to username, and (cfg/get :password) to password. ...) #+END_SRC

*** Special operations for EDN options

Sometimes you don't want to completely overwrite an EDN value, but append to
it. For this case two special operations, --- =^:concat= and =^:merge= ---
can be attached to a map or a list when setting them from any source.
Example:

#+BEGIN_SRC clojure
(cfg/define {:emails {:type :edn
                      :default ["[email protected]" "[email protected]"]}
             :roles  {:type :edn
                      :default {"[email protected]" :admin
                                "[email protected]" :admin}}})
...
$ my-app --emails '^:concat ["[email protected]"]' --roles '^:merge {"[email protected]" :user}'
#+END_SRC

*** Custom logging for Omniconf

By default, Omniconf prints errors and final configuration map to standard
output. If you want it to use a special logging solution, call
=cfg/set-logging-fn= and provide a vararg function for Omniconf to use
it instead of =println=. For example:

#+BEGIN_SRC clojure

(require '[taoensso.timbre :as log]) (cfg/set-logging-fn (fn [& args] (log/info (str/join " " args)))) #+END_SRC

** License

© Copyright 2016-2020 Grammarly, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.