ironhide 
Ironhide, the data transformer.
Idea
Create a runtime agnostic bidirectional data-driven transformation domain-specific language for fun and profit.
Problem
There are a lot of data, which has to be represented in different shapes. For this reason created a lot of query/transformation languages such as XSLT, AWK, etc, but most of them have a significant disadvantage: they work only in one direction and you can't get original data from the result of transformation.
It worth noting that there are other languages like boomerang, which doesn't have this significant (in some cases) weakness, but have others : )
Simplified real life example of representation person name in different systems:
"form": {
"name": "Firstname Lastname"
}"fhir": {
"name": {
"given": [
"Firstname"
],
"family": "Lastname"
}
}By different reasons both respresentations should be availiable and moreover syncronized. Syncronization can be done by implementing and applying when needed two function f and f -1 for each field or subset of fields, but you already probably know how hard to maintain such code?)
It is hard to implement such functions for big nested tree data structures and much harder to keep f -1 in sync with f.
Solution
ironhide is an attempt to create a
bidirectional data transformation language described by a data structure stored
in EDN (you can think about edn like a
better JSON). ironhide still in early stage of
development, but already covers some practical usecases. It's also declarative,
bidirectional, data-driven and simple.
Following code in ironhide solves example above:
#:ih{:direction [:fhir :form]
:rules [{:form [:name :ihs/str<->vector [0]]
:fhir [:name [0] :given [0]]}
{:form [:name :ihs/str<->vector [1]]
:fhir [:name [0] :family]}]}The direction of transformation controlled by :ih/direction key and this
simple snippet allows to transform data in both ways out of the box.
Full grammar defined using clojure.spec in core namespace.
Usage
This section contains examples of clojure ironhide interpreter usage with a little explanation, to get the taste of dsl capabilities. More detailed info provided in Description section.
Add to :deps in deps.edn:
healthsamurai/ironhide {:mvn/version "RELEASE"}hello_world.clj:
(ns hello-world.core
(:require [ironhide.core :as ih]))
;; (ih/execute shell)
;; (ih/get-data shell)Field to field mapping
(def update-name-shell
#:ih{:direction [:form :form-2]
:rules [{:form [:name]
:form-2 [:fullname]}]
:data {:form {:name "Full Name"}
:form-2 {:fullname "Old Name"}}})
(get-data update-name-shell)
;; => {:form {:name "Full Name"}, :form-2 {:fullname "Full Name"}}Create missing fields
(def create-name-shell
#:ih{:direction [:form :form-2]
:rules [{:form [:name]
:form-2 [:fullname]}]
:data {:form {:name "Full Name"}}})
(get-data create-name-shell)
;; => {:form {:name "Full Name"}, :form-2 {:fullname "Full Name"}}Default values
(def default-name-shell
#:ih{:direction [:form :form-2]
:values {:person/name "Name not provided by the form"}
:rules [{:form [:name]
:form-2 [:fullname]
:ih/value {:form-2 [:ih/values :person/name]}}]
:data {:form {}
:form-2 {:fullname "Old Name"}}})
(get-data default-name-shell)
;; => {:form {}, :form-2 {:fullname "Name not provided by the form"}}Sight for string
(def sight-name-shell
#:ih{:direction [:form :fhir]
:rules [{:form [:name :ihs/str<->vector [0]]
:fhir [:name [0] :given [0]]}]
:data {:form {:name "Full Name"}}})
(get-data sight-name-shell)
;; => {:form {:name "Full Name"}, :fhir {:name [{:given ["Full"]}]}}See Sight section for the detailed explanation.
Update and create if not-exists
(def create-and-update-phone-shell
#:ih{:direction [:form :fhir]
:rules [{:form [:phones [:*]]
:fhir [:telecom [:* {:system "phone"}] :value]}]
:data {:form {:phones ["+1 111" "+2 222"]}
:fhir {:telecom [{:system "phone"
:use "home"
:value "+3 333"}
{:system "email"
:value "[email protected]"}]}}})
(get-data create-and-update-phone-shell)
;; =>
;; {:form {:phones ["+1 111" "+2 222"]},
;; :fhir {:telecom [{:system "phone", :use "home", :value "+1 111"}
;; {:system "email", :value "[email protected]"}
;; {:system "phone", :value "+2 222"}]}}Micro
(def micro-name-shell
#:ih{:direction [:fhir :form] ;; !!!
:micros #:ihm {:name<->vector [:name {:ih/sight :ihs/str<->vector
:separator ", "}]}
:rules [{:form [:ihm/name<->vector [0]]
:fhir [:name [0] :given [0]]}
{:form [:ihm/name<->vector [1]]
:fhir [:name [0] :family]}]
:data {:form {:name "Full, Name"}
:fhir {:name [{:given ["First"] :family "Family"}]}}})
(get-data micro-name-shell :form)
;; => {:name "First, Family"}Micros can be parametrized in the same way as sights.
Description
shell is a tree datastructure, which contains declaration of transformation
rules + data itself. ironhide interpreter can execute shell's.
It consists of few main parts:
:ih/dataa data for transformation:ih/valuessimilar to previous one, but used mostly for default values:ih/microsshortcuts for long repetitive pathes in rules:ih/directiondefault transformation direction (rule can define its own):ih/rulesvector of transformation rules
Simple shell executed with get-data:
(get-data
#:ih{:direction [:form :fhir]
:data {:form {:first-name "Firstname"}
:fhir {}}
:rules [{:form [:first-name]
:fhir [:name [0] :given [0]]}]})
;; => {:form {:first-name "Firstname"}, :fhir {:name [{:given ["Firstname"]}]}}Path and pelem
Path is a vector consist of pelems (path elements), which describes how to
get to some node in data-sources. Something similar to
XPath,
JsonPath, but not exactly.
There are few types of pelems:
mkeyvnavsightmicro
| term | definition |
|---|---|
mkey |
a simple edn :keyword, which tells shell executor to navigate to specific key in the map. |
vnav |
a vector, which consists of vkey and optional vfilter. |
vkey |
an index (some non-negative integer) or wildcard :* (keyword). |
vfilter |
a map used for pattern matching and templating |
sight |
:ihs/ namespaced keyword or {:ih/sight :ihs/sight-name :arg1 :value1} |
micro |
:ihm/ namespaced keyword or {:ih/micro :ihm/micro-name :arg1 :value1} |
Full grammar defined in core namespace.
Example of paths and get-values results:
;; {:k1 {:k2 :v3}}
[:k1] ;; => [[{:k2 :v3}]]
[:k1 :k2] ;; => [[:v3]]
;; [{:a :b} {:k :v :k1 :v2}]
[[0]] ;; => [[{:a :b}]]
[[:*]] ;; => [[0 {:a :b}] [1 {:k :v :k1 :v1}]]
[[1 {:a :b}]] ;; => [[nil]]
[[:* {:k :v}]] ;; => [[0 {:k :v :k1 :v1}]]
;; {:name "Firstname, Secondname"}
[:name :ihs/str<->vector [0]] ;; => [["Firstname,"]]
;; [:name {:ih/sight :ihs/str<->vector :separator ", "} [0]]
[:ihm/first-name] ;; => [["Firstname"]]
;; [[1 2] [3 4 5]]get-values always returns a vector of indexed values. Each wildcard inside
the path creates one dimension of index. Source and sink path should have same
number of wildcards to make transformation possible. ironhide interpreter
will align the shape automatically (without deleting existing data).
(get-values
[[:v1 :v2] [:v3 :v4 :v5]]
[[:*] [:*]])
;; => [[0 0 :v1] [0 1 :v2] [1 0 :v3] [1 1 :v4] [1 2 :v5]]
;; the result of get-values is a vector of indexed values
;; 1 2 - is a multi-dimensional indexSight
sight is a special type of pelem, which allows to percieve current node of
data-source differently. It's useful when you want to treat a string as a
vector of words for example:
;; {:name "Firstname Secondname"}
[:name :ihs/str<->vector [0]] ;; => [["Firstname"]]It allows to navigate inside node of data-source differently and more
preciesly, but don't change original structure of it.
It's possible to extend sights by defining
ironhide.core/get-global-sight method or using
nested sights.
Micro
micro is a parametrized shortcut for part of the path.
(microexpand-path
#:ih{:micros #:ihm {:name [:name [:index] :given [0]]}}
[:ihm/name])
;; => [:name [:index] :given [0]]
(microexpand-path
#:ih{:micros #:ihm {:name [:name [:index] :given [0]]}}
[{:ih/micro :ihm/name :index 10}])
;; => [:name [10] :given [0]]Default values for micros not supported yet.
Rule
Rule specifies relation between parts of data-sources. It is a map, which can
contain few different key types:
data-sourcename, which associated with path to exact part ofdata-source:ih/direction, which associated with a pair of source and sinkdata-sources:ih/defaults, which associated with map ofdata-sourcename keys and path-to-default-value values
{:form [:firstname]
:fhir [:name [0] :given [0]]
:ih/defaults {:fhir [:ih/values :firstname]}
:ih/direction [:form :fhir]}Thanks
Special thanks to:
- Nathan Marz for specter
- Nikolai Ryzhikov for matcho and 2way
Contribution
PRs are welcome, but merging not guaranteed. Create issue or contact abcdw if you need or want.
License
Copyright © 2018 HealthSamurai
Distributed under the Eclipse Public License either version 1.0 or (at your option) any later version.