into-docker / Clj Docker Client

clj-docker-client

An idiomatic, data-driven, REPL friendly Clojure Docker client inspired from Cognitect's AWS client.

See this for documentation for versions before 0.4.0.

The README here is for the current master branch and may not reflect the released version.

Please raise issues here for any new feature requests!

Installation

Leiningen/Boot

[lispyclouds/clj-docker-client "1.0.3"]

Clojure CLI/deps.edn

{lispyclouds/clj-docker-client {:mvn/version "1.0.3"}}

Gradle

compile 'lispyclouds:clj-docker-client:1.0.3'

Maven

<dependency>
  <groupId>lispyclouds</groupId>
  <artifactId>clj-docker-client</artifactId>
  <version>1.0.3</version>
</dependency>

Build Requirements

• Leiningen 2.8+
• JDK 1.8+

Running tests locally

• Install leiningen
• Install Docker
• lein kaocha to run all tests. (needs Docker and working internet)

Auto generated code docs can be found here

Developing with Reveal and Leiningen

Since this is fully data driven, using Reveal is really beneficial as it allows us to walk through the output from Docker, see potential errors and be more productive with instant visual feedback.

• Clone this repo.
• In the root, start the leiningen REPL with: lein with-profile +reveal repl. This fires up the the Reveal UI alongside the usual REPL.
• Connect your editor of choice to this REPL or start using Reveal REPL directly.
• Then repeat after me 3 times: ALL HAIL THE DATA! 🙏🏽

The Docker API

This uses Docker's HTTP REST API to run. See the API version matrix to find the corresponding API version for the Docker daemon you're running.

clj-docker-client works by parsing the Swagger 2.0 YAMLs from the docker client API and vendors it in this directory. This defaults to using the latest version available there if no versions are pinned. It is recommended to use a pinned version to have consistent behavior across different engine versions.

See the page about the docker REST API to learn more about the usage and params to pass.

Usage

(require '[clj-docker-client.core :as docker])

This library aims to be a as thin layer as possible between you and Docker. This consists of following public functions:

categories

Lists the categories of operations supported. Can be bound to an API version.

(docker/categories) ; Latest version

(docker/categories "v1.40") ; Locked to v1.40

#_=> #{:system
       :exec
       :images
       :secrets
       :events
       :_ping
       :containers
       :auth
       :tasks
       :volumes
       :networks
       :build
       :nodes
       :commit
       :plugins
       :info
       :swarm
       :distribution
       :version
       :services
       :configs
       :session}

client

Connect to the docker daemon's UNIX socket and create a client scoped to the operations of a given category. Can be bound to an API version.

(def images (docker/client {:category :images
                            :conn     {:uri "unix:///var/run/docker.sock"}})) ; Latest version

(def containers (docker/client {:category    :containers
                                :conn        {:uri "unix:///var/run/docker.sock"}
                                :api-version "v1.40"})) ; Container client for v1.40

Using a timeout for the connections. Thanks olymk2 for the suggestion. Docker actions can take quite a long time so set the timeout accordingly. When you don't provide timeouts then there will be no timeout on the client side.

(def ping (docker/client {:category :_ping
                          :conn     {:uri      "unix:///var/run/docker.sock"
                                     :timeouts {:connect-timeout 10
                                                :read-timeout    30000
                                                :write-timeout   30000
                                                :call-timeout    30000}}}))

Alternatively if connecting to a remote docker daemon over TCP supply the :uri as http://your.docker.host:2376. NOTE: unix://, http://, tcp:// and https:// are the currently supported protocols.

ops

Lists the supported ops by a client.

(docker/ops images)

#_=> (:ImageList
      :ImageCreate
      :ImageInspect
      :ImageHistory
      :ImagePush
      :ImageTag
      :ImageDelete
      :ImageSearch
      :ImagePrune
      :ImageGet
      :ImageGetAll
      :ImageLoad)

doc

Returns the doc of an operation in a client.

(docker/doc images :ImageList)

#_=> {:doc
      "List Images\nReturns a list of images on the server. Note that it uses a different, smaller representation of an image than inspecting a single image.",
      :params
      ({:name "all", :type "boolean"}
       {:name "filters", :type "string"}
       {:name "digests", :type "boolean"})}

invoke

Invokes an operation via the client and a given operation map and returns the result data.

; Pulls the busybox:musl image from Docker hub
(docker/invoke images {:op     :ImageCreate
                       :params {:fromImage "busybox:musl"}})

; Creates a container named conny from it
(docker/invoke containers {:op     :ContainerCreate
                           :params {:name "conny"
                                    :body {:Image "busybox:musl"
                                           :Cmd   "ls"}}})

The operation map is of the following structure:

{:op     :NameOfOp
 :params {:param-1 "value1"
          :param-2 true}}

Takes an optional key as. Defaults to :data. Returns an InputStream if passed as :stream, the raw underlying network socket if passed as :socket. :stream is useful for streaming responses like logs, events etc, which run till the container is up. :socket is useful for events when bidirectional streams are returned by docker in operations like :ContainerAttach.

{:op     :NameOfOp
 :params {:param-1 "value1"
          :param-2 true}
 :as     :stream}

Takes another optional key :throw-exception?. Defaults to false. If set to true will throw an exception for exceptional status codes from the Docker API i.e. status >= 400. Throws an java.lang.RuntimeException with the message.

{:op               :NameOfOp
 :throw-exception? true}

General guidelines

• Head over to the Docker API docs to get more info on the type of parameters you should be sending. eg: this page for v1.40 API docs.
• The type stream is mapped to java.io.InputStream and when the API needs a stream as an input, send an InputStream. When it returns a stream, the call can possibly block till the container or source is up and it's recommended to pass the as param as :stream to the invoke call and read it asynchronously. See this section for more info.

Sample code for common scenarios

Pulling an image

(def images (docker/client {:category :images
                            :conn     {:uri "unix:///var/run/docker.sock"}}))

(docker/invoke images {:op     :ImageCreate
                       :params {:fromImage "busybox:musl"}})

Creating a container

(def containers (docker/client {:category :containers
                                :conn     {:uri "unix:///var/run/docker.sock"}}))

(docker/invoke containers {:op     :ContainerCreate
                           :params {:name "conny"
                                    :body {:Image "busybox:musl"
                                           :Cmd   ["sh"
                                                   "-c"
                                                   "i=1; while :; do echo $i; sleep 1; i=$((i+1)); done"]}}})

Starting a container

(docker/invoke containers {:op     :ContainerStart
                           :params {:id "conny"}})

Creating a network

(def networks (docker/client {:category    :networks
                              :conn        {:uri "unix:///var/run/docker.sock"}
                              :api-version "v1.40"}))

(docker/invoke networks {:op     :NetworkCreate
                         :params {:networkConfig {:Name "conny-network"}}})

Streaming logs

; fn to react when data is available
(defn react-to-stream
  [stream reaction-fn]
  (future
    (with-open [rdr (clojure.java.io/reader stream)]
      (loop [r (java.io.BufferedReader. rdr)]
        (when-let [line (.readLine r)]
          (reaction-fn line)
          (recur r))))))

(def log-stream (docker/invoke containers {:op     :ContainerLogs
                                           :params {:id     "conny"
                                                    :follow true
                                                    :stdout true}
                                           :as     :stream}))

(react-to-stream log-stream println) ; prints the logs line by line when they come.

Attach to a container and send data to stdin

;; This is a raw bidirectional java.net.Socket, so both reads and writes are possible.
;; conny-reader has been started with: docker run -d -i --name conny-reader alpine:latest sh -c "cat - >/out"
(def sock (docker/invoke containers {:op     :ContainerAttach
                                     :params {:id     "conny-reader"
                                              :stream true
                                              :stdin  true}
                                     :as     :socket}))

(clojure.java.io/copy "hello" (.getOutputStream sock))

(.close sock) ; Important for freeing up resources.

Using registries that need authentication

Thanks @AustinC for this example.

(ns dclj.core
  (:require [clj-docker-client.core :as d]
            [cheshire.core :as json])
  (:import [java.util Base64]))

(defn b64-encode
  [to-encode]
  (.encodeToString (Base64/getEncoder) (.getBytes to-encode)))

(def auth
  (-> {"username"      "un"
       "password"      "pw"
       "serveraddress" "docker.acme.com"}
      json/encode
      b64-encode))

(def images
  (d/client {:category    :images
             :conn        {:uri "unix:///var/run/docker.sock"}
             :api-version "v1.40"}))

(d/invoke images
          {:op               :ImageCreate
           :params           {:fromImage       "docker.acme.com/eg:2.1.995"
                              :X-Registry-Auth auth}
           :throw-exception? true})

HTTPS and Mutual TLS(mTLS)

Since both https and unix sockets are suppported, and generally docker deamons exposed over HTTPS are protected via mTLS, here is an example using mTLS to connect to docker via HTTPS:

;; Create a client using https
;; The ca.pem, key.pem and cert.pem are produced by the docker daemon when protected via mTLS
(def http-tls-ping
  (client {:category :_ping
           :conn     {:uri  "https://my.remote.docker.host:8000"
                      :mtls {:ca   "ca.pem"
                             :key  "key.pem"
                             :cert "cert.pem"}}}))

(invoke http-tls-ping {:op :SystemPing}) ;; => Returns "OK"

The caveat here is password protected PEM files aren't supported yet. Please raise an issue if there is a need for it.

Not so common scenarios

Accessing undocumented/experimental Docker APIs

There are some cases where you may need access to an API that is either experimental or is not in the swagger docs. Docker checkpoint is one such example. Thanks @mk for bringing it up!

Since this uses the published APIs from the swagger spec, the way to access them is to use the lower level fn fetch from the clj-docker-client/requests ns. The caveat is the response will be totally raw(data, stream or the socket itself).

fetch takes the following params as a map:

• conn: the connection to the daemon. Required.
• url: the relative path to the operation. Required.
• method: the method of the HTTP request as a keyword. Default: :get.
• query: the map of key-values to be passed as query params.
• path: the map of key-values to be passed as path params. Needed for interpolated path values like /v1.40/containers/{id}/checkpoints. Pass {:id "conny"} here.
• header: the map of key-values to be passed as HEADER params.
• body: the stream or map(will be converted to JSON) to be passed as body.
• as: takes the kind of response expected. One of :stream, :socket or :data. Same as invoke. Default: :data.

(require '[clj-docker-client.requests :as req])
(require '[clj-docker-client.core :as docker])

;; This is the undocumented API in the Docker Daemon.
;; See https://github.com/moby/moby/pull/22049/files#diff-8038ade87553e3a654366edca850f83dR11
(req/fetch {:conn (req/connect* {:uri "unix:///var/run/docker.sock"})
            :url  "/v1.40/containers/conny/checkpoints"})

More examples of low level calls:

;; Ping the server
(req/fetch {:conn (req/connect* {:uri "unix:///var/run/docker.sock"})
            :url  "/v1.40/_ping"})

;; Copy a folder to a container
(req/fetch {:conn   (req/connect* {:uri "unix:///var/run/docker.sock"})
            :url    "/v1.40/containers/conny/archive"
            :method :put
            :query  {:path "/root/src"}
            :body   (-> "src.tar.gz"
                        io/file
                        io/input-stream)})

Reading a streaming output in case of an exception being thrown

When :throw-exception? is passed as true and the :as is set to :stream, to read the response stream, pass throw-entire-message? as true to the invoke. The stream is available as :body in the ex-data of the exception.

(try
  (invoke containers
          {:op                    :ContainerArchive
           :params                {:id   "conny"
                                   :path "/this-does-not-exist"}
           :as                    :stream
           :throw-exception?      true
           :throw-entire-message? true})
  (catch Exception e
    (-> e ex-data :body slurp println))) ; Prints out the body of error from docker.

And anything else is possible!

License

Copyright © 2020 Rahul De and contributors.

Distributed under the LGPLv3+ License. See LICENSE.