Lines

A pure bash clojureish CI pipeline.

Main Features

• Complete CI engine
• Execute local or remote
• Pure data syntax edn
• Clojure script syntax
• Easy command line integration like (docker, kubectl, apt)
• Concurrency execution
• Modular and extensible

Why?

• Alternative method to install clojure, using "clojure", but without clojure
• Tired to write yaml every day
• Create something cool with Fleck

Table of contents

• Installation
• Job keywords
• Modules
  • shell
  • docker
  • scp
  • template
  • user module
• EDN Pipeline
  • Targets file
• Clojure Pipeline
  • Lines functions
• Environment vars
• Extensions
• Development

Installation

requirements

• bash 4
• coreutils
• git
• docker ¹
• ssh ²
• scp ²

¹ docker module
² remote execution

install script

# download

curl -L https://github.com/rosineygp/lines-sh/releases/latest/download/lines > lines
chmod +x lines
sudo mv lines /usr/bin/lines

minimal usage

Create a file .lines.edn with followin data.

[{:apply ["echo hello world!"]}]

Just execute command lines

name: lines *******************************************************************
target: local
start: ter 22 dez 2020 21:38:01 -03
  cmd: echo hello world!
    hello world!
  exit-code: 0
status: true
finished: ter 22 dez 2020 21:38:04 -03 ****************************************

Job keywords

A job is defined as a hashmap of keywords. The commons keywords available for job are:

apply

Is the only required keyword that job needs. It's an array of objects that will be executed by a module. Each element of array is a tasks and the job can handler N tasks. If any tasks has a exit code different than 0, the job will stop and throw the error. ignore-error and retries helps to handler errors.

{:apply ["uname -a"
         "make build"]}

using default module shell, apply is a array of strings.

{:module "scp"
 :apply [{:src "program.tar.gz" :dest "/tmp/program.tar.gz"}
         {:src "script.sh"      :dest "/tmp/script.sh"}]}

scp uses a list of hashmaps

name

name is just a label for the job.

{:name "install curl"
 :module "shell"
 :apply ["apk add curl"]}

module

module is the method executed by job (default is shell)

{:module "shell"
 :apply ["whoami"]}

Docker module

{:module "docker"
 :args {:image "node"}
 :apply ["npm test"]}

the builtin modules are:

is possible create custom modules

target

Host target is the location where the job will run. If any target passed the job will run at localhost.

{:target {:label "web-server"
          :host "web.local.net"
          :port 22
          :method "ssh"}}

Targets can be defined in separated file, during execution is possible to merge data with job and execute the same job in n hosts.

Is possible set another keywords for filter like group, dc or any other value you need to organizer targets.

After job executed it return themself with result values

vars

Variables will be inject in environment during tasks execution.

{:vars {MY_VAR_0 "lines"
        MY_VAR_1 "go"}
 :apply ["echo $MY_VAR_0"
         "echo $MY_VAR_1"]}

BRANCH_NAME and BRANCH_NAME_SLUG are inject in environment.

args

Args is the parameters of modules.

{:args {:sudo true}
 :apply ["apt-get update"
         "apt-get install htop -y]}

All tasks will run with sudo

ignore-error

If some task fail, lines will not stop the pipeline, just return the current task failed.

{:ignore-error true
 :apply ["whoami"
         "exit 1"
         "dpkg -l"]}

The tasks after error will not be executed.

retries

If some task fail, retry will run it again.

{:retries 2
 :apply ["ping -c 1 my-host"]}

The max retries are 2, but it can be increase setting LINES_JOB_MAX_ATTEMPTS at environment vars.

Modules

shell

Is it the default module, just spawn scripts to shell.

{:module "shell"
 :apply ["date"]}

Command will be executed inside a heredoc block

<command_header> bash -s <<-'LINES-BLOCK-EOF'
export <env vars>;
commands
LINES-BLOCK-EOF

arguments

¹ needs pre configured sudoers (without password)

docker

Create a docker instance and execute commands inside it.

single instance

{:module "docker"
 :apply ["whoami"]}

• start docker instance with default image (alpine)
• run command whoami inside container

services

{:module "docker"
 :args {:image "ubuntu"
        :services [{:image "nginx"
                    :alias "nginx"}]}
 :apply ["apt-get update"
         "apt-get install curl -y"
         "curl http://nginx"]}

• start docker instance with nginx image as a service and set network alias as nginx
• start another docker instance with ubuntu image
• install ubuntu packages
• execute curl at service from ubuntu instance

download artifacts

Download files or directory from a docker instance.

{:module "docker"
 :args {:artifacts {:paths ["file"
                            "directory"]}}
 :apply ["touch file"
         "mkdir directory"
         "touch directory/file"]}

arguments

services description

Services is an array of hashmaps, is possible up N services with docker module, the following keywords can be used to build a service.

artifacts description

Download a files or directories from a docker instance.

scp

Copy files and folders to remote host over scp.

{:module "scp"
 :apply [{:src "./dist/command.bin"
          :dest "/usb/bin/command"}]}

apply arguments

template

Simple template file that only replace values inside double brackets {{ varname }}.

Hello {{ NAME }}!

template file

{:module "template"
 :vars {NAME "lines"}
 :apply [{:src "./hello-world.j2"
          :dest "/tmp/hello-world.txt"}]}

user module

Lines provides interface with custom user module.

Just put additional clojure scripts at .lines/modules/<module_name>/module.clj, like following example.

; .lines/modules/git/module.clj

; create a boilerplate function for git command
(str-use ["git"])

; custom user function, params: job (receive job definition), i (apply index)
; the function to return a string eg. `git clone -v [email protected]:rosineygp/mkdkr.git mkdkr`
(defn str-git-command-line [job i]
  (git ["clone"
        "-v"
        (get i :repos)
        (get i :dest)]))

; lines will call this function `lines-module-<module_name>`
(defn lines-module-git [job]
  (lines-task-loop job str-git-command-line)) ; loop handler

Using user module.

{:module "git"
 :apply [{:repos "[email protected]:rosineygp/lines.git" :dest "lines"}
         {:repos "[email protected]:rosineygp/mkdkr.git" :dest "mkdkr"}]}

EDN Pipeline

Pipelines is an array of jobs, and can be described using only edn.

; file: node.edn

[{:name "build"
  :apply ["npm install"]}
 [{:name "unit test"
   :group ["test"]
   :apply ["npm unit"]}
  {:name "mocha test"
   :group ["test"]
   :apply ["npm mocha"]}]
 {:name "deploy"
  :apply ["npm deploy"]}]

Jobs inside pipeline can be executed in parallel, just group then with [ array ], is possible use custom keywords for better notation like :groups.

# execute all jobs
lines -p node.edn

# filter only tests
lines -p node.edn -j group=test

Targets file

Is possible describe hosts targets using edn files, like pipeline.

; file: hosts.edn

[{:label "vm-0" :host "192.168.1.4" :method "ssh" :user "ubuntu"}
 {:label "vm-1" :host "192.168.1.5" :method "ssh" :user "ubuntu"}]

ssh only works with authorized_keys pre-configured, command ssh-copy-id can help configure it.

# execute pipeline in all hosts
lines -p node.edn -i hosts.edn

# filter only wm-0 host
lines -p node.edn -i hosts.edn -l label=wm-0

# filter jobs deploy and host wm-1
lines -p node.edn -i hosts.edn -l label=wm-1 -j name=deploy

EDN files are not safe is possible call functions dynamically.

; unsafe pipeline, extensions are based on this behavior

[{:apply [(str "echo " (time-ms))]}]

Clojure Pipeline

Powerful and dynamic pipelines with complex scenarios.

; generate single job for each yml found and test with docker image yamllint

(lines-pp (parallel
    (map (fn [x] (assoc {}
                        :name (get x :object)
                        :module "docker"
                        :args {:image "my/yamllint"}
                        :apply [(str "yamllint " (get x :object))]))
            (filter (fn [i] (= "yml" (get i :type))) (list-dir "src/")))))

This project use this case to test itself at .lines.clj.

Lines functions

The following functions will help to build lines clojure scripts.

src/core.clj and src/includes/lang-utils.clj has more useful functions.

Additional functions included in Lines.

filter | pmap | println_stderr | exit! | trap! | unset | str-join | str-subs | range | str-ident | mod | file-exists | file-write | unlink | or | and | hashmap-list | merge | key-name | call | callable? | get-in | even? | odd? | load-once | spit

The result of function job is the following data:

({:attempts 1
  :args {}
  :module "shell"
  :status true
  :apply ["echo hello world!"]
  :name "lines"
  :retries 0
  :target {:label "local" :method "local"}
  :pipestatus ((0))
  :finished 1608684590507
  :vars {"BRANCH_NAME_SLUG" "master" "BRANCH_NAME" "master"}
  :ignore-error false
  :start 1608684587657
  :result (({:exit-code 0
             :finished 1608684590253
             :cmd "echo hello world!"
             :stdout "hello world!"
             :stderr ""
             :start 1608684590233
             :debug "  bash -s <<-'LINES-BLOCK-EOF'\n export BRANCH_NAME_SLUG=\"master\" BRANCH_NAME=\"master\" ;\n echo hello world! \nLINES-BLOCK-EOF"}))})

Environment vars

Lines variables definition.

Extensions

Is possible create functions and extend usability, all extensions must be keep at .lines/ext/, it is just a clojure script with useful functions.

; file: .lines/ext/apt-helper.clj

(str-use ["apt-get"])

(defn apt-update []
  (apt-get ["update"]))

(defn apt-args [options packages]
   (apt-get  (concat options packages)))

(defn apt-install [packages]
  (apt-args ["install" "-y"] packages))

(defn apt-remove [packages]
  (apt-args ["remove" "-y"] packages))

Now edn files or pipeline can call those functions.

; edn pipeline

[{:name "apt with extensions"
  :apply [(apt-update)
          (apt-install ["python"
                        "mysql-server"
                        "memcached"])]}]

Development

Build this project

./build.sh

> result
  flk lines

flk is same of Fleck with patches/ applied and lines is flk with all src/ included.

Testing changes in src/ without build.

./flk src/main.clj <params>