Tricks

Running an experiment should be as easy as describing it in an YAML file.

Tricks presentation

Here's a presentation about Tricks.

About Tricks

A Tricks entry (TE) is a set of replicas/pods. An experiment is one or more TE's.

Features

• Register occurrence of events. Each event has a counter associated.
• Implicit events (start and stop of each pod in a TE)
• Subscribe to events
  • can be used to implement synchronization barrier (e.g. make sure all clients start at the same time)
• Workflow (e.g. only start a given TE once event X counter is Y)
• Sequences of experiments
• Pod discovery
• Log aggregation
• Plotting from logs (e.g. latency/throughput, CDF, bar, line)
• Detect coordination omission from logs
• Fault injection
  • (when) could be defined in the same way workflow is
  • (what) maybe provide some sort of pod selector for process faults, and link selector for network faults
• Federation support (run across multiple Kubernetes clusters)
• Spot/preemptible instances support (if an instance is killed, the experiment is restarted)

Example

In this example, event client1_stop and client2_stop are implicit events, while server-ready is an event registered by replicas in the server TE.

apiVersion: v1
experiment:
- tag: server
  image: vitorenesduarte/tricks-example
  replicas: 3
  env:
  - name: TYPE
    value: loop
  workflow:
    stop:
      name: client2_stop
      value: 6
- tag: client1
  image: vitorenesduarte/tricks-example
  replicas: 3
  env:
  - name: TYPE
    value: loop
  - name: SECONDS 
    value: 5
  workflow:
    start:
      name: server-ready
      value: 3
- tag: client2
  image: vitorenesduarte/tricks-example
  replicas: 6
  env:
  - name: TYPE
    value: loop
  - name: SECONDS
    value: 10
  workflow:
    start:
      name: client1_stop
      value: 3

Other examples

• examples/hello-world.yaml
• examples/implicit-events.yaml
• examples/explicit-events.yaml

Running Tricks

Assuming there's a Kubernetes cluster running:

$ tricks start
$ tricks logs

And then in another terminal, run one of the examples:

$ tricks exp examples/explicit-events.yaml

Environment variables

Configuration of pods is achieved through environment variables. Some variable names are reserved and always defined in every pod:

• TAG: from configuration file
• REPLICAS: from configuration file
• EXP_ID: an experiment identifier generated by Tricks
• POD_ID: unique (no other pod with the same TAG and EXP_ID has the same id)
• POD_IP: pod IP from Kubernetes
• TRICKS_IP: pod IP of Tricks
• TRICKS_PORT: port of Tricks

Drivers API

Replicas in experiments can be written in any language, as long as there's a driver available. Drivers open a socket (TRICKS_IP and TRICKS_PORT) and talk with Tricks using the following API ([DT] is used if it's a message from a Driver to Tricks, or with [TD] otherwise).

• Register events [DT]

{
  "expId": "123456",
  "type": "event",
  "eventName": "connected",
}

• Subscription of events [DT]

{
  "expId": "123456",
  "type": "subscription",
  "eventName": "connected",
  "value": 10
}

• Notification of events [TD]

{
  "expId": "123456",
  "type": "notification",
  "eventName": "connected",
  "value": 10
}

• Pod discovery [DT]

{
  "expId": "123456",
  "type": "discovery",
  "tag": "server",
  "min": 2
}

The argument min in the previous message is optional (default value is 0). Tricks will only reply when it has at least min pods (sending all, even if more than min).

• Pod discovery [TD]

{
  "expId": "123456",
  "type": "pods",
  "tag": "server",
  "pods": [
    {
      "id": 1,
      "ip": "10.12.13.15"
    },
    {
      "id": 2,
      "ip": "10.12.13.16"
    },
    {
      "id": 3,
      "ip": "10.12.13.17"
    }
  ]
}

Sequences of experiments

Typically we want, not to run a single experiment, but several, and in the end compare metrics collected.

In order to support sequences of experiments, configuration values in the YAML file can be replaced by variables (e.g. $var), that are defined in a list of configurations. Each configuration will be used to create an experiment.

An example with two variables ($op_number and $client_number) that are used to define number of replicas, workflow configuration and environment variables:

apiVersion: v1
config:
  - $op_number: 100
    $client_number: 3
  - $op_number: 200
    $client_number: 3
  - $op_number: 100
    $client_number: 6
  - $op_number: 200
    $client_number: 6
experiment:
  - tag: server
    image: vitorenesduarte/tricks-example
    replicas: 3
    env:
    - name: TYPE
      value: server
    workflow:
      stop:
        name: client_stop
        value: $client_number
  - tag: client
    image: vitorenesduarte/tricks-example
    replicas: $client_number
    env:
    - name: TYPE
      value: client
    - name: OP_NUMBER
      value: $op_number
    workflow:
      start:
        name: server_start
        value: 3