Collective Knowledge framework (CK)

Linux/MacOS: Windows:

News

• See the ACM TechTalk from the CK author (2021.Feb.11).
• We are preparing a major redesign of the CK framework to make it more Pythonic and easier to use (CK2) - feel free to contact Grigori Fursin for more details and subscribe for our newsletter!
• We have finished prototyping the CK concept and successfully used it to automate MLPerf inference benchmark submissions for edge devices and the design space exploration of efficient ML/SW/HW stacks in terms of speed, accuracy, energy and costs.

Overview

Collective Knowledge framework (CK) helps to organize software projects as a database of reusable components with common automation actions and extensible meta descriptions based on FAIR principles (findability, accessibility, interoperability and reusability) as described in this article.

Our goal is to help researchers and practitioners share, reuse and extend their knowledge in the form of portable workflows, automation actions and reusable artifacts with a common API, CLI, and meta description. See how CK supports collaborative and reproducible research:

• MLPerf automation workflows (GitHub);
• Real-world use cases from MLPerf, Arm, General Motors, IBM, the Raspberry Pi foundation, ACM, dividiti and other great partners;
• Reddit discussion about reproducing 150 papers;
• Our reproducibility initiatives: [methodology], checklist, [events].

Documentation

• Online CK documentation
  • Why CK?
  • CK Basics
  • Try CK
• Publications
• All news

Installation

Follow this guide to install CK framework on your platform.

CK supports the following platforms:

Example (without Docker)

Here we show how to pull a GitHub repo in the CK format and use a unified CK interface to compile and run any program (image corner detection in our case) with any compatible data set on any compatible platform:

python3 -m pip install ck

ck pull repo --url=https://github.com/ctuning/ck-crowdtuning

ck ls program:*susan*

ck search dataset --tags=jpeg

ck detect soft --tags=compiler,gcc
ck detect soft --tags=compiler,llvm

ck show env --tags=compiler

ck compile program:cbench-automotive-susan --speed

ck run program:cbench-automotive-susan --cmd_key=corners --repeat=1 --env.MY_ENV=123 --env.TEST=xyz

You can check output of this program in the following directory:

cd `ck find program:cbench-automotive-susan`/tmp
ls -l

tmp-output.tmp - image with detected corners (rename to ppm to view it)

Check CK docs for further details.

Example (with Docker)

We have prepared a CK container with AI and ML components: [Docker], [CK meta]

You can start it as follows:

docker run --rm -it ctuning/ck-ai:ubuntu-20.04

You can then prepare and run portable AI/ML workflows and program pipelines.

More examples of CK workflows, automation actions and reusable artifacts for

• Showroom (public projects powered by CK):
  • MLPerf automation
  • Student Cluster Competition automation: SCC18, digital artifacts
  • Jupyter notebooks: ML/SW/HW DSE for edge devices
  • ML-based autotuning project: reproducible paper demo, MILEPOST
  • Stable Docker containers with CK workflows: MLPerf example, cKnowledge.io, Docker Hub
  • Quantum hackathons
  • ACM SW/HW co-design tournaments for Pareto-efficient deep learning
  • Portable CK workflows and components for:
    • TensorFlow
    • PyTorch
    • TensorRT
    • OpenVino
    • individual NN operators
    • object detection
  • GUI to automate ML/SW/HW benchmarking with MLPerf example (under development)
  • Reproduced papers
  • Live scoreboards for reproduced papers
• Examples of CK components (automations, API, meta descriptions):
  • program : image-classification-tflite-loadgen [cKnowledge.io] [GitHub]
  • program : image-classification-tflite [GitHub]
  • soft : lib.mlperf.loadgen.static [GitHub]
  • package : lib-mlperf-loadgen-static [GitHub]
  • package : model-onnx-mlperf-mobilenet [GitHub]
  • package : lib-tflite [cKnowledge.io] [GitHub]
  • docker : object-detection-tf-py.tensorrt.ubuntu-18.04 [cKnowledge.io]
  • docker [GitHub]
  • docker : speech-recognition.rnnt [GitHub]
  • package : model-tf-* [GitHub]
  • script : mlperf-inference-v0.7.image-classification [cKnowledge.io]
  • jnotebook : object-detection [GitHub]

CK portal

cKnowledge.io: organizing ML and Systems knowledge in the form of portable CK workflows, automation actions and reusable components:

• All CK ML&systems components
• CK compatible repositories
• CK portable solutions (including MLPerf inference on edge)
• CK-based adaptive containers
• CK modules
• Automation actions
• Portable program workflows
• Meta packages
• Software detection (code, data, models)
• Platform detection
• Shared experiments
• Reproduced results from papers

Contributions

Users can extend the CK functionality via CK modules or external GitHub reposities in the CK format as described here.

Please check this documentation if you want to extend the CK core functionality and modules.

Note, that we plan to redesign the CK core to be more pythonic (we wrote the first prototype without OO to be able to port to bare-metal devices in C but we decided not to do it at the end). We also plan to relicense the framework to Apache 2.0.

Author

• Grigori Fursin (cTuning foundation and cKnowledge SAS)

Acknowledgments

We would like to thank all contributors and collaborators for their support, fruitful discussions, and useful feedback! See more acknowledgments in the CK journal article.