All Projects → ctuning → Ck Tensorflow

ctuning / Ck Tensorflow

Licence: bsd-3-clause
Collective Knowledge components for TensorFlow (code, data sets, models, packages, workflows):

Programming Languages

python
139335 projects - #7 most used programming language

Projects that are alternatives of or similar to Ck Tensorflow

Pipedream
Connect APIs, remarkably fast. Free for developers.
Stars: ✭ 2,068 (+2197.78%)
Mutual labels:  apis, automation
Itunes store transporter
Upload and manage your assets in the iTunes Store using the iTunes Store’s Transporter (iTMSTransporter).
Stars: ✭ 117 (+30%)
Mutual labels:  automation, metadata
Streamsheets
An open-source tool for processing stream data using a spreadsheet-like interface.
Stars: ✭ 281 (+212.22%)
Mutual labels:  apis, automation
Habitat
Modern applications with built-in automation
Stars: ✭ 2,334 (+2493.33%)
Mutual labels:  automation, package-management
N8n
Free and open fair-code licensed node based Workflow Automation Tool. Easily automate tasks across different services.
Stars: ✭ 19,252 (+21291.11%)
Mutual labels:  apis, automation
Cenit
🚀 Cenit IO - 100% open source integration Platform (iPaaS)
Stars: ✭ 186 (+106.67%)
Mutual labels:  apis, automation
Auto App
Crie um aplicativo com todas as tabelas de um dos seus bancos sem uma linha de código.
Stars: ✭ 18 (-80%)
Mutual labels:  automation, metadata
Tower Operator
DEPRECATED: This project was moved and renamed to: https://github.com/ansible/awx-operator
Stars: ✭ 87 (-3.33%)
Mutual labels:  automation
Vkbot
Простой разговорный бот на PHP
Stars: ✭ 88 (-2.22%)
Mutual labels:  automation
Image search
Python Library to download images and metadata from popular search engines.
Stars: ✭ 86 (-4.44%)
Mutual labels:  metadata
Google It Automation With Python
Repository to keep track of Google IT Automation with Python provided by Coursera
Stars: ✭ 86 (-4.44%)
Mutual labels:  automation
Auto Py To Exe
Converts .py to .exe using a simple graphical interface
Stars: ✭ 1,272 (+1313.33%)
Mutual labels:  automation
Pybuilder
Software build automation tool for Python.
Stars: ✭ 1,290 (+1333.33%)
Mutual labels:  automation
Watomatic
Auto reply app helping you move away from less private messengers like WhatsApp and soon Facebook Messenger
Stars: ✭ 83 (-7.78%)
Mutual labels:  automation
Blueprint
solid framework for building APIs and backend services
Stars: ✭ 87 (-3.33%)
Mutual labels:  apis
Medusa
Automatic Video Library Manager for TV Shows. It watches for new episodes of your favorite shows, and when they are posted it does its magic.
Stars: ✭ 1,268 (+1308.89%)
Mutual labels:  metadata
Patrowlhears
PatrowlHears - Vulnerability Intelligence Center / Exploits
Stars: ✭ 89 (-1.11%)
Mutual labels:  automation
Install Unity
Script to install Unity 3D versions from the command line on macOS
Stars: ✭ 89 (-1.11%)
Mutual labels:  automation
Nps
NPM Package Scripts -- All the benefits of npm scripts without the cost of a bloated package.json and limits of json
Stars: ✭ 1,285 (+1327.78%)
Mutual labels:  automation
Flexget
The official FlexGet repository
Stars: ✭ 1,279 (+1321.11%)
Mutual labels:  automation

Collective Knowledge components for TensorFlow

All CK components can be found at cKnowledge.io and in one GitHub repository!

This project is hosted by the cTuning foundation.

compatibility automation workflow

DOI License

Linux/MacOS: Travis Build Status Windows: AppVeyor Build status

Introduction

CK-TensorFlow repository provides automation components in the CK format for tedious and repetitive tasks such as detecting and installing different TensorFlow versions, models and data sets across diverse platforms and running AI/ML workflows in a unified way.

Note that if some third-party automation fails or misses some functionality (software detection, package installation, bechmarking and autotuning workflow, etc), the CK concept is to continuously and collaboratively improve such reusable components! Please provide your feedback and report bugs via GitHub issues or get in touch with the community using this public CK mailing list!

Installation

Prerequisites for Ubuntu

  • Python 2.x:
$ sudo apt-get install python-dev python-pip python-setuptools python-opencv git

or

  • Python 3.x:
$ sudo apt-get install python3-dev python3-pip python3-setuptools

Note that CK will automatically install the following dependencies into CK TF virtual space: protobuf easydict joblib image wheel numpy scipy absl-py

Optional dependencies depending on your use cases:

  • CUDA/cuDNN if you have CUDA-enabled GPU

  • Android NDK if you want to compile and run TF for Android devices

CK installation

Follow these instructions to install CK.

Installation of ck-tensorflow repository

$ ck pull repo:ck-tensorflow

Basic usage

Example of a unified TensorFlow installation on Ubuntu or Windows via CK (pre-build versions)

$ ck install package:lib-tensorflow-1.8.0-cpu
 and/or (CK enables easy co-existance of different versions of tools
$ ck install package:lib-tensorflow-1.8.0-cuda

Check that TF is installed locally and registered in the CK:

$ ck show env --tags=lib,tensorflow

Use CK virtual environment to test it (similar to Python virtual env but for any binary package installed via CK):

$ ck virtual env --tags=lib,tensorflow

Install other TF versions available in the CK:

$ ck install package --tags=lib,tensorflow

Test unified image classification workflow via CK using above TF

$ ck run program:tensorflow --cmd_key=classify

Note, that you will be asked to select a jpeg image from available CK data sets. We added standard demo images (cat.jpg, catgrey.jpg, fish-bike.jpg, computer_mouse.jpg) to the 'ctuning-datasets-min' repository. You can list them via

 $ ck pull repo:ctuning-datasets-min
 $ ck search dataset --tags=dnn

Customize builds for different platforms

You can find more details about customized TensorFlow builds via CK for Android, Linux, Windows, Raspberry Pi, odroid, etc here.

Benchmarking

 $ ck run program:tensorflow (--env.BATCH_SIZE=10) (--env.NUM_BATCHES=5)

Select one of the test_cpu and test_cuda commands; select an available version of TensorFlow, if prompted (more than one choice); select an available benchmark, if prompted (more than one choice), and select TensorFlow model.

Crowd-benchmarking

It is now possible to participate in crowd-benchmarking of TensorFlow (early prototype):

$ ck crowdbench tensorflow --user={your email or ID to acknowledge contributions} (--env.BATCH_SIZE=128 --env.NUM_BATCHES=100)

You can see continuously aggregated results in the public Collective Knowledge repository under 'crowd-benchmark TensorFlow library' scenario.

Note, that this is an on-going, heavily evolving and long-term project to enable collaborative and systematic benchmarking and tuning of realistic workloads across diverse hardware (ARM TechCon'16 talk and demo, DATE'16, CPC'15). We also plan to add crowd-benchmarking and crowd-tuning of Caffe, TensorFlow and other DNN frameworks to our Android application soon - please, stay tuned!

Unified, multi-dimensional and multi-objective autotuning

It is now possible to take advantage of our universal multi-objective CK autotuner to optimize TensorFlow. As a first simple example, we added batch size tuning via CK. You can invoke it as follows:

$ ck autotune tensorflow

All results will be recorded in the local CK repository and you will be given command lines to plot graphs or replay experiments such as:

$ ck plot graph:{experiment UID}
$ ck replay experiment:{experiment UID} --point={specific optimization point}

Collaborative and unified DNN optimization

We are now working to extend above autotuner and crowdsource optimization of the whole SW/HW/model/data set stack (paper 1, paper 2).

We would like to thank the community for their interest and feedback about this collaborative AI optimization approach powered by CK at ARM TechCon'16 and the Embedded Vision Summit'17 - so please stay tuned ;) !

logo

Using other DNN via unified CK API

CK allows us to unify AI interfaces while collaboratively optimizing underneath engines. For example, we added similar support to install, use and evaluate Caffe/Caffe2, CK-PyTorch and MXNet via CK:

$ ck pull repo:ck-caffe2
$ ck pull repo --url=https://github.com/dividiti/ck-caffe
$ ck pull repo:ck-mxnet

$ ck install package:lib-caffe-bvlc-master-cpu-universal --env.CAFFE_BUILD_PYTHON=ON
$ ck install package:lib-caffe2-master-eigen-cpu-universal --env.CAFFE_BUILD_PYTHON=ON
$ ck install package --tags=mxnet

$ ck run program:caffe --cmd_key=classify
$ ck run program:caffe2 --cmd_key=classify
$ ck run program:mxnet --cmd_key=classify

$ ck crowdbench caffe --env.BATCH_SIZE=5
$ ck crowdbench caffe2 --env.BATCH_SIZE=5 --user=i_want_to_ack_my_contribution

$ ck autotune caffe
$ ck autotune caffe2

Realistic/representative training sets

We provided an option in all our AI crowd-tuning tools to let the community report and share mispredictions (images, correct label and wrong misprediction) to gradually and collaboratively build realistic data/training sets:

Online demo of a unified CK-AI API

  • Simple demo to classify images with continuous optimization of DNN engines underneath, sharing of mispredictions and creation of a community training set; and to predict compiler optimizations based on program features.

Open R&D challenges

We use crowd-benchmarking and crowd-tuning of such realistic workloads across diverse hardware for open academic and industrial R&D challenges - join this community effort!

Publications

Troublesooting

  • SqueezeDet demo currently work well with Python 3.5 and package:squeezedetmodel-squeezedet, so install it first:
$ ck install package:squeezedetmodel-squeezedet
$ ck run program:squeezedet

Feedback

Get in touch with ck-tensorflow developers via CK mailing list: http://groups.google.com/group/collective-knowledge !

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].