All Projects → Toumash → mlflow-docker

Toumash / mlflow-docker

Licence: MIT license
Ready to run docker-compose configuration for ML Flow with Mysql and Minio S3

Programming Languages

python
139335 projects - #7 most used programming language
shell
77523 projects
Dockerfile
14818 projects

Projects that are alternatives of or similar to mlflow-docker

Helm S3
Helm plugin that allows to set up a chart repository in AWS S3.
Stars: ✭ 372 (+154.79%)
Mutual labels:  s3, minio
mlflow-tracking-server
MLFLow Tracking Server based on Docker and AWS S3
Stars: ✭ 59 (-59.59%)
Mutual labels:  mlflow, mlflow-tracking-server
Stock Analysis Engine
Backtest 1000s of minute-by-minute trading algorithms for training AI with automated pricing data from: IEX, Tradier and FinViz. Datasets and trading performance automatically published to S3 for building AI training datasets for teaching DNNs how to trade. Runs on Kubernetes and docker-compose. >150 million trading history rows generated from +5000 algorithms. Heads up: Yahoo's Finance API was disabled on 2019-01-03 https://developer.yahoo.com/yql/
Stars: ✭ 605 (+314.38%)
Mutual labels:  s3, minio
s3x
s3x is a minio gateway providing an S3 API powered by TemporalX that uses IPFS as the data storage layer. It lets you turn any S3 application into an IPFS application with no change in application design
Stars: ✭ 85 (-41.78%)
Mutual labels:  s3, minio
Drone Cache
A Drone plugin for caching current workspace files between builds to reduce your build times
Stars: ✭ 194 (+32.88%)
Mutual labels:  s3, minio
minio-rclone-webdav-server
A @rclone served WebDAV server with @minio as the s3 storage backend docker example
Stars: ✭ 17 (-88.36%)
Mutual labels:  s3, minio
Minio Go
MinIO Client SDK for Go
Stars: ✭ 1,231 (+743.15%)
Mutual labels:  s3, minio
esop
Cloud-enabled backup and restore tool for Apache Cassandra
Stars: ✭ 40 (-72.6%)
Mutual labels:  s3, minio
Rust S3
Rust library for interfacing with AWS S3 and other API compatible services
Stars: ✭ 177 (+21.23%)
Mutual labels:  s3, minio
S3contents
A S3 backed ContentsManager implementation for Jupyter
Stars: ✭ 175 (+19.86%)
Mutual labels:  s3, minio
terraform-provider-minio
Terraform provider for managing minio S3 buckets and IAM Users
Stars: ✭ 123 (-15.75%)
Mutual labels:  s3, minio
mindav
A self-hosted file backup server which bridges WebDAV protocol with @minio written in @totoval. Webdav ❤️ Minio
Stars: ✭ 64 (-56.16%)
Mutual labels:  s3, minio
fss3
FSS3 is an S3 filesystem abstraction layer for Golang
Stars: ✭ 52 (-64.38%)
Mutual labels:  s3, minio
Burry.sh
Cloud Native Infrastructure BackUp & RecoveRY
Stars: ✭ 260 (+78.08%)
Mutual labels:  s3, minio
minback-postgres
A container which provides the ability to backup a PostgreSQL database to Minio on demand
Stars: ✭ 18 (-87.67%)
Mutual labels:  s3, minio
Rome
Carthage cache for S3, Minio, Ceph, Google Storage, Artifactory and many others
Stars: ✭ 724 (+395.89%)
Mutual labels:  s3, minio
minio-dart
Unofficial MinIO Dart Client SDK that provides simple APIs to access any Amazon S3 compatible object storage server.
Stars: ✭ 42 (-71.23%)
Mutual labels:  s3, minio
gosquito
gosquito ("go" + "mosquito") is a pluggable tool for data gathering, data processing and data transmitting to various destinations.
Stars: ✭ 25 (-82.88%)
Mutual labels:  s3, minio
Dataengineeringproject
Example end to end data engineering project.
Stars: ✭ 82 (-43.84%)
Mutual labels:  s3, minio
backup-repository
Backup storage for E2E GPG-encrypted files, with multi-user, quotas, versioning, using a object storage (S3/Min.io/GCS etc.) and deployed on Kubernetes or standalone.
Stars: ✭ 21 (-85.62%)
Mutual labels:  s3, minio

MLFlow Docker Setup Actions Status

If you want to boot up mlflow project with one-liner - this repo is for you. The only requirement is docker installed on your system and we are going to use Bash on linux/windows.

🚀 1-2-3! Setup guide

  1. Configure .env file for your choice. You can put there anything you like, it will be used to configure you services
  2. Run docker compose up
  3. Open up http://localhost:5000 for MlFlow, and http://localhost:9001/ to browse your files in S3 artifact store

👇Video tutorial how to set it up + BONUS with Microsoft Azure 👇

Youtube tutorial

Features

  • One file setup (.env)
  • Minio S3 artifact store with GUI
  • MySql mlflow storage
  • Ready to use bash scripts for python development!
  • Automatically-created s3 buckets

How to use in ML development in python

Click to show
  1. Configure your client-side

For running mlflow files you need various environment variables set on the client side. To generate them user the convienience script ./bashrc_install.sh, which installs it on your system or ./bashrc_generate.sh, which just displays the config to copy & paste.

$ ./bashrc_install.sh
[ OK ] Successfully installed environment variables into your .bashrc!

The script installs this variables: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, MLFLOW_S3_ENDPOINT_URL, MLFLOW_TRACKING_URI. All of them are needed to use mlflow from the client-side.

  1. Test the pipeline with below command with conda. If you dont have conda installed run with --no-conda
mlflow run [email protected]:databricks/mlflow-example.git -P alpha=0.5
# or
python ./quickstart/mlflow_tracking.py
  1. (Optional) If you are constantly switching your environment you can use this environment variable syntax
MLFLOW_S3_ENDPOINT_URL=http://localhost:9000 MLFLOW_TRACKING_URI=http://localhost:5000 mlflow run [email protected]:databricks/mlflow-example.git -P alpha=0.5

Licensing

Copyright (c) 2021 Tomasz Dłuski

Licensed under the MIT License (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License by reviewing the file LICENSE in the repository.

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].