All Projects → ludwig-ai → Ludwig

ludwig-ai / Ludwig

Licence: apache-2.0
Data-centric declarative deep learning framework

Programming Languages

python
139335 projects - #7 most used programming language
Dockerfile
14818 projects

Projects that are alternatives of or similar to Ludwig

Bidaf Keras
Bidirectional Attention Flow for Machine Comprehension implemented in Keras 2
Stars: ✭ 60 (-99.25%)
Mutual labels:  natural-language-processing, deep-neural-networks, deeplearning, natural-language-understanding
Awesome Cybersecurity Datasets
A curated list of amazingly awesome Cybersecurity datasets
Stars: ✭ 380 (-95.26%)
Mutual labels:  learning, deeplearning, machinelearning, deep
Coursera Natural Language Processing Specialization
Programming assignments from all courses in the Coursera Natural Language Processing Specialization offered by deeplearning.ai.
Stars: ✭ 39 (-99.51%)
Mutual labels:  natural-language-processing, deeplearning, natural-language-understanding, natural-language
Ai Series
📚 [.md & .ipynb] Series of Artificial Intelligence & Deep Learning, including Mathematics Fundamentals, Python Practices, NLP Application, etc. 💫 人工智能与深度学习实战,数理统计篇 | 机器学习篇 | 深度学习篇 | 自然语言处理篇 | 工具实践 Scikit & Tensoflow & PyTorch 篇 | 行业应用 & 课程笔记
Stars: ✭ 702 (-91.24%)
Mutual labels:  natural-language-processing, deeplearning, machinelearning, datascience
Makine Ogrenmesi
Makine Öğrenmesi Türkçe Kaynak
Stars: ✭ 82 (-98.98%)
Mutual labels:  learning, machinelearning, machine, ml
Action Recognition Visual Attention
Action recognition using soft attention based deep recurrent neural networks
Stars: ✭ 350 (-95.63%)
Mutual labels:  deep-neural-networks, deeplearning, deep
Nlp Conference Compendium
Compendium of the resources available from top NLP conferences.
Stars: ✭ 349 (-95.65%)
Mutual labels:  natural-language-processing, natural-language-understanding, natural-language-generation
Botlibre
An open platform for artificial intelligence, chat bots, virtual agents, social media automation, and live chat automation.
Stars: ✭ 412 (-94.86%)
Mutual labels:  natural-language-processing, natural-language-understanding, natural-language
Ffdl
Fabric for Deep Learning (FfDL, pronounced fiddle) is a Deep Learning Platform offering TensorFlow, Caffe, PyTorch etc. as a Service on Kubernetes
Stars: ✭ 640 (-92.02%)
Mutual labels:  deep-neural-networks, deeplearning, ml
gan deeplearning4j
Automatic feature engineering using Generative Adversarial Networks using Deeplearning4j and Apache Spark.
Stars: ✭ 19 (-99.76%)
Mutual labels:  datascience, machinelearning, deeplearning
Amazon Sagemaker Examples
Example 📓 Jupyter notebooks that demonstrate how to build, train, and deploy machine learning models using 🧠 Amazon SageMaker.
Stars: ✭ 6,346 (-20.85%)
Mutual labels:  learning, machine, deep
Pytorch Forecasting
Time series forecasting with PyTorch
Stars: ✭ 849 (-89.41%)
Mutual labels:  learning, machine, deep
Oie Resources
A curated list of Open Information Extraction (OIE) resources: papers, code, data, etc.
Stars: ✭ 283 (-96.47%)
Mutual labels:  natural-language-processing, natural-language-understanding, datascience
Data-Scientist-In-Python
This repository contains notes and projects of Data scientist track from dataquest course work.
Stars: ✭ 23 (-99.71%)
Mutual labels:  datascience, machinelearning, deeplearning
dst
yet another custom data science template via cookiecutter
Stars: ✭ 59 (-99.26%)
Mutual labels:  datascience, machinelearning, deeplearning
Speech Emotion Analyzer
The neural network model is capable of detecting five different male/female emotions from audio speeches. (Deep Learning, NLP, Python)
Stars: ✭ 633 (-92.11%)
Mutual labels:  natural-language-processing, deep-neural-networks, natural-language-understanding
Nlp Recipes
Natural Language Processing Best Practices & Examples
Stars: ✭ 5,783 (-27.87%)
Mutual labels:  natural-language-processing, natural-language-understanding, natural-language
Deepj
A deep learning model for style-specific music generation.
Stars: ✭ 681 (-91.51%)
Mutual labels:  learning, machine, deep
Opencog
A framework for integrated Artificial Intelligence & Artificial General Intelligence (AGI)
Stars: ✭ 2,132 (-73.41%)
Mutual labels:  learning, natural-language-understanding, natural-language
awesome-conformal-prediction
A professionally curated list of awesome Conformal Prediction videos, tutorials, books, papers, PhD and MSc theses, articles and open-source libraries.
Stars: ✭ 998 (-87.55%)
Mutual labels:  datascience, machinelearning, deeplearning

Ludwig logo

PyPI version Build Status CII Best Practices Slack

DockerHub Downloads License Twitter

Translated in 🇰🇷 Korean/

Ludwig is a data-centric deep learning framework that allows users to train and test deep learning models by specifying a declarative configuration tht matches the schema of the data. It is built on top of PyTorch.

To train a model all you need to provide is a file containing your data, a list of columns to use as inputs, and a list of columns to use as outputs, Ludwig will do the rest. Simple commands can be used to train models both locally and in a distributed way, and to use them to predict new data.

A programmatic API is also available to use Ludwig from Python. A suite of visualization tools allows you to analyze models' training and test performance and to compare them.

Ludwig is built with extensibility principles in mind and is based on datatype abstractions, making it easy to add support for new datatypes as well as new model architectures.

It can be used by practitioners to quickly train and test deep learning models as well as by researchers to obtain strong baselines to compare against and have an experimentation setting that ensures comparability by performing the same data processing and evaluation.

Ludwig provides a set of model architectures that can be combined together to create an end-to-end model for a given use case. As an analogy, if deep learning libraries provide the building blocks to make your building, Ludwig provides the buildings to make your city, and you can choose among the available buildings or add your own building to the set of available ones.

The core design principles baked into the toolbox are:

  • No coding required: no coding skills are required to train a model and use it for obtaining predictions.
  • Generality: a new datatype-based approach to deep learning model design makes the tool usable across many different use cases.
  • Flexibility: experienced users have extensive control over model building and training, while newcomers will find it easy to use.
  • Extensibility: easy to add new model architecture and new feature datatypes.
  • Understandability: deep learning model internals are often considered black boxes, but Ludwig provides standard visualizations to understand their performance and compare their predictions.
  • Open Source: Apache License 2.0

LF AI & Data logo

Ludwig is hosted by the Linux Foundation as part of the LF AI & Data Foundation. For details about who's involved and how Ludwig fits into the larger open source AI landscape, read the Linux Foundation announcement.

Installation

Ludwig requires you to use Python 3.6+. If you don’t have Python 3 installed, install it by running:

sudo apt install python3  # on ubuntu
brew install python3      # on mac

You may want to use a virtual environment to maintain an isolated Python environment.

virtualenv -p python3 venv && source venv/bin/activate

In order to install Ludwig just run:

pip install ludwig

This will install only Ludwig's basic requirements, different feature types require different dependencies. We divided them as different extras so that users could install only the ones they actually need:

  • ludwig[text] for text dependencies.
  • ludwig[audio] for audio and speech dependencies.
  • ludwig[image] for image dependencies.
  • ludwig[hyperopt] for hyperparameter optimization dependencies.
  • ludwig[horovod] for distributed training dependencies.
  • ludwig[serve] for serving dependencies.
  • ludwig[viz] for visualization dependencies.
  • ludwig[test] for dependencies needed for testing.

Distributed training is supported with Horovod, which can be installed with pip install ludwig[horovod] or HOROVOD_GPU_OPERATIONS=NCCL pip install ludwig[horovod] for GPU support. See Horovod's installation guide for full details on available installation options.

Any combination of extra packages can be installed at the same time with pip install ludwig[extra1,extra2,...] like for instance pip install ludwig[text,viz]. The full set of dependencies can be installed with pip install ludwig[full].

For developers who wish to build the source code from the repository:

git clone [email protected]:ludwig-ai/ludwig.git
cd ludwig
virtualenv -p python3 venv
source venv/bin/activate
pip install -e '.[test]'

Basic Principles

Ludwig provides three main functionalities: training models and using them to predict and evaluate them. It is based on datatype abstraction, so that the same data preprocessing and postprocessing will be performed on different datasets that share datatypes and the same encoding and decoding models developed can be re-used across several tasks.

Training a model in Ludwig is pretty straightforward: you provide a dataset file and a config YAML file.

The config contains a list of input features and output features, all you have to do is specify names of the columns in the dataset that are inputs to your model alongside with their datatypes, and names of columns in the dataset that will be outputs, the target variables which the model will learn to predict. Ludwig will compose a deep learning model accordingly and train it for you.

Currently, the available datatypes in Ludwig are:

  • binary
  • numerical
  • category
  • set
  • bag
  • sequence
  • text
  • timeseries
  • image
  • audio
  • date
  • h3
  • vector

By choosing different datatype for inputs and outputs, users can solve many different tasks, for instance:

  • text input + category output = text classifier
  • image input + category output = image classifier
  • image input + text output = image captioning
  • audio input + binary output = speaker verification
  • text input + sequence output = named entity recognition / summarization
  • category, numerical and binary inputs + numerical output = regression
  • timeseries input + numerical output = forecasting model
  • category, numerical and binary inputs + binary output = fraud detection

take a look at the Examples to see how you can use Ludwig for several more tasks.

The config can contain additional information, in particular how to preprocess each column in the data, which encoder and decoder to use for each one, architectural and training parameters, hyperparameters to optimize. This allows ease of use for novices and flexibility for experts.

Training

For example, given a text classification dataset like the following:

doc_text class
Former president Barack Obama ... politics
Juventus hired Cristiano Ronaldo ... sport
LeBron James joins the Lakers ... sport
... ...

you want to learn a model that uses the content of the doc_text column as input to predict the values in the class column. You can use the following config:

{input_features: [{name: doc_text, type: text}], output_features: [{name: class, type: category}]}

and start the training typing the following command in your console:

ludwig train --dataset path/to/file.csv --config "{input_features: [{name: doc_text, type: text}], output_features: [{name: class, type: category}]}"

where path/to/file.csv is the path to a UTF-8 encoded CSV file containing the dataset in the previous table (many other data formats are supported). Ludwig will:

  1. Perform a random split of the data.
  2. Preprocess the dataset.
  3. Build a ParallelCNN model (the default for text features) that decodes output classes through a softmax classifier.
  4. Train the model on the training set until the performance on the validation set stops improving.

Training progress will be displayed in the console, but the TensorBoard can also be used.

If you prefer to use an RNN encoder and increase the number of epochs to train for, all you have to do is to change the config to:

{input_features: [{name: doc_text, type: text, encoder: rnn}], output_features: [{name: class, type: category}], training: {epochs: 50}}

Refer to the User Guide to find out all the options available to you in the config and take a look at the Examples to see how you can use Ludwig for several different tasks.

After training, Ludwig will create a results directory containing the trained model with its hyperparameters and summary statistics of the training process. You can visualize them using one of the several visualization options available in the visualize tool, for instance:

ludwig visualize --visualization learning_curves --training_statistics path/to/training_statistics.json

This command will display a graph like the following, where you can see loss and accuracy during the training process:

Learning Curves

Several more visualizations are available, please refer to Visualizations for more details.

Distributed Training

You can distribute the training of your models using Horovod, which allows training on a single machine with multiple GPUs as well as on multiple machines with multiple GPUs. Refer to the User Guide for full details.

Prediction and Evaluation

If you want your previously trained model to predict target output values on new data, you can type the following command in your console:

ludwig predict --dataset path/to/data.csv --model_path /path/to/model

Running this command will return model predictions.

If your dataset also contains ground truth values of the target outputs, you can compare them to the predictions obtained from the model to evaluate the model performance.

ludwig evaluate --dataset path/to/data.csv --model_path /path/to/model

This will produce evaluation performance statistics that can be visualized by the visualize tool, which can also be used to compare performances and predictions of different models, for instance:

ludwig visualize --visualization compare_performance --test_statistics path/to/test_statistics_model_1.json path/to/test_statistics_model_2.json

will return a bar plot comparing the models on different metrics:

Performance Comparison

A handy ludwig experiment command that performs training and prediction one after the other is also available.

Programmatic API

Ludwig also provides a simple programmatic API that allows you to train or load a model and use it to obtain predictions on new data:

from ludwig.api import LudwigModel

# train a model
config = {...}
model = LudwigModel(config)
train_stats = model.train(training_data)

# or load a model
model = LudwigModel.load(model_path)

# obtain predictions
predictions = model.predict(test_data)

config containing the same information of the YAML file provided to the command line interface. More details are provided in the User Guide and in the API documentation.

Extensibility

Ludwig is built from the ground up with extensibility in mind. It is easy to add an additional datatype that is not currently supported by adding a datatype-specific implementation of abstract classes that contain functions to preprocess the data, encode it, and decode it.

Furthermore, new models, with their own specific hyperparameters, can be easily added by implementing a class that accepts tensors (of a specific rank, depending on the datatype) as inputs and provides tensors as output. This encourages reuse and sharing new models with the community. Refer to the Developer Guide for further details.

Full documentation

You can find the full documentation here.

License

FOSSA Status

Getting Involved

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].