Overview
Versatile Data Kit (VDK) is an open source framework that enables anybody with basic SQL or Python knowledge to create their own data pipelines.
Versatile Data Kit enables Data Engineers to develop, deploy, run and manage Data Jobs. A Data Job is a data processing workload and can be written in Python, SQL, or both at the same time. A Data Job enables Data Engineers to implement automated pull ingestion (E in ELT) and batch data transformation (T in ELT) into a database or any type of data storage.
Versatile Data Kit consists of two main components:
- A Data SDK provides all tools for the automation of data extraction, transformation, and loading, as well as a plugin framework that allows users to extend the framework according to their specific requirements.
- A Control Service allows users to create, deploy, manage and execute Data Jobs in Kubernetes runtime environment.
To help solve common data engineering problems Versatile Data Kit:
- allows ingestion of data from different sources, including CSV files, JSON objects, data provided by REST API services, etc.;
- ensures data applications are packaged, versioned, and deployed correctly while dealing with credentials, retries, reconnects, etc.;
- provides built-in monitoring and smart notification capabilities;
- tracks both code and data modifications and the relations between them, enabling engineers to troubleshoot faster and providing an easy revert to a stable version.
Data Journey and where VDK fits in
Installation and Getting Started
Install Versatile Data Kit SDK
pip install -U pip setuptools wheel
pip install quickstart-vdk
Note that Versatile Data Kit requires Python 3.7+.
See the Installation page for more details.
Use
# see Help to see what you can do
vdk --help
Check out the Getting Started page to create and run your first Data Job.
Documentation
Official documentation for Versatile Data Kit can be found here.
Contributing
If you are interested in contributing as a developer, visit CONTRIBUTING.md.
Contacts
Feedback is very welcome via the GitHub site as issues or pull requests
- Join our mailing list
- Follow us on twitter.
- Subscribe to the Versatile Data Kit YouTube Channel.
- Join our dedicated Slack channel on the CNCF Slack workspace - simply search for #versatile-data-kit or click here to join the channel
How to use Versatile Data Kit?
- Video Data Ingestion with Versatile Data Kit
- Video Data Transformation with Versatile Data Kit
- Blog Post A complete example using the Versatile Data Kit and Trino DB
For the full list of resources go to Community and Resources
Code of Conduct
Everyone involved in working on the project's source code, or engaging in any issue trackers, Slack channels and mailing lists is expected to follow the Code of Conduct.