All Projects → schubergphilis → data-migrator

schubergphilis / data-migrator

Licence: MIT license
A declarative data-migration package

Programming Languages

python
139335 projects - #7 most used programming language
Makefile
30231 projects

Projects that are alternatives of or similar to data-migrator

kodex
A privacy and security engineering toolkit: Discover, understand, pseudonymize, anonymize, encrypt and securely share sensitive and personal data: Privacy and security as code.
Stars: ✭ 70 (+366.67%)
Mutual labels:  gdpr, anonymization
pgantomizer
Anonymize data in your PostgreSQL dabatase with ease
Stars: ✭ 95 (+533.33%)
Mutual labels:  gdpr, anonymization
pganonymize
A commandline tool for anonymizing PostgreSQL databases
Stars: ✭ 20 (+33.33%)
Mutual labels:  gdpr, anonymization
pynonymizer
A universal tool for translating sensitive production database dumps into anonymized copies.
Stars: ✭ 58 (+286.67%)
Mutual labels:  gdpr, anonymization
database-anonymizer
CLI tool an PHP library to anonymize data in various databases
Stars: ✭ 23 (+53.33%)
Mutual labels:  gdpr, anonymization
Rekord
A javascript REST ORM that is offline and real-time capable
Stars: ✭ 171 (+1040%)
Mutual labels:  migration
Fluentmigrator
Fluent migrations framework for .NET
Stars: ✭ 2,636 (+17473.33%)
Mutual labels:  migration
Imapsync
Imapsync is an IMAP transfers tool. The purpose of imapsync is to migrate IMAP accounts or to backup IMAP accounts. IMAP is one of the three current standard protocols to access mailboxes, the two others are POP3 and HTTP with webmails, webmails are often tied to an IMAP server. Upstream website is
Stars: ✭ 2,201 (+14573.33%)
Mutual labels:  migration
Mongolastic
🚥 A dataset migration tool from MongoDB to Elasticsearch and vice versa.
Stars: ✭ 131 (+773.33%)
Mutual labels:  migration
monolog-gdpr
Some Monolog processors that help with GDPR compliance
Stars: ✭ 49 (+226.67%)
Mutual labels:  gdpr
hugo-component-matomo
Matomo user tracking and optout scripts for Hugo
Stars: ✭ 38 (+153.33%)
Mutual labels:  gdpr
Migrate
Database migrations. CLI and Golang library.
Stars: ✭ 2,315 (+15333.33%)
Mutual labels:  migration
Lunatech Scala 2 To Scala3 Course
Lunatech course - "Moving forward from Scala 2 to Scala 3"
Stars: ✭ 174 (+1060%)
Mutual labels:  migration
Scala 3 Migration Guide
The Scala 3 migration guide for everyone.
Stars: ✭ 220 (+1366.67%)
Mutual labels:  migration
Undermoon
Mordern Redis Cluster solution for easy operation.
Stars: ✭ 166 (+1006.67%)
Mutual labels:  migration
CosmicClone
Cosmic Clone is a utility that can backup\clone\restore a azure Cosmos database Collection. It can also anonymize cosmos documents and helps hide personally identifiable data.
Stars: ✭ 113 (+653.33%)
Mutual labels:  anonymization
C2rust
Migrate C code to Rust
Stars: ✭ 2,111 (+13973.33%)
Mutual labels:  migration
Ibm Z Zos
The helpful and handy location for finding and sharing z/OS files, which are not included in the product.
Stars: ✭ 198 (+1220%)
Mutual labels:  migration
frizzle
The magic message bus
Stars: ✭ 14 (-6.67%)
Mutual labels:  kinesis
Migration
《系统重构与迁移指南》手把手教你分析、评估现有系统、制定重构策略、探索可行重构方案、搭建测试防护网、进行系统架构重构、服务架构重构、模块重构、代码重构、数据库重构、重构后的架构守护
Stars: ✭ 2,753 (+18253.33%)
Mutual labels:  migration

Data-migrator (version 0.6.3.dev2) is a simple data-migration package for python lovers.

https://circleci.com/gh/schubergphilis/data-migrator.svg?style=shield
Updates Documentation Status https://api.codacy.com/project/badge/Grade/bf6030e9e7e248979607802880336611 https://api.codacy.com/project/badge/Coverage/bf6030e9e7e248979607802880336611

Data-migrator is a declarative DSL for table driven data transformations, set up as an open and extensive system. Use this to create data transformations for changing databases as a result of changing code, initial loads to datalakes (it contains a Kinesis provider)and more.

Example

Data-migrator assumes data is extracted and loaded with client access.

$ mysql source_db -E 'select id,a,b from table' -B  | python my_filter.py | mysql target_db

It than offers a wide range of primitives with default settings to build complex transformations fast, readable and extendable

from data_migrator import models, transform

class Result(models.Model):
  id   = models.IntField(pos=0) # keep id
  uuid = models.UUIDField()     # generate new uuid4 field
  a    = models.StringField(pos=1, default='NO_NULL', max_length=5, nullable='NULL', replacement=lambda x:x.upper())
  b    = models.StringField(pos=2, name='my_b')

if __name__ == "__main__":
  transform.Transformer(models=[Result]).process()

Installation

Execute the following command to install data-migrator with pip:

pip install data-migrator

See the Installation Instructions in Documentation for more instructions on installing, upgrading, and uninstalling data-migrator.

The project is maintained at GitHub.

Support and contribute

Questions, comments, bug reports and especially tested patches may be submitted directly to the issue tracker.

Everyone interacting with this codebase, issue trackers, chat rooms, and mailing lists is expected to follow the Code of Conduct.

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].