Database Migration Tool
Don't use this tool in production databases!! At this time tool under development!!
🧐 About
The Database Migration Tool was designed for those looking to migrate their data from one database to another. Basically, the tool is focused on transferring data from different database structures. Currently, the MySQL, Mariadb and Postgres driver related tool allows us to add NoSQL to SQL databases and vice versa. Our main goal is to make data migration possible in all environments.
Documentation: Documentation
🏁 Getting Started
Installing
pip install madmigration🎈 Usage
After installation you should define YAML file where configuration will be taken in order to apply data to target database. Yaml file with list and dictionaries may contain following structures:
Connection Config
- SourceConfig is intented to be data from source database
- DestinationConfig is intented to be transfered data to target database
version: 0.1.6
Configs:
- SourceConfig:
dbURI: "postgres://root:[email protected]/oldDB"
- DestinationConfig:
dbURI: "mysql://root:[email protected]/newDB"
migrationTables:
- migrationTable:
SourceTable:
name: users
DestinationTable:
name: persons
create: True
MigrationColumns:
- sourceColumn:
name: id
destinationColumn:
name: id
options:
type_cast: bigint
primary_key: true
autoincrement: true
- sourceColumn:
name: name
destinationColumn:
name: fname
options:
type_cast: varchar
length: 32
- sourceColumn:
name: surname
destinationColumn:
name: lname
options:
type_cast: varchar
length: 32
index: true
- sourceColumn:
name: age
destinationColumn:
name: age
options:
type_cast: int
- sourceColumn:
name: createdAT
destinationColumn:
name: created_at
options:
type_cast: datetime
- sourceColumn:
name: updatedAT
destinationColumn:
name: updated_at
options:
type_cast: datetime
Configs section
SourceConfigset the source database database configurationsdbURIsource database URI
DestinationConfigset the destination database configurationsdbURIdestination database URI
Configs:
- SourceConfig:
dbURI: "postgres://root:[email protected]/oldDB" # set source database uri
- DestinationConfig:
dbURI: "mysql://root:[email protected]/newDB" # set destination database urimigrationTables section
migrationTablesin this configuration, you will write the source of the table that you have to migrate and the destination tables that will migrate the data.migrationTablespecify the source and destination table nameSourceTableinformation about source tablenamesource table name
DestinationTableinformation about destination tablenamedestination table namecreatebool value. This parameter tells the program whether it should create a table or not. (default false)
migrationTables:
- migrationTable:
SourceTable:
name: users
DestinationTable:
name: persons
create: TrueMigrationColumns section
MigrationColumnsspecify source and destination columnsourceColumninformation about source columnnamesource column name
destinationColumninformation about destination columnnamedestination column nameoptionscolumn optionstype_castdestination column type name varchar,integer etc. (when we convert data we use this parameter)
MigrationColumns:
- sourceColumn:
name: id
destinationColumn:
name: id
options:
type_cast: bigint
primary_key: true
autoincrement: trueIf you want to create a foreign key you can specify it in the column parameters
- sourceColumn:
name: USERID
destinationColumn:
name: user_id
options:
type_cast: uuid
foreign_key:
table_name: users
column_name: id
ondelete: CASCADEYou can split your .yaml files or import .json file into .yaml file.
You must create the main .yaml file and importing other files into main .yaml file.
main.yaml file
version: 1.1
Configs:
- SourceConfig:
dbURI: "mysql://root:[email protected]/old"
- DestinationConfig:
dbURI: "postgresql://root:[email protected]/new"
migrationTables:
- migrationTable: !import company.yaml
- migrationTable: !import op_cond.json
company.yaml file
SourceTable:
name: company
DestinationTable:
name: company
create: true
MigrationColumns:
- sourceColumn:
name: id
destinationColumn:
name: id
options:
primary_key: true
type_cast: uuid
- sourceColumn:
name: name
destinationColumn:
name: name
options:
length: 120
type_cast: varchar
nullable: false
- sourceColumn:
name: created
destinationColumn:
name: created
options:
type_cast: datetime
- sourceColumn:
name: updated
destinationColumn:
name: updated
options:
type_cast: datetime
nullable: trueop_conds.json file
{
"SourceTable": {
"name": "operation_conditions"
},
"DestinationTable": {
"name": "operation_conditions",
"create": true
},
"MigrationColumns": [
{
"sourceColumn": {
"name": "id"
},
"destinationColumn": {
"name": "id",
"options": {
"primary_key": true,
"type_cast": "uuid"
}
}
},
{
"sourceColumn": {
"name": "interest"
},
"destinationColumn": {
"name": "interest",
"options": {
"type_cast": "varchar",
"length": 30,
"nullable": false
}
}
},
{
"sourceColumn": {
"name": "FIFD"
},
"destinationColumn": {
"name": "FIFD",
"options": {
"type_cast": "varchar",
"length": 30,
"nullable": false
}
}
},
{
"sourceColumn": {
"name": "comission"
},
"destinationColumn": {
"name": "comission",
"options": {
"type_cast": "varchar",
"length": 30,
"nullable": false
}
}
}
]
}Currently new feature PostgreSql to MongoDB has been added.
version: 1.1
Configs:
- SourceConfig:
dbURI: "postgresql://sabuhi:sabuhi12345@localhost:5432/company_service"
- DestinationConfig:
dbURI: "mongodb://localhost:27017/mydb"
migrationTables:
- migrationTable:
SourceTable: #postgresql table name
name: company
DestinationTable: #the collection name:
name: Company
MigrationColumns:
- sourceColumn:
name: id
destinationColumn:
name: id
options:
type_cast: uuid
- sourceColumn:
name: name
destinationColumn:
name: NAME
options:
type_cast: varchar
- sourceColumn:
name: created
destinationColumn:
name: CREATED
options:
type_cast: datetime
- sourceColumn:
name: email
destinationColumn:
name: EMAIL
options:
type_cast: string
- sourceColumn:
name: updated
destinationColumn:
name: UPDATED
options:
type_cast: datetime
- sourceColumn:
name: code
destinationColumn:
name: CODE
options:
type_cast: stringWork on PostgreSQl to MongoDB still goes, we will add other futures as soon as possible.
We will create all tables and database on the destination server if they do not exist
madmigrate -f migration_schema.yaml
✍️ Authors
See also the list of contributors who participated in this project.
Contributing
We are open to new ideas, additions. If you have any we would be happy to recieve and diccuss.