Django Horizon
Purpose
Simple database sharding (horizontal partitioning) library for Django applications.
- Free software: MIT license
- Documentation: https://django-horizon.readthedocs.io.
- Inspired by django-sharding. Thank you so much for your cool solution :)
Features
- Shard (horizontal partitioning) by some ForeignKey field like user account.
Installation
To install Django Horizon, run this command in your terminal:
$ pip install django-horizonThis is the preferred method to install Django Horizon, as it will always install the most recent stable release.
If you don't have pip installed, this Python installation guide can guide you through the process.
Usage
Setup
Add database router configuration in your settings.py:
Horizontal database groups and a metadata store
HORIZONTAL_CONFIG = {
'GROUPS': {
'group1': { # The name of database horizontal partitioning group
'DATABASES': {
1: {
'write': 'member1-primary',
'read': ['member1-replica-1', 'member1-replica-2'], # Pick randomly by router
},
2: {
'write': 'member2-primary',
'read': ['member2-replica'],
},
3: {
'write': 'a3', # Used by 'read' too
},
},
'PICKABLES': [2, 3], # Group member keys to pick new database
},
},
'METADATA_MODEL': 'app.HorizontalMetadata', # Metadata store for horizontal partition key and there database
}Database router
DATABASE_ROUTERS = (
'horizon.routers.HorizontalRouter',
...
)Example models
Horizontal partitioning by user
Metadata store
from horizon.models import AbstractHorizontalMetadata
class HorizontalMetadata(AbstractHorizontalMetadata):
passIn the example, metadata store save followings.
group: Group name for horizontal partitioning.key: Determines the distribution of the table's records among the horizontal partitioning group.index: Choosed database index in horizontal partitioning groups.
Sharded model
from django.conf import settings
from horizon.manager import HorizontalManager # For Django<1.10
from horizon.models import AbstractHorizontalModel
class SomeLargeModel(AbstractHorizontalModel):
user = models.ForeignKey(
settings.AUTH_USER_MODEL,
on_delete=models.DO_NOTHING,
db_constraint=False, # May be using anothor database
)
...
objects = HorizontalManager() # For Django<1.10
class Meta(object):
horizontal_group = 'group1' # Group name
horizontal_key = 'user' # Field name to use group keyIn many cases use UUIDField field for id.
The AbstractHorizontalModel uses UUIDField as a them id field in default.
Using a model
from django.contrib.auth import get_user_model
user_model = get_user_model()
user = user_model.objects.get(pk=1)
# Get by foreign instance
SomeLargeModel.objects.filter(uses=user)
# Get by foreign id
SomeLargeModel.objects.filter(uses_id=user.id)Model limitations
django.db.utils.IntegrityErroroccured when not specify horizontal key field to filterSomeLargeModel.objects.all()
Cannot lookup by foreign key field, cause there are other (like
default) databaselist(user.somelargemodel_set.all())