Metroid

Subscribe, act, publish.

Metroid - Metro for Django

This app is intended to streamline integration with Metro for all Django+Celery users by:

• Asynchronous handling of subscriptions and messages with one command
• Execute Celery tasks based on message topics, defined in settings.py
• Retry failed tasks through your admin dashboard when using the MetroidTask base

Overview

• python >= 3.8
• django >= 3.1.1 - For asgiref, settings
• django-guid >= 3.2.0 - Storing correlation IDs for failed tasks in the database, making debugging easy
• Choose one:
  • celery >= 5.0.0 - Execute tasks based on a subject
  • rq >= 2.4.1 - Execute tasks based on a subject

Implementation

The python manage.py metroid app is fully asynchronous, and has no blocking code. It utilizes Celery to execute tasks.

It works by:

1. Going through all your configured subscriptions and start a new async connection for each one of them
2. Metro sends messages on the subscriptions
3. This app filters out messages matching subjects you have defined, and queues a celery task to execute the function as specified for that subject
   3.1. If no task is found for that subject, the message is marked as complete
4. The message is marked as complete after the Celery task has successfully been queued
5. If the task is failed, an entry is automatically created in your database
6. All failed tasks can be retried manually through the admin dashboard

Configure and install this package

Note For a complete example, have a look in demoproj/settings.py.

1. Create a METROID key in settings.py with all your subscriptions and handlers. Example settings:

METROID = {
    'subscriptions': [
        {
            'topic_name': 'metro-demo',
            'subscription_name': 'sub-metrodemo-metrodemoerfett',
            'connection_string': config('CONNECTION_STRING_METRO_DEMO', None),
            'handlers': [
               {
                  'subject': 'MetroDemo/Type/GeekJokes',
                  'regex': False,
                  'handler_function': 'demoproj.demoapp.services.my_func'
                }
            ],
        },
    ],
   'worker_type': 'celery', # default
}

The handler_function is defined by providing the full dotted path as a string. For example,from demoproj.demoapp.services import my_func is provided as 'demoproj.demoapp.services.my_func'.

The handlers subject can be a regular expression or a string. If a regular expression is provided, the variable regex must be set to True. Example:

'handlers': [{'subject': r'^MetroDemo/Type/.*$','regex':True,'handler_function': my_func}],

2. Configure Django-GUID by adding the app to your installed apps, to your middlewares and configuring logging as described here. Make sure you enable the CeleryIntegration:

from django_guid.integrations import CeleryIntegration

DJANGO_GUID = {
    'INTEGRATIONS': [
        CeleryIntegration(
            use_django_logging=True,
            log_parent=True,
        )
    ],
}

Creating your own handler functions

Your functions will be called with keyword arguments for

message, topic_name, subscription_name and subject. You function should in other words look something like this:

Celery

@app.task(base=MetroidTask)
def my_func(*, message: dict, topic_name: str, subscription_name: str, subject: str) -> None:

rq

def my_func(*, message: dict, topic_name: str, subscription_name: str, subject: str) -> None:

Running the project

1. Ensure you have redis running:

docker-compose up

2. Run migrations

python manage.py migrate

3. Create an admin account

python manage.py createsuperuser

4. Start a worker:

celery -A demoproj worker -l info

5. Run the subscriber:

python manage.py metroid

6. Send messages to Metro. Example code can be found in demoproj/demoapp/services.py
7. Run the webserver:

python manage.py runserver 8000

8. See failed messages under http://localhost:8080/admin

To contribute, please see CONTRIBUTING.md