joowani / Kq
Licence: mit
Kafka-based Job Queue for Python
Stars: ✭ 530
Programming Languages
Projects that are alternatives of or similar to Kq
Karafka
Framework for Apache Kafka based Ruby and Rails applications development.
Stars: ✭ 1,223 (+130.75%)
Mutual labels: apache-kafka, kafka, kafka-client, kafka-producer, kafka-consumer
Apachekafkatutorials
Example Code for Kafka Tutorials @ Learning Journal
Stars: ✭ 155 (-70.75%)
Mutual labels: apache-kafka, kafka-client, kafka-producer, kafka-consumer
Strimzi Kafka Bridge
Apache Kafka bridge
Stars: ✭ 137 (-74.15%)
Mutual labels: kafka, kafka-client, kafka-producer, kafka-consumer
Rafka
Kafka proxy with a simple API, speaking the Redis protocol
Stars: ✭ 49 (-90.75%)
Mutual labels: kafka, kafka-client, kafka-producer, kafka-consumer
Librdkafka
The Apache Kafka C/C++ library
Stars: ✭ 5,617 (+959.81%)
Mutual labels: apache-kafka, kafka, kafka-producer, kafka-consumer
Kattlo Cli
Kattlo CLI Project
Stars: ✭ 58 (-89.06%)
Mutual labels: apache-kafka, kafka, kafka-producer, kafka-consumer
Kukulcan
A REPL for Apache Kafka
Stars: ✭ 103 (-80.57%)
Mutual labels: apache-kafka, kafka, kafka-producer, kafka-consumer
Kafka Ui
Open-Source Web GUI for Apache Kafka Management
Stars: ✭ 230 (-56.6%)
Mutual labels: apache-kafka, kafka, kafka-client, kafka-producer
kafka-0.11-examples
Code snippets that demonstrate how to leverage the new Kafka 0.11 APIs
Stars: ✭ 17 (-96.79%)
Mutual labels: kafka-consumer, kafka-producer, kafka-client
Debezium
Change data capture for a variety of databases. Please log issues at https://issues.redhat.com/browse/DBZ.
Stars: ✭ 5,937 (+1020.19%)
Mutual labels: apache-kafka, kafka, kafka-producer
Trubka
A CLI tool for Kafka
Stars: ✭ 296 (-44.15%)
Mutual labels: kafka, kafka-producer, kafka-consumer
Qbusbridge
The Apache Kafka Client SDK
Stars: ✭ 272 (-48.68%)
Mutual labels: kafka, kafka-producer, kafka-consumer
kafka-net-core
kafka .net core library
Stars: ✭ 23 (-95.66%)
Mutual labels: apache-kafka, kafka-client
Kafka-quickstart
Kafka Examples focusing on Producer, Consumer, KStreams, KTable, Global KTable using Spring, Kafka Cluster Setup & Monitoring. Implementing Event Sourcing and CQRS Design Pattern using Kafka
Stars: ✭ 31 (-94.15%)
Mutual labels: kafka-consumer, kafka-producer
python-fake-data-producer-for-apache-kafka
Kafka Python Fake Data Producer is a complete demo app allowing you to quickly produce a Python fake Pizza-based streaming dataset and push it to an Apache Kafka topic. It gives an example on how easy is to create great fake streaming data to feed Apache Kafka.
Stars: ✭ 34 (-93.58%)
Mutual labels: apache-kafka, kafka-producer
qwery
A SQL-like language for performing ETL transformations.
Stars: ✭ 28 (-94.72%)
Mutual labels: kafka-consumer, kafka-producer
Node Sinek
🎩 Most advanced high level Node.js Kafka client
Stars: ✭ 262 (-50.57%)
Mutual labels: kafka, kafka-client
Insulator
A client UI to inspect Kafka topics, consume, produce and much more
Stars: ✭ 53 (-90%)
Mutual labels: kafka-consumer, apache-kafka
Kafka Webview
Full-featured web-based Apache Kafka consumer UI
Stars: ✭ 319 (-39.81%)
Mutual labels: kafka, kafka-consumer
KQ: Kafka Job Queue for Python
KQ (Kafka Queue) is a lightweight Python library which lets you enqueue and execute jobs asynchronously using Apache Kafka. It uses kafka-python under the hood.
Announcements
- Support for Python 3.5 will be dropped from KQ version 3.0.0.
- See releases for latest updates.
Requirements
- Apache Kafka 0.9+
- Python 3.6+
Installation
Install using pip:
pip install kq
Getting Started
Start your Kafka instance. Example using Docker:
docker run -p 9092:9092 -e ADV_HOST=127.0.0.1 lensesio/fast-data-dev
Define your KQ worker.py
module:
import logging
from kafka import KafkaConsumer
from kq import Worker
# Set up logging.
formatter = logging.Formatter("[%(levelname)s] %(message)s")
stream_handler = logging.StreamHandler()
stream_handler.setFormatter(formatter)
logger = logging.getLogger("kq.worker")
logger.setLevel(logging.DEBUG)
logger.addHandler(stream_handler)
# Set up a Kafka consumer.
consumer = KafkaConsumer(
bootstrap_servers="127.0.0.1:9092",
group_id="group",
auto_offset_reset="latest"
)
# Set up a worker.
worker = Worker(topic="topic", consumer=consumer)
worker.start()
Start your worker:
python my_worker.py
[INFO] Starting Worker(hosts=127.0.0.1:9092 topic=topic, group=group) ...
Enqueue a function call:
import requests
from kafka import KafkaProducer
from kq import Queue
# Set up a Kafka producer.
producer = KafkaProducer(bootstrap_servers="127.0.0.1:9092")
# Set up a queue.
queue = Queue(topic="topic", producer=producer)
# Enqueue a function call.
job = queue.enqueue(requests.get, "https://google.com")
# You can also specify the job timeout, Kafka message key and partition.
job = queue.using(timeout=5, key=b"foo", partition=0).enqueue(requests.get, "https://google.com")
The worker executes the job in the background:
python my_worker.py
[INFO] Starting Worker(hosts=127.0.0.1:9092, topic=topic, group=group) ...
[INFO] Processing Message(topic=topic, partition=0, offset=0) ...
[INFO] Executing job c7bf2359: requests.api.get("https://www.google.com")
[INFO] Job c7bf2359 returned: <Response [200]>
See the documentation for more information.
Note that the project description data, including the texts, logos, images, and/or trademarks,
for each open source project belongs to its rightful owner.
If you wish to add or remove any projects, please contact us at [email protected].