All Projects → aio-libs → Aiokafka

aio-libs / Aiokafka

Licence: apache-2.0
asyncio client for kafka

Programming Languages

python
139335 projects - #7 most used programming language

Projects that are alternatives of or similar to Aiokafka

Strimzi Kafka Bridge
Apache Kafka bridge
Stars: ✭ 137 (-74.82%)
Mutual labels:  kafka, kafka-client
Kq
Kafka-based Job Queue for Python
Stars: ✭ 530 (-2.57%)
Mutual labels:  kafka, kafka-client
Kafkajs
A modern Apache Kafka client for node.js
Stars: ✭ 2,315 (+325.55%)
Mutual labels:  kafka, kafka-client
Alpakka Kafka
Alpakka Kafka connector - Alpakka is a Reactive Enterprise Integration library for Java and Scala, based on Reactive Streams and Akka.
Stars: ✭ 1,295 (+138.05%)
Mutual labels:  kafka, kafka-client
Faust
Python Stream Processing
Stars: ✭ 5,899 (+984.38%)
Mutual labels:  asyncio, kafka
Java Kafka Client
OpenTracing Instrumentation for Apache Kafka Client
Stars: ✭ 101 (-81.43%)
Mutual labels:  kafka, kafka-client
Phobos
Simplifying Kafka for ruby apps
Stars: ✭ 176 (-67.65%)
Mutual labels:  kafka, kafka-client
Ruby Kafka
A Ruby client library for Apache Kafka
Stars: ✭ 1,039 (+90.99%)
Mutual labels:  kafka, kafka-client
Kafka Ui
Open-Source Web GUI for Apache Kafka Management
Stars: ✭ 230 (-57.72%)
Mutual labels:  kafka, kafka-client
Ksql Udf Deep Learning Mqtt Iot
Deep Learning UDF for KSQL for Streaming Anomaly Detection of MQTT IoT Sensor Data
Stars: ✭ 219 (-59.74%)
Mutual labels:  kafka, kafka-client
Karafka
Framework for Apache Kafka based Ruby and Rails applications development.
Stars: ✭ 1,223 (+124.82%)
Mutual labels:  kafka, kafka-client
Node Sinek
🎩 Most advanced high level Node.js Kafka client
Stars: ✭ 262 (-51.84%)
Mutual labels:  kafka, kafka-client
Fs2 Kafka
Kafka client for functional streams for scala (fs2)
Stars: ✭ 75 (-86.21%)
Mutual labels:  kafka, kafka-client
Php Rdkafka
Production-ready, stable Kafka client for PHP
Stars: ✭ 1,703 (+213.05%)
Mutual labels:  kafka, kafka-client
Rafka
Kafka proxy with a simple API, speaking the Redis protocol
Stars: ✭ 49 (-90.99%)
Mutual labels:  kafka, kafka-client
Confluent Kafka Dotnet
Confluent's Apache Kafka .NET client
Stars: ✭ 2,110 (+287.87%)
Mutual labels:  kafka, kafka-client
Kafka Streams Machine Learning Examples
This project contains examples which demonstrate how to deploy analytic models to mission-critical, scalable production environments leveraging Apache Kafka and its Streams API. Models are built with Python, H2O, TensorFlow, Keras, DeepLearning4 and other technologies.
Stars: ✭ 661 (+21.51%)
Mutual labels:  kafka, kafka-client
Sarama
Sarama is a Go library for Apache Kafka 0.8, and up.
Stars: ✭ 7,964 (+1363.97%)
Mutual labels:  kafka, kafka-client
Franz Go
franz-go contains a high performance, pure Go library for interacting with Kafka from 0.8.0 through 2.7.0+. Producing, consuming, transacting, administrating, etc.
Stars: ✭ 199 (-63.42%)
Mutual labels:  kafka, kafka-client
Faust
Python Stream Processing. A Faust fork
Stars: ✭ 124 (-77.21%)
Mutual labels:  asyncio, kafka

aiokafka

.. image:: https://travis-ci.com/aio-libs/aiokafka.svg?branch=master :target: https://travis-ci.com/aio-libs/aiokafka :alt: |Build status| .. image:: https://codecov.io/github/aio-libs/aiokafka/coverage.svg?branch=master :target: https://codecov.io/gh/aio-libs/aiokafka/branch/master :alt: |Coverage| .. image:: https://badges.gitter.im/Join%20Chat.svg :target: https://gitter.im/aio-libs/Lobby :alt: |Chat on Gitter|

asyncio client for Kafka

AIOKafkaProducer


AIOKafkaProducer is a high-level, asynchronous message producer.

Example of AIOKafkaProducer usage:

.. code-block:: python

from aiokafka import AIOKafkaProducer
import asyncio

async def send_one():
    producer = AIOKafkaProducer(bootstrap_servers='localhost:9092')
    # Get cluster layout and initial topic/partition leadership information
    await producer.start()
    try:
        # Produce message
        await producer.send_and_wait("my_topic", b"Super message")
    finally:
        # Wait for all pending messages to be delivered or expire.
        await producer.stop()

asyncio.run(send_one())

AIOKafkaConsumer


AIOKafkaConsumer is a high-level, asynchronous message consumer. It interacts with the assigned Kafka Group Coordinator node to allow multiple consumers to load balance consumption of topics (requires kafka >= 0.9.0.0).

Example of AIOKafkaConsumer usage:

.. code-block:: python

from aiokafka import AIOKafkaConsumer
import asyncio

async def consume():
    consumer = AIOKafkaConsumer(
        'my_topic', 'my_other_topic',
        bootstrap_servers='localhost:9092',
        group_id="my-group")
    # Get cluster layout and join group `my-group`
    await consumer.start()
    try:
        # Consume messages
        async for msg in consumer:
            print("consumed: ", msg.topic, msg.partition, msg.offset,
                  msg.key, msg.value, msg.timestamp)
    finally:
        # Will leave consumer group; perform autocommit if enabled.
        await consumer.stop()

asyncio.run(consume())

Running tests

Docker is required to run tests. See https://docs.docker.com/engine/installation for installation notes. Also note, that lz4 compression libraries for python will require python-dev package, or python source header files for compilation on Linux. NOTE: You will also need a valid java installation. It's required for the keytool utility, used to generate ssh keys for some tests.

Setting up tests requirements (assuming you're within virtualenv on ubuntu 14.04+)::

sudo apt-get install -y libsnappy-dev
make setup

Running tests with coverage::

make cov

To run tests with a specific version of Kafka (default one is 1.0.2) use KAFKA_VERSION variable::

make cov KAFKA_VERSION=0.10.2.1

Test running cheatsheat:

  • make test FLAGS="-l -x --ff" - run until 1 failure, rerun failed tests fitst. Great for cleaning up a lot of errors, say after a big refactor.
  • make test FLAGS="-k consumer" - run only the consumer tests.
  • make test FLAGS="-m 'not ssl'" - run tests excluding ssl.
  • make test FLAGS="--no-pull" - do not try to pull new docker image before test run.
Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].