Kafka Toolkit
Welcome to the Kafka Toolkit. This toolkit enables SPL applications to integrate with Apache Kafka.
This version of the toolkit currently supports: Apache Kafka 0.10.2, 0.11, 1.0, 1.1, and 2.0 to 2.7. The older Kafka versions 0.9 and 0.10.0, are not supported any more. Using the Kafka operators from the com.ibm.streamsx.messaging toolkit for these server versions is not recommended.
Event Streams cloud service (Message Hub)
For building applications that connect to the IBM Cloud Event Streams service, it is recommended that the com.ibm.streamsx.messagehub toolkit be used. This toolkit provides functionality on top of the Kafka toolkit to simplify the configuration needed to connect to the public cloud service.
Migrating from com.ibm.streamsx.messaging
To migrate applications using the old Kafka operators in the com.ibm.streamsx.messaging toolkit, refer to the Migration Document.
Documentation
Documentation on GitHub
- Toolkit documentation can be found here: SPLDoc
- Standard use cases for Consumers and Partitions can be found on the overview page of the user documentation
- Toolkit reference for latest and older versions
Build the toolkit
cd com.ibm.streamsx.kafka
../gradlew build
Build SPLDoc
./gradlew spldoc
NOTE: SPLDocs will be generated in the docs/spldoc directory.
Release
./gradlew release
NOTE: The release will be available in the build/release/output folder.
Test
cd tests/KafkaTests
./setup.sh -b <list_of_bootstrap_servers>
../../gradlew test
NOTE 1: setup.sh will add a domain-level app config called "kafka-tests", as well as create a properties file containing the bootstrap.servers property.
NOTE 2: Tests will run using the local domain specified by the STREAMS_DOMAIN_ID env var. All tests run in Distributed mode.
Following topics are required for the tests
| topic name | number of partitions |
|---|---|
| test | 3 |
| other1 | 1 |
| other2 | 1 |
| position | 1 |
Create them by hand before or make sure that they are automatically created with the required number of partitions.
Hint: The test case kafkaStartPositionTest requires that the position topic is re-created before every test run.
Samples
Each sample contains a build.gradle file and a Makefile. The samples can be built/compiled by running ../../gradlew build or make from the sample directory.