vaquarkhan / Apache Kafka Poc And Notes

Installing Kafka and Zookeeper Using Containers

Installing a Kafka Cluster using containers is a quick way to get up and running. It's portable and lightweight, so we can use this on any machine running Docker. You'll see in this lesson, it takes much less time to get to the point where we can create our first topic. See the below commands for easily copying and pasting into your own terminal:

Add Docker to Your Package Repository

		curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

		sudo add-apt-repository    "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
		   $(lsb_release -cs) \
		   stable"

Update Packages and Install Docker

		sudo apt update

		sudo apt install -y docker-ce=18.06.1~ce~3-0~ubuntu

Add Your User to the Docker Group

  sudo usermod -a -G docker cloud_user

Install Docker Compose

		sudo -i

		curl -L https://github.com/docker/compose/releases/download/1.24.0/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose

		chmod +x /usr/local/bin/docker-compose

Clone the Repository That Has Our Docker Compose File

  git clone https://github.com/linuxacademy/content-kafka-deep-dive.git

Change Directory and Run the Compose YAML File

  cd content-kafka-deep-dive

  docker-compose up -d --build

Install Java

  sudo apt install -y default-jdk

Get the Kafka Binaries

		wget http://mirror.cogentco.com/pub/apache/kafka/2.2.0/kafka_2.12-2.2.0.tgz

		tar -xvf kafka_2.12-2.2.0.tgz

Create Your First Topic

    ./bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic test --partitions 3 --replication-factor 1

Describe the Topic

  ./bin/kafka-topics.sh --zookeeper localhost:2181 --topic test --describe

Now that we've setup our Kafka cluster, let's explore some of the various commands for creating topics, and producing and consuming messages. In this lesson, we'll go over how to determine what flag to use, as well as how to use a combination of flags. Overall, the command line is friendly, giving verbose explanation when someone does something wrong.

Detail for the topics command

    bin/kafka-topics.sh

Creating a topic will all the required arguments

    bin/kafka-topics.sh --zookeeper zookeeper1:2181/kafka --topic test1 --create --partitions 3 --replication-factor 3

Creating a topic including all of the zookeeper servers (not required)

    bin/kafka-topics.sh --zookeeper zookeeper1:2181,zookeeper2:2181,zookeeper3:2181/kafka --topic test1 --create --partitions 3 --replication-factor 3

List all topics

bin/kafka-topics.sh --zookeeper zookeeper1:2181/kafka --list

Describing a topic

 bin/kafka-topics.sh --zookeeper zookeeper1:2181/kafka --topic test2 --describe

Delete a topic

 bin/kafka-topics.sh --zookeeper zookeeper1:2181/kafka --topic test2 --delete

Detail for the producer command

   bin/kafka-console-producer.sh

Detail for the consumer command

   bin/kafka-console-consumer.sh

Detail for the consumer groups command

 bin/kafka-consumer-groups.sh

By using a Producer, you can publish messages to the Kafka cluster. In this lesson we'll produce some messages to the topics that we've created thus far. There are a few items to remember when creating topics and default partitions.

Start a console producer to topic 'test'

	bin/kafka-console-producer.sh --broker-list kafka1:9092 --topic test

Add the acks=all flag to your producer

	bin/kafka-console-producer.sh --broker-list kafka1:9092 --topic test --producer-property acks=all

Create a topic with the console producer (not recommended)

	bin/kafka-console-producer.sh --broker-list kafka1:9092 --topic test4

List the newly created topic

	bin/kafka-topics.sh --zookeeper zookeeper1:2181/kafka --list

View the partitions for a topic

	bin/kafka-topics.sh --zookeeper zookeeper1:2181/kafka --topic test5 --describe

Consumers are the only way to get messages out of a Kafka cluster. In this lesson, we'll retreive some of the messages that we've produced in the last lesson and learn a bit about how consumers keep track of their offset.

Start a console consumer to a topic

	bin/kafka-console-consumer.sh --bootstrap-server kafka3:9092 --topic test

Consuming messages from the beginning

	bin/kafka-console-consumer.sh --bootstrap-server kafka3:9092 --topic test --from-beginning

Kafka was meant to read multiple messages at once using consumer groups. This way, the speed at which messages are read increases. The consumers work very intelligently, in that they never read the same messages, and keep track of where they left off using the offset. In this lesson, we'll discover the power of consumer groups and how to describe their characteristics.

Start a consumer group for a topic

	bin/kafka-console-consumer.sh --bootstrap-server kafka3:9092 --topic test --group application1

Start producing new messages to a topic

	bin/kafka-console-producer.sh --broker-list kafka1:9092 --topic test

Start a consumer group and read messages from the beginning

	bin/kafka-console-consumer.sh --bootstrap-server kafka3:9092 --topic test --group application1 --from-beginning

List the consumer groups

	bin/kafka-consumer-groups.sh --bootstrap-server kafka3:9092 --list

Describe a consumer group bin/kafka-consumer-groups.sh --bootstrap-server kafka3:9092 --describe --group application1