Support Online
MirrorLooking Glass
GenixNodeGenixNode
Client Panel
Skip to main content

Apache Kafka Installation-Ubuntu/Debian Guide

What You Will Learn in This Guide​

In this guide, you will learn the following topics step by step:

  • Installing Apache Kafka With KRaft mode without ZooKeeper
  • Cluster ID creation and storage configuration for Kafka
  • Running Kafka in the background as a systemd service**
  • Creating a topic, sending messages with producer and reading messages with consumer
  • Kafka's Differences with alternatives such as RabbitMQ and Pulsar

At the end of this guide, you will be able to use Kafka's basic event streaming architecture end-to-end.**

What is Apache Kafka?​

Apache Kafka; It is a distributed event streaming platform used to process high volume data streams in real time, reliably and scalably.

Thanks to the KRaft (Kafka Raft) architecture that comes with Kafka 3.x versions, ZooKeeper dependency has now been eliminated. ZooKeeper will be completely removed with the release of Kafka 4.

1. Kafka Installation and KRaft Configuration​

1.1 Creating a custom user for Kafka​

sudo adduser kafka

It is recommended to run Kafka with a separate user for security reasons.

Switch to user:

su kafka

1.2 Downloading and extracting the Kafka package​

Download the Kafka package:

curl -o /tmp/kafka.tgz https://downloads.apache.org/kafka/3.7.0/kafka_2.13-3.7.0.tgz

Extract the archive:

mkdir ~/kafka
tar -xzf /tmp/kafka.tgz -C ~/kafka --strip-components=1

This extracts the Kafka files directly to the ~/kafka directory.

1.3 Editing the KRaft configuration file​

Open configuration file:

nano ~/kafka/config/kraft/server.properties

Edit the following line:

log.dirs=/home/kafka/kafka-logs

This setting ensures that Kafka logs are stored in a permanent directory.

1.4 Cluster ID creation and storage formatting​

Create Kafka cluster ID:

KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)"

Format storage:

bin/kafka-storage.sh format -t $KAFKA_CLUSTER_ID -c config/kraft/server.properties

1.5 Starting Kafka for the first time​

Start Kafka server:

bin/kafka-server-start.sh config/kraft/server.properties

Once you see that the server is running, you can stop it with CTRL + C.

2. Running Kafka as Systemd Service​

2.1 Creating the service file​

Create a new systemd service file:

sudo nano /etc/systemd/system/kafka.service

File content:

[Unit]
Description=Kafka Server
Requires=network.target

[Service]
Type=simple
User=kafka
ExecStart=/bin/sh -c '/home/kafka/kafka/bin/kafka-server-start.sh /home/kafka/kafka/config/kraft/server.properties > /home/kafka/kafka/kafka.log 2>&1'
ExecStop=/home/kafka/kafka/bin/kafka-server-stop.sh
Restart=on-abnormal

[Install]
WantedBy=multi-user.target

2.2 Activating the service​

Enable and start the service:

sudo systemctl daemon-reload
sudo systemctl start kafka
sudo systemctl status kafka
sudo systemctl enable kafka

3. Kafka Topic Creation and Message Processing​

3.1 Creating a topic​

Create a new topic:

bin/kafka-topics.sh --create --topic first-topic --bootstrap-server localhost:9092

To view the topic list:

bin/kafka-topics.sh --list --bootstrap-server localhost:9092

To see topic details:

bin/kafka-topics.sh --describe --topic first-topic --bootstrap-server localhost:9092

3.2 Sending messages with Producer​

Start producer:

bin/kafka-console-producer.sh --topic first-topic --bootstrap-server localhost:9092

Example messages:

ilk mesajım ikinci mesaj

3.3 Reading messages with Consumer​

To listen to real-time messages:

bin/kafka-console-consumer.sh --topic first-topic --bootstrap-server localhost:9092

To read past messages including:

bin/kafka-console-consumer.sh --topic first-topic --from-beginning --bootstrap-server localhost:9092

3.4 Topic deletion​

bin/kafka-topics.sh --delete --topic first-topic --bootstrap-server localhost:9092

Frequently Asked Questions​

Does Kafka work without ZooKeeper?

Yes. Thanks to the KRaft architecture, Kafka can now run without a ZooKeeper dependency.

Can Kafka be installed on a single server?

A single server can be used in development and test environments. However, at least 3 nodes are recommended in production environments.

Why can Kafka messages be re-read?

Since Kafka stores messages in a disk-based log structure, past messages can be read again.

Why is partition important?

Partition structure provides parallel data processing and performance increase.

Is KRaft or ZooKeeper better?

Since ZooKeeper will be completely removed with Kafka version 4, it is recommended to use KRaft.

Result​

In this guide, you learned how to install Apache Kafka on Ubuntu and Debian servers with KRaft mode, create a systemd service, and how to generate and consume topic-based messages.

You can use Kafka's powerful event streaming architecture in your projects and manage data flows in real time.

You can immediately implement your Kafka projects with powerful servers in GenixNode infrastructure.