Support Online
MirrorLooking Glass
GenixNodeGenixNode
Client Panel
Skip to main content

Apache Kafka Installation – Kafka Server Installation Guide on Ubuntu 20.04

In this guide, you will learn Installing Apache Kafka 2.8.2 on Ubuntu 20.04, Configuring with ZooKeeper and Verifying the installation with Producer-Consumer tests. You will also run Kafka in the background as a systemd service and configure security settings.

What You Will Learn in This Guide​

In this document, you will perform the following steps step by step:

  • Creating a special user account for Kafka
  • Download and install Apache Kafka 2.8.2
  • Creating ZooKeeper and Kafka systemd services
  • Creating a topic, sending and reading messages
  • Making Kafka user account more secure

What is Apache Kafka?​

Apache Kafka is a distributed event streaming platform used to process high-volume data streams in real time.

Kafka is typically used in the following scenarios:

  • Real-time data streaming
  • Messaging between microservices
  • Log collection and analysis
  • Event-based architectures

Kafka 2.x versions have a ZooKeeper dependency for metadata management. Therefore, in this guide, ZooKeeper will be installed along with Kafka.

1. Creating a Kafka User​

Running Kafka with a separate user is recommended for security reasons.

sudo adduser kafka

Give sudo permission to the Kafka user:

sudo adduser kafka sudo

Switch to Kafka user:

su -l kafka

2. Downloading Kafka 2.8.2​

First, create the download directory:

mkdir ~/Downloads

Download the Kafka package:

curl "https://downloads.apache.org/kafka/2.8.2/kafka_2.13-2.8.2.tgz" -o ~/Downloads/kafka.tgz

Create home directory for Kafka:

mkdir ~/kafka && cd ~/kafka

Extract the archive:

tar -xvzf ~/Downloads/kafka.tgz --strip 1

This extracts the Kafka files directly to the ~/kafka directory.

3. Kafka Configuration (server.properties)​

Open the Kafka configuration file:

nano ~/kafka/config/server.properties

Enable topic deletion feature:

delete.topic.enable=true

Edit the log directory:

log.dirs=/home/kafka/logs

This setting ensures that Kafka logs are stored in a permanent directory.

4. ZooKeeper and Kafka systemd Services​

Kafka 2.8.2 version works dependent on ZooKeeper. Therefore, we will create two services.

Quit Kafka user:

exit

4.1 Creating ZooKeeper Service​

sudo nano /etc/systemd/system/zookeeper.service

Service file content:

[Unit]
Requires=network.target remote-fs.target
After=network.target remote-fs.target

[Service]
Type=simple
User=kafka
ExecStart=/home/kafka/kafka/bin/zookeeper-server-start.sh /home/kafka/kafka/config/zookeeper.properties
ExecStop=/home/kafka/kafka/bin/zookeeper-server-stop.sh
Restart=on-abnormal

[Install]
WantedBy=multi-user.target

4.2 Creating Kafka Service​

sudo nano /etc/systemd/system/kafka.service

Service content:

[Unit]
Requires=zookeeper.service
After=zookeeper.service

[Service]
Type=simple
User=kafka
ExecStart=/bin/sh -c '/home/kafka/kafka/bin/kafka-server-start.sh /home/kafka/kafka/config/server.properties > /home/kafka/kafka/kafka.log 2>&1'
ExecStop=/home/kafka/kafka/bin/kafka-server-stop.sh
Restart=on-abnormal

[Install]
WantedBy=multi-user.target

4.3 Starting Services​

Refresh systemd configuration:

sudo systemctl daemon-reload

Start services:

sudo systemctl start zookeeper
sudo systemctl start kafka

Check Kafka service status:

sudo systemctl status kafka

To start automatically at server startup:

sudo systemctl enable zookeeper
sudo systemctl enable kafka

5. Testing Kafka Installation (Producer – Consumer)​

Switch to the Kafka user again:

su -l kafka

5.1 Creating a Topic​

~/kafka/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic TutorialTopic

5.2 Sending Messages with Producer​

echo "Merhaba, Dünya" | ~/kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic TutorialTopic > /dev/null

5.3 Reading Messages with Consumer​

~/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic TutorialTopic --from-beginning

You should see the following output in the terminal:

Merhaba, Dünya

6. Securing Kafka User​

Quit Kafka user:

exit

Remove Kafka user's sudo privilege:

sudo deluser kafka sudo

Lock out user:

sudo passwd kafka -l

This process prevents direct user login and increases security.

Frequently Asked Questions​

Why does Kafka 2.8.2 use ZooKeeper?

In Kafka 2.x versions, metadata management is done by ZooKeeper.

Why is Topic deletion turned off by default?

Kafka asks you to manually enable the topic deletion feature to protect data security.

Which port does Kafka use?

Kafka broker uses port 9092 by default.

How can Consumer read past messages?

All history messages can be read using the --from-beginning parameter.

Why is Kafka user account locked?

By preventing direct entry, the potential attack surface is reduced.

Result​

In this guide, you learned how to install Apache Kafka 2.8.2 on Ubuntu 20.04, configure it with ZooKeeper, and manage it with systemd services. You have also verified that the installation is working correctly by performing a producer–consumer test.

You can safely use Apache Kafka on GenixNode servers to manage high volume data streams.