Java Kafka Producer Setup: Programmatic Data Sending

In this guide, we will develop a producer application that sends messages to the Kafka cluster using Java and Apache Maven.
The application will send the message asynchronously and record the metadata information (time, segment, offset) of the sent message within the cluster via the Callback method.

🧠 Technical Summary

Main topic: Developing Apache Kafka Producer application with Java.
The problem it solves: It allows sending messages to Kafka topics in a flexible and programmatic way instead of command line scripts.
User steps:

1. Creating a Maven project
2. Adding Kafka dependencies
3. Writing the Producer class
4. Sending the message to Kafka
5. Getting metadata information with callback

🎯 This structure is especially ideal for developers who want microservices to produce data via Kafka.

---

🧩 Prerequisites

• Virtual Server: At least 4 GB RAM, 2 CPUs (example: tr1-node01.ornek.com)
• Java JDK: Version 8 or later
• Apache Kafka: Must be installed and configured
• Maven: Package management tool installed
• Kafka Knowledge: Familiarity with basic topic, section, offset concepts

💡 If you haven't installed Kafka yet, you can start with Kafka Installation Guide.

---

⚙️ Step 1 – Creating a Maven Project

First update your system and install Maven:

sudo apt update
sudo apt install maven -y

Verify installation:

mvn --version

Create a new folder and go into it:

mkdir ~/kafka-projeleri
cd ~/kafka-projeleri

Start a new project:

mvn archetype:generate \
-DgroupId=com.rabisucloud \
-DartifactId=genixnode-kafka-uyg \
-DarchetypeArtifactId=maven-archetype-quickstart \
-DarchetypeVersion=1.4 \
-DinteractiveMode=false

Enter the created project folder:

cd genixnode-kafka-uyg

---

📦 Step 2 – Adding Kafka Dependencies

Add the Kafka client libraries to the pom.xml file:

<dependencies>
  <dependency>
      <groupId>org.apache.kafka</groupId>
      <artifactId>kafka-clients</artifactId>
      <version>3.7.0</version>
  </dependency>

  <dependency>
      <groupId>org.slf4j</groupId>
      <artifactId>slf4j-api</artifactId>
      <version>2.0.12</version>
  </dependency>
  <dependency>
      <groupId>org.slf4j</groupId>
      <artifactId>slf4j-simple</artifactId>
      <version>2.0.12</version>
  </dependency>
</dependencies>

Also add the following plugin to the build section to ensure that dependencies are packaged with the project:

<build>
  <plugins>
      <plugin>
          <artifactId>maven-dependency-plugin</artifactId>
          <executions>
              <execution>
                  <phase>package</phase>
                  <goals>
                      <goal>copy-dependencies</goal>
                  </goals>
                  <configuration>
                      <outputDirectory>${project.build.directory}/lib</outputDirectory>
                  </configuration>
              </execution>
          </executions>
      </plugin>
  </plugins>
</build>

Compile the project and download the dependencies:

mvn package

✅ Output: BUILD SUCCESS — The project has been compiled successfully.

---

☕ Step 3 – Writing the Kafka Producer Class

Delete the default App.java file:

rm src/main/java/com/rabisucloud/App.java

Create the new file:

nano src/main/java/com/rabisucloud/ProducerDemo.java

Add the following code:

package com.rabisucloud;

import org.apache.kafka.clients.producer.*;
import org.apache.kafka.common.serialization.StringSerializer;
import java.util.Properties;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class ProducerDemo {
  private static final Logger log = LoggerFactory.getLogger(ProducerDemo.class);

  public static void main(String[] args) {
    String bootstrapServers = "localhost:9092";
    String topicName = "java_demo";

    Properties properties = new Properties();
    properties.setProperty(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
    properties.setProperty(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
    properties.setProperty(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());

    KafkaProducer<String, String> producer = new KafkaProducer<>(properties);
    ProducerRecord<String, String> record = new ProducerRecord<>(topicName, "GenixNode'tan selamlar 👋");

    // Mesajı asenkron olarak gönder
    producer.send(record);

    // Gönderim için bekleyen tüm mesajları zorla gönder
    producer.flush();

    // Producer nesnesini kapat
    producer.close();

    log.info("Mesaj başarıyla gönderildi!");
  }
}

💬 This class is a simple Kafka producer that sends messages to the topic named “java_demo”.

---

🧱 Step 4 – Creating a Runtime Script

Create a bash script to easily run the application:

nano run-producer.sh

Add content:

#!/bin/bash
mvn clean
mvn package
java -cp target/genixnode-kafka-uyg-1.0-SNAPSHOT.jar:target/lib/* com.rabisucloud.ProducerDemo

Set script permissions and run:

chmod +x run-producer.sh
./run-producer.sh

💡 You are now sending a message to Kafka. 🎉

---

🔄 Step 5 – Getting Metadata with Callback

Update send() method with callback support:

producer.send(record, new Callback() {
  public void onCompletion(RecordMetadata metadata, Exception e) {
    if (e != null) {
      log.error("Hata oluştu!", e);
      return;
    }

    log.info(String.format("Zaman: %s | Bölüt: %s | Ofset: %s",
      metadata.timestamp(),
      metadata.partition(),
      metadata.offset()));
  }
});

💬 This callback logs the time, segment and offset the message reached the cluster.

Output example:

[ProducerDemo] Zaman: 1710181831814 | Bölüt: 0 | Ofset: 3

---

❓ Frequently Asked Questions (FAQ)

1.What does flush() do?

It allows pending messages to be sent instantly. Prevents data loss in asynchronous operations.

2. Why is it important to use callback?

It verifies whether the message actually reached the cluster and provides error management.

3. What is ProducerRecord?

It is the data model that represents the message (topic, key, value) to be sent to Kafka.

4. What does the “UNKNOWN_TOPIC_OR_PARTITION” error mean?

Topic does not exist. You need to create it with kafka-topics.sh.

5. How can I improve performance?

You can increase throughput by optimizing acks, batch.size and linger.ms settings.

---

🏁 Conclusion

In this guide:

Creating a Java Kafka project with Maven,

Adding Kafka client,

You learned how to write producers and use callbacks.

💡 Now it's your turn! Test your data flows by setting up Kafka clusters on GenixNode.