Support Online
MirrorLooking Glass
GenixNodeGenixNode
Client Panel
Skip to main content

OpenSearch Log Redirection with Fluent Bit

Being able to analyze system logs on Linux servers in real time is critical for performance monitoring and security monitoring.

In this guide, you will learn how to securely route the system logs in the /var/log directory to the OpenSearch cluster** by installing the Fluent Bit tool, which is known for its low resource consumption.

Thanks to this structure, you can collect and analyze your logs on a central platform and quickly detect possible problems.

What Will You Learn in This Guide?​

The following topics will be covered in this document:

  • Fluent Bit installation
  • Reading Linux system logs
  • Configuring the OpenSearch Output plugin
  • Starting the Fluent Bit service
  • Connection verification and basic troubleshooting methods

What is Fluent Bit?​

Fluent Bit is a lightweight and high-performance log routing tool** that collects log data from servers, processes it and forwards it to different target systems.

Main features:

  • Low CPU and RAM usage
  • Real-time log processing
  • Support for many services such as OpenSearch, Elasticsearch, Kafka
  • Efficient work in small VPS and container environments

Fluent Bit Installation​

You can run the command below to quickly install Fluent Bit on Debian, Ubuntu, RHEL and CentOS based systems.

curl https://raw.githubusercontent.com/fluent/fluent-bit/master/install.sh | sh

This command:

Downloads the installation script Installs Fluent Bit packages Adds the service to the system

Fluent Bit Configuration​

Fluent Bit's main configuration file is located in the following directory:

/etc/fluent-bit/fluent-bit.conf

Two basic sections are configured in this file:

INPUT ->	Okunacak log dosyalarını belirtir
OUTPUT ->	Logların gönderileceği hedef sistemi tanımlar

INPUT Configuration – Reading System Logs​

Most Linux system logs are located under the /var/log directory.

In particular, the following logs contain critical information:

auth.log → Kimlik doğrulama kayıtları
syslog → Sistem servis olayları
journal logları → Kernel ve servis aktiviteleri

Add the following configuration for Fluent Bit to read these logs.

[INPUT]
    Name   tail
    Path   /var/log/auth.log,/var/log/syslog,/var/log/journal/*.log

This setting:

Monitors specified log files. Transfers newly added lines to Fluent Bit in real time

OUTPUT Configuration – Sending Logs to OpenSearch​

The following configuration ensures that logs are sent to the OpenSearch cluster encrypted with TLS.

[OUTPUT]
    Name        opensearch
    Match       *
    Host        tr1-opensearch.ornek.com
    Port        25060
    HTTP_User   doadmin
    HTTP_Passwd SIFRENIZ
    Index       tr1_sistem_loglari
    tls         On
    Suppress_Type_Name On

Parameter Descriptions​

Match ->	Tüm input kaynaklarından gelen logları eşleştirir
Host ->	OpenSearch sunucu adresi
Port ->	OpenSearch servis portu
Index ->	Logların yazılacağı indeks adı
tls	-> TLS şifrelemesini aktif eder

Starting Fluent Bit Service​

After completing the configuration, start the Fluent Bit service.

sudo systemctl enable fluent-bit.service
sudo systemctl start fluent-bit.service
sudo systemctl status fluent-bit.service

These commands:

Adds the service to the system startup Starts the service Checks the running status

Verification and Troubleshooting​

Testing OpenSearch Connection​

To test connectivity to the OpenSearch cluster:

curl -u doadmin:SIFRE -X GET "https://tr1-opensearch.ornek.com:25060/_cat/indices?v"

This command lists the indexes found in the cluster.

Checking Whether Logs Are Indexed:​

curl -u doadmin:SIFRE -X GET "https://tr1-opensearch.ornek.com:25060/tr1_sistem_loglari/_search?pretty"

This query displays log records within the specified index.

Examining Fluent Bit Logs​

To see service errors:

sudo journalctl -u fluent-bit

This command allows you to quickly detect configuration or connection problems.

Testing the Configuration File (Dry Run)​

To check if the Fluent Bit configuration is correct:

/opt/fluent-bit/bin/fluent-bit -c /etc/fluent-bit/fluent-bit.conf --dry-run

This process checks for syntax errors before running the configuration.

Frequently Asked Questions (FAQ)​

Why are auth.log and syslog used?

These log files contain critical information about authentication, service activities and system behavior.

Can I change the index name?

Yes. You can create different indexes for different services or projects.

Logs are delayed, why could this be?

Possible reasons:

Network latency Low CPU or RAM Fluent Bit buffer settings on the server Can logs be sent to more than one OpenSearch cluster?

Yes. By adding additional OUTPUT blocks, you can route the same logs to different OpenSearch clusters.

Result​

In this guide, you learned how to centrally and securely route the system logs on your Linux server to the OpenSearch infrastructure using Fluent Bit.

Thanks to this structure:

You can analyze server logs from a single point. You can facilitate performance and error tracking. You can quickly detect security incidents.

To try the infrastructure right away, you can test the integration by creating a virtual server and Managed OpenSearch cluster on GenixNode.