Support Online
MirrorLooking Glass
GenixNodeGenixNode
Client Panel
Skip to main content

Logstash Apache Log Forwarding: OpenSearch Integration Guide

In this guide, you will learn how to send access and error logs on your Apache server to OpenSearch by processing them with Logstash.
We show step by step the steps of parsing (grok), filtering and indexing the data.

What Will You Learn in This Guide?​

  • Install Logstash (Ubuntu/Debian & CentOS)
  • Installing the OpenSearch output plugin
  • Creating a pipeline to read Apache access & error logs
  • Parse logs with Grok filters
  • Verifying Logstash → OpenSearch connection
  • Error resolution and log analysis methods

1. Phase — Technical Analysis Summary​

This guide explains how to collect the access.log and error.log files of the Apache web server with Logstash, parse them with extended grok patterns and send them to OpenSearch.

Steps followed by the user:

  1. Logstash installation (APT/YUM)
  2. Installing the OpenSearch output plugin
  3. Pipeline creation (input → filter → output)
  4. Log parsing with Grok
  5. Starting the Logstash service
  6. OpenSearch connection test
  7. Troubleshooting: firewall, SSL, log analysis

The goal is to make Apache logs centralized, fast, secure and analyzable.

2. Logstash Setup (Localized and Simplified)​

First, find out the operating system type:

cat /etc/os-release
  • Indicates which package manager to use.

2.1 Installation for Ubuntu / Debian​

1. Add GPG key:


wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elastic-keyring.gpg

2. Add the repository:


echo "deb [signed-by=/usr/share/keyrings/elastic-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-8.x.list

3. Install Logstash:


sudo apt-get update && sudo apt-get install logstash

2.2 Installation for CentOS/RHEL​

1. Get Elastic GPG key:


sudo rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

2. Create a repo:


sudo tee /etc/yum.repos.d/logstash.repo > /dev/null <<EOF
[logstash-8.x]
name=Elastic repository 8.x
baseurl=https://artifacts.elastic.co/packages/8.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
EOF

Install:


sudo yum install logstash

3. Creating a Pipeline for Apache Logs​

1. First install the OpenSearch output plugin:


/usr/share/logstash/bin/logstash-plugin install logstash-output-opensearch
  • This plugin allows Logstash to send data to OpenSearch.

2. Create the pipeline file


sudo nano /etc/logstash/conf.d/apache_pipeline.conf

Add the following configuration:


input {
  file {
    path => "/var/log/apache2/access.log"
    start_position => "beginning"
    sincedb_path => "/dev/null"
    tags => "apache_access"
  }

  file {
    path => "/var/log/apache2/error.log"
    start_position => "beginning"
    sincedb_path => "/dev/null"
    tags => "apache_error"
  }
}

filter {
  if "apache_access" in [tags] {
    grok {
        match => { "message" => "%&#123;HTTPD_COMBINEDLOG&#125;" }#
    }
    mutate {
        remove_field => [ "message","[log][file][path]","[event][original]" ]
    }
   } else {
    grok {
        match => { "message" => "%&#123;HTTPD24_ERRORLOG&#125;" }
    }
   }
}

output {
  if "apache_access" in [tags] {
    opensearch {
      hosts       => "https://tr1-opensearch.ornek.com:25060"
      user        => "doadmin"
      password    => "<OPENSEARCH_SIFRE>"
      index       => "apache_access"
      ssl_certificate_verification => true
    }
  } else {
    opensearch {
      hosts       => "https://tr1-opensearch.ornek.com:25060"
      user        => "doadmin"
      password    => "<OPENSEARCH_SIFRE>"
      index       => "apache_error"
      ssl_certificate_verification => true
    }
  }
}

4. Starting the Service​


systemctl enable logstash.service
systemctl start logstash.service
systemctl status logstash.service

5. Phase — Troubleshooting and Verification​

  • Test OpenSearch connection:

curl -u doadmin:PAROLA -X GET "https://tr1-opensearch.ornek.com:25060/_cat/indices?v"

5.1 Are the logs included in the index?​


curl -u doadmin:PAROLA -X GET "https://tr1-opensearch.ornek.com:25060/apache_access/_search?pretty"

5.2 Firewall control​

  • Port 25060 must be open.

1. Logstash error logs


/var/log/logstash/logstash-plain.log

Frequently Asked Questions (FAQ)​

1. Why does Grok parser give an error? If your Apache log format is not standard, it cannot parse the log.

2. Why doesn't Logstash ever read the file? Because the sincedb location requires permissions or the Logstash user doesn't have read permissions.

3. Logs are not uploaded to OpenSearch, why? Usually there is SSL verification or firewall blocking.

4. How do I process multiple domain logs? You apply separate tag → separate index logic for each domain.

5. Why does Error.log come up empty? Because Apache error levels may be off (LogLevel warn is recommended).

Result​

This guide helps you parse Apache logs through the Logstash pipeline and transmit them securely to OpenSearch. Your central logging structure is now faster, trackable and analysable.

With GenixNode, you can immediately create your own OpenSearch cluster and implement your log architecture in minutes.