Support Online
MirrorLooking Glass
GenixNodeGenixNode
Client Panel
Skip to main content

Transferring Nginx Logs to OpenSearch: Logstash Installation

What will you learn in this guide?​

In this guide, you will learn how to install Logstash on Ubuntu.
You will send Nginx access and error logs centrally to OpenSearch.
In the final stage, you will view the logs via OpenSearch Dashboard.

Technical Summary​

Main topic: Transferring Nginx logs to OpenSearch with Logstash
Solved problem: Inability to track scattered logs and difficulty in analysis
Gain: Centralized logging, search and visualization

Steps followed:

  1. Logstash installation
  2. Adding the OpenSearch output plugin
  3. Logstash pipeline configuration
  4. OpenSearch Dashboard settings
  5. Troubleshooting

Prerequisites​

  • A server with Ubuntu installed
  • Nginx must be installed and producing logs
  • A working OpenSearch cluster
  • OpenSearch access information must be ready

1. Logstash Installation (Ubuntu)​

Verify the operating system first.

cat /etc/os-release
  • This command shows the Linux distribution of the server.
  1. Add the repository key

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elastic-keyring.gpg
  • This command adds the signature key for Logstash packages.
  1. Install the necessary packages

sudo apt-get install apt-transport-https
  • This package enables repository usage over HTTPS.
  1. Add the Logstash repository

echo "deb [signed-by=/usr/share/keyrings/elastic-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list
  • This command identifies the Logstash package source.
  1. Complete the installation

sudo apt-get update
sudo apt-get install logstash
  • These commands install the Logstash service on the system.
  1. Start the service

sudo systemctl start logstash
sudo systemctl enable logstash
  • These commands run Logstash and launch it at boot.

2. Installing the OpenSearch Output Plugin​

  1. Plugin required for Logstash to send data to OpenSearch.

/usr/share/logstash/bin/logstash-plugin install logstash-output-opensearch
  • This command installs the OpenSearch output plugin.

3. Logstash Pipeline Configuration​

  1. Logstash pipeline consists of three parts: input, filter, output.

  2. Create the configuration file


sudo nano /etc/logstash/conf.d/nginx-to-opensearch.conf
  • This file determines how Nginx logs will be processed.

input {
  file {
    path => "/var/log/nginx/access.log"
    start_position => "beginning"
    sincedb_path => "/dev/null"
    tags => ["nginx_access"]
  }
  file {
    path => "/var/log/nginx/error.log"
    start_position => "beginning"
    sincedb_path => "/dev/null"
    tags => ["nginx_error"]
  }
}

filter {
  if "nginx_access" in [tags] {
    grok {
      match => { "message" => "%{IPORHOST:client_ip} - %{USER:ident} \[%{HTTPDATE:timestamp}\] \"%{WORD:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:http_version}\" %{NUMBER:response} %{NUMBER:bytes} \"%{DATA:referrer}\" \"%{DATA:user_agent}\"" }
    }
    mutate {
      remove_field => ["message", "[log][file][path]", "[event][original]"]
    }
  } else if "nginx_error" in [tags] {
    grok {
      match => { "message" => "\[%{TIMESTAMP_ISO8601:timestamp}\] \[%{LOGLEVEL:level}\] \[%{DATA:pid}\] \[%{DATA:tid}\] %{GREEDYDATA:error_message}" }
    }
    mutate {
      remove_field => ["message", "[log][file][path]", "[event][original]"]
    }
  }
}

output {
  if "nginx_access" in [tags] {
    opensearch {
      hosts => ["https://opensearch.ornek.com:25060"]
      user => "doadmin"
      password => "OPENSEARCH_SIFRE"
      index => "nginx_access-%{+YYYY.MM.dd}"
      ssl => true
      ssl_certificate_verification => true
    }
  } else if "nginx_error" in [tags] {
    opensearch {
      hosts => ["https://opensearch.ornek.com:25060"]
      user => "doadmin"
      password => "OPENSEARCH_SIFRE"
      index => "nginx_error-%{+YYYY.MM.dd}"
      ssl => true
      ssl_certificate_verification => true
    }
  }
}
  • This configuration parses Nginx logs and sends them to OpenSearch.
  1. Apply the configuration

sudo systemctl restart logstash
  • This command activates the new pipeline settings.
  1. Check logs

sudo tail -f /var/log/logstash/logstash-plain.log
  • This command shows whether Logstash is processing data or not.

4. OpenSearch Dashboard Configuration​

  1. Enter the OpenSearch interface from the browser.

https://opensearch.ornek.com
  1. Create Index Pattern:
nginx_access-*

nginx_error-*
  • Verify that the logs are flowing in the Discover section.

Frequently Asked Questions​

1. Why use Logstash? It collects, processes and transmits logs to central systems.

2. Can access and error logs be separated? Yes, they are sent to separate indexes with the tag structure.

3. Why are the logs not visible? Authorizations, connection or firewall settings should be checked.

4. Is SSL verification mandatory? It must be turned on in production environments.

Troubleshooting​

  1. Test OpenSearch connection

curl -u doadmin:OPENSEARCH_SIFRE -X GET "https://opensearch.ornek.com:25060/_cat/indices?v"
  • This command verifies OpenSearch access.
  1. Firewall control
  • Logstash → OpenSearch port: 25060
  • Make sure exit traffic is allowed

Result​

With this guide, you transferred Nginx logs to a central OpenSearch structure. Logs were parsed and indexed thanks to Logstash. Now you can search, analyze and visualize your logs.

You can safely implement this architecture on the GenixNode infrastructure.