Remote syslog to Logstash

Introduction

This guide describes how you can send syslog messages from a Halon cluster to Logstash and then onwards to for example Elasticsearch. If you are looking for ways to send over structured logs of the mail history similar to whats on the "History and queue" page on a Halon cluster have a look at our Remote logging to Elasticsearch guide instead.

Prerequisites

• A server with Logstash installed (Version >= 6.2)
• A server with Elasticsearch installed (Version >= 6.2, Optional)
• A server with Kibana installed (Version >= 6.2, Optional)
• The syslog port you plan to use in Logstash must be accessible from the Halon cluster

Configuring Logstash

Before you start sending the logs from a Halon cluster to Logstash it's important that Logstash is configured correctly. Below you will find a sample configuration that configures a TCP listener that uses the Syslog input plugin, provides a grok and date filter that works with the structure of the Halon syslog messages and also configures it to use the Elasticsearch output plugin. The Halon syslog messages follows the RFC 3164 standard and can be configured to use up to 3 decimals for the timestamp value. In this guide we will use 3 decimals to ensure that the logs are always logged in the correct order in Elasticsearch. Note that we specify which timezone the logs will be stored as which is highly recommended since the timestamp from a Halon node does not contain a timezone and since it's sent in it's local time the logs could otherwise be stored with an incorrect timezone.

input {
  tcp {
    port => 5000
    type => syslog
  }
}

filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss.SSS", "MMM dd HH:mm:ss.SSS" ]
      timezone => "UTC"
    }
  }
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    ilm_enabled => false
  }
  stdout { codec => rubydebug }
}

Configuring the Halon cluster

Now you're ready to start sending syslog messages to Logstash. To do this, begin by going in under Hosts -> Services -> Syslog in the Halon web interface and configure each node in the cluster to use 3 decimals for the timestamp value like we mentioned before.

After this we can add a remote syslog destination for each node in the cluster that points to the Logstash server. Note how we specify the same port as we used in the sample configuration we provided for Logstash earlier. If you used a different port you should specify that port instead.

Viewing the logs

If everything went well you should start to see logs coming into Logstash. If you have also configured Elasticsearch and Kibana you should be able to see the logs showing up there as well.