This is the 2020 edition of my most read blog entry about syslog-ng web-based graphical user interfaces (web GUIs). Many things have changed in the past few years. In 2011, only a single logging as a service solution was available, while nowadays, I regularly run into others. Also, while some software disappeared, the number of logging-related GUIs is growing. This is why in this post, I will mostly focus on generic log management and open source instead of highly specialized software, like SIEMs.

Why grep is not enough?

Centralized event logging has been an important part of IT for many years for many reasons. Firstly, it is more convenient to browse logs in a central location rather than viewing them on individual machines. Secondly, central storage is also more secure. Even if logs stored locally are altered or removed, you can still check the logs on the central log server. Finally, compliance with different regulations also makes central logging necessary.

System administrators often prefer to use the command line. Utilities such as grep and AWK are powerful tools, but complex queries can be completed much faster with logs indexed in a database and a web interface. In the case of large amounts of messages, a web-based database solution is not just convenient, it is a necessity. With tens of thousands of incoming messages per second, the indexes of log databases still give Google-like response times even for the most complex queries, while traditional text-based tools are not able to scale as efficiently.

Why still syslog-ng?

Many software used for log analysis come with their own log aggregation agents. So why should you still use syslog-ng then? As organizations grow, so does the IT staff starts to diversify. Separate teams are created for operations, development and security, each with its own specialized needs in log analysis. And even the business side often needs log analysis as an input for business decisions. You can quickly end up with 4-5 different log analysis and aggregation systems running in parallel and working from the very same log messages.

This is where syslog-ng can come handy: creating a dedicated log management layer, where syslog-ng collects all of the log messages centrally, does initial basic log analysis, and feeds all the different log analysis software with relevant log messages. This can save you time and resources in multiple ways:

• You only have to learn one tool instead of many.

• Only a single tool to push through security and operations teams.

• There are less computing resources on clients.

• Logs travel only once over the network.

• Long term archival in a single location with syslog-ng instead of using multiple log analysis software.

• Filtering on the syslog-ng side can save significantly on the hardware costs of the log analysis software, and also on licensing in case of a commercial solution.

The syslog-ng application can collect both system and application logs, and can be installed both as a client and a server. Thus, you have a single application to install for log management everywhere on your network. It can reliably collect and transport huge amounts of log messages, parse (“look into”) your log messages, enrich them with geographical location and other extra data, making filters and thus, log routing, much more accurate.

Logging as a Service (LaaS)

A couple years ago, Loggly was the pioneer of logging as a service (LaaS). Today, there are many other LaaS providers (Papertrail, Logentries, Sumo Logic, and so on) and syslog-ng works perfectly with all of them.

Structured fields and name-value pairs in logs are increasingly important, as they are easier to search, and it is easier to create meaningful reports from them. The more recent IETF RFC 5424 syslog standard supports structured data, but it is still not in widespread use.

People started to use JSON embedded into legacy (RFC 3164) syslog messages. The syslog-ng application can send JSON-formatted messages – for example, you can convert the following messages into structured JSON messages:

• RFC5424-formatted log messages.

• Windows EventLog messages received from the syslog-ng Agent for Windows application.

• Name-value pairs extracted from a log message with PatternDB or the CSV parser.

Loggly and other services can receive JSON-formatted messages, and make them conveniently available from the web interface.

A number of LaaS providers are already supported by syslog-ng out of the box. If your service of choice is not yet directly supported, the following blog can help you create a new LaaS destination: https://www.syslog-ng.com/community/b/blog/posts/how-to-use-syslog-ng-with-laas-and-why

Some non-syslog-ng-based solutions

Before focusing on the solutions with syslog-ng at their heart, I would like to say a few words about the others, some which were included in the previous edition of the blog.

LogAnalyzer from the makers of Rsyslog was a simple, easy to use PHP application a few years ago. While it has developed quite a lot, recently I could not get it to work with syslog-ng. Some of the popular monitoring software have syslog support to some extent, for example, Nagios, Cacti and several others. I have tested some of these, I have even sent patches and bug reports to enhance their syslog-ng support, but syslog is clearly not their focus, just one of the possible inputs.

The ELK stack (Elasticsearch + Logstash + Kibana) and Graylog2 have become popular recently, but they have their own log collectors instead of syslog-ng, and syslog is just one of many log sources. Syslog support is quite limited both in performance and protocol support. They recommend using file readers for collecting syslog messages, but that increases complexity, as it is an additional software on top of syslog(-ng), and filtering still needs to be done on the syslog side. Note that syslog-ng can send logs to Elasticsearch natively, which can greatly simplify your logging architecture.

Collecting and displaying metrics data

You can collect metrics data using syslog-ng. Examples include netdata or collectd. You can send the collected data to Graphite or Elasticsearch. Graphite has its own web interface, while you can use Kibana to query and visualize data collected to Elasticsearch.

Another option is to use Grafana. Originally, it was developed as an alternative web interface to the Graphite databases, but now it can also visualize data from many more data sources, including Elasticsearch. It can combine multiple data sources to a single dashboard and provides fine-grained access control.

Loki by Grafana is one of the latest applications that lets you aggregate and query log messages, and of course, to visualize logs using Grafana. It does not index the contents of log messages, only the labels associated with logs. This way, processing and storing log messages requires less resources, making Loki more cost-effective. Promtail, the log collector component of Loki, can collect log messages using the new, RFC 5424 syslog protocol. Learn here how syslog-ng can send its log messages to Loki.

Splunk

One of the most popular web-based interfaces for log messages is Splunk. A returning question is whether to use syslog-ng or Splunk. Well, the issue is a bit of apples vs. oranges: they do not replace, but rather complement each other. As I already mentioned in the introduction, syslog-ng is good at reliably collecting and processing huge amounts of data. Splunk, on the other hand, is good at analyzing log messages for various purposes. Learn more about how you can integrate syslog-ng with Splunk from our white paper!

Syslog-ng based solutions

Here I show a number of syslog-ng based solutions. While every software described below is originally based on syslog-ng Open Source Edition (except for One Identity’s own syslog-ng Store Box (SSB)), there are already some large-scale deployments available also with syslog-ng Premium Edition as their syslog server.

• The syslog-ng application and SSB focus on generic log management tasks and compliance.

• LogZilla focuses on logs from Cisco devices.

• Security Onion focuses on network and host security.

• Recent syslog-ng releases are also able to store log messages directly into Elasticsearch, a distributed, scalable database system popular in DevOps environments, which enables the use of Kibana for analyzing log messages.

Benefits of using syslog-ng PE with these solutions include the logstore, a tamper-proof log storage (even if it means that your logs are stored twice), Windows support, and enterprise grade support.

LogZilla

LogZilla is the commercial reincarnation of one of the oldest syslog-ng web GUIs: PHP-Syslog-NG. It provides the familiar user interface of its predecessor, but also includes many new features. The user interface supports Cisco Mnemonics, extended graphing capabilities, and e-mail alerts. Behind the scenes, LDAP integration, message de-duplication, and indexing for quick searching were added for large datasets.

Over the past years, it received many small improvements. It became faster, and role-based access control was added, as well as the live tailing of log messages. Of course, all these new features come with a price; the free edition, which I have often recommended for small sites with Cisco logs is completely gone now.

A few years ago, a complete rewrite became available with many performance improvements under the hood and a new dashboard on the surface. Development never stopped, and now LogZilla can parse and enrich log messages, and can also automatically respond to events.

Therefore, it is an ideal solution for a network operations center (NOC) full of Cisco devices.

Web site: http://logzilla.net/

Security Onion

One of the most interesting projects utilizing syslog-ng is Security Onion, a free and open source Linux distribution for threat hunting, enterprise security monitoring, and log management. It is utilizing syslog-ng for log collection and log transfer, and uses the Elastic stack to store and search log messages. Even if you do not use its advanced security features, you can still use it for centralized log collection and as a nice web interface for your logs. But it is also worth getting acquainted with its security monitoring features, as it can provide you some useful insights about your network. Best of all, Security Onion is completely free and open source, with commercial support available for it.

You can learn more about it at https://www.syslog-ng.com/community/b/blog/posts/syslog-ng-and-security-onion

Elastisearch and Kibana

Elasticsearch is gaining momentum as the ultimate destination for log messages. There are two major reasons for this:

• You can store arbitrary name-value pairs coming from structured logging or message parsing.

• You can use Kibana as a search and visualization interface.

The syslog-ng application can send logs directly into Elasticsearch. We call this an ESK stack (Elasticsearch + syslog-ng + Kibana).

Learn how you can simplify your logging to Elasticsearch by using syslog-ng: https://www.syslog-ng.com/community/b/blog/posts/logging-to-elasticsearch-made-simple-with-syslog-ng

syslog-ng Store Box (SSB)

SSB is a log management appliance built on syslog-ng Premium Edition. SSB adds a powerful indexing engine, authentication and access control, customized reporting capabilities, and an easy-to-use web-based user interface.

Recent versions introduced AWS and Azure cloud support and horizontal scalability using remote logspaces. The new content-based alerting can send an e-mail alert whenever a match between the contents of a log message and a search expression is found.

SSB is really fast when it comes to indexing and searching log data. To put this scalability in context, the largest SSB appliance stores up to 10 terabytes of uncompressed, raw logs. With SSB’s current indexing performance of 100,000 events per second, that equates to approximately 8.6 billion logs per day or 1.7 terabytes of log data per day (calculating with an average event size of 200 bytes). Using compression, a single, large SSB appliance could store approximately one month of log data for an enterprise generating 1.7 terabytes of event data a day. This compares favorably to other solutions that require several nodes for collecting this amount of messages, and even more additional nodes for storing them. While storing logs to the cloud is getting popular, on-premise log storage is still a lot cheaper for a large amount of logs.

The GUI makes searching logs, configuring and managing the SSB easy. The search interface allows you to use wildcards and Boolean operators to perform complex searches, and drill down on the results. You can gain a quick overview and pinpoint problems fast by generating ad-hoc charts from the distribution of the log messages.

Configuring the SSB is done through the user interface. Most of the flexible filtering, classification and routing features in the syslog-ng Open Source and Premium Editions can be configured with the UI. Access and authentication policies can be set to integrate with Microsoft Active Directory, LDAP and RADIUS servers. The web interface is accessible through a network interface dedicated to the management traffic. This management interface is also used for backups, sending alerts, and other administrative traffic.

SSB is a ready-to-use appliance, which means that no software installation is necessary. It is easily scalable, because SSB is available both as a virtual machine and as a physical appliance, ranging from entry-level servers to multiple-unit behemoths. For mission critical applications, you can use SSB in High Availability mode. Enterprise-level support for SSB and syslog-ng PE is also available.

■ Read more about One Identity’s syslog-ng and SSB products here.

■ Request evaluation version / callback.

Until now collecting logs behind proxies or load balancers needed some compromises. You either trusted the host information included in the log messages or you could only see the proxy as the sender host. Starting with syslog-ng 3.30 there is a third option available: using the PROXY protocol. While not an official Internet standard, it is supported by a number of popular software, like HAProxy. Other software can be extended to use it, like F5 load balancers using iRules. This way crucial information about the original network connection is not lost, but it is forwarded to the server by the proxy.

From this blog you can learn about the PROXY protocol, how to enable it in the syslog-ng configuration, and how to send test messages using loggen directly and through HAProxy.

Before you begin

You need to use at least sylog-ng version 3.30 (or syslog-ng PE 7.0.23 of the commercial version) to utilize PROXY protocol support. Most Linux distributions still carry older versions. You can find information about unofficial 3^rd party syslog-ng repositories with up-to-date syslog-ng packages at https://www.syslog-ng.com/3rd-party-binaries. At the moment these versions are not yet released, so I used git snapshot packages for testing.

In my blog I will show you a simple configuration for HAProxy, as it is available for free and it is included in most Linux distributions. I ran my tests on three openSUSE virtual machines separately for the client sending logs, for HAProxy and for the syslog-ng server. But you can use any platform that HAProxy and syslog-ng supports and can actually have all three on a single host.

The PROXY protocol

Before we take a deep dive into syslog-ng configuration, let’s take a closer look at the PROXY protocol. The PROXY protocol was created by HAProxy developers and it is available on their website: http://www.haproxy.org/download/1.8/doc/proxy-protocol.txt While it resembles an RFC, it is not an official standard, yet many devices and software support it because it solves a very common problem. When a TCP connection goes through a proxy or a load balancer, the original TCP information, including the source IP address, is lost. The PROXY protocol makes sure that this information reaches the servers behind the proxy. The PROXY protocol has two versions. The first version is text-based, while the second version forwards information in a binary form. The syslog-ng implementation supports the first version of the protocol.

When the text-based first version of the PROXY protocol is enabled, the proxy starts each new TCP connection with a similar line:

PROXY TCP4 192.168.0.1 192.168.0.11 56324 443

As you can see, it starts with a fixed text – PROXY – followed by the version of the TCP protocol, the source and destination IP addresses and ports of the connection. The PROXY protocol is not auto detected, it does not involve any heuristics. If the first line does not follow this format, the server side rejects the connection. The only exception is when the proxy sends the following line:

PROXY UNKNOWN

What happens here depends on the implementation: some servers simply reject these connections, but most accept them, including syslog-ng.

Configuring syslog-ng

Append the following configuration snippet to your syslog-ng.conf or place it in a file with .conf extension under the /etc/syslog-ng/conf.d/ directory, if syslog-ng on your host is configured to use it.

source s_tcp_pp {
    network(
        port(7777)
        ip(0.0.0.0)
        transport("proxied-tcp")
    );
};

destination d_file {
    file("/var/log/pp.log" template("$(format-json --scope nv-pairs)\n"));
};

log {
    source(s_tcp_pp);
    destination(d_file);
};

The source listens on port 7777 and expects incoming connections to use the PROXY protocol. You can also use encrypted connections here by replacing “proxied-tcp” with “proxied-tls” and adding the TLS-related options, just like with a regular encrypted source.

The file destination uses JSON formatting. This way you can see the name-value pairs created by this source: PROXIED_SRCPORT, PROXIED_SRCIP, PROXIED_IP_VERSION, PROXIED_DSTPORT and PROXIED_DSTIP. Note that these name-value pairs are not created with PROXY UNKNOWN.

Finally, the log statement connects the source and destination into a pipeline together. Reload syslog-ng for the new configuration to take effect.

Testing with loggen

The easiest way to test the above configuration is to use the loggen utility of syslog-ng. First try to send a few logs without enabling PROXY protocol support:

loggen -i -S localhost 7777

You will not find any new log messages in the new file destination. However, /var/log/messages will contain messages similar to these (if logging of the internal() source is enabled):

Nov  6 16:16:07 localhost syslog-ng[891]: PROXY proto header with invalid header length; max_parsable_length='216', max_length_by_spec='108', length='255', header='<38>2020-11-06T16:16:07 localhost prg00000[1234]: seq: 0000000000, thread: 0000, runid: 1604675767, stamp: 2020-11-06T16:16:07 PADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADD\x0a<38>2020-11-06T16:16:07 localhost prg00000[1234]: seq: 0000000001, thread: 0000, runid: 1604675767, stamp: 2020-11-06T16:16:07 [...] PADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADD\x0a-client.c'
Nov  6 16:16:07 localhost syslog-ng[891]: Error parsing PROXY protocol header;
Nov  6 16:16:07 localhost syslog-ng[891]: Syslog connection closed; fd='16', client='AF_INET(127.0.0.1:41066)', local='AF_INET(0.0.0.0:7777)'

It means that loggen did not use the PROXY header and thus the connection was rejected. Let’s try again, this time using the new -H option of loggen:

loggen -i -S -H localhost 7777

This time /var/log/messages show a successful connection:

Nov  6 16:37:25 localhost syslog-ng[891]: Initializing PROXY protocol source driver; driver='0x560b2fb9b310'
Nov  6 16:37:25 localhost syslog-ng[891]: Syslog connection accepted; fd='16', client='AF_INET(127.0.0.1:41068)', local='AF_INET(0.0.0.0:7777)'
Nov  6 16:37:25 localhost syslog-ng[891]: PROXY protocol header parsed successfully;
Nov  6 16:37:29 localhost syslog-ng[891]: Syslog connection closed; fd='16', client='AF_INET(127.0.0.1:41068)', local='AF_INET(0.0.0.0:7777)'

And in /var/log/pp.log you will find similar messages:

{"SOURCE":"s_tcp_pp","PROXIED_SRCPORT":"7075","PROXIED_SRCIP":"192.168.1.48","PROXIED_IP_VERSION":"4","PROXIED_DSTPORT":"514","PROXIED_DSTIP":"192.168.1.47","PROGRAM":"prg00000","PID":"1234","MESSAGE":"seq: 0000003961, thread: 0000, runid: 1604677045, stamp: 2020-11-06T16:37:29 PADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADD","LEGACY_MSGHDR":"prg00000[1234]: ","HOST_FROM":"127.0.0.1","HOST":"localhost"}
{"SOURCE":"s_tcp_pp","PROXIED_SRCPORT":"7075","PROXIED_SRCIP":"192.168.1.48","PROXIED_IP_VERSION":"4","PROXIED_DSTPORT":"514","PROXIED_DSTIP":"192.168.1.47","PROGRAM":"prg00000","PID":"1234","MESSAGE":"seq: 0000003962, thread: 0000, runid: 1604677045, stamp: 2020-11-06T16:37:29 PADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADD","LEGACY_MSGHDR":"prg00000[1234]: ","HOST_FROM":"127.0.0.1","HOST":"localhost"}

The PROXY protocol related name-value pairs contain random IP addresses and ports by default, but you can specify your own values as well if you want to test your configuration with specific values.

Installing and configuring HAProxy

HAProxy is part of most Linux distributions and it is also available on FreeBSD. On openSUSE the HAProxy package comes with a sample configuration. All it needs is appending four lines at the end. Append these lines:

listen sng
  bind *:6666
  mode tcp
  server server1 172.16.167.153:7777 maxconn 32 send-proxy

Of course, you also need to change the IP address to the address of your syslog-ng server. The above configuration snippet listens on port 6666 and forwards connections to port 7777 on the given IP address. The “mode tcp” makes sure that HAProxy handles the connection as a generic TCP connection instead of as a HTTP connection. The “send-proxy” keyword enables PROXY protocol for this destination. If it is not included, syslog-ng rejects the connections coming from this proxy. Once you saved the new configuration and reloaded HAProxy, you are ready for testing!

Testing through HAProxy

You are not longer limited to using loggen when testing through HAProxy. You can use any software that can send logs to port 6666 through a TCP connection, but for testing, the generic logger and the loggen utility from syslog-ng are the easiest to use.

logger --tcp --port 6666 --server 172.16.167.139 --rfc3164 this is a test

or

loggen -i -S 172.16.167.139 6666

Of course, replace the IP address with the IP address of your HAProxy server. You should see logs in /var/log/pp.log with real IP addresses of your hosts:

{"SOURCE":"s_tcp_pp","PROXIED_SRCPORT":"59516","PROXIED_SRCIP":"172.16.167.1","PROXIED_IP_VERSION":"4","PROXIED_DSTPORT":"6666","PROXIED_DSTIP":"172.16.167.139","PROGRAM":"prg00000","PID":"1234","MESSAGE":"seq: 0000002318, thread: 0000, runid: 1604679827, stamp: 2020-11-06T17:23:48 PADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADD","LEGACY_MSGHDR":"prg00000[1234]: ","HOST_FROM":"172.16.167.139","HOST":"localhost"}
{"SOURCE":"s_tcp_pp","PROXIED_SRCPORT":"59518","PROXIED_SRCIP":"172.16.167.1","PROXIED_IP_VERSION":"4","PROXIED_DSTPORT":"6666","PROXIED_DSTIP":"172.16.167.139","PROGRAM":"czanik","MESSAGE":"This is a test","LEGACY_MSGHDR":"czanik: ","HOST_FROM":"172.16.167.139","HOST":"czplaptop"}

One more thing

In your log analysis software, you most likely want to use the real source IP instead of the IP of the proxy server / load balancer. You can train your analytics software about the PROXIED_SRCIP name-value pair, but it is easier to handle this on the syslog-ng side, so I suggest rewriting HOST_FROM with the value of PROXIED_SRCIP. Here is a slightly modified version of the previous configuration with a rewrite rule added to it:

source s_tcp_pp {
    network(
        port(7777)
        ip(0.0.0.0)
        transport("proxied-tcp")
    );
};

rewrite r_fixfrom {
    set("$PROXIED_SRCIP", value("HOST_FROM"));
};

destination d_file {
    file("/var/log/pp.log" template("$(format-json --scope nv-pairs)\n"));
};

log {
    source(s_tcp_pp);
    rewrite(r_fixfrom);
    destination(d_file);
};

When you send another test message, HOST_FROM will now contain the real source IP address instead of the proxy IP address:

{"SOURCE":"s_tcp_pp","PROXIED_SRCPORT":"52532","PROXIED_SRCIP":"172.16.167.1","PROXIED_IP_VERSION":"4","PROXIED_DSTPORT":"6666","PROXIED_DSTIP":"172.16.167.139","PROGRAM":"czanik","MESSAGE":"This is a test fixed","LEGACY_MSGHDR":"czanik: ","HOST_FROM":"172.16.167.1","HOST":"czplaptop"}

What is next?

From this blog you could learn how to configure syslog-ng for the PROXY protocol and how to validate your configuration using loggen directly. I also showed you a very basic HAProxy configuration and an example for sending logs to syslog-ng through HAProxy. This setup was sufficient to test the PROXY protocol, but using a single server in a production environment does not make much sense.

If you need commercial level support to integrate syslog-ng with a proxy or load balancer like HAProxy or F5, consider buying syslog-ng PE which not only providesenterprise support, but also comes with a number of extra features. Do not hesitate to contact us at https://www.syslog-ng.com/products/log-management-software/

If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/syslog-ng/syslog-ng. On Twitter, I am available as @PCzanik.