Everything you need to know about HAProxy log format

HAProxy is one of today’s fastest and most widely used load balancing solutions. If you’re already using HAProxy or considering using it in your environment, understanding HAProxy logging is essential.

Let’s discuss why HAProxy logging is vital to the load balancer implementation, the logging HAProxy offers, and how to manage and configure HAProxy logs to suit your unique needs.

Why load balancer logging is critical

A load balancer is essential when connecting your application to the outside world. It analyzes all inbound requests, validates them, and then passes them to the appropriate backend server. But load balancers do more than just route traffic. They help ensure high availability by distributing requests across healthy instances of your application and preventing requests from reaching unhealthy ones.

The load balancer handles all requests, so it often becomes the focal point of investigations when connectivity issues arise. That’s why HAProxy logging is crucial. It provides detailed visibility into request flows, performance metrics, and potential failures. By analyzing your HAProxy logs, you can detect anomalies and ensure smooth operations.

Why is HAProxy log monitoring important?

HAProxy logs offer millisecond precision, providing actionable data about your infrastructure at a macro and a micro level. Let’s look at some of the information you can glean from a HAProxy log and how each log entry helps provide comprehensive observability for your application.

Traffic metrics

Tracking the traffic volume moving through your application can help you understand resource allocation needs and provide critical insights about how, why, and when failures might occur.

Request and response details

If you’re investigating error conditions affecting targeted users or regions, inspecting the request header, response header, payloads, and status codes can help you isolate the exact cause of a problem. It also provides detailed information about the behaviors clients may experience.

Routing decisions

HAProxy logs also include information about how requests are routed, which backend they were assigned to, and whether any filters or persistence rules were applied. This data helps ensure efficient traffic distribution and security compliance.

Error tracing

Finally, HAProxy logs contain information engineers can use to identify where request life cycle failures occur. These logs can include information about active sessions and their termination status.

HAProxy logging: Default and pre-configured logging formats

HAProxy derives its logging format from settings within the HAProxy configuration file. You can use the default logging format by excluding the option directive from the configuration or setting one of two pre-configured formats.

The two primary logging options in HAProxy are:

• For TCP or Layer 4 operation mode, the directive to include is option tcplog.

• For HTTP or Layer 7 operation mode, the configuration directive is option httplog.

Let’s take a closer look at how each log format works and what data it provides.

These example logs below are from an instance of HAProxy version 2.4.2, installed on an AWS instance running AWS Linux 4.14 and configured to manage traffic between two HTTP servers within the same subnet. The logs were captured using RSyslog and stored locally on the host server.

Default log format (No option configured)

Jul 12 06:32:30 localhost haproxy[2679]: Connect from 67.171.183.156:50871 to 172.31.30.201:80 (http_front/TCP)

TCP / Layer 4 log format (option tcplog)

Jul 12 06:24:02 localhost haproxy[2590]: 67.171.183.156:54500 [12/Jul/2021:06:23:21.058] http_front http_back/webserver1 1/0/40996 383 -- 2/2/1/0/0 0/0

HTTP / Layer 7 (option httplog)

Jul 12 05:54:55 localhost haproxy[1060]: 67.171.183.156:64978 [12/Jul/2021:05:54:55.077] http_front http_back/webserver1 0/0/0/1/1 200 288 - - ---- 2/2/0/0/0 0/0 "GET / HTTP/1.1"

Customizing the HAProxy log format

While the default and pre-configured HAProxy log formats provide substantial insights, they may lack critical metrics and log elements related to activities supporting SSL encryption, headers, and the payload itself. These elements can significantly increase the amount of data in the log and help resolve problems.

If you want to use a customized log format, you can replace the option direction with a log-format directive, followed by a string indicating the contents and format of the desired log messages. If we were to replicate the HTTP log format described above, the directive would be as follows:

log-format "%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r"

A complete reference of available options and their correct usage is listed in the HAProxy Documentation. Some of the available options that you may want to include to support advanced troubleshooting include:

• Connection handshake time (%Th)

• SSL ciphers and versions (%sslc / %sslv)

• Request headers (%hr or %hrl for CLF formatting)

• Response headers (%hs or %hsl for CLF formatting)

• The full HTTP request (%r)

Managing HAProxy log volume at scale

HAProxy is one of the most efficient load balancing solutions currently available, capable of handling thousands of requests per second. While HAProxy can easily create a log entry for each request, as your traffic scales, so does the volume of your log data, potentially overwhelming disk storage and making manual log analysis impractical.

To manage this efficiently, consider leveraging a centralized log management solution like Sumo Logic and implementing automated anomaly detection to alert your team when issues occur. With real-time log aggregation and automated alerts, your team can proactively detect issues, handle your volume of log data, and optimize performance.

Learn more about integrating HAProxy with Sumo Logic and sign up for a free trial to see how it can streamline your log management.