HAProxy

Monitor HAProxy load balancer logs with advanced parsing and real-time analytics using the File Stream plugin

Monitor and analyze HAProxy load balancer logs in real-time using LogFlux Agent’s File Stream plugin. This configuration-based approach provides comprehensive log parsing, backend health monitoring, and advanced load balancing analytics.

Overview

The HAProxy integration leverages LogFlux Agent’s File Stream plugin to:

Real-time monitoring of HTTP and TCP traffic logs
Backend health tracking with server response monitoring
Load balancing analytics with distribution metrics
Performance monitoring with connection and response times
SSL/TLS termination logging and certificate monitoring
Custom log format support for specific use cases

Installation

The File Stream plugin is included with LogFlux Agent. Enable it for HAProxy log monitoring:

1
2
3
4
5


# Enable File Stream plugin
sudo systemctl enable --now logflux-filestream

# Verify plugin status
sudo systemctl status logflux-filestream

HAProxy Configuration

First, configure HAProxy to enable detailed logging in /etc/haproxy/haproxy.cfg:

global
    log 127.0.0.1:514 local0
    log-tag haproxy
    
defaults
    log global
    option httplog
    option dontlognull
    option log-health-checks
    
frontend web_frontend
    bind *:80
    bind *:443 ssl crt /etc/ssl/certs/example.com.pem
    option httplog
    capture request header Host len 64
    capture request header User-Agent len 128
    default_backend web_servers
    
backend web_servers
    balance roundrobin
    option httpchk GET /health
    server web1 192.168.1.10:8080 check
    server web2 192.168.1.11:8080 check

Configure rsyslog to write HAProxy logs to a file (/etc/rsyslog.d/49-haproxy.conf):

$ModLoad imudp
$UDPServerRun 514
$UDPServerAddress 127.0.0.1

# HAProxy logs
local0.* /var/log/haproxy.log
& stop

Basic Configuration

Configure the File Stream plugin to monitor HAProxy logs by creating /etc/logflux-agent/plugins/filestream-haproxy.toml:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


[filestream.haproxy_http]
paths = ["/var/log/haproxy.log"]
format = "haproxy_http"
tags = ["haproxy", "http", "loadbalancer"]
fields = {
  service = "haproxy",
  log_type = "http"
}

[filestream.haproxy_tcp]
paths = ["/var/log/haproxy-tcp.log"] 
format = "haproxy_tcp"
tags = ["haproxy", "tcp", "loadbalancer"]
fields = {
  service = "haproxy",
  log_type = "tcp"
}

HAProxy Log Formats

HTTP Log Format (Default)

HAProxy HTTP format captures comprehensive request details:

1
2
3
4
5
6
7


[filestream.haproxy_http]
paths = ["/var/log/haproxy.log"]
format = "regex"
regex = '^(?P<timestamp>\w{3} \d{1,2} \d{2}:\d{2}:\d{2}) (?P<hostname>\S+) (?P<process>\S+)\[(?P<pid>\d+)\]: (?P<client_ip>[\d\.]+):(?P<client_port>\d+) \[(?P<accept_date>[^\]]+)\] (?P<frontend_name>\S+) (?P<backend_name>\S+)/(?P<server_name>\S+) (?P<time_request>\d+)/(?P<time_queue>\d+)/(?P<time_connect>\d+)/(?P<time_response>\d+)/(?P<time_active>\d+) (?P<http_status>\d+) (?P<bytes_read>\d+) - - ---- (?P<actconn>\d+)/(?P<feconn>\d+)/(?P<beconn>\d+)/(?P<srvconn>\d+)/\+?(?P<retries>\d+) (?P<srv_queue>\d+)/(?P<backend_queue>\d+) "(?P<http_method>\S+) (?P<http_uri>\S+) (?P<http_version>[^"]*)"$'
parse_timestamp = true
timestamp_field = "timestamp"
timestamp_format = "Jan 2 15:04:05"

TCP Log Format

For TCP mode load balancing:

1
2
3
4
5
6
7


[filestream.haproxy_tcp]
paths = ["/var/log/haproxy-tcp.log"]
format = "regex"
regex = '^(?P<timestamp>\w{3} \d{1,2} \d{2}:\d{2}:\d{2}) (?P<hostname>\S+) (?P<process>\S+)\[(?P<pid>\d+)\]: (?P<client_ip>[\d\.]+):(?P<client_port>\d+) \[(?P<accept_date>[^\]]+)\] (?P<frontend_name>\S+) (?P<backend_name>\S+)/(?P<server_name>\S+) (?P<time_connect>\d+)/(?P<time_session>\d+)/(?P<time_active>\d+) (?P<bytes_read>\d+)/\+?(?P<bytes_sent>\d+) (?P<actconn>\d+)/(?P<feconn>\d+)/(?P<beconn>\d+)/(?P<srvconn>\d+)/\+?(?P<retries>\d+) (?P<srv_queue>\d+)/(?P<backend_queue>\d+)$'
parse_timestamp = true
timestamp_field = "timestamp"
timestamp_format = "Jan 2 15:04:05"

Enhanced HTTP Format with SSL

For HTTPS traffic with SSL information:

1
2
3
4
5
6
7
8
9


[filestream.haproxy_https]
paths = ["/var/log/haproxy-ssl.log"]
format = "regex"
regex = '^(?P<timestamp>\w{3} \d{1,2} \d{2}:\d{2}:\d{2}) (?P<hostname>\S+) (?P<process>\S+)\[(?P<pid>\d+)\]: (?P<client_ip>[\d\.]+):(?P<client_port>\d+) \[(?P<accept_date>[^\]]+)\] (?P<frontend_name>\S+)~ (?P<backend_name>\S+)/(?P<server_name>\S+) (?P<time_request>\d+)/(?P<time_queue>\d+)/(?P<time_connect>\d+)/(?P<time_response>\d+)/(?P<time_active>\d+) (?P<http_status>\d+) (?P<bytes_read>\d+) - - ---- (?P<actconn>\d+)/(?P<feconn>\d+)/(?P<beconn>\d+)/(?P<srvconn>\d+)/\+?(?P<retries>\d+) (?P<srv_queue>\d+)/(?P<backend_queue>\d+) "(?P<http_method>\S+) (?P<http_uri>\S+) (?P<http_version>[^"]*)" (?P<ssl_version>\S+) (?P<ssl_ciphers>\S+)$'
tags = ["haproxy", "https", "ssl"]
fields = {
  service = "haproxy",
  log_type = "https"
}

Advanced Configuration

Multi-Frontend Monitoring

Monitor multiple HAProxy frontends separately:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


[filestream.haproxy_web]
paths = ["/var/log/haproxy-web.log"]
format = "haproxy_http"
tags = ["haproxy", "web", "frontend"]
fields = {
  service = "haproxy",
  frontend = "web",
  log_type = "http"
}

[filestream.haproxy_api]
paths = ["/var/log/haproxy-api.log"]
format = "haproxy_http" 
tags = ["haproxy", "api", "frontend"]
fields = {
  service = "haproxy",
  frontend = "api",
  log_type = "http"
}

Backend Health Monitoring

Track backend server health and availability:

1
2
3
4
5
6
7
8
9


[filestream.haproxy_health]
paths = ["/var/log/haproxy.log"]
format = "regex"
regex = '^(?P<timestamp>\w{3} \d{1,2} \d{2}:\d{2}:\d{2}) (?P<hostname>\S+) (?P<process>\S+)\[(?P<pid>\d+)\]: Health check for server (?P<backend_name>\S+)/(?P<server_name>\S+) (?P<health_status>\w+).*$'
tags = ["haproxy", "health", "monitoring"]
fields = {
  service = "haproxy",
  log_type = "health_check"
}

Performance Analytics

Detailed performance tracking with calculated metrics:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


[filestream.haproxy_performance]
paths = ["/var/log/haproxy.log"]
format = "haproxy_http"
tags = ["haproxy", "performance"]
fields = {
  service = "haproxy",
  log_type = "performance"
}

# Add calculated fields for analysis
[filestream.haproxy_performance.processors.add_fields]
fields = {
  total_time = "{{ add .time_request .time_queue .time_connect .time_response }}",
  backend_efficiency = "{{ div .time_response .time_active | mul 100 }}",
  queue_ratio = "{{ div .time_queue .time_active | mul 100 }}"
}

# Filter slow requests
[filestream.haproxy_performance.processors.grep]
patterns = [
  "time_active:[5-9][0-9]{3,}",  # >5 seconds
  "http_status:5[0-9]{2}"        # 5xx errors
]

Load Balancing Analytics

Backend Distribution Tracking

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


[filestream.haproxy_distribution]
paths = ["/var/log/haproxy.log"]
format = "haproxy_http"
tags = ["haproxy", "distribution", "analytics"]
fields = {
  service = "haproxy",
  log_type = "distribution"
}

# Count requests per backend server
[filestream.haproxy_distribution.processors.counter]
field = "server_name"
window = "1m"

Connection Pool Monitoring

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


[filestream.haproxy_connections]
paths = ["/var/log/haproxy.log"]
format = "haproxy_http"
tags = ["haproxy", "connections"]
fields = {
  service = "haproxy",
  log_type = "connections"
}

# Alert on high connection counts
[filestream.haproxy_connections.processors.threshold]
field = "actconn"
threshold = 1000
action = "alert"
message = "High active connection count: {{ .actconn }}"

Usage Examples

Monitor Load Balancer Traffic

1
2
3
4
5
6
7
8


# Stream all HAProxy logs
logflux-cli stream --filter 'service:haproxy'

# Monitor specific frontend
logflux-cli stream --filter 'service:haproxy AND frontend_name:web_frontend'

# Track backend server health
logflux-cli stream --filter 'service:haproxy AND log_type:health_check'

Performance Analysis

1
2
3
4
5
6
7
8


# Monitor slow requests (>5 seconds)
logflux-cli stream --filter 'service:haproxy AND time_active:>5000'

# Track 5xx errors
logflux-cli stream --filter 'service:haproxy AND http_status:>=500'

# Monitor queue times
logflux-cli stream --filter 'service:haproxy AND time_queue:>100'

Security Monitoring

1
2
3
4
5
6
7
8


# Track failed requests
logflux-cli stream --filter 'service:haproxy AND http_status:403'

# Monitor SSL/TLS connections
logflux-cli stream --filter 'service:haproxy AND tags:ssl'

# Detect potential DDoS
logflux-cli stream --filter 'service:haproxy AND actconn:>500'

HAProxy Configuration Best Practices

Logging Configuration

global
    log 127.0.0.1:514 local0 info
    log-tag haproxy
    
defaults
    log global
    mode http
    option httplog
    option dontlognull
    option log-health-checks
    option log-separate-errors
    
    # Capture important headers
    capture request header Host len 64
    capture request header User-Agent len 128
    capture request header X-Forwarded-For len 64
    capture response header Content-Type len 64

Custom Log Format

# Enhanced log format with timing breakdown
log-format "%ci:%cp [%t] %ft %b/%s %Tq/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r"

SSL/TLS Logging

# Frontend with SSL logging
frontend https_frontend
    bind *:443 ssl crt /etc/ssl/certs/
    option httplog
    option log-separate-errors
    capture request header Host len 64
    
    # Log SSL version and cipher
    http-request set-header X-SSL-Version %[ssl_fc_protocol]
    http-request set-header X-SSL-Cipher %[ssl_fc_cipher]

Monitoring and Alerting

Key Metrics to Monitor

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27


# Backend server down alert
[alerts.haproxy_server_down]
query = "service:haproxy AND health_status:DOWN"
threshold = 1
window = "30s"
message = "HAProxy backend server is down: {{ .server_name }}"

# High response time alert
[alerts.haproxy_slow_response]
query = "service:haproxy AND time_response:>10000"
threshold = 5
window = "2m"
message = "HAProxy slow response times detected"

# High error rate alert
[alerts.haproxy_error_rate]
query = "service:haproxy AND http_status:>=500"
threshold = 10
window = "1m"  
message = "High error rate in HAProxy: {{ .backend_name }}"

# Queue buildup alert
[alerts.haproxy_queue_buildup]
query = "service:haproxy AND srv_queue:>10"
threshold = 5
window = "1m"
message = "Queue buildup detected: {{ .server_name }}"

Dashboard Metrics

Monitor these key HAProxy metrics:

Request rate (requests per second)
Response times (Tq, Tw, Tc, Tr, Ta timing breakdown)
Backend distribution (requests per server)
Error rates (4xx, 5xx responses by backend)
Connection pools (active, frontend, backend, server connections)
Queue depths (server and backend queues)
SSL/TLS usage (protocol versions, cipher suites)
Health check status (server availability)

Troubleshooting

Common Issues

HAProxy logs not appearing:

1
2
3
4
5
6
7
8
9


# Check rsyslog configuration
sudo systemctl status rsyslog
sudo grep haproxy /etc/rsyslog.d/49-haproxy.conf

# Test UDP logging
logger -p local0.info "Test HAProxy log"

# Verify log file creation
sudo tail -f /var/log/haproxy.log

Log format parsing errors:

1
2
3
4
5
6
7
8


# Test regex against sample log line
echo "sample haproxy log line" | grep -P "your_regex_pattern"

# Check LogFlux Agent logs
sudo journalctl -u logflux-filestream -f

# Validate HAProxy log format
haproxy -c -f /etc/haproxy/haproxy.cfg

Missing backend health information:

1
2
3
4
5
6


# Enable health check logging in HAProxy
# Add to defaults section:
option log-health-checks

# Restart HAProxy
sudo systemctl restart haproxy

SSL/TLS log parsing issues:

1
2
3
4
5
6
7
8


# Verify SSL certificates
sudo haproxy -c -f /etc/haproxy/haproxy.cfg

# Check SSL log capture
grep "ssl_fc" /etc/haproxy/haproxy.cfg

# Test SSL connection logging
openssl s_client -connect localhost:443 -servername example.com

Best Practices

Performance

Enable compression in HAProxy for better bandwidth utilization
Use sticky sessions when appropriate for stateful applications
Monitor connection limits to prevent resource exhaustion
Implement proper timeouts for client and server connections

Security

Hide HAProxy version in responses with option httpclose
Implement rate limiting to prevent abuse
Monitor for suspicious patterns in client IPs and user agents
Use SSL/TLS everywhere with proper certificate management

High Availability

Monitor backend health with appropriate health checks
Implement graceful failover with proper server weights
Use multiple HAProxy instances behind a VIP for redundancy
Monitor resource usage (CPU, memory, file descriptors)

Log Management

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


# Optimize log rotation for HAProxy
/var/log/haproxy*.log {
    daily
    rotate 30
    missingok
    notifempty
    compress
    delaycompress
    postrotate
        /bin/systemctl reload rsyslog.service > /dev/null 2>&1 || true
        /bin/systemctl reload logflux-filestream.service > /dev/null 2>&1 || true
    endpostrotate
}

Integration Examples

With Kubernetes

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


# HAProxy Kubernetes integration
apiVersion: v1
kind: ConfigMap
metadata:
  name: haproxy-config
data:
  haproxy.cfg: |
    global
        log stdout local0
    defaults
        mode http
        option httplog
        log global

With Docker

1
2
3
4
5


# HAProxy Docker logging setup
FROM haproxy:2.4
COPY haproxy.cfg /usr/local/etc/haproxy/haproxy.cfg
EXPOSE 80 443
# Forward logs to stdout for container logging

With Prometheus

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


# HAProxy stats for Prometheus
[filestream.haproxy_stats]
paths = ["/var/log/haproxy.log"]
format = "haproxy_http"

# Export metrics in Prometheus format
[filestream.haproxy_stats.processors.prometheus]
metrics = [
  {name = "haproxy_requests_total", type = "counter", field = "http_method"},
  {name = "haproxy_response_time_seconds", type = "histogram", field = "time_response"},
  {name = "haproxy_backend_servers", type = "gauge", field = "server_name"}
]

This comprehensive HAProxy integration provides real-time load balancer monitoring, backend health tracking, and performance analytics using LogFlux Agent’s File Stream plugin. The configuration-based approach offers detailed insights into load balancing performance, SSL/TLS usage, and backend server health.

Disclaimer

HAProxy and the HAProxy logo are trademarks of HAProxy Technologies. LogFlux is not affiliated with, endorsed by, or sponsored by HAProxy Technologies or the HAProxy project. The HAProxy logo is used solely for identification purposes to indicate compatibility with HAProxy load balancer logs.