Dashboard GitHub

Squid Proxy

Monitor Squid proxy server logs including access logs, cache logs, and store logs for web traffic analysis and security monitoring using the File Stream plugin

Squid Proxy Integration

Squid

Monitor and analyze Squid proxy server logs in real-time using LogFlux Agent’s File Stream plugin. This configuration-based approach provides comprehensive web traffic monitoring, cache performance analysis, and security monitoring for Squid proxy deployments.

Overview

The Squid Proxy integration leverages LogFlux Agent’s File Stream plugin to:

  • Real-time monitoring of access logs, cache logs, and store logs
  • Web traffic analysis with client requests, response codes, and bandwidth tracking
  • Cache performance monitoring with hit/miss ratios and cache efficiency metrics
  • Security monitoring with blocked requests, malware detection, and access violations
  • Bandwidth analysis with data transfer tracking and usage patterns
  • Authentication monitoring for proxy authentication events and user access

Installation

The File Stream plugin is included with LogFlux Agent. Enable it for Squid log monitoring:

1
2
3
4
5

# Enable File Stream plugin
sudo systemctl enable --now logflux-filestream

# Verify plugin status
sudo systemctl status logflux-filestream

Squid Configuration

Configure Squid logging in /etc/squid/squid.conf:

Basic Logging Configuration

# Access log configuration
access_log daemon:/var/log/squid/access.log squid
access_log_file_daemon /var/log/squid/access.log

# Cache log
cache_log /var/log/squid/cache.log

# Store log
cache_store_log /var/log/squid/store.log

# Debug logging (for troubleshooting)
debug_options ALL,1 33,2 28,9

# Log format options
logformat squid %ts.%03tu %6tr %>a %Ss/%03>Hs %<st %rm %ru %[un %Sh/%<a %mt
logformat common %>a %[ui %[un [%tl] "%rm %ru HTTP/%rv" %>Hs %<st %Ss:%Sh
logformat combined %>a %[ui %[un [%tl] "%rm %ru HTTP/%rv" %>Hs %<st "%{Referer}>h" "%{User-Agent}>h" %Ss:%Sh

Enhanced Logging Configuration

# Custom log format with detailed information
logformat detailed %ts.%03tu %6tr %>a %Ss/%03>Hs %<st %rm %ru %[un %Sh/%<a %mt "%{Referer}>h" "%{User-Agent}>h" %{X-Forwarded-For}>h

# Multiple access logs for different purposes
access_log daemon:/var/log/squid/access.log detailed
access_log daemon:/var/log/squid/access-denied.log detailed deny_info
access_log daemon:/var/log/squid/access-cache-hits.log detailed !MISS

# Error page logging
error_log_languages off
error_directory /usr/share/squid/errors/English

# Authentication logging
auth_param basic program /usr/lib/squid/basic_ncsa_auth /etc/squid/passwd
auth_param basic realm Squid Proxy
auth_param basic credentialsttl 2 hours
auth_param basic casesensitive off

# SSL bump logging (for HTTPS inspection)
access_log daemon:/var/log/squid/ssl-bump.log detailed ssl::bump_mode

Cache and Store Logging

# Cache configuration with detailed logging
cache_dir ufs /var/spool/squid 100 16 256
cache_access_log /var/log/squid/cache-access.log
cache_log /var/log/squid/cache.log
cache_store_log /var/log/squid/store.log

# Cache performance logging
logformat cache_perf %ts.%03tu %tr %>a %Ss %03>Hs %<st %rm %ru %Sh %mt

# Memory cache logging
cache_mem 256 MB
maximum_object_size_in_memory 512 KB
memory_cache_shared on

Basic Configuration

Configure the File Stream plugin to monitor Squid logs by creating /etc/logflux-agent/plugins/filestream-squid.toml:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

[filestream.squid_access]
paths = ["/var/log/squid/access.log"]
format = "squid_access"
tags = ["squid", "proxy", "access", "web"]
fields = {
  service = "squid",
  log_type = "access"
}

[filestream.squid_cache]
paths = ["/var/log/squid/cache.log"]
format = "squid_cache"
tags = ["squid", "proxy", "cache"]
fields = {
  service = "squid",
  log_type = "cache"
}

[filestream.squid_store]
paths = ["/var/log/squid/store.log"]
format = "squid_store"
tags = ["squid", "proxy", "store"]
fields = {
  service = "squid",
  log_type = "store"
}

[filestream.squid_denied]
paths = ["/var/log/squid/access-denied.log"]
format = "squid_access"
tags = ["squid", "proxy", "denied", "security"]
fields = {
  service = "squid",
  log_type = "access_denied"
}

Squid Log Formats

Standard Access Log Format

1
2
3
4
5
6
7

[filestream.squid_access_standard]
paths = ["/var/log/squid/access.log"]
format = "regex"
regex = '^(?P<timestamp>\d+\.\d+)\s+(?P<response_time>\d+)\s+(?P<client_ip>\S+)\s+(?P<result_code>\w+)/(?P<http_status>\d+)\s+(?P<bytes>\d+)\s+(?P<request_method>\w+)\s+(?P<url>\S+)\s+(?P<user>\S+)\s+(?P<hierarchy_code>\w+)/(?P<server_ip>\S+)\s+(?P<content_type>\S+)$'
parse_timestamp = true
timestamp_field = "timestamp"
timestamp_format = "unix_decimal"

Detailed Access Log Format

1
2
3
4
5
6
7

[filestream.squid_access_detailed]
paths = ["/var/log/squid/access.log"]
format = "regex"
regex = '^(?P<timestamp>\d+\.\d+)\s+(?P<response_time>\d+)\s+(?P<client_ip>\S+)\s+(?P<result_code>\w+)/(?P<http_status>\d+)\s+(?P<bytes>\d+)\s+(?P<request_method>\w+)\s+(?P<url>\S+)\s+(?P<user>\S+)\s+(?P<hierarchy_code>\w+)/(?P<server_ip>\S+)\s+(?P<content_type>\S+)\s+"(?P<referer>[^"]*)"\s+"(?P<user_agent>[^"]*)"\s+(?P<x_forwarded_for>\S+)$'
parse_timestamp = true
timestamp_field = "timestamp"
timestamp_format = "unix_decimal"

Cache Log Format

1
2
3
4
5
6
7

[filestream.squid_cache_log]
paths = ["/var/log/squid/cache.log"]
format = "regex"
regex = '^(?P<timestamp>\d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2})\|\s*(?P<level>\w+)\|\s*(?P<message>.*)$'
parse_timestamp = true
timestamp_field = "timestamp"
timestamp_format = "2006/01/02 15:04:05"

Store Log Format

1
2
3
4
5
6
7

[filestream.squid_store_log]
paths = ["/var/log/squid/store.log"]
format = "regex"
regex = '^(?P<timestamp>\d+\.\d+)\s+(?P<action>\w+)\s+(?P<swap_file_number>\d+)\s+(?P<hash>\w+)\s+(?P<status_code>\d+)\s+(?P<timestamp_last_mod>\d+)\s+(?P<timestamp_expires>\d+)\s+(?P<timestamp_last_ref>\d+)\s+(?P<swap_file_size>\d+)\s+(?P<reference_count>\d+)\s+(?P<flags>\w+)$'
parse_timestamp = true
timestamp_field = "timestamp"
timestamp_format = "unix_decimal"

Advanced Configuration

Web Traffic Analysis

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

[filestream.squid_traffic_analysis]
paths = ["/var/log/squid/access.log"]
format = "squid_access"
tags = ["squid", "traffic", "analysis"]
fields = {
  service = "squid",
  log_type = "traffic_analysis"
}

# Add traffic analysis metrics
[filestream.squid_traffic_analysis.processors.add_fields]
fields = {
  response_time_seconds = "{{ div .response_time 1000 }}",
  bytes_mb = "{{ div .bytes 1048576 }}",
  is_cache_hit = "{{ if eq .result_code \"TCP_HIT\" }}true{{ else }}false{{ end }}",
  is_error = "{{ if ge .http_status 400 }}true{{ else }}false{{ end }}"
}

Security Monitoring

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

[filestream.squid_security]
paths = ["/var/log/squid/access.log", "/var/log/squid/access-denied.log"]
format = "squid_access"
tags = ["squid", "security", "monitoring"]
fields = {
  service = "squid",
  log_type = "security"
}

# Filter for security-related events
[filestream.squid_security.processors.grep]
patterns = [
  "TCP_DENIED",
  "TCP_MISS/403",
  "TCP_MISS/404",
  "CONNECT",
  "result_code:TCP_DENIED",
  "http_status:40[0-9]",
  "http_status:50[0-9]"
]

Cache Performance Monitoring

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

[filestream.squid_cache_performance]
paths = ["/var/log/squid/access.log"]
format = "squid_access"
tags = ["squid", "cache", "performance"]
fields = {
  service = "squid",
  log_type = "cache_performance"
}

# Add cache performance metrics
[filestream.squid_cache_performance.processors.add_fields]
fields = {
  cache_hit = "{{ if or (eq .result_code \"TCP_HIT\") (eq .result_code \"TCP_MEM_HIT\") (eq .result_code \"TCP_REFRESH_HIT\") }}1{{ else }}0{{ end }}",
  cache_miss = "{{ if or (eq .result_code \"TCP_MISS\") (eq .result_code \"TCP_CLIENT_REFRESH_MISS\") }}1{{ else }}0{{ end }}",
  bandwidth_saved = "{{ if or (eq .result_code \"TCP_HIT\") (eq .result_code \"TCP_MEM_HIT\") }}{{ .bytes }}{{ else }}0{{ end }}"
}

SSL/HTTPS Traffic Monitoring

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

[filestream.squid_ssl]
paths = ["/var/log/squid/ssl-bump.log"]
format = "squid_access"
tags = ["squid", "ssl", "https"]
fields = {
  service = "squid",
  log_type = "ssl_traffic"
}

# Filter for SSL/HTTPS requests
[filestream.squid_ssl.processors.grep]
patterns = [
  "CONNECT",
  "443",
  "ssl",
  "https"
]

Bandwidth and Usage Analysis

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

[filestream.squid_bandwidth]
paths = ["/var/log/squid/access.log"]
format = "squid_access"
tags = ["squid", "bandwidth", "usage"]
fields = {
  service = "squid",
  log_type = "bandwidth"
}

# Add bandwidth calculations
[filestream.squid_bandwidth.processors.add_fields]
fields = {
  bytes_kb = "{{ div .bytes 1024 }}",
  bytes_mb = "{{ div .bytes 1048576 }}",
  bandwidth_class = "{{ if gt .bytes 10485760 }}large{{ else if gt .bytes 1048576 }}medium{{ else }}small{{ end }}"
}

Usage Examples

Monitor Proxy Traffic

1
2
3
4
5
6
7
8

# Stream all Squid logs
logflux-cli stream --filter 'service:squid'

# Monitor access logs only
logflux-cli stream --filter 'service:squid AND log_type:access'

# Track denied requests
logflux-cli stream --filter 'service:squid AND log_type:access_denied'

Performance Analysis

1
2
3
4
5
6
7
8

# Monitor cache hit ratio
logflux-cli stream --filter 'service:squid AND is_cache_hit:true'

# Track slow responses (>5 seconds)
logflux-cli stream --filter 'service:squid AND response_time_seconds:>5'

# Monitor large downloads (>10MB)
logflux-cli stream --filter 'service:squid AND bytes_mb:>10'

Security Monitoring

1
2
3
4
5
6
7
8

# Track blocked requests
logflux-cli stream --filter 'service:squid AND result_code:TCP_DENIED'

# Monitor error responses
logflux-cli stream --filter 'service:squid AND is_error:true'

# Track CONNECT requests (potential tunneling)
logflux-cli stream --filter 'service:squid AND request_method:CONNECT'

Traffic Analysis

1
2
3
4
5
6
7
8

# Monitor top client IPs
logflux-cli stream --filter 'service:squid' --aggregate 'client_ip'

# Track bandwidth usage by client
logflux-cli stream --filter 'service:squid AND bytes_mb:>1'

# Monitor SSL/HTTPS traffic
logflux-cli stream --filter 'service:squid AND log_type:ssl_traffic'

Squid Metrics Collection

Cache Statistics

Create cache statistics collection script:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

#!/bin/bash
# Collect Squid cache statistics
SQUID_CACHE_MGR="http://localhost:3128/squid-internal-mgr/"

collect_cache_stats() {
    local timestamp=$(date -u +%Y-%m-%dT%H:%M:%S.000Z)
    
    # Cache manager info (requires configuration)
    squidclient -h localhost -p 3128 mgr:info 2>/dev/null | \
    awk -v ts="$timestamp" '
    /Storage LRU Expiration Age:/ { lru_age = $5 }
    /Storage Swap size:/ { swap_size = $4 }
    /Storage Swap capacity:/ { swap_capacity = $4 }
    /Mean Object Size:/ { mean_object_size = $4 }
    /Requests given to unlinkd:/ { unlink_requests = $5 }
    END {
        printf "{\"timestamp\": \"%s\", \"lru_age\": \"%s\", \"swap_size\": \"%s\", \"swap_capacity\": \"%s\", \"mean_object_size\": \"%s\", \"unlink_requests\": \"%s\", \"metric_type\": \"cache_stats\"}\n",
               ts, lru_age, swap_size, swap_capacity, mean_object_size, unlink_requests
    }' >> /var/log/squid/cache-stats.log
}

while true; do
    collect_cache_stats
    sleep 60
done

Client Connection Statistics

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

#!/bin/bash
# Monitor active client connections
while true; do
    squidclient -h localhost -p 3128 mgr:client_list 2>/dev/null | \
    awk -v ts="$(date -u +%Y-%m-%dT%H:%M:%S.000Z)" '
    /^Address/ { next }
    NF > 0 {
        printf "{\"timestamp\": \"%s\", \"client_ip\": \"%s\", \"requests\": %s, \"bytes_in\": %s, \"bytes_out\": %s, \"metric_type\": \"client_connection\"}\n",
               ts, $1, $2, $3, $4
    }' >> /var/log/squid/client-stats.log
    sleep 120
done

Performance Metrics

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

#!/bin/bash
# Collect Squid performance metrics
collect_performance() {
    local timestamp=$(date -u +%Y-%m-%dT%H:%M:%S.000Z)
    
    squidclient -h localhost -p 3128 mgr:counters 2>/dev/null | \
    awk -v ts="$timestamp" '
    /client_http.requests =/ { client_requests = $3 }
    /client_http.hits =/ { client_hits = $3 }
    /client_http.errors =/ { client_errors = $3 }
    /server.all.requests =/ { server_requests = $3 }
    /server.all.errors =/ { server_errors = $3 }
    /swap.outs =/ { swap_outs = $3 }
    /swap.ins =/ { swap_ins = $3 }
    END {
        hit_ratio = (client_requests > 0) ? (client_hits / client_requests * 100) : 0
        printf "{\"timestamp\": \"%s\", \"client_requests\": %s, \"client_hits\": %s, \"client_errors\": %s, \"server_requests\": %s, \"server_errors\": %s, \"hit_ratio\": %.2f, \"swap_outs\": %s, \"swap_ins\": %s, \"metric_type\": \"performance\"}\n",
               ts, client_requests, client_hits, client_errors, server_requests, server_errors, hit_ratio, swap_outs, swap_ins
    }' >> /var/log/squid/performance-stats.log
}

while true; do
    collect_performance
    sleep 300  # Every 5 minutes
done

Monitoring and Alerting

Key Metrics to Monitor

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

# Low cache hit ratio alert
[alerts.squid_low_cache_hit_ratio]
query = "service:squid AND metric_type:performance AND hit_ratio:<60"
threshold = 1
window = "5m"
message = "Squid cache hit ratio is low: {{ .hit_ratio }}%"

# High error rate alert
[alerts.squid_high_error_rate]
query = "service:squid AND is_error:true"
threshold = 50
window = "1m"
message = "High error rate in Squid proxy"

# Large number of denied requests
[alerts.squid_high_deny_rate]
query = "service:squid AND result_code:TCP_DENIED"
threshold = 20
window = "1m"
message = "High number of denied requests in Squid"

# Slow response time alert
[alerts.squid_slow_response]
query = "service:squid AND response_time_seconds:>10"
threshold = 10
window = "2m"
message = "Squid slow response times detected"

# Cache storage full alert
[alerts.squid_cache_storage_full]
query = "service:squid AND message:store AND (message:full OR message:space)"
threshold = 1
window = "30s"
message = "Squid cache storage approaching capacity"

# SSL bump errors
[alerts.squid_ssl_errors]
query = "service:squid AND log_type:ssl_traffic AND http_status:>=400"
threshold = 10
window = "2m"
message = "SSL bump errors detected in Squid"

Dashboard Metrics

Monitor these key Squid metrics:

  • Cache performance (hit ratio, miss ratio, memory usage)
  • Traffic metrics (requests per second, bytes transferred)
  • Response times (average, 95th percentile, slow requests)
  • Error rates (4xx, 5xx errors, denied requests)
  • Bandwidth usage (total bandwidth, bandwidth saved by caching)
  • Client statistics (active connections, top clients)
  • Security events (blocked requests, malware detection)
  • Storage metrics (cache size, swap usage, disk space)

Troubleshooting

Common Issues

Squid logs not appearing:

1
2
3
4
5
6
7
8

# Check Squid is running
sudo systemctl status squid

# Verify log configuration
sudo grep -E "(access_log|cache_log)" /etc/squid/squid.conf

# Check log file permissions
sudo ls -la /var/log/squid/

Log parsing errors:

1
2
3
4
5
6
7
8

# Check log format
sudo tail -n 10 /var/log/squid/access.log

# Verify log format configuration
sudo grep "logformat" /etc/squid/squid.conf

# Test log rotation
sudo squid -k rotate

Cache performance issues:

1
2
3
4
5
6
7
8

# Check cache directory
sudo du -sh /var/spool/squid

# Verify cache configuration
sudo squid -k parse

# Check cache statistics
squidclient -h localhost -p 3128 mgr:info

Access control problems:

1
2
3
4
5
6
7
8

# Check ACL configuration
sudo grep -E "(acl|http_access)" /etc/squid/squid.conf

# Test access rules
squidclient -h localhost -p 3128 http://example.com

# Check denied requests
sudo tail -f /var/log/squid/access.log | grep DENIED

Best Practices

Performance

  • Configure appropriate cache size based on available disk space
  • Monitor cache hit ratio and tune cache policies
  • Use memory cache for frequently accessed small objects
  • Implement cache hierarchies for large deployments

Security

  • Configure access control lists (ACLs) properly
  • Enable SSL bump for HTTPS inspection when appropriate
  • Monitor blocked requests and update security policies
  • Implement authentication for user accountability

High Availability

  • Configure multiple Squid instances behind a load balancer
  • Implement cache hierarchies with parent/sibling relationships
  • Monitor parent proxy health and implement failover
  • Use cache digests for efficient cache sharing

Log Management

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

# Optimize log rotation for Squid
/var/log/squid/*.log {
    daily
    rotate 30
    missingok
    notifempty
    compress
    delaycompress
    postrotate
        /usr/sbin/squid -k rotate
        systemctl reload logflux-filestream.service > /dev/null 2>&1 || true
    endpostrotate
}

Cache Optimization

Configure cache policies for optimal performance:

# Cache memory settings
cache_mem 1024 MB
maximum_object_size_in_memory 1 MB
memory_cache_shared on

# Disk cache settings
cache_dir ufs /var/spool/squid 10240 16 256
maximum_object_size 100 MB

# Cache replacement policy
cache_replacement_policy heap LFUDA
memory_replacement_policy heap GDSF

Integration Examples

Docker Deployment

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

version: '3.8'
services:
  squid:
    image: ubuntu/squid:latest
    volumes:
      - ./squid.conf:/etc/squid/squid.conf
      - squid_cache:/var/spool/squid
      - squid_logs:/var/log/squid
    ports:
      - "3128:3128"
    command: squid -NYC -d1
    
  logflux-agent:
    image: logflux/agent:latest
    volumes:
      - squid_logs:/var/log/squid:ro
      - ./logflux-config:/etc/logflux-agent/plugins
    depends_on:
      - squid

volumes:
  squid_cache:
  squid_logs:

Kubernetes Deployment

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

apiVersion: apps/v1
kind: Deployment
metadata:
  name: squid-proxy
spec:
  replicas: 2
  selector:
    matchLabels:
      app: squid-proxy
  template:
    metadata:
      labels:
        app: squid-proxy
    spec:
      containers:
      - name: squid
        image: ubuntu/squid:latest
        ports:
        - containerPort: 3128
        volumeMounts:
        - name: squid-config
          mountPath: /etc/squid/squid.conf
          subPath: squid.conf
        - name: squid-cache
          mountPath: /var/spool/squid
        - name: squid-logs
          mountPath: /var/log/squid
      volumes:
      - name: squid-config
        configMap:
          name: squid-config
      - name: squid-cache
        emptyDir:
          sizeLimit: 10Gi
      - name: squid-logs
        emptyDir: {}

Transparent Proxy Configuration

# Transparent proxy setup
http_port 3128 transparent
https_port 3129 transparent ssl-bump cert=/etc/ssl/certs/squid.crt key=/etc/ssl/private/squid.key

# SSL bump configuration
sslcrtd_program /usr/lib/squid/ssl_crtd -s /var/lib/squid/ssl_db -M 4MB
sslcrtd_children 5

# Intercept HTTPS traffic
ssl_bump peek step1
ssl_bump bump all

# Logging for transparent mode
access_log daemon:/var/log/squid/transparent.log detailed

This comprehensive Squid Proxy integration provides real-time web traffic monitoring, cache performance analysis, and security monitoring using LogFlux Agent’s File Stream plugin. The configuration-based approach offers detailed insights into proxy operations, client behavior, and cache efficiency across different Squid deployment scenarios.