journald Integration

Collect system service logs via systemd journald with LogFlux Agent

The LogFlux journald integration provides deep integration with systemd’s journal daemon (journald) to collect structured system and service logs. This plugin leverages systemd’s native logging infrastructure to provide comprehensive visibility into system events, service states, and application logs.

Overview

The journald plugin provides:

Native Journal Integration: Direct access to systemd journal via journalctl
Structured Data: Always uses JSON output for rich metadata extraction
Service Filtering: Filter logs by specific systemd units and services
Priority Levels: Map systemd priorities to LogFlux log levels
Rich Metadata: Extract PIDs, UIDs, hostnames, boot IDs, and more
Real-time Monitoring: Follow journal entries in real-time
Historical Access: Access historical logs with flexible time ranges
Batch Processing: Efficient batching for high-volume log collection

Installation

The journald plugin is included with the LogFlux Agent but disabled by default.

Prerequisites

LogFlux Agent installed (see Installation Guide)
systemd-based Linux distribution
journalctl command available
Appropriate permissions to read system journal

Enable the Plugin

1
2
3
4
5


# Enable and start the journald plugin
sudo systemctl enable --now logflux-journald

# Check status
sudo systemctl status logflux-journald

Configuration

Basic Configuration

Create or edit the journald plugin configuration:

1

sudo nano /etc/logflux-agent/plugins/journald.yaml

Basic configuration:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37


# journald Plugin Configuration
journal:
  # Follow journal in real-time
  follow: true
  
  # Output format (always JSON for structured data)
  output: "json"
  
  # Time range for initial collection
  since: "1 hour ago"
  
  # Priority filter (optional)
  priority: ""  # empty = all levels

# Agent connection
agent:
  socket_path: "/tmp/logflux-agent.sock"
  timeout: "30s"
  
# Batch processing
batching:
  size: 100
  flush_interval: "5s"
  max_wait: "30s"

# Metadata extraction
metadata:
  include_all: true
  extract_fields:
    - "unit"
    - "hostname" 
    - "syslog_identifier"
    - "pid"
    - "uid"
    - "executable"
    - "boot_id"
    - "machine_id"

Advanced Configuration

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107


# Advanced journald Plugin Configuration
journal:
  # Real-time following
  follow: true
  output: "json"
  
  # Time filtering
  since: "boot"          # Since last boot
  until: ""              # Until specific time (optional)
  
  # Unit filtering
  units:
    - "nginx.service"
    - "postgresql.service"
    - "docker.service"
  
  # Priority filtering
  priority: "info"       # debug, info, notice, warning, err, crit, alert, emerg
  
  # Kernel messages
  dmesg: false          # Include kernel messages
  
  # System vs user journal
  system: true          # System journal
  user: false           # User journal
  
  # Journal directory (optional)
  directory: "/var/log/journal"

# Advanced agent settings
agent:
  socket_path: "/tmp/logflux-agent.sock"
  timeout: "60s"
  retry_attempts: 3
  retry_delay: "10s"
  
  # Connection pooling
  pool_size: 5
  keep_alive: true

# Enhanced batching
batching:
  size: 500
  flush_interval: "10s"
  max_wait: "60s"
  
  # Memory limits
  max_memory: "100MB"
  compression: true

# Metadata processing
metadata:
  include_all: true
  
  # Field mapping
  field_mapping:
    "systemd_unit": "unit"
    "process_id": "pid"
    "user_id": "uid"
    "group_id": "gid"
    "hostname": "host"
    "executable_path": "executable"
    "boot_session": "boot_id"
    "machine_identifier": "machine_id"
  
  # Additional labels
  labels:
    source_type: "plugin"
    source_name: "journald"
    collection_method: "systemd"

# Filtering and processing
filtering:
  # Exclude noisy services
  exclude_units:
    - "systemd-networkd.service"
    - "systemd-resolved.service"
  
  # Message filtering
  exclude_patterns:
    - "^Started Session \\d+ of user"
    - "^Removed session"
  
  # Priority remapping
  priority_mapping:
    "emerg": "critical"
    "alert": "critical" 
    "crit": "error"
    "err": "error"
    "warning": "warn"
    "notice": "info"
    "info": "info"
    "debug": "debug"

# Performance tuning
performance:
  # Journal reading
  max_entries_per_read: 1000
  read_timeout: "30s"
  
  # Processing workers
  worker_count: 2
  queue_size: 5000
  
  # Resource limits
  max_cpu_percent: 10
  max_memory_mb: 256

Usage Examples

Basic Service Monitoring

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


# Monitor specific service
sudo systemctl enable --now logflux-journald
sudo journalctl -u logflux-journald -f

# Configuration for web services
# journald.yaml
journal:
  follow: true
  units:
    - "nginx.service"
    - "apache2.service"
  priority: "warning"

System-wide Monitoring

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


# Monitor all system services
journal:
  follow: true
  system: true
  since: "boot"
  priority: "info"

filtering:
  # Focus on important services
  exclude_units:
    - "systemd-*.service"
  exclude_patterns:
    - "^Starting.*"
    - "^Started.*"
    - "^Stopping.*"
    - "^Stopped.*"

Application-specific Collection

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


# Monitor application services
journal:
  units:
    - "myapp.service"
    - "myapp-worker.service"
    - "myapp-scheduler.service"
  priority: "debug"  # Include all levels

metadata:
  labels:
    application: "myapp"
    environment: "production"

Metadata Extraction

Available Journal Fields

The plugin automatically extracts rich metadata from systemd journal:

Field	Description	Example
`_SYSTEMD_UNIT`	Systemd unit name	`nginx.service`
`_HOSTNAME`	System hostname	`web-server-01`
`_PID`	Process ID	`1234`
`_UID`	User ID	`1000`
`_GID`	Group ID	`1000`
`_EXECUTABLE`	Executable path	`/usr/sbin/nginx`
`_BOOT_ID`	Boot session ID	`a1b2c3d4...`
`_MACHINE_ID`	Machine identifier	`e5f6g7h8...`
`_TRANSPORT`	Transport method	`journal`
`SYSLOG_IDENTIFIER`	Syslog identifier	`nginx`
`PRIORITY`	Log priority	`6`
`MESSAGE`	Log message	`Server started`

LogFlux Output Format

Journal entries are transformed into structured LogFlux logs:

Input Journal Entry:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


{
  "__REALTIME_TIMESTAMP": "1640995850000000",
  "_SYSTEMD_UNIT": "nginx.service",
  "_HOSTNAME": "web-server-01",
  "_PID": "1234",
  "PRIORITY": "6",
  "MESSAGE": "HTTP request completed",
  "SYSLOG_IDENTIFIER": "nginx",
  "_EXECUTABLE": "/usr/sbin/nginx"
}

Output LogFlux Log:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


{
  "timestamp": "2024-01-20T14:30:50.000Z",
  "level": "info",
  "message": "HTTP request completed",
  "node": "system",
  "metadata": {
    "source_type": "plugin",
    "source_name": "journald",
    "unit": "nginx.service",
    "hostname": "web-server-01",
    "pid": 1234,
    "syslog_identifier": "nginx",
    "executable": "/usr/sbin/nginx",
    "priority": 6,
    "boot_id": "a1b2c3d4...",
    "machine_id": "e5f6g7h8..."
  }
}

Service Management

Plugin Control Commands

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


# Start journald collection
sudo systemctl start logflux-journald

# Enable automatic startup
sudo systemctl enable logflux-journald

# Check plugin status
sudo systemctl status logflux-journald

# View plugin logs
sudo journalctl -u logflux-journald -f

# Stop collection
sudo systemctl stop logflux-journald

# Restart with new configuration
sudo systemctl restart logflux-journald

Manual Testing

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


# Test plugin directly
sudo logflux-agent-journald -config /etc/logflux-agent/plugins/journald.yaml

# Filter specific service
sudo logflux-agent-journald -unit nginx.service

# Show recent errors only
sudo logflux-agent-journald -priority err -since "1 hour ago"

# Follow real-time logs
sudo logflux-agent-journald -follow

Common Use Cases

Web Server Monitoring

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


# nginx/apache monitoring
journal:
  units:
    - "nginx.service"
    - "apache2.service"
  priority: "warning"

filtering:
  exclude_patterns:
    - "^\\[notice\\]"  # Exclude nginx notice messages

metadata:
  labels:
    service_type: "web_server"

Database Monitoring

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


# Database service monitoring
journal:
  units:
    - "postgresql.service"
    - "mysql.service"
    - "redis.service"
  priority: "info"

metadata:
  labels:
    service_type: "database"
    tier: "data"

Container Runtime Monitoring

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


# Docker/containerd monitoring
journal:
  units:
    - "docker.service"
    - "containerd.service"
    - "kubelet.service"
  priority: "warning"

filtering:
  exclude_patterns:
    - "^container .* died"  # Exclude normal container exits

metadata:
  labels:
    service_type: "container_runtime"

System Service Monitoring

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


# Critical system services
journal:
  units:
    - "sshd.service"
    - "chronyd.service"
    - "NetworkManager.service"
  priority: "warning"

metadata:
  labels:
    service_type: "system"
    criticality: "high"

Security and Permissions

Journal Access Permissions

1
2
3
4
5
6
7
8


# Add logflux user to systemd-journal group
sudo usermod -a -G systemd-journal logflux

# Verify permissions
sudo journalctl --verify

# Check journal files
ls -la /var/log/journal/

Privilege Configuration

1
2
3
4
5
6
7


# systemd service file override
sudo systemctl edit logflux-journald

# Add capability for journal access
[Service]
CapabilityBoundingSet=CAP_DAC_READ_SEARCH CAP_SYSLOG
AmbientCapabilities=CAP_DAC_READ_SEARCH CAP_SYSLOG

Security Best Practices

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


# Secure configuration
journal:
  # Limit access to system journal only
  user: false
  system: true
  
filtering:
  # Exclude sensitive information
  exclude_patterns:
    - "password"
    - "token"
    - "key"
    - "secret"

metadata:
  # Sanitize executable paths
  sanitize_paths: true
  exclude_fields:
    - "_EXECUTABLE"  # May contain sensitive paths

Performance Optimization

High-Volume Configuration

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


# High-throughput configuration
journal:
  follow: true
  output: "json"

batching:
  size: 2000
  flush_interval: "30s"
  max_memory: "500MB"
  compression: true

performance:
  worker_count: 4
  queue_size: 20000
  max_entries_per_read: 5000

Resource Management

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


# Resource-constrained environments
journal:
  follow: true
  priority: "warning"  # Only warnings and above

batching:
  size: 100
  flush_interval: "10s"
  max_memory: "50MB"

performance:
  worker_count: 1
  max_cpu_percent: 5
  max_memory_mb: 128

Monitoring and Alerting

Plugin Health Monitoring

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


#!/bin/bash
# check-journald-plugin.sh

if ! systemctl is-active --quiet logflux-journald; then
    echo "CRITICAL: LogFlux journald plugin is not running"
    exit 2
fi

# Check if plugin is processing logs
if ! journalctl -u logflux-journald --since="5 minutes ago" | grep -q "entries processed"; then
    echo "WARNING: No log processing detected in last 5 minutes"
    exit 1
fi

echo "OK: LogFlux journald plugin is healthy"
exit 0

Journal Health Checks

1
2
3
4
5
6
7
8
9


# Check journal integrity
sudo journalctl --verify

# Monitor journal disk usage
sudo journalctl --disk-usage

# Check journal rotation
sudo journalctl --rotate
sudo journalctl --vacuum-time=7d

Troubleshooting

Common Issues

Permission Denied:

1
2
3
4
5


# Add user to systemd-journal group
sudo usermod -a -G systemd-journal logflux

# Verify group membership
groups logflux

High CPU Usage:

1
2
3
4
5
6
7


# Reduce processing load
journal:
  priority: "warning"  # Only important messages

performance:
  max_cpu_percent: 5
  worker_count: 1

Memory Issues:

1
2
3
4
5
6
7
8


# Limit memory usage
batching:
  max_memory: "50MB"
  size: 50
  
filtering:
  exclude_patterns:
    - ".*"  # Very aggressive filtering if needed

Missing Journal Entries:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


# Check journal service
sudo systemctl status systemd-journald

# Verify journal storage
sudo ls -la /var/log/journal/

# Check persistent storage
sudo systemctl edit systemd-journald
# Add:
[Service]
Environment=Storage=persistent

Debugging

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


# Enable debug logging
sudo systemctl edit logflux-journald
# Add:
[Service]
Environment="LOGFLUX_LOG_LEVEL=debug"

# Test journalctl access
sudo -u logflux journalctl --no-pager -n 10

# Monitor real-time collection
sudo journalctl -u logflux-journald -f

Integration Examples

Kubernetes Node Monitoring

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


# Monitor kubelet and container runtime
journal:
  units:
    - "kubelet.service"
    - "docker.service"
    - "containerd.service"
  priority: "info"

metadata:
  labels:
    node_type: "kubernetes"
    cluster: "production"

Database Cluster Monitoring

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


# PostgreSQL cluster monitoring
journal:
  units:
    - "postgresql@main.service"
    - "postgresql@replica.service"
  priority: "notice"

filtering:
  exclude_patterns:
    - "^LOG:  duration:"  # Exclude query duration logs

metadata:
  labels:
    database_type: "postgresql"
    cluster_role: "primary"

Web Application Stack

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


# Full web stack monitoring
journal:
  units:
    - "nginx.service"
    - "php8.1-fpm.service"
    - "mysql.service"
    - "redis.service"
  priority: "warning"

metadata:
  labels:
    stack: "lamp"
    application: "webstore"

Best Practices

Configuration Management

Service-specific configs: Create separate configs for different service groups
Priority filtering: Use appropriate priority levels to reduce noise
Unit filtering: Monitor only relevant services to improve performance
Batch sizing: Optimize batch sizes based on log volume

Performance

Resource limits: Set appropriate CPU and memory limits
Worker scaling: Scale workers based on log volume
Priority filtering: Use priority filtering to reduce processing load
Compression: Enable compression for high-volume environments

Security

Least privilege: Use minimal required permissions
Field sanitization: Exclude sensitive fields from metadata
Pattern filtering: Filter out sensitive information patterns
Access controls: Restrict journal access appropriately

Monitoring

Health checks: Monitor plugin and journal daemon health
Resource usage: Track CPU, memory, and disk usage
Log volume: Monitor log ingestion rates and patterns
Error tracking: Set up alerts for plugin errors

Disclaimer

systemd and the systemd logo are trademarks of The systemd project. LogFlux is not affiliated with, endorsed by, or sponsored by the systemd project or its maintainers. The systemd logo is used solely for identification purposes to indicate compatibility with systemd’s journald service.

Next Steps

Docker Integration - Container log collection
Kubernetes Integration - Pod log monitoring
LogFlux Inspector - Analyze collected system logs
API Reference - Query system logs programmatically