Dashboard GitHub

Prometheus Integration

Ingest metrics via Prometheus Remote Write protocol with LogFlux Agent

Prometheus

The LogFlux Prometheus plugin provides a Prometheus Remote Write protocol-compatible endpoint that accepts metrics from Prometheus servers and forwards them to LogFlux as structured log entries. This enables centralized metrics and logs in a single platform while maintaining compatibility with existing Prometheus monitoring infrastructure.

Overview

The Prometheus plugin provides:

  • Remote Write Protocol: Full support for Prometheus Remote Write API v1
  • Snappy Compression: Support for Snappy-compressed payloads
  • Protocol Buffers: Native protobuf message parsing
  • High Throughput: Optimized for high-volume metrics ingestion
  • Metric Transformation: Convert Prometheus metrics to LogFlux log format
  • Label Preservation: Maintain all Prometheus labels as structured metadata
  • Batch Processing: Efficient batching for large metric payloads
  • Authentication: Support for HTTP Basic Auth and custom headers

Installation

The Prometheus plugin is included with the LogFlux Agent but disabled by default.

Enable the Service

1
2
3
4
5

# Enable and start the Prometheus plugin
sudo systemctl enable --now logflux-prometheus

# Check status
sudo systemctl status logflux-prometheus

Configuration

Basic Configuration

Create or edit the Prometheus plugin configuration:

1

sudo nano /etc/logflux-agent/plugins/prometheus.yaml

Configuration Options

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

# Prometheus Remote Write receiver configuration
server:
  host: "0.0.0.0"      # Listen address
  port: 9201           # Default Prometheus Remote Write port
  path: "/api/v1/write" # Remote Write endpoint path

# LogFlux connection
logflux:
  api_key: "your-api-key-here"
  server_url: "/tmp/logflux-agent.sock"
  node: "prometheus-receiver"
  prefix: "PROMETHEUS"

# Processing options
processing:
  batch_size: 1000     # Metrics per batch
  batch_timeout: "5s"  # Maximum batch wait time
  compression: true    # Enable snappy compression
  
# Security
security:
  require_tls: false   # Enable TLS (recommended for production)
  cert_file: ""        # TLS certificate path
  key_file: ""         # TLS private key path

Prometheus Configuration

Configure your Prometheus server to send metrics to LogFlux:

prometheus.yml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

global:
  scrape_interval: 15s

remote_write:
  - url: "http://your-logflux-agent:9201/api/v1/write"
    # Optional: Basic authentication
    basic_auth:
      username: "logflux"
      password: "your-api-key"
    
    # Optional: Custom headers
    headers:
      X-LogFlux-Node: "prometheus-metrics"
    
    # Retry configuration
    queue_config:
      capacity: 10000
      max_samples_per_send: 1000
      batch_send_deadline: 5s
      max_retries: 3
      min_backoff: 30ms
      max_backoff: 100ms

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

Usage Examples

Send Test Metrics

Test the integration using curl:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# Create a sample metric payload
cat > sample_metrics.pb << 'EOF'
# This would contain protobuf-encoded Prometheus metrics
EOF

# Send to LogFlux Prometheus endpoint
curl -X POST \
  -H "Content-Type: application/x-protobuf" \
  -H "Content-Encoding: snappy" \
  --data-binary @sample_metrics.pb \
  http://localhost:9201/api/v1/write

Query Ingested Metrics

Metrics ingested through the Prometheus plugin appear in LogFlux with the configured prefix:

1
2
3
4
5

# Search for Prometheus metrics
logflux-inspector search --prefix PROMETHEUS --since 1h

# Filter by metric name
logflux-inspector search --contains "prometheus_build_info" --since 30m

Metric Format

Prometheus metrics are transformed into LogFlux log entries:

Input (Prometheus)

prometheus_http_requests_total{method="GET",handler="/metrics"} 142

Output (LogFlux)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

{
  "timestamp": "2024-09-07T01:30:00Z",
  "level": "info",
  "node": "prometheus-receiver",
  "prefix": "PROMETHEUS",
  "message": "prometheus_http_requests_total{method=\"GET\",handler=\"/metrics\"} 142",
  "fields": {
    "metric_name": "prometheus_http_requests_total",
    "metric_value": 142,
    "metric_type": "counter",
    "labels": {
      "method": "GET",
      "handler": "/metrics"
    }
  }
}

Security Configuration

TLS Setup

For production deployments, enable TLS:

1
2
3
4

security:
  require_tls: true
  cert_file: "/etc/logflux-agent/certs/prometheus.crt"
  key_file: "/etc/logflux-agent/certs/prometheus.key"

Generate certificates:

1
2
3
4
5
6
7

# Generate self-signed certificate (for testing)
openssl req -x509 -newkey rsa:4096 -keyout prometheus.key -out prometheus.crt -days 365 -nodes

# Move to LogFlux directory
sudo mkdir -p /etc/logflux-agent/certs
sudo mv prometheus.{crt,key} /etc/logflux-agent/certs/
sudo chown logflux:logflux /etc/logflux-agent/certs/prometheus.*

Authentication

The plugin supports multiple authentication methods:

1
2
3
4
5
6
7
8
9

security:
  # API key in header
  api_key_header: "X-API-Key"
  
  # Basic authentication
  basic_auth:
    enabled: true
    username: "logflux"
    password: "your-secure-password"

Monitoring and Troubleshooting

Check Plugin Status

1
2
3
4
5
6
7
8

# View plugin logs
sudo journalctl -u logflux-prometheus -f

# Check plugin health
curl http://localhost:9201/health

# View configuration
sudo cat /etc/logflux-agent/plugins/prometheus.yaml

Common Issues

Port Already in Use

1
2
3
4

# Check what's using port 9201
sudo netstat -tlnp | grep 9201

# Use a different port if needed

High Memory Usage

1
2
3
4

# Reduce batch size in configuration
processing:
  batch_size: 500      # Reduce from default 1000
  batch_timeout: "3s"  # Reduce timeout

Connection Refused

1
2
3
4
5

# Verify LogFlux Agent is running
sudo systemctl status logflux-agent

# Check Unix socket permissions
ls -la /tmp/logflux-agent.sock

Performance Tuning

High-Volume Deployments

For environments with high metric volume:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

processing:
  batch_size: 2000         # Increase batch size
  batch_timeout: "10s"     # Longer timeout for larger batches
  worker_pool_size: 4      # Multiple processing workers
  buffer_size: 50000       # Increase internal buffer

server:
  read_timeout: "30s"      # Longer read timeout
  write_timeout: "30s"     # Longer write timeout
  max_concurrent: 100      # Max concurrent connections

Resource Limits

Set appropriate system limits:

1
2
3
4

# /etc/systemd/system/logflux-prometheus.service.d/override.conf
[Service]
LimitNOFILE=65536
LimitNPROC=4096

Integration Patterns

Grafana Dashboards

Create dashboards using metrics ingested via LogFlux:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

{
  "dashboard": {
    "title": "Prometheus Metrics via LogFlux",
    "panels": [
      {
        "title": "HTTP Request Rate",
        "type": "graph",
        "datasource": "LogFlux",
        "targets": [
          {
            "query": "prefix:PROMETHEUS AND prometheus_http_requests_total",
            "metric": "rate"
          }
        ]
      }
    ]
  }
}

Alerting Rules

Set up alerts based on ingested metrics:

1
2
3
4
5
6
7

# LogFlux alert configuration
alerts:
  - name: "high_error_rate"
    query: "prefix:PROMETHEUS AND prometheus_http_requests_total AND status:5xx"
    threshold: 100
    window: "5m"
    notification: "slack"

Best Practices

Configuration

  1. Use specific node names for different Prometheus instances
  2. Enable compression for better network efficiency
  3. Set appropriate batch sizes based on metric volume
  4. Use TLS in production environments

Monitoring

  1. Monitor plugin resource usage with system metrics
  2. Set up alerts for plugin failures
  3. Regularly check logs for error patterns
  4. Monitor ingestion latency and throughput

Security

  1. Use API key authentication for all connections
  2. Enable TLS encryption for data in transit
  3. Restrict network access to the plugin port
  4. Rotate credentials regularly

Troubleshooting Guide

Plugin Won’t Start

1
2
3
4
5
6
7
8

# Check configuration syntax
sudo logflux-prometheus --config /etc/logflux-agent/plugins/prometheus.yaml --validate

# Check port availability
sudo netstat -tlnp | grep 9201

# Review system logs
sudo journalctl -u logflux-prometheus --since "1 hour ago"

Metrics Not Appearing

1
2
3
4
5
6
7
8

# Verify Prometheus is sending data
curl -X GET http://localhost:9201/metrics

# Check LogFlux Agent connectivity
sudo systemctl status logflux-agent

# Search for recent metrics
logflux-inspector search --prefix PROMETHEUS --since 5m

High Latency

  1. Increase batch size to process more metrics per operation
  2. Add more worker threads if CPU usage is low
  3. Check network connectivity between Prometheus and LogFlux
  4. Monitor disk I/O if using persistent storage

Disclaimer

Prometheus and the Prometheus logo are trademarks of The Linux Foundation. LogFlux is not affiliated with, endorsed by, or sponsored by The Linux Foundation or the Prometheus project. The Prometheus logo is used solely for identification purposes to indicate compatibility with the Prometheus Remote Write protocol.

Next Steps