Kubernetes Integration

Collect logs from Kubernetes pods automatically with LogFlux Agent

The LogFlux Kubernetes plugin automatically discovers and collects logs from all pods running in your Kubernetes cluster. It provides seamless integration with Kubernetes’ native logging infrastructure and enriches logs with comprehensive pod and cluster metadata.

Overview

The Kubernetes plugin provides:

Automatic Pod Discovery: Detects all pods across all namespaces automatically
Real-time Log Streaming: Streams logs as they’re generated by containers
Rich Metadata Enrichment: Adds pod, container, namespace, and deployment information
Multi-container Support: Handles pods with multiple containers
Namespace Filtering: Include/exclude specific namespaces or pods
Label-based Filtering: Filter based on Kubernetes labels and annotations
Cluster-wide Monitoring: Monitor logs across entire cluster from single agent

Installation

The Kubernetes plugin is included with the LogFlux Agent but disabled by default.

Prerequisites

LogFlux Agent installed (see Installation Guide)
Kubernetes cluster access
Appropriate RBAC permissions for log collection

Enable the Plugin

1
2
3
4
5


# Enable and start the Kubernetes plugin
sudo systemctl enable --now logflux-kubernetes

# Check status
sudo systemctl status logflux-kubernetes

Kubernetes RBAC Setup

Create the required RBAC permissions:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31


# logflux-rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: logflux-agent
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: logflux-agent
rules:
- apiGroups: [""]
  resources: ["pods", "pods/log", "namespaces"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["apps"]
  resources: ["deployments", "replicasets", "daemonsets", "statefulsets"]
  verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: logflux-agent
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: logflux-agent
subjects:
- kind: ServiceAccount
  name: logflux-agent
  namespace: kube-system

Apply the RBAC configuration:

1

kubectl apply -f logflux-rbac.yaml

Configuration

Basic Configuration

Create or edit the Kubernetes plugin configuration:

1

sudo nano /etc/logflux-agent/plugins/kubernetes.yaml

Basic configuration:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29


# Kubernetes Plugin Configuration
kubernetes:
  # Kubernetes API configuration
  kubeconfig: "/etc/kubernetes/admin.conf"  # Path to kubeconfig
  
  # Include all namespaces by default
  include_all_namespaces: true
  
  # Add Kubernetes metadata to logs
  add_metadata: true

# Log processing
batching:
  # Batch size for log entries
  size: 200
  # Flush interval
  interval: 10s

# Filtering (optional)
filters:
  # Include only specific namespaces
  include_namespaces:
    - "production"
    - "staging"
  
  # Exclude system namespaces
  exclude_namespaces:
    - "kube-system"
    - "kube-public"

Advanced Configuration

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108


# Advanced Kubernetes Plugin Configuration
kubernetes:
  # Kubernetes API settings
  kubeconfig: "/etc/kubernetes/admin.conf"
  api_server: "https://kubernetes.default.svc:443"  # Override API server
  
  # Authentication
  service_account_token_file: "/var/run/secrets/kubernetes.io/serviceaccount/token"
  ca_cert_file: "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
  
  # Collection settings
  collect_existing: false    # Only new logs
  follow: true              # Follow log streams
  tail_lines: 100           # Number of existing lines to collect
  
  # Pod discovery settings
  resync_period: 300s       # How often to resync pod list
  retry_attempts: 3
  timeout: 30s

# Namespace and pod filtering
filters:
  # Include specific namespaces
  include_namespaces:
    - "production"
    - "staging"
    - "default"
  
  # Exclude namespaces
  exclude_namespaces:
    - "kube-system"
    - "kube-public"
    - "local-path-storage"
  
  # Include pods by labels
  include_labels:
    - "app.kubernetes.io/managed-by=logflux"
    - "logging.enabled=true"
  
  # Exclude pods by labels
  exclude_labels:
    - "logging.exclude=true"
  
  # Include specific pod name patterns
  include_pods:
    - "^web-.*"
    - "^api-.*"
    - "^worker-.*"
  
  # Exclude pod name patterns
  exclude_pods:
    - "^test-.*"
    - "^debug-.*"

# Multi-line log handling
multiline:
  patterns:
    # Java stack traces
    - pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
      match: after
      negate: true
    - pattern: '^\s+at\s+'
      match: after
    
    # Python stack traces  
    - pattern: '^Traceback'
      match: after
      negate: true
    - pattern: '^\s+'
      match: after

# Performance settings
performance:
  # Buffer size per pod
  buffer_size: 128KB
  # Maximum concurrent pods
  max_pods: 1000
  # Log line size limit
  max_line_size: 32KB
  # Worker threads for log collection
  worker_threads: 10

# Metadata enrichment
metadata:
  # Pod metadata
  pod_name: true
  pod_namespace: true
  pod_uid: true
  pod_ip: true
  
  # Container metadata
  container_name: true
  container_id: true
  container_image: true
  
  # Node metadata
  node_name: true
  node_ip: true
  
  # Kubernetes objects
  include_labels: true
  include_annotations: true
  label_prefix: "k8s."
  annotation_prefix: "k8s.annotation."
  
  # Deployment/ReplicaSet metadata
  deployment_name: true
  replicaset_name: true

Kubernetes Deployment

DaemonSet Deployment

Deploy LogFlux Agent as a DaemonSet to collect logs from all nodes:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86


# logflux-daemonset.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: logflux-agent
  namespace: kube-system
spec:
  selector:
    matchLabels:
      name: logflux-agent
  template:
    metadata:
      labels:
        name: logflux-agent
    spec:
      serviceAccountName: logflux-agent
      containers:
      - name: logflux-agent
        image: logflux/agent:latest
        env:
        - name: LOGFLUX_API_KEY
          valueFrom:
            secretKeyRef:
              name: logflux-secret
              key: api-key
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        volumeMounts:
        - name: config
          mountPath: /etc/logflux-agent
        - name: varlog
          mountPath: /var/log
          readOnly: true
        - name: dockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 500m
            memory: 512Mi
      volumes:
      - name: config
        configMap:
          name: logflux-config
      - name: varlog
        hostPath:
          path: /var/log
      - name: dockercontainers
        hostPath:
          path: /var/lib/docker/containers
      tolerations:
      - operator: Exists
        effect: NoSchedule
---
apiVersion: v1
kind: Secret
metadata:
  name: logflux-secret
  namespace: kube-system
type: Opaque
data:
  api-key: <base64-encoded-api-key>
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: logflux-config
  namespace: kube-system
data:
  agent.yaml: |
    api_key: ${LOGFLUX_API_KEY}
    node: ${NODE_NAME}
  kubernetes.yaml: |
    kubernetes:
      kubeconfig: ""  # Use in-cluster config
      include_all_namespaces: true
      add_metadata: true
    filters:
      exclude_namespaces:
        - "kube-system"
        - "kube-public"

Helm Chart

Create a Helm chart for easier deployment:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35


# values.yaml
image:
  repository: logflux/agent
  tag: latest
  pullPolicy: IfNotPresent

config:
  apiKey: "your-api-key-here"
  
kubernetes:
  includeAllNamespaces: true
  excludeNamespaces:
    - "kube-system"
    - "kube-public"

resources:
  requests:
    cpu: 100m
    memory: 128Mi
  limits:
    cpu: 500m
    memory: 512Mi

tolerations:
  - operator: Exists
    effect: NoSchedule

nodeSelector: {}

rbac:
  create: true

serviceAccount:
  create: true
  name: logflux-agent

Usage Examples

Monitor Production Namespace

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


# production-only.yaml
kubernetes:
  kubeconfig: "/etc/kubernetes/admin.conf"
  include_all_namespaces: false

filters:
  include_namespaces:
    - "production"
  
  include_labels:
    - "app.kubernetes.io/env=production"

Exclude System Components

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# exclude-system.yaml
filters:
  exclude_namespaces:
    - "kube-system"
    - "kube-public"
    - "kube-node-lease"
    - "local-path-storage"
  
  exclude_labels:
    - "app.kubernetes.io/component=system"

Multi-environment Filtering

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


# multi-env.yaml
filters:
  include_namespaces:
    - "production"
    - "staging"
    - "development"
  
  include_labels:
    - "logging.enabled=true"
  
  exclude_pods:
    - "^.*-test-.*$"
    - "^.*-debug-.*$"

Pod Log Metadata

The Kubernetes plugin automatically enriches logs with comprehensive metadata:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28


{
  "timestamp": "2024-01-20T14:30:45.123Z",
  "level": "info", 
  "message": "Request processed successfully",
  "node": "k8s-worker-01",
  "k8s": {
    "pod_name": "web-app-7d4b6c8f9-x5z2k",
    "pod_namespace": "production",
    "pod_uid": "abc123-def456-ghi789",
    "pod_ip": "10.244.1.15",
    "container_name": "web-app",
    "container_id": "docker://abc123def456...",
    "container_image": "myapp:v1.2.3",
    "node_name": "worker-node-1",
    "node_ip": "192.168.1.100",
    "deployment_name": "web-app",
    "replicaset_name": "web-app-7d4b6c8f9",
    "labels": {
      "app": "web-app",
      "version": "v1.2.3",
      "environment": "production"
    },
    "annotations": {
      "deployment.kubernetes.io/revision": "5",
      "prometheus.io/scrape": "true"
    }
  }
}

Performance Optimization

High-Volume Clusters

For large clusters with many pods:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


# high-volume.yaml
performance:
  buffer_size: 1MB
  max_pods: 5000
  max_line_size: 128KB
  worker_threads: 20

batching:
  size: 1000
  interval: 5s

# Reduce metadata collection
metadata:
  include_annotations: false
  include_labels: true

Resource Management

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


# Pod resource limits
resources:
  requests:
    cpu: 200m
    memory: 256Mi
  limits:
    cpu: 1000m
    memory: 1Gi

# JVM settings for Java apps
env:
- name: JAVA_OPTS
  value: "-Xmx768m -XX:MaxMetaspaceSize=256m"

Troubleshooting

Check Plugin Status

1
2
3
4
5
6
7
8


# Check if plugin is running
sudo systemctl status logflux-kubernetes

# View plugin logs
sudo journalctl -u logflux-kubernetes -f

# Check Kubernetes connectivity
kubectl get pods -A

RBAC Issues

1
2
3
4
5
6
7
8


# Check service account permissions
kubectl auth can-i get pods --as=system:serviceaccount:kube-system:logflux-agent

# Check ClusterRoleBinding
kubectl get clusterrolebinding logflux-agent -o yaml

# Test API access
kubectl get pods --as=system:serviceaccount:kube-system:logflux-agent

Pod Discovery Problems

1
2
3
4
5
6
7
8


# Check namespace access
kubectl get namespaces

# Verify label selectors
kubectl get pods -l "logging.enabled=true" -A

# Test pod log access
kubectl logs <pod-name> -n <namespace>

Performance Issues

1
2
3
4
5
6
7
8


# Check memory usage
kubectl top pods -n kube-system | grep logflux

# Monitor log collection rate
sudo journalctl -u logflux-kubernetes | grep "collected"

# Check for throttling
kubectl describe pod -n kube-system | grep -i throttl

Common Error Solutions

Permission Denied:

1
2
3
4
5


# Update RBAC permissions
kubectl apply -f logflux-rbac.yaml

# Restart plugin
sudo systemctl restart logflux-kubernetes

Too Many Pods:

1
2
3
4


# Increase limits in configuration
performance:
  max_pods: 2000
  worker_threads: 15

Memory Issues:

1
2
3
4


# Reduce buffer sizes
performance:
  buffer_size: 256KB
  max_line_size: 16KB

Security Considerations

Service Account Security

Use dedicated service account with minimal permissions:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


# Minimal RBAC
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: logflux-agent-minimal
rules:
- apiGroups: [""]
  resources: ["pods", "pods/log"]
  verbs: ["get", "list", "watch"]
- apiGroups: [""]
  resources: ["namespaces"]
  verbs: ["get", "list"]

Network Policies

Restrict network access:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: logflux-agent-policy
  namespace: kube-system
spec:
  podSelector:
    matchLabels:
      name: logflux-agent
  policyTypes:
  - Egress
  egress:
  - to:
    - namespaceSelector: {}
    ports:
    - protocol: TCP
      port: 443  # Kubernetes API
  - to: []
    ports:
    - protocol: TCP
      port: 443  # LogFlux API

Sensitive Data Protection

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


# Avoid collecting sensitive logs
filters:
  exclude_labels:
    - "security.sensitive=true"
  
  exclude_namespaces:
    - "secrets"
    - "vault"

# Disable environment variable collection
metadata:
  include_env: false

Monitoring and Alerting

Plugin Health Monitoring

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


#!/bin/bash
# /usr/local/bin/check-k8s-plugin.sh

if ! systemctl is-active --quiet logflux-kubernetes; then
    echo "CRITICAL: LogFlux Kubernetes plugin is not running"
    exit 2
fi

# Check if plugin is discovering pods
if ! journalctl -u logflux-kubernetes --since="10 minutes ago" | grep -q "discovered.*pods"; then
    echo "WARNING: No pod discovery in last 10 minutes"
    exit 1
fi

echo "OK: LogFlux Kubernetes plugin is healthy"
exit 0

Kubernetes Events

Monitor Kubernetes events for issues:

1
2
3
4
5


# Watch for LogFlux-related events
kubectl get events -A --field-selector involvedObject.name=logflux-agent -w

# Check for resource issues
kubectl get events -A --field-selector reason=FailedScheduling

Metrics Collection

Set up monitoring for the LogFlux agent:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


# ServiceMonitor for Prometheus
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: logflux-agent
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: logflux-agent
  endpoints:
  - port: metrics
    path: /metrics

Best Practices

Cluster Organization

Use Namespaces: Organize applications in separate namespaces
Label Consistently: Use consistent labeling for filtering
Resource Limits: Set appropriate resource limits for LogFlux pods
Node Selection: Use node selectors for dedicated log collection nodes

Performance

Filter Early: Use namespace and label filters to reduce load
Batch Sizing: Configure appropriate batch sizes for your cluster
Multi-line Handling: Configure patterns for your application stack traces
Buffer Management: Monitor and adjust buffer sizes based on log volume

Security

Minimal RBAC: Use least-privilege RBAC permissions
Network Policies: Restrict network access with NetworkPolicies
Secret Management: Use Kubernetes secrets for API keys
Regular Updates: Keep LogFlux agent updated with security patches

Integration Examples

GitOps with ArgoCD

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


# argocd-application.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: logflux-agent
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/company/k8s-manifests
    targetRevision: HEAD
    path: logflux-agent
  destination:
    server: https://kubernetes.default.svc
    namespace: kube-system
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Istio Service Mesh

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


# istio-sidecar.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: logflux-istio-config
data:
  kubernetes.yaml: |
    kubernetes:
      include_all_namespaces: true
      add_metadata: true
    
    filters:
      include_labels:
        - "service.istio.io/canonical-name"
    
    metadata:
      include_annotations: true
      annotation_prefix: "istio."

Multi-cluster Setup

1
2
3
4
5
6
7
8


# cluster-1.yaml
kubernetes:
  kubeconfig: "/etc/kubeconfig/cluster-1"
  add_metadata: true

metadata:
  cluster_name: "us-west-2"
  environment: "production"

Disclaimer

Kubernetes and the Kubernetes logo are trademarks of The Linux Foundation. LogFlux is not affiliated with, endorsed by, or sponsored by The Linux Foundation or the Kubernetes project. The Kubernetes logo is used solely for identification purposes to indicate compatibility with Kubernetes clusters and container orchestration.

Next Steps

Docker Integration - Container log collection
Nginx Integration - Web server log analysis
LogFlux Inspector - Analyze collected Kubernetes logs
API Reference - Query Kubernetes logs programmatically