OpenTelemetry Collector Exporter

The LogFlux OTel Collector Exporter is a native OpenTelemetry Collector component that sends logs, traces, and metrics to LogFlux with end-to-end encryption. It uses the LogFlux Go SDK for transport, encryption, and retry – no separate agent required.

Features

All three signals – Logs, traces, and metrics in a single exporter
End-to-end encryption – AES-256-GCM via the Go SDK (zero-knowledge)
Automatic retry – Exponential backoff with configurable limits
Non-blocking – Async queue with background workers
Service discovery – Automatic endpoint resolution by zone

Installation

Build a custom collector using the OpenTelemetry Collector Builder (ocb):

1. Create a builder manifest:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
# builder-config.yaml
dist:
  name: logflux-otel-collector
  output_path: ./bin

exporters:
  - gomod: github.com/logflux-io/logflux-otel-collector v1.0.0

receivers:
  - gomod: go.opentelemetry.io/collector/receiver/otlpreceiver v0.115.0

processors:
  - gomod: go.opentelemetry.io/collector/processor/batchprocessor v0.115.0

2. Build the collector:

1
2
3
4
5
# Install the builder
go install go.opentelemetry.io/collector/cmd/builder@latest

# Build
ocb --config builder-config.yaml

3. Run it:

1
./bin/logflux-otel-collector --config collector-config.yaml

Configuration

Basic

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
exporters:
  logflux:
    api_key: "${LOGFLUX_API_KEY}"
    zone: "eu"
    source: "otel-collector"
    environment: "production"

service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [logflux]
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [logflux]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [logflux]

Full Configuration Reference

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
exporters:
  logflux:
    # Required: LogFlux API key
    api_key: "${LOGFLUX_API_KEY}"

    # Zone for automatic endpoint discovery (e.g., "eu", "us")
    # Ignored if endpoint is set
    zone: "eu"

    # Direct endpoint URL (bypasses discovery)
    # endpoint: "https://custom.ingest.logflux.io"

    # Source identifier attached to all entries
    source: "otel-collector"

    # Environment tag
    environment: "production"

    # Gzip compress payloads before encryption (default: true)
    compression: true

    # In-memory buffer capacity (default: 1000)
    queue_size: 1000

    # Entries per HTTP request (default: 100)
    batch_size: 100

    # Background send goroutines (default: 2)
    worker_count: 2

Option	Type	Default	Description
`api_key`	string	(required)	LogFlux API key (`<region>-lf_<key>`)
`zone`	string		Discovery zone (`eu`, `us`). Ignored if `endpoint` is set
`endpoint`	string		Direct endpoint URL (bypasses discovery)
`source`	string	hostname	Source identifier for all entries
`environment`	string		Environment tag (`production`, `staging`, etc.)
`compression`	bool	`true`	Gzip compress before encryption
`queue_size`	int	`1000`	In-memory buffer capacity
`batch_size`	int	`100`	Entries per HTTP request
`worker_count`	int	`2`	Background send goroutines

Signal Mapping

Logs (OTel -> LogFlux Type 1)

OTel Field	LogFlux Field	Notes
`SeverityNumber`	`level`	Mapped: TRACE/DEBUG->8, INFO->7, WARN->5, ERROR->4, FATAL->2
`Body`	`message`	String representation
`Timestamp`	`timestamp`	ISO8601
`Attributes`	`attributes`	All values stringified
`Resource.Attributes`	`meta`	Service name, version, environment

Traces (OTel -> LogFlux Type 3)

OTel Field	LogFlux Field	Notes
`TraceID`	`trace_id`	32-char hex string
`SpanID`	`span_id`	16-char hex string
`ParentSpanID`	`parent_span_id`	16-char hex or empty
`Name`	`operation`	Span name
`StartTimestamp`	`start_time`	ISO8601
`EndTimestamp`	`end_time`	ISO8601
Duration	`duration_ms`	Computed from start/end
`Status`	`status`	`"ok"` or `"error"`
`Attributes`	`attributes`	All values stringified
`Resource.Attributes`	`meta`	Service context

Metrics (OTel -> LogFlux Type 2)

OTel Field	LogFlux Field	Notes
`Name`	`name`	Metric name
`Unit`	`unit`	Metric unit
Data type	`metric_type`	`"gauge"`, `"counter"`, or `"histogram"`
Data point value	`value`	Float64 (gauge value, sum, or histogram sum)
Data point attributes	`attributes`	All values stringified
`Resource.Attributes`	`meta`	Service context

Application Examples

Once the collector is running, point your OpenTelemetry SDKs at it:

Go

1
2
3
4
5
6
7
8
import (
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
)

exporter, _ := otlptracegrpc.New(ctx,
    otlptracegrpc.WithEndpoint("localhost:4317"),
    otlptracegrpc.WithInsecure(),
)

Python

1
2
3
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

exporter = OTLPSpanExporter(endpoint="localhost:4317", insecure=True)

Node.js

1
2
3
4
5
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-grpc');

const exporter = new OTLPTraceExporter({
    url: 'http://localhost:4317',
});

Java

1
2
3
OtlpGrpcSpanExporter exporter = OtlpGrpcSpanExporter.builder()
    .setEndpoint("http://localhost:4317")
    .build();

Deployment

Docker

1
2
3
4
5
6
7
8
9
FROM golang:1.23-alpine AS builder
RUN go install go.opentelemetry.io/collector/cmd/builder@latest
COPY builder-config.yaml .
RUN ocb --config builder-config.yaml

FROM alpine:3.19
COPY --from=builder /go/bin/logflux-otel-collector /usr/local/bin/
COPY collector-config.yaml /etc/otel/config.yaml
ENTRYPOINT ["logflux-otel-collector", "--config", "/etc/otel/config.yaml"]

Kubernetes

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
apiVersion: apps/v1
kind: Deployment
metadata:
  name: logflux-otel-collector
spec:
  replicas: 2
  selector:
    matchLabels:
      app: logflux-otel-collector
  template:
    metadata:
      labels:
        app: logflux-otel-collector
    spec:
      containers:
      - name: collector
        image: your-registry/logflux-otel-collector:latest
        env:
        - name: LOGFLUX_API_KEY
          valueFrom:
            secretKeyRef:
              name: logflux-credentials
              key: api-key
        ports:
        - containerPort: 4317
          name: otlp-grpc
        - containerPort: 4318
          name: otlp-http
        resources:
          requests:
            cpu: 200m
            memory: 256Mi
          limits:
            cpu: 1
            memory: 1Gi
---
apiVersion: v1
kind: Service
metadata:
  name: logflux-otel-collector
spec:
  selector:
    app: logflux-otel-collector
  ports:
  - name: otlp-grpc
    port: 4317
  - name: otlp-http
    port: 4318

Security

All data is encrypted client-side with AES-256-GCM before transmission to LogFlux. The encryption key is negotiated via RSA-2048 OAEP handshake. The LogFlux server never sees plaintext – zero-knowledge architecture.

The exporter uses the LogFlux Go SDK for all cryptographic operations.

Troubleshooting

Collector won’t start:

1
2
3
4
5
# Validate config
./logflux-otel-collector validate --config collector-config.yaml

# Check logs
./logflux-otel-collector --config collector-config.yaml 2>&1 | head -50

No data arriving:

Verify api_key is correct and has the region prefix (e.g., eu-lf_...)
Check the zone matches your LogFlux region
Ensure the collector can reach the LogFlux ingestor endpoint

High memory usage:

Reduce queue_size and batch_size
Add a memory_limiter processor before the exporter
Reduce worker_count

Source Code

GitHub: logflux-io/logflux-otel-collector
License: Elastic License 2.0

Disclaimer

OpenTelemetry and the OpenTelemetry logo are trademarks of The Linux Foundation. LogFlux is not affiliated with, endorsed by, or sponsored by The Linux Foundation or the OpenTelemetry project.

Next Steps

OpenTelemetry Agent Plugin – Use with the LogFlux Agent instead
Go SDK – Direct SDK integration for Go applications
API Reference – Query your data programmatically