Dashboard GitHub

Architecture Overview

Understanding the LogFlux architecture and its core components

LogFlux is designed as a distributed log management system with three main components that work together to provide comprehensive log collection, storage, and analysis capabilities.

System Architecture

The LogFlux architecture consists of three primary components:

┌─────────────────────────────────┐     ┌─────────────────────────────────┐
│         COLLECTOR               │     │      INGESTOR/BACKEND           │
│                                 │     │     (Hosted by LogFlux)         │
├─────────────────────────────────┤     ├─────────────────────────────────┤
│ • CLI Tool                      │     │ • Ingestor Service              │
│ • SDKs (Go, Python, JS, etc.)   │────▶│ • Backend Service               │
│ • Direct API Integration        │     │ • Data Storage                  │
│                                 │     │ • Authentication                │
└─────────────────────────────────┘     └─────────────────────────────────┘
                                                      │
                                                      ▼
                                        ┌─────────────────────────────────┐
                                        │         INSPECTOR               │
                                        ├─────────────────────────────────┤
                                        │ • CLI (Command Line)            │
                                        │ • TUI (Terminal UI)             │
                                        │ • GUI (Desktop App)             │
                                        └─────────────────────────────────┘

Component Overview

1. Collector - Log Collection Layer

The Collector is responsible for gathering logs from your applications and infrastructure. It provides multiple integration options to suit different use cases:

Collection Methods

  • CLI Tool: A command-line utility for collecting logs from files, system logs, or piping output
  • SDKs: Native libraries for popular programming languages
    • Go SDK
    • Python SDK
    • JavaScript/Node.js SDK
    • Java SDK (coming soon)
  • Direct API Integration: RESTful API for custom implementations

Key Features

  • Flexible Integration: Choose the method that best fits your stack
  • Batching: Automatically batches log entries for efficient transmission
  • Retry Logic: Built-in retry mechanisms for network failures
  • Local Buffering: Temporarily stores logs during network outages
  • Minimal Overhead: Designed for low resource consumption

2. Ingestor/Backend - Core Services

The Ingestor and Backend services form the heart of LogFlux and are fully managed by us. This component handles all the heavy lifting of log processing and storage.

Ingestor Service

  • High-Performance Ingestion: Handles thousands of log entries per second
  • Data Validation: Ensures log integrity and format compliance
  • Rate Limiting: Protects the system from abuse
  • Geographic Distribution: Available in EU and US regions for data residency
  • Customer-Specific URLs: Dedicated subdomains for enhanced security and isolation

Backend Service

  • Log Storage: Efficient storage with compression and indexing
  • Query Engine: Fast full-text and structured search capabilities
  • Retention Management: Automatic data lifecycle management
  • Access Control: Multi-tenant architecture with secure data isolation

Benefits of Hosted Services

  • Zero Maintenance: We handle all updates, scaling, and operations
  • High Availability: Built-in redundancy and failover
  • Security: Enterprise-grade security with encryption at rest and in transit
  • Compliance: GDPR compliant with data residency options

3. Inspector - Log Analysis Tools

The Inspector provides various interfaces for searching, analyzing, and monitoring your logs. Choose the interface that best suits your workflow:

Inspector CLI

  • Command-Line Interface: Perfect for automation and scripting
  • Powerful Queries: Complex search queries with filters
  • Export Capabilities: Export results in various formats
  • Integration Ready: Easy to integrate with other tools
1
2

# Example: Search for errors in the last hour
logflux search --level=error --since=1h --format=json

Inspector TUI

  • Terminal UI: Interactive interface in your terminal
  • Real-Time Updates: Live log streaming and monitoring
  • Keyboard Navigation: Efficient navigation without leaving the terminal
  • Split Views: View multiple log streams simultaneously

Inspector GUI

  • Desktop Application: Native desktop app for Windows, macOS, and Linux
  • Visual Analytics: Charts and graphs for log trends
  • Advanced Filtering: Point-and-click filter creation
  • Saved Searches: Save and share common queries

Data Flow

Understanding how data flows through LogFlux helps in optimizing your logging strategy:

  1. Log Generation: Your application generates log entries
  2. Collection: The Collector captures logs using your chosen method
  3. Transmission: Logs are sent to the LogFlux API (with automatic retries)
  4. Processing: The Ingestor validates, enriches, and stores the logs
  5. Indexing: Logs are indexed for fast searching
  6. Analysis: Use Inspector tools to search and analyze your logs

Security Architecture

LogFlux implements multiple layers of security:

Authentication & Authorization

  • API Keys: Unique keys for each application/service
  • Scoped Access: Keys are tied to specific customers and applications
  • Role-Based Access: Different permission levels for team members

Data Protection

  • Encryption in Transit: TLS 1.3 for all API communications
  • Encryption at Rest: AES-256 encryption for stored logs
  • Data Isolation: Complete logical separation between customers
  • Customer-Specific URLs: Dedicated subdomains for enhanced isolation
  • Access Logging: Comprehensive audit trail of all access

Compliance

  • Data Residency: Choose between EU and US regions
  • Customer-Specific URLs: Enhanced data isolation for compliance requirements
  • GDPR Compliant: Right to deletion, data portability
  • SOC 2 Type II: (In progress)

Scalability & Performance

LogFlux is built to scale with your needs:

Horizontal Scaling

  • Collectors: Deploy as many collectors as needed
  • Ingestors: Automatically scale based on load
  • Storage: Virtually unlimited log storage

Performance Optimization

  • Compression: Logs are compressed for efficient storage
  • Indexing: Smart indexing for sub-second search results
  • Caching: Intelligent caching for frequently accessed data
  • Load Balancing: Automatic distribution across multiple servers

Best Practices

Collector Deployment

  1. Use SDKs when possible: Native integration provides the best performance
  2. Configure batching: Send logs in batches to reduce network overhead
  3. Implement retry logic: Handle temporary network failures gracefully
  4. Monitor collector health: Track metrics like queue size and error rates

Log Design

  1. Structured Logging: Use JSON format for better searchability
  2. Consistent Fields: Maintain consistent field names across services
  3. Appropriate Levels: Use correct log levels (DEBUG, INFO, WARN, ERROR)
  4. Contextual Information: Include request IDs, user IDs, etc.

Inspector Usage

  1. Save Common Queries: Create saved searches for frequent investigations
  2. Use Filters: Narrow down results before searching
  3. Time Windows: Always specify time ranges for better performance
  4. Export Important Findings: Export critical logs for compliance or debugging

Getting Started

Ready to implement LogFlux in your infrastructure?

  1. Create an Account: Get your API keys
  2. Choose Your Collector: Pick the best integration method
  3. Send Your First Logs: Start collecting logs
  4. Explore with Inspector: Analyze your log data

Next Steps