Retention Policies

LogFlux.io provides flexible retention policies that automatically manage your log data lifecycle, balancing storage costs with data accessibility needs.

Storage Tiers Overview

LogFlux uses a tiered storage approach to optimize performance and costs:

┌─────────────┐     ┌─────────────┐     ┌─────────────┐
│  Hot Tier   │ ──> │  Warm Tier  │ ──> │  Cold Tier  │
│   7 days    │     │   30 days   │     │   90 days   │
│ Instant     │     │ Sub-second  │     │ 1-5 seconds │
└─────────────┘     └─────────────┘     └─────────────┘
        │                   │                   │
        └───────────────────┴───────────────────┘
                     Automatic Migration

Default Retention Settings

Hot Storage (7 days)

• Purpose: Recent logs requiring instant access
• Performance: < 100ms query response
• Storage: High-performance SSD
• Use Cases: Real-time debugging, active monitoring

Warm Storage (30 days)

• Purpose: Recent historical data
• Performance: < 1 second query response
• Storage: Standard SSD
• Use Cases: Trend analysis, incident investigation

Cold Storage (90 days)

• Purpose: Compliance and long-term analysis
• Performance: 1-5 seconds query response
• Storage: Object storage (S3)
• Use Cases: Compliance audits, historical analysis

Configuring Retention

Navigate to Settings → Retention Policies in the dashboard to configure:

1
2
3
4
5

retention_policies:
  hot_days: 7          # 1-14 days
  warm_days: 30        # 7-60 days
  cold_days: 90        # 30-365 days
  auto_delete: true    # Delete after cold period

Data Lifecycle

1. Ingestion (Day 0)

Application → Collector → Ingestor → Hot Storage
                                          │
                                    Immediately
                                    Queryable

2. Hot to Warm Migration (Day 7)

• Automatic migration occurs daily at 00:00 UTC
• No manual intervention required
• Data remains fully queryable during migration

3. Warm to Cold Migration (Day 30)

• Compression applied during migration
• Original: 100GB → Compressed: ~10GB (90% reduction)
• Maintains full text search capability

4. Auto-Deletion (Day 90)

• Permanent deletion after cold period
• Configurable per customer requirements
• Audit log maintained for compliance

Custom Retention Rules

By Application

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

{
  "application_rules": [
    {
      "application": "payment-service",
      "hot_days": 14,
      "warm_days": 60,
      "cold_days": 365  // Extended for compliance
    },
    {
      "application": "web-frontend",
      "hot_days": 3,
      "warm_days": 14,
      "cold_days": 30   // Reduced for cost savings
    }
  ]
}

By Log Level

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

{
  "level_rules": [
    {
      "level": "ERROR",
      "hot_days": 14,    // Keep errors accessible longer
      "warm_days": 60
    },
    {
      "level": "DEBUG",
      "hot_days": 1,     // Minimize debug log retention
      "warm_days": 7
    }
  ]
}

Query Performance by Tier

Hot Storage Queries

• Response Time: < 100ms
• Use Case: Real-time log search for recent events
• Time Range: Last 7 days
• Performance: Instant results with full-text search

Warm Storage Queries

• Response Time: < 1 second
• Use Case: Recent historical analysis and aggregations
• Time Range: 7-30 days ago
• Performance: Fast results with minimal latency

Cold Storage Queries

• Response Time: 1-5 seconds
• Use Case: Compliance audits and long-term analysis
• Time Range: Beyond 30 days
• Performance: Complete access with acceptable delay

Cost Optimization

Storage Costs by Tier

Example Cost Calculation

For 1TB of daily log ingestion:

Month 1 (Hot):    1TB × $0.10 = $100
Month 2 (Warm):   333GB × $0.05 = $16.65
Month 3+ (Cold):  100GB × $0.01 = $1.00
Total 90-day cost: $117.65 (vs $300 if all hot)

Compliance Considerations

Data Residency

Configure retention with regional compliance:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

{
  "region_policies": {
    "eu": {
      "storage_location": "eu-central-1",
      "retention_days": 180,  // GDPR considerations
      "deletion_policy": "immediate"
    },
    "us": {
      "storage_location": "us-east-1",
      "retention_days": 90,
      "deletion_policy": "standard"
    }
  }
}

Backup and Recovery

Automatic Backups

All tiers include automatic backups:

• Hot: Continuous replication
• Warm: Daily snapshots
• Cold: Weekly backups to glacier

Recovery Time Objectives

Best Practices

1. Right-Size Your Retention

1
2
3
4
5
6
7
8
9

# Development environment
hot_days: 3
warm_days: 7
cold_days: 30

# Production environment
hot_days: 7
warm_days: 30
cold_days: 90

2. Use Application-Specific Rules

1
2
3
4
5
6
7
8
9

# Critical services need longer retention
payment-service:
  hot_days: 14
  cold_days: 365

# Non-critical services can have shorter retention
static-website:
  hot_days: 1
  cold_days: 30

3. Monitor Storage Usage

Use the dashboard to monitor:

• Storage distribution across tiers
• Average query times per tier
• Growth trends and capacity planning

4. Plan for Growth

Current: 100GB/day
Growth: 20% monthly
Plan: Adjust retention before hitting limits

FAQ

Can I retrieve deleted logs?

No, logs are permanently deleted after the retention period. Consider extending retention or implementing backups if you need longer access.

How quickly are logs migrated between tiers?

Migration happens automatically at midnight UTC. The process typically completes within 1-2 hours depending on data volume.

Can I query across multiple tiers?

Yes, LogFlux automatically queries all relevant tiers based on your time range. Performance varies by tier.

What happens if I change retention settings?

• Shortening: Existing data beyond new limits is queued for deletion
• Extending: Only affects new data; historical data follows original policy

Is data compressed in all tiers?

• Hot: No compression (instant access)
• Warm: 3:1 compression (zstd)
• Cold: 10:1 compression (zstd + columnar storage)