Log Management and Centralized Logging

Logs Scattered Across Servers

A bug occurs in production. You SSH into 5 different servers, grep log files manually, piece together what happened.

This is archaeology. It's slow and error-prone.

Centralized logging aggregates logs from all services into one searchable place. Find the bug in seconds.

Structured Logging Foundation

First, fix your logs. Don't log plain text:

Bad:

2025-03-15 14:23:45 User login failed
2025-03-15 14:23:46 Database error
2025-03-15 14:23:47 Request timeout

Good (structured):

{"timestamp": "2025-03-15T14:23:45Z", "level": "error", "service": "auth", "user_id": "usr_123", "error": "invalid_password", "duration_ms": 150}
{"timestamp": "2025-03-15T14:23:46Z", "level": "error", "service": "db", "query": "SELECT * FROM users", "error": "connection_timeout", "duration_ms": 5000}

Structured logs are queryable by any field.

Libraries:

• Node.js: winston, pino
• Python: structlog, python-json-logger
• Go: zerolog, zap
• Java: logback with JSON encoder

The ELK Stack (Elasticsearch, Logstash, Kibana)

Traditional log aggregation:

App logs
  ↓
Logstash (parse, transform logs)
  ↓
Elasticsearch (index, store)
  ↓
Kibana (search, visualize)

Elasticsearch: Inverted index database. Fast full-text search. Logstash: ETL for logs. Parse formats, extract fields, enrich data. Kibana: Web UI for searching and dashboarding.

Setup (self-hosted):

• Docker Compose or Kubernetes
• 3-node Elasticsearch cluster (HA)
• Logstash to receive logs
• Kibana for UI
• Total cost: infrastructure + operations time

Cost: $500-5k/month depending on log volume (self-hosted infrastructure).

Advantages:

• Full control, no vendor lock-in
• Powerful query language (Lucene)
• Highly customizable

Disadvantages:

• Operational overhead (cluster management, upgrades)
• Elasticsearch is resource-hungry
• Logstash is complex

Loki: Lightweight Alternative

Loki is Prometheus for logs. Simple, lightweight.

App logs
  ↓
Promtail (agent, ships logs)
  ↓
Loki (stores logs, indexed by labels)
  ↓
Grafana (search, visualize)

Advantages:

• Minimal resource usage (40MB vs 1GB for Elasticsearch)
• Simple operational model
• Integrates with Prometheus (same query language)
• Cheap ($100-500/month self-hosted)

Disadvantages:

• Limited query language (no full-text search on all fields)
• Smaller ecosystem than ELK

Best for: Startups, Kubernetes environments.

Cloud Logging Services

Datadog, New Relic, AWS CloudWatch, GCP Logging:

• Managed SaaS
• Zero operational overhead
• Easy setup (install agent, ship logs)
• Expensive ($1-5k/month at scale)
• Limited customization

Best for: Teams that want logging without ops headache.

Log Retention and Costs

Logs grow fast. 1000 services × 1GB logs per day = 1TB daily.

Storage costs:

• Hot storage (recent, searchable): $0.50-2.00 per GB/month
• Cold storage (archived, slow): $0.05 per GB/month

Retention strategy:

• Keep 7-14 days of logs hot (searchable)
• Archive older logs to cold storage
• Delete after 90 days (or per compliance requirements)

Example:

10GB logs/day × $1/GB hot storage × 14 days = $140/month
Older logs archived to S3 ($0.05/GB) = negligible

Querying Logs

Kibana (ELK):

service:payment AND error:timeout AND duration_ms:[1000 TO *]

Find payment service errors with "timeout" and duration > 1s.

Loki/Grafana:

{job="api-server", status="error"} | duration_ms > 1000

CloudWatch Logs Insights:

fields @timestamp, @message, user_id
| filter error = "payment_failed"
| stats count() by user_id

Log Aggregation Checklist

• Structured logging (JSON, not plain text)
• Centralized log aggregation (ELK, Loki, or managed service)
• Retention policy (hot: 14 days, cold: 90 days)
• Log search and filtering
• Dashboards for common queries
• Monitoring on log volume (alert if volume spikes unexpectedly)
• Access control (who can see logs?)
• PII redaction (remove sensitive data from logs)
• Integration with alerting (alert based on log patterns)