Observability in distributed systems
From console.log to distributed tracing with OpenTelemetry and the LGTM stack
A practical guide to implementing observability in a distributed system. It starts from the distinction between monitoring and observability, builds the full LGTM stack, and gets to handling tail sampling, compliance routing and PII filtering in production. Every article comes with working code on GitHub and real debugging scenarios.
Monitoring and observability are not the same thing
Monitoring answers questions you anticipated: CPU is above threshold, the disk is filling up, the service is not responding. It works as long as failures look like the ones you imagined when you wrote the alerts.
In a distributed system, failures stop looking like what you anticipated. The question becomes why did this request, from this customer, take twelve seconds. That is not a question you can put in a dashboard in advance. Observability is the property of a system that lets you answer questions nobody prepared for.
There are three signals, and on their own they are worth little. Metrics tell you something changed, logs tell you what happened at one point, traces tell you where the time went along the path. The value is in moving from one to the next without losing the thread: from the alert to the trace, from the trace to the log line that failed.
This series starts from that thread and follows it into production, where the problems stop being conceptual and become data volume, storage cost and compliance constraints.
What you will learn
- ✓ Configure OpenTelemetry on Node.js without changing application code
- ✓ Build the LGTM stack (Loki, Grafana, Tempo, Mimir) with Docker Compose
- ✓ Debug real distributed scenarios with distributed tracing and log correlation
- ✓ Cut data volume by 90% with tail sampling without losing visibility on errors
- ✓ Separate audit data from technical data for GDPR/SOC 2 compliance
Articles in the series
- 01Where the cost of observability is decided 7 min
Where you put the Collector and what you index: the two choices that decide what observing a system costs. Topologies, OTLP transport, LGTM storage.
→ - 02Hands-On Distributed Tracing with OpenTelemetry and the LGTM Stack 18 min
A practical guide to distributed tracing with OpenTelemetry and the LGTM stack. Three real debug scenarios: silent failure, latency spike, fan-out.
→ - 03OpenTelemetry in Production: Tail Sampling and Retention 17 min
How to cut trace volume by 90% without losing visibility into errors. A practical guide with config templates and a runnable demo scenario.
→ - 04OpenTelemetry in Production: Data Routing for Compliance and Cost Control 18 min
Separate audit logs from technical logs using the OTel Collector routing connector. Demo, GDPR/SOC 2 compliance, and differentiated retention strategies.
→ - 05PII Filtering in OpenTelemetry: Protecting Sensitive Data from Keycloak 13 min
How to instrument Keycloak and third-party services that handle sensitive data, reducing PII exposure through OTel Collector filtering. GDPR-compliant techniques.
→ - 06From console.log to Grafana: Structured and Centralized Logging with Node.js 8 min
How to move from console.log to a structured, centralized logging system using Pino, OpenTelemetry, Loki, and Grafana on Node.js — in three incremental steps.
→