sydepsystem design atlas

Observability: Logs, Metrics & Traces

Instrument every service with the three complementary signal types.

IntermediatePlatform
ServiceComputeLog AggregationObservabilityMetrics StoreObservabilityDistributed TracingObservabilityDashboardsObservabilityAlertingObservability

Services emit structured logs, time-series metrics and distributed traces to dedicated backends, unified in dashboards and alerting -- so an incident can be diagnosed from symptom (metric) to root cause (trace) to detail (log).

When to use it

  • More than a couple of services are involved, and 'ssh in and check the logs' no longer works
  • You need to answer both 'is something wrong' and 'why' quickly during an incident

Trade-offs

  • Three collection paths (logs, metrics, traces) to instrument, ship and retain
  • High-cardinality metrics or 100% trace sampling get expensive fast at scale

Components used

Managed App ServiceLog AggregationMetrics StoreDistributed TracingDashboardsAlerting & On-call

How it works

  • Metrics are cheap numeric aggregates over time, ideal for alerting and dashboards but unable to explain any individual request.
  • Logs are detailed discrete records, ideal for explaining a specific event but expensive to store and slow to aggregate.
  • Traces stitch a single request's path across services, showing where the time actually went. Correlation ids link all three together.

Used in the wild

  • Any distributed system where a slow request could be caused by any of a dozen services.
  • SLO-based alerting driven by metrics, with traces and logs used for the follow-up investigation.
  • Capacity planning and cost attribution across shared infrastructure.

Good to know

  • Monitoring answers questions you predicted; observability is about answering ones you did not. A dashboard for every known failure mode is monitoring, not observability.
  • Traces are usually sampled because storing every span is prohibitive — which means the one pathological request you actually care about was frequently not recorded.