sydepsystem design atlas

Metrics Monitoring Platform

Collect, store and query time-series metrics from every service.

AdvancedPlatform
ServicesComputeMetrics CollectorObservabilityMetrics StoreObservabilityDashboardsObservabilityAlertingObservability

Services emit metrics to a central collector, which writes them into a metrics store. Dashboards and alerting query that store for operational visibility, while retention and downsampling policies control cost.

When to use it

  • The fleet is large enough that a single service's logs are no longer sufficient to understand system health
  • You need alerting on thresholds and trends, not just manual inspection

Trade-offs

  • High-cardinality labels and long retention can make storage very expensive very quickly
  • Traffic spikes during incidents can overwhelm the collection layer if it is not designed for it

Components used

Managed App ServiceMetrics StoreDashboardsAlerting & On-call

How it works

  • Agents collect metrics from every service and ship them to a time-series database optimised for many small appends and range queries.
  • Data is downsampled as it ages — full resolution recently, coarser rollups for older windows — because nobody queries per-second data from six months ago.
  • An alerting engine evaluates rules continuously against the same store that powers dashboards.

Used in the wild

  • Infrastructure and application monitoring.
  • SLO tracking and error-budget burn alerts.
  • Capacity planning from historical trends.

Good to know

  • Cardinality is the thing that kills these systems. Adding a label like user id or request id multiplies the number of stored series into the millions and will exhaust memory long before request volume does.
  • Pull-based collection (the server scrapes targets) versus push-based (agents send) is a long-running argument; pull makes target health obvious, push handles short-lived jobs and network boundaries better.