Monitoring and Observability: Building Visibility Into Your Systems

The distinction between monitoring and observability matters for how you architect your systems and respond to incidents. Monitoring asks predefined questions: Is the server up? Is the response time under 500ms? Is disk usage below 80%? Observability enables you to ask new questions about system behavior based on the data your systems emit — questions you did not think to ask before an incident revealed you needed to.

The Three Pillars: Metrics, Logs, Traces

Metrics are numeric measurements over time — CPU usage, request rates, error rates, response latencies. They are efficient to store and excellent for alerting and dashboards. Tools: Prometheus, Graphite, InfluxDB, CloudWatch, Datadog. The four golden signals from Google SRE — latency, traffic, errors, saturation — are the baseline metrics every service should expose.

Logs are timestamped records of discrete events — application errors, access requests, audit trails. They provide context that metrics cannot: which user triggered the error, what parameters were passed, what the exact error message was. Tools: ELK stack (Elasticsearch, Logstash, Kibana), Loki, Splunk, CloudWatch Logs.

Traces track requests as they flow through distributed systems, showing latency at each step and how services call each other. Essential for diagnosing performance issues in microservices architectures where a slow response might result from a slow database call three hops downstream. Tools: Jaeger, Zipkin, AWS X-Ray, Honeycomb.

Building an Alerting Strategy

Alert fatigue kills on-call effectiveness. Every alert that fires and requires no action trains the on-call engineer to dismiss alerts. Build alerts around SLA/SLO violations (user-visible impact), not internal system states. Set alert thresholds based on historical baseline data, not guesses. Ensure every alert has a runbook that answers: what is happening, what is the user impact, and what are the initial diagnostic steps.

Infrastructure Monitoring with Prometheus and Grafana

Prometheus scrapes metrics from instrumented applications and exporters (node_exporter for system metrics, blackbox_exporter for endpoint monitoring). PromQL provides a powerful query language for aggregating and calculating metrics across many instances. Grafana builds dashboards from Prometheus (and other) data sources, with pre-built dashboard templates available for common infrastructure components. AlertManager handles alert routing, grouping, and notification delivery to PagerDuty, Slack, email, or other channels.

Read our guide on Bash scripting for sysadmins or explore our tools page for recommended monitoring platforms and comparison guides.