Four Golden Signals

What are the four golden signals of monitoring?

The Four Golden Signals, defined in Google's Site Reliability Engineering book, are the four most important metrics for monitoring any user-facing distributed service. If you can only instrument four things, these are the ones.

The Four Signals

1. Latency

The time it takes to serve a request.

• Measure: p50, p95, p99 (percentiles, not averages - averages hide tail latency)
• Distinguish: successful request latency vs error latency separately
• Example: http_request_duration_seconds{status="200"} histogram
• Alert: p99 latency > 500ms for > 5 minutes

Why it matters: slow responses are a failure mode. A request that times out is worse than one that errors quickly - it holds connections and makes the user wait.

2. Traffic

A measure of how much demand is being placed on your system.

• Measure: requests per second, concurrent users, messages per second, queries per second
• Helps distinguish: is the issue increased load or degraded performance?
• Example: http_requests_total counter rate
• Alert: traffic spike >3× baseline (may indicate an attack or viral event)

Why it matters: traffic changes are the context for interpreting all other signals. Error rate going up while traffic also goes up is different from errors going up with constant traffic.

3. Errors

The rate of requests that fail - explicitly (5xx), implicitly (wrong content), or by policy (latency SLO violations).

• Measure: error rate as a percentage of total requests
• Types: HTTP 5xx, application-level errors, timeout errors, data validation failures
• Example: rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m])
• Alert: error rate > 1% of requests

Why it matters: error rate directly correlates with user impact. A 5% error rate means 1 in 20 users is getting a broken experience.

4. Saturation

How "full" your service is - how close to capacity are your constrained resources.

• Measure: CPU utilisation %, memory usage %, disk I/O %, connection pool usage %
• Leading indicator: saturation often precedes latency and error increases
• Example: node_cpu_seconds_total, pg_stat_activity connection count
• Alert: CPU > 80% sustained for > 10 minutes

Why it matters: saturation predicts problems before they materialise as errors. A service at 95% CPU won't have headroom to absorb a traffic spike.

Comparison: RED and USE Methods

RED Method (for services)

• Rate - request rate (= Traffic)
• Errors - error rate (= Errors)
• Duration - latency (= Latency)

RED is a subset of the four golden signals, focused on service-level metrics.

USE Method (for resources)

• Utilisation - % time busy (= Saturation aspect)
• Saturation - queue depth / backlog
• Errors - error rate for the resource

USE is for infrastructure-level metrics (CPU, disk, network).

Implementation

Prometheus + Grafana Stack

# FastAPI + prometheus_client
from prometheus_client import Counter, Histogram

REQUEST_COUNT = Counter('http_requests_total', 'Total requests', ['method', 'endpoint', 'status'])
REQUEST_LATENCY = Histogram('http_request_duration_seconds', 'Request latency', ['endpoint'])

@app.middleware("http")
async def metrics_middleware(request, call_next):
    with REQUEST_LATENCY.labels(request.url.path).time():
        response = await call_next(request)
    REQUEST_COUNT.labels(request.method, request.url.path, response.status_code).inc()
    return response

Dashboard Design

For each service, one dashboard page with:

• Traffic: requests/sec
• Error rate: % 5xx
• Latency: p50, p95, p99 over time
• Saturation: CPU, memory, connection pool %

Interview Angle

For observability questions: name the four golden signals and explain what each catches. Then describe implementation: Prometheus metrics, histograms for latency (not averages), and Grafana dashboards. Show that you understand percentile latency - p99 is what your worst-case users experience, not the median.