SRE in Practice | Day 5: Four Golden Signals

What you will learn

• The four golden signals and how to read them together
• Why percentiles beat averages, and why to separate failed from successful latency
• The core reliability principles from Google's SRE books
• Six hands-on labs that calculate each signal with expected output

Safety first: All commands are read-only or touch only files under /tmp/sre-lab that you create. Run them on a system you are authorized to use. Never run cleanup or deletion commands on real data until you have reviewed a dry run, and do not paste logs containing credentials or customer data into public posts.

1. You cannot watch everything: start with four signals

A modern service produces thousands of metrics. If you try to alert on all of them, you will drown in noise. Google's SRE book recommends four signals that cover the user-facing health of almost any service. They are called the Four Golden Signals.

A subtle point: Latency of failed requests can be misleading. A request that fails instantly looks fast, and an error that waits for a timeout looks slow. Track the latency of successful requests on their own, and watch failures with the error signal.

2. A production-style example

This is an illustrative scenario. At 10:01 users report that a checkout API is slow and failing. The four signals tell a story that a CPU graph alone would not:

Together the signals point the investigation toward a faulty dependency or release, and away from "add more servers". That is the value of looking at all four.

3. Reliability principles that guide SRE work

The golden signals are one tool. Behind them sit a few principles from Google's SRE books that you will meet throughout this series:

4. Hands-on labs

Safety notes: All commands are read-only or touch only files under /tmp/sre-lab that you create. Run them on a system you are authorized to use. Never run cleanup or deletion commands on real data until you have reviewed a dry run, and do not paste logs containing credentials or customer data into public posts.

Setup: create a sample access log

The sample log has 6,000 requests over three minutes. Field 9 is the HTTP status and field 11 is the request time in seconds. Minute 10:01 contains a deliberate error spike so the later calculations have something to find.

mkdir -p /tmp/sre-lab && cd /tmp/sre-lab

awk 'BEGIN{
  for(i=1;i<=6000;i++){
    m=int((i-1)/2000); s=200
    if(i%200==0) s=500
    else if(m==1 && i%20==0) s=500
    else if(i%97==0) s=404
    t=0.05+(i%20)*0.01
    if(i%50==0) t=1.5
    ts=sprintf("[01/Nov/2026:10:%02d:00 +0000]", m)
    printf "10.0.0.%d - - %s \"GET /api HTTP/1.1\" %d 512 %.3f\n",
           (i%250)+1, ts, s, t
  }}' > access.log

Check it: wc -l access.log should print 6000 access.log.

Lab 1: traffic

Count requests per minute, using the timestamp in field 4.

awk '{split($4,a,":"); c[a[3]]++}
     END{for(m in c) print "10:" m, c[m], "requests"}' access.log | sort

Expected output

10:00 2000 requests
10:01 2000 requests
10:02 2000 requests

Lab 2: errors

awk '{t++; if ($9>=500) e++}
     END{printf "errors=%d of %d (%.2f%%)\n", e, t, e/t*100}' access.log

Expected output

errors=120 of 6000 (2.00%)

Lab 3: latency, and why the average is not enough

Take only successful requests (status below 500), sort their times, and read the percentiles.

awk '$9<500{print $11}' access.log | sort -n | awk '{a[NR]=$1; s+=$1}
END{printf "n=%d avg=%.3fs p50=%.3fs p95=%.3fs p99=%.3fs max=%.3fs\n",
    NR, s/NR, a[int(NR*0.50)], a[int(NR*0.95)], a[int(NR*0.99)], a[NR]}'

Expected output

n=5880 avg=0.166s p50=0.150s p95=0.240s p99=1.500s max=1.500s

The average (0.166 s) looks healthy, and p95 (0.240 s) looks fine. Only p99 (1.5 s) reveals that about one request in a hundred is very slow. Percentiles show what your slowest users experience. This uses a simple nearest-rank calculation, which is fine for learning.

Now compare failed and successful requests, as the signal rules advise:

awk '{k=($9>=500) ? "failed" : "ok"; n[k]++; s[k]+=$11}
     END{for(k in n)
         printf "%-6s n=%d avg=%.3fs\n", k, n[k], s[k]/n[k]}' access.log | sort

Expected output

failed n=120 avg=0.533s
ok     n=5880 avg=0.166s

Lab 4: saturation (Linux)

Saturation is about the most constrained resource. These read-only commands give a first view of CPU load, memory, and disk.

awk -v c=$(nproc) '{printf "load1=%s cores=%d per_core=%.2f\n",
                          $1, c, $1/c}' /proc/loadavg
free -m | awk 'NR==2{printf "memory used: %.0f%%\n", $3/$2*100}'
df -P / | awk 'NR==2{print "disk used: " $5}'

Expected output

load1=0.02 cores=1 per_core=0.02
memory used: 5%
disk used: 49%

Your numbers will differ. A per_core load that stays well above 1 means work is queueing for CPU. Rising memory or disk use close to 100% is a saturation warning that can show up later as latency or errors. Other saturation signals matter too, such as connection pools and queue depth.

Lab 5: one table with the signals together

printf "%-6s %9s %8s %8s %8s\n" min requests errors% avg\(s\) max\(s\)
awk '{split($4,a,":"); m=a[3]; t[m]++; if ($9>=500) e[m]++
      s[m]+=$11; if ($11>mx[m]) mx[m]=$11}
END{for(m in t) printf "10:%s %9d %8.2f %8.3f %8.3f\n",
    m, t[m], e[m]/t[m]*100, s[m]/t[m], mx[m]}' access.log | sort

Expected output

min     requests  errors%   avg(s)   max(s)
10:00      2000     0.50    0.173    1.500
10:01      2000     5.00    0.173    1.500
10:02      2000     0.50    0.173    1.500

The spike at 10:01 is obvious in the error column. Notice that traffic stayed the same, so this is not a demand problem. In real systems you would see these as time-series graphs and alerts, not text tables.

Lab 6 (worksheet): your own service

Fill in the table for a service you support. If a cell is blank, you have found a monitoring gap.

5. Common mistakes

• Looking only at averages, and missing the slow tail
• Watching CPU and memory but not request errors or latency
• Mixing failed and successful request latency in one number
• Alerting on every metric instead of the signals users feel
• Treating 100% reliability as the goal
• Reading one signal alone instead of all four together

6. Quick check: test yourself

• Name the four golden signals and one example measure for each.
• Why can the average latency look healthy while users are unhappy?
• In the example, why is adding servers the wrong fix?
• Why measure the latency of failed requests separately?
• Which signal is missing from your own service's monitoring?

Suggested answers: latency, traffic, errors, saturation; the average hides the slow tail that percentiles reveal; traffic and saturation were normal, so capacity was not the problem; fast failures and slow timeouts distort the latency picture; check your worksheet.

Glossary

The first ten days at a glance

Coming next, Day 6: SLIs, measuring what users experience.

Follow @leadwithsree  for the daily posts in this 30-day journey.