SRE in Practice | Day 11: Metrics, Logs, and Traces
Understand what happens inside a system from what it outputs. Learn metrics, logs, and traces and follow one problem through all three. Four hands-on labs, each explained step by step.

✍️ Translate this blog into Hindi, Tamil or Telugu
What you will learn
- What observability is, and how it differs from monitoring
- What metrics, logs, and traces each do best
- How to follow one problem from a metric to a log to a trace
- Four labs where every command is explained in plain words, with how to read the result
Safety first: All commands are read-only or touch only files under /tmp/sre-lab that you create. Run them on a system you are authorized to use. Never run cleanup or deletion commands on real data until you have reviewed a dry run, and do not paste logs containing credentials or customer data into public posts.
In one minute: Observability means you can understand what is happening inside a system by looking at what it gives out. Three kinds of output do most of the work: metrics (numbers over time), logs (written events), and traces (the path of one request). Think of a car: metrics are the dashboard dials, logs are the diary of what happened, and traces are the GPS trail of one journey.
1. Monitoring and observability
Monitoring answers questions you already know to ask: "Is CPU above 80%?" "Is the error rate above 1%?" Observability is wider. It lets you ask questions you did not plan for, such as "Why are only mobile checkout requests slow on Tuesdays?" To do that, the system must emit enough useful data. Monitoring is something you do with that data. Observability is a property of the system.
2. The three pillars

How they work together
🛍️✨ From the author · ByteStoreCareer gap analysis and Mentoring - Basic PlanβΉ299View offer →
A good investigation moves from wide to narrow. Metrics tell you something is wrong. Logs tell you what happened. Traces tell you where. In this guide's scenario, a checkout page becomes slow:
3. Good habits that make data useful
- Write structured logs. Use key=value or JSON, so tools can filter by field instead of guessing from sentences.
- Put a trace or request ID in every log line. It is the thread that ties logs, metrics, and traces together.
- Keep metric labels few and bounded. Labels like "region" are fine. A label with a different value for every user (high cardinality) makes metrics slow and expensive.
- Measure what users feel, not only what servers do.
- Never log secrets or personal data such as passwords, tokens, or card numbers.
- Keep time consistent. Use one time zone (usually UTC) across systems.
4. Hands-on labs
Safety notes: All commands are read-only or touch only files under /tmp/sre-lab that you create. Run them on a system you are authorized to use. Never run cleanup or deletion commands on real data until you have reviewed a dry run, and do not paste logs containing credentials or customer data into public posts.
Setup: create two practice files
You will make a pretend application log (app.log) with 60 checkout requests, and a pretend trace record (spans.csv) that says how long each service spent on each request. Every tenth request is slow, every twelfth one fails, and nothing real is touched.
mkdir -p /tmp/sre-lab && cd /tmp/sre-labawk 'BEGIN{
for(r=1;r<=60;r++){
m=int((r-1)/20); sec=(r*3)%60
st=200; lvl="INFO"; msg="ok"
d=100+(r%7)*10
if(r%10==0) d=2500
if(r%12==0){st=500; lvl="ERROR"; msg="db_timeout"}
printf "time=10:%02d:%02d level=%s trace=t%03d status=%d dur_ms=%d msg=%s\n",
m, sec, lvl, r, st, d, msg
}}' > app.log
awk 'BEGIN{print "trace,service,parent,duration_ms"
for(r=1;r<=60;r++){ db=40+(r%5)*5; if(r%10==0) db=2400
api=db+70; web=api+30
printf "t%03d,web,-,%d\nt%03d,api,web,%d\n", r,web,r,api
printf "t%03d,db,api,%d\n", r,db }}' > spans.csvCheck what you made:
head -2 app.log
wc -l app.log spans.csvExpected output
time=10:00:03 level=INFO trace=t001 status=200 dur_ms=110 msg=ok
time=10:00:06 level=INFO trace=t002 status=200 dur_ms=120 msg=ok
60 app.log
181 spans.csv
241 totalEach log line is a set of key=value pairs. That is a structured log. The 181 lines in spans.csv are 1 heading plus 60 requests x 3 services (web, api, db).
Lab 1: Metrics: turn log lines into numbers
Goal: Count requests, errors, and response times for each minute.
Why it matters: A metric is just an event log squeezed into a number per time slot. This is what dashboards and alerts are built on.
printf "%-7s %8s %6s %7s %7s\n" minute requests errors avg_ms max_ms
awk '{ for(i=1;i<=NF;i++){ split($i,kv,"="); v[kv[1]]=kv[2] }
split(v["time"],t,":"); m=t[2]; n[m]++
if (v["status"]>=500) e[m]++
s[m]+=v["dur_ms"]; if (v["dur_ms"]>mx[m]) mx[m]=v["dur_ms"] }
END{ for(m in n)
printf "10:%-4s %8d %6d %7.0f %7d\n",
m, n[m], e[m], s[m]/n[m], mx[m] }' app.log | sort
Expected output
minute requests errors avg_ms max_ms
10:00 20 1 367 2500
10:01 20 2 365 2500
10:02 20 2 366 2500How to read the result: Each row is one minute. Every minute has 20 requests and 1 or 2 errors. The average response time is about 366 ms, which sounds fine. But the max is 2,500 ms every minute, so some users wait two and a half seconds. This is the same lesson as Day 5: a single average hides slow requests.
Try it yourself: Change
v["status"]>=500tov["dur_ms"]>1000to count slow requests per minute instead of errors.
Lab 2: Logs: find the story behind the numbers
Goal: Filter the log to see what the errors were, and follow one request.
Why it matters: Metrics say how many. Logs say what. This is the second step of an investigation.
grep 'level=ERROR' app.log | head -3
awk '{split($2,a,"="); c[a[2]]++} END{for(k in c) print k, c[k]}' app.log | sort
grep 'trace=t012' app.logExpected output
time=10:00:36 level=ERROR trace=t012 status=500 dur_ms=150 msg=db_timeout
time=10:01:12 level=ERROR trace=t024 status=500 dur_ms=130 msg=db_timeout
time=10:01:48 level=ERROR trace=t036 status=500 dur_ms=110 msg=db_timeout
ERROR 5
INFO 55
time=10:00:36 level=ERROR trace=t012 status=500 dur_ms=150 msg=db_timeoutHow to read the result: All three errors say msg=db_timeout. The database is the common factor. Out of 60 requests, 5 are errors (about 8%). Because every line has a trace= ID, you could pull out the whole story of one request. In a real system, that ID would be shared across all services.
Try it yourself: Run
grep -c 'msg=db_timeout' app.log. The -c option prints a count instead of lines.
Lab 3: Traces: see where the time went
Goal: Look at one slow request and find which service used the time.
🛍️✨ From the author · ByteStoreLinux & Cloud Administration Consultation | AWS & Azure Infrastructure SupportβΉ499View offer →Why it matters: A trace breaks one request into steps called spans. It answers "where?", which logs and metrics usually cannot.
awk -F, '$2=="web" && $4>1000 {print $1, $4 " ms"}' spans.csv | tr '\n' ';'; echo
grep '^t010,' spans.csv
awk -F, '$1=="t010"{d[$2]=$4}
END{printf "web self=%d ms, api self=%d ms, db self=%d ms\n",
d["web"]-d["api"], d["api"]-d["db"], d["db"]}' spans.csv
Expected output
t010 2500 ms;t020 2500 ms;t030 2500 ms;t040 2500 ms;t050 2500 ms;t060 2500 ms;
t010,web,-,2500
t010,api,web,2470
t010,db,api,2400
web self=30 ms, api self=70 ms, db self=2400 msHow to read the result: Six requests were slow (every tenth one). For t010, the web span lasts 2,500 ms in total, but that includes the api, which includes the db. After subtracting, the web tier spent only 30 ms and the api 70 ms. The database used 2,400 ms. You would not know this from the web server's own timing alone.
Try it yourself: Try t020 instead of t010. The pattern is the same, because every slow request in this sample is a slow database call.
Lab 4: Put it together: metric to log to trace
Goal: Follow one problem through all three pillars, as you would in a real incident.
Why it matters: This is the real skill. Each signal points to the next.
slow=$(awk '{ for(i=1;i<=NF;i++){ split($i,kv,"="); v[kv[1]]=kv[2] }
if (v["dur_ms"]>mx) { mx=v["dur_ms"]; id=v["trace"] } } END{print id}' app.log)
echo "slowest trace: $slow"
grep "$slow" app.log
grep "^$slow," spans.csv
Expected output
slowest trace: t010
time=10:00:30 level=INFO trace=t010 status=200 dur_ms=2500 msg=ok
t010,web,-,2500
t010,api,web,2470
t010,db,api,2400How to read the result: Metric: the max response time is 2,500 ms. Log: that request succeeded, so a metric based on errors alone would have missed it. Trace: 2,400 ms was spent in the database. Each signal added something the others could not.
Lab 5 (worksheet): your own service
Fill in this table for a service you support. Any blank cell is an observability gap.

5. Common mistakes
- Collecting everything and understanding nothing. Start from the questions you need to answer.
- Logs in free text that cannot be filtered by field
- No request ID, so logs and traces cannot be matched
- Metric labels with endless unique values
- Only watching averages
- Logging secrets or personal data
6. Quick check: test yourself
- In one sentence each, what are metrics, logs, and traces?
- Which signal would you use first to notice a problem, and which to locate it?
- Why is a trace ID in every log line useful?
- In Lab 3, why is the db self time larger than the web self time?
- Which of the three is missing for the service you support?
Suggested answers: numbers over time, timestamped event records, and the path of one request; metrics to notice and traces to locate; it lets you join logs, metrics, and traces for one request; web includes api which includes db, and after subtracting children the database used most of the time; check your worksheet.
Glossary

The first fifteen days at a glance

Your turn: Which of metrics, logs, and traces is weakest for the service you support today?
Coming next, Day 12: Linux monitoring with the USE method.
Follow @leadwithsreeΒ for the daily posts in this 30-day journey.
🛍️ Srilakshmi T's shelf
Courses, cohorts and more from the authorGet Srilakshmi T's next blog by email
One email a week when they publish. Add the weekly Best of ByteBlog too. No spam, unsubscribe with one click.
We email you once to confirm. By subscribing you agree to the privacy policy.
More from Srilakshmi T

SRE in Practice: DAY 1 OF 30 What is SRE?
Why a running server does not always mean a working service. Concept, production example,and a safe hands-on lab.
#aws
SRE in Practice | Day 3: The SRE Mindset
Look beyond "Is the server up?" and ask whether the service works reliably for the people who depend on it. Concept, infrastructure example, and a hands-on lab.

SRE in Practice | Day 2: SRE vs DevOps
Two complementary perspectives on delivering and running reliable software. Concept, production example, and a hands-on release-mapping lab.

SRE in Practice | Day 9: Error Budgets
Turn the SLO into a budget you can spend. Learn burn rates, burn-rate alerts, and error budget policies. Six hands-on labs with verified expected results.

💬 Comments (0)
Top · NewestSign in to comment