CareerByteCode
ByteBlogby CareerByteCode

SRE in Practice | Day 11: Metrics, Logs, and Traces

Understand what happens inside a system from what it outputs. Learn metrics, logs, and traces and follow one problem through all three. Four hands-on labs, each explained step by step.

SRE in Practice | Day 11: Metrics, Logs, and Traces
📚 Part 11 of 11 in SRE in Practise - 30 days

✍️ Translate this blog into Hindi, Tamil or Telugu

What you will learn

  • What observability is, and how it differs from monitoring
  • What metrics, logs, and traces each do best
  • How to follow one problem from a metric to a log to a trace
  • Four labs where every command is explained in plain words, with how to read the result

Safety first: All commands are read-only or touch only files under /tmp/sre-lab that you create. Run them on a system you are authorized to use. Never run cleanup or deletion commands on real data until you have reviewed a dry run, and do not paste logs containing credentials or customer data into public posts.


In one minute: Observability means you can understand what is happening inside a system by looking at what it gives out. Three kinds of output do most of the work: metrics (numbers over time), logs (written events), and traces (the path of one request). Think of a car: metrics are the dashboard dials, logs are the diary of what happened, and traces are the GPS trail of one journey.

1. Monitoring and observability

Monitoring answers questions you already know to ask: "Is CPU above 80%?" "Is the error rate above 1%?" Observability is wider. It lets you ask questions you did not plan for, such as "Why are only mobile checkout requests slow on Tuesdays?" To do that, the system must emit enough useful data. Monitoring is something you do with that data. Observability is a property of the system.

2. The three pillars

How they work together

🛍️✨ From the author · ByteStoreCareer gap analysis and Mentoring - Basic Planβ‚Ή299View offer →

A good investigation moves from wide to narrow. Metrics tell you something is wrong. Logs tell you what happened. Traces tell you where. In this guide's scenario, a checkout page becomes slow:

3. Good habits that make data useful

  • Write structured logs. Use key=value or JSON, so tools can filter by field instead of guessing from sentences.
  • Put a trace or request ID in every log line. It is the thread that ties logs, metrics, and traces together.
  • Keep metric labels few and bounded. Labels like "region" are fine. A label with a different value for every user (high cardinality) makes metrics slow and expensive.
  • Measure what users feel, not only what servers do.
  • Never log secrets or personal data such as passwords, tokens, or card numbers.
  • Keep time consistent. Use one time zone (usually UTC) across systems.

4. Hands-on labs

Safety notes: All commands are read-only or touch only files under /tmp/sre-lab that you create. Run them on a system you are authorized to use. Never run cleanup or deletion commands on real data until you have reviewed a dry run, and do not paste logs containing credentials or customer data into public posts.

Setup: create two practice files

You will make a pretend application log (app.log) with 60 checkout requests, and a pretend trace record (spans.csv) that says how long each service spent on each request. Every tenth request is slow, every twelfth one fails, and nothing real is touched.

mkdir -p /tmp/sre-lab && cd /tmp/sre-lab
awk 'BEGIN{
  for(r=1;r<=60;r++){
    m=int((r-1)/20); sec=(r*3)%60
    st=200; lvl="INFO"; msg="ok"
    d=100+(r%7)*10
    if(r%10==0) d=2500
    if(r%12==0){st=500; lvl="ERROR"; msg="db_timeout"}
    printf "time=10:%02d:%02d level=%s trace=t%03d status=%d dur_ms=%d msg=%s\n",
           m, sec, lvl, r, st, d, msg
  }}' > app.log

awk 'BEGIN{print "trace,service,parent,duration_ms"
  for(r=1;r<=60;r++){ db=40+(r%5)*5; if(r%10==0) db=2400
    api=db+70; web=api+30
    printf "t%03d,web,-,%d\nt%03d,api,web,%d\n", r,web,r,api
    printf "t%03d,db,api,%d\n", r,db }}' > spans.csv

Check what you made:

head -2 app.log
wc -l app.log spans.csv

Expected output

time=10:00:03 level=INFO trace=t001 status=200 dur_ms=110 msg=ok
time=10:00:06 level=INFO trace=t002 status=200 dur_ms=120 msg=ok
  60 app.log
 181 spans.csv
 241 total

Each log line is a set of key=value pairs. That is a structured log. The 181 lines in spans.csv are 1 heading plus 60 requests x 3 services (web, api, db).

Lab 1: Metrics: turn log lines into numbers

Goal: Count requests, errors, and response times for each minute.

Why it matters: A metric is just an event log squeezed into a number per time slot. This is what dashboards and alerts are built on.

printf "%-7s %8s %6s %7s %7s\n" minute requests errors avg_ms max_ms
awk '{ for(i=1;i<=NF;i++){ split($i,kv,"="); v[kv[1]]=kv[2] }
       split(v["time"],t,":"); m=t[2]; n[m]++
       if (v["status"]>=500) e[m]++
       s[m]+=v["dur_ms"]; if (v["dur_ms"]>mx[m]) mx[m]=v["dur_ms"] }
END{ for(m in n)
       printf "10:%-4s %8d %6d %7.0f %7d\n",
              m, n[m], e[m], s[m]/n[m], mx[m] }' app.log | sort

Expected output

minute  requests errors  avg_ms  max_ms
10:00         20      1     367    2500
10:01         20      2     365    2500
10:02         20      2     366    2500

How to read the result: Each row is one minute. Every minute has 20 requests and 1 or 2 errors. The average response time is about 366 ms, which sounds fine. But the max is 2,500 ms every minute, so some users wait two and a half seconds. This is the same lesson as Day 5: a single average hides slow requests.

Try it yourself: Change v["status"]>=500 to v["dur_ms"]>1000 to count slow requests per minute instead of errors.

Lab 2: Logs: find the story behind the numbers

Goal: Filter the log to see what the errors were, and follow one request.

Why it matters: Metrics say how many. Logs say what. This is the second step of an investigation.

grep 'level=ERROR' app.log | head -3
awk '{split($2,a,"="); c[a[2]]++} END{for(k in c) print k, c[k]}' app.log | sort
grep 'trace=t012' app.log

Expected output

time=10:00:36 level=ERROR trace=t012 status=500 dur_ms=150 msg=db_timeout
time=10:01:12 level=ERROR trace=t024 status=500 dur_ms=130 msg=db_timeout
time=10:01:48 level=ERROR trace=t036 status=500 dur_ms=110 msg=db_timeout
ERROR 5
INFO 55
time=10:00:36 level=ERROR trace=t012 status=500 dur_ms=150 msg=db_timeout

How to read the result: All three errors say msg=db_timeout. The database is the common factor. Out of 60 requests, 5 are errors (about 8%). Because every line has a trace= ID, you could pull out the whole story of one request. In a real system, that ID would be shared across all services.

Try it yourself: Run grep -c 'msg=db_timeout' app.log. The -c option prints a count instead of lines.

Lab 3: Traces: see where the time went

Goal: Look at one slow request and find which service used the time.

Why it matters: A trace breaks one request into steps called spans. It answers "where?", which logs and metrics usually cannot.

🛍️✨ From the author · ByteStoreLinux & Cloud Administration Consultation | AWS & Azure Infrastructure Supportβ‚Ή499View offer →
awk -F, '$2=="web" && $4>1000 {print $1, $4 " ms"}' spans.csv | tr '\n' ';'; echo
grep '^t010,' spans.csv
awk -F, '$1=="t010"{d[$2]=$4}
  END{printf "web self=%d ms, api self=%d ms, db self=%d ms\n",
      d["web"]-d["api"], d["api"]-d["db"], d["db"]}' spans.csv

Expected output

t010 2500 ms;t020 2500 ms;t030 2500 ms;t040 2500 ms;t050 2500 ms;t060 2500 ms;
t010,web,-,2500
t010,api,web,2470
t010,db,api,2400
web self=30 ms, api self=70 ms, db self=2400 ms

How to read the result: Six requests were slow (every tenth one). For t010, the web span lasts 2,500 ms in total, but that includes the api, which includes the db. After subtracting, the web tier spent only 30 ms and the api 70 ms. The database used 2,400 ms. You would not know this from the web server's own timing alone.

Try it yourself: Try t020 instead of t010. The pattern is the same, because every slow request in this sample is a slow database call.

Lab 4: Put it together: metric to log to trace

Goal: Follow one problem through all three pillars, as you would in a real incident.

Why it matters: This is the real skill. Each signal points to the next.

slow=$(awk '{ for(i=1;i<=NF;i++){ split($i,kv,"="); v[kv[1]]=kv[2] }
  if (v["dur_ms"]>mx) { mx=v["dur_ms"]; id=v["trace"] } } END{print id}' app.log)
echo "slowest trace: $slow"
grep "$slow" app.log
grep "^$slow," spans.csv

Expected output

slowest trace: t010
time=10:00:30 level=INFO trace=t010 status=200 dur_ms=2500 msg=ok
t010,web,-,2500
t010,api,web,2470
t010,db,api,2400

How to read the result: Metric: the max response time is 2,500 ms. Log: that request succeeded, so a metric based on errors alone would have missed it. Trace: 2,400 ms was spent in the database. Each signal added something the others could not.

Lab 5 (worksheet): your own service

Fill in this table for a service you support. Any blank cell is an observability gap.

5. Common mistakes

  • Collecting everything and understanding nothing. Start from the questions you need to answer.
  • Logs in free text that cannot be filtered by field
  • No request ID, so logs and traces cannot be matched
  • Metric labels with endless unique values
  • Only watching averages
  • Logging secrets or personal data

6. Quick check: test yourself

  • In one sentence each, what are metrics, logs, and traces?
  • Which signal would you use first to notice a problem, and which to locate it?
  • Why is a trace ID in every log line useful?
  • In Lab 3, why is the db self time larger than the web self time?
  • Which of the three is missing for the service you support?

Suggested answers: numbers over time, timestamped event records, and the path of one request; metrics to notice and traces to locate; it lets you join logs, metrics, and traces for one request; web includes api which includes db, and after subtracting children the database used most of the time; check your worksheet.

Glossary

The first fifteen days at a glance

Your turn: Which of metrics, logs, and traces is weakest for the service you support today?

Coming next, Day 12: Linux monitoring with the USE method.

Follow @leadwithsreeΒ for the daily posts in this 30-day journey.

Read next →SRE in Practice: DAY 1 OF 30 What is SRE?⏱ 6 min

🛍️ Srilakshmi T's shelf

Courses, cohorts and more from the author
Srilakshmi T
Written by Srilakshmi T
Cloud and Linux SRE Engineer
🏅 Contributor writer

Cloud & Linux SRE Engineer with 5 years of IT experience, including Linux/Unix system administration and cloud platforms (AWS & Azure). Strong expertise in production support, reliability engineering, monitoring, automation, patching, and incident management across large-scale e

Visit my ByteStoreFreelancer - Cloud Engineer
Open to workHire me →

Get Srilakshmi T's next blog by email

One email a week when they publish. Add the weekly Best of ByteBlog too. No spam, unsubscribe with one click.

We email you once to confirm. By subscribing you agree to the privacy policy.

💬 Comments (0)

Top · Newest

Sign in to comment

More from Srilakshmi T

🛍️✨ From the author · ByteStoreCareer gap analysis and Mentoring - Basic Planβ‚Ή299View offer →