CareerByteCode
ByteBlogby CareerByteCode

SRE in Practice | Day 10: MTTR, MTTD, and MTBF

Measure incidents without fooling yourself. Calculate MTTD, MTTR, and MTBF from an incident log, compare mean and median, and connect them to availability. Six hands-on labs.

SRE in Practice | Day 10: MTTR, MTTD, and MTBF
📚 Part 10 of 10 in SRE in Practise - 30 days

✍️ Translate this blog into Hindi, Tamil or Telugu

What you will learn

  • The incident timeline and the four measures built on it
  • Why MTTR must be defined carefully, and why averages can mislead
  • How MTBF and MTTR connect to availability
  • Six hands-on labs, including a calculation from real timestamps

Safety first: All commands are read-only or touch only files under /tmp/sre-lab that you create. Run them on a system you are authorized to use. Never run cleanup or deletion commands on real data until you have reviewed a dry run, and do not paste logs containing credentials or customer data into public posts.


1. The life of an incident

Every incident has a timeline. The measures we cover today are just time intervals along it:

2. The four measures

🛍️✨ From the author · ByteStoreCareer gap analysis and Mentoring - Basic Planโ‚น299View offer →

A word of warning: The "R" in MTTR is ambiguous. It is used for repair, recovery, response, restore, and resolve, and these can differ a lot. Pick one definition, write it down, and measure it the same way every time. In this guide MTTR means start to service restored.

How they connect to availability

Availability = MTBF / (MTBF + MTTR)

Availability improves if failures get rarer (higher MTBF) or if each failure is shorter (lower MTTR). Lab 4 checks this formula against real numbers.

3. Limits of these measures

  • Averages are fragile. One very long incident can distort the mean. Look at the median and the spread too.
  • Small samples. With three incidents in a month, the numbers are anecdotes, not statistics.
  • Incidents differ. A database failure and a typo in a config file are not comparable.
  • Start times are estimates. Many incidents are only noticed afterwards, so the "start" is reconstructed from logs.
  • They are lagging. They tell you what happened, not what is happening now. SLOs and burn rates show current user impact.
  • Do not use them to rank people. That encourages hiding or reshaping incidents instead of learning from them.

4. How to improve each measure

5. A production-style example

This is an illustrative scenario. Over a month a team has three incidents. The third was found only after 11 minutes, because the main alert watched server CPU and not user errors. The postmortem adds an error-rate alert on the user-facing endpoint. The lab quantifies what a faster detection time would have saved.

6. Hands-on labs

Safety notes: All commands are read-only or touch only files under /tmp/sre-lab that you create. Run them on a system you are authorized to use. Never run cleanup or deletion commands on real data until you have reviewed a dry run, and do not paste logs containing credentials or customer data into public posts. The date commands in Lab 1 use GNU date, which is standard on Linux. On macOS install coreutils and use gdate instead.

Setup: create an incident log

mkdir -p /tmp/sre-lab && cd /tmp/sre-lab
cat > incidents.csv <<'EOF'
id,started,detected,resolved
INC-1,2026-10-03T02:10:00,2026-10-03T02:14:00,2026-10-03T02:40:00
INC-2,2026-10-11T14:00:00,2026-10-11T14:03:00,2026-10-11T14:25:00
INC-3,2026-10-19T09:30:00,2026-10-19T09:41:00,2026-10-19T10:30:00
EOF

Here "resolved" is the time service was restored. Times are in one time zone, so any offset cancels out in the differences.

Lab 1: turn timestamps into durations

while IFS=, read id started detected resolved; do
  [ "$id" = "id" ] && continue
  ts=$(date -d "$started" +%s); td=$(date -d "$detected" +%s)
  tr=$(date -d "$resolved" +%s)
  echo "$id $(( (td-ts)/60 )) $(( (tr-ts)/60 ))"
done < incidents.csv > durations.txt

cat durations.txt

Expected output

INC-1 4 30
INC-2 3 25
INC-3 11 60

The columns are the incident id, minutes from start to detection, and minutes from start to restoration.

Lab 2: MTTD and MTTR

awk '{d+=$2; r+=$3}
  END{printf "incidents=%d  MTTD=%.1f min  MTTR(mean)=%.1f min\n",
             NR, d/NR, r/NR}' durations.txt

Expected output

incidents=3  MTTD=6.0 min  MTTR(mean)=38.3 min

Lab 3: why the median matters

awk '{print $3}' durations.txt | sort -n | awk '{a[NR]=$1}
  END{m=(NR%2) ? a[(NR+1)/2] : (a[NR/2]+a[NR/2+1])/2
      printf "MTTR(median)=%.1f min\n", m}'

Expected output

MTTR(median)=30.0 min

The mean (38.3 min) is pulled up by the single 60-minute incident, while the median (30.0 min) shows a typical incident. If the third incident had lasted 600 minutes, the mean would jump but the median would not. Report both.

Lab 4: MTBF, and check the availability formula

🛍️✨ From the author · ByteStoreLinux & Cloud Administration Consultation | AWS & Azure Infrastructure Supportโ‚น499View offer →
awk -v period=43200 '{down+=$3; r+=$3; n++}
  END{up=period-down; mtbf=up/n; mttr=r/n
      printf "downtime=%d min  MTBF=%.0f min (%.1f h)\n", down, mtbf, mtbf/60
      printf "availability=%.4f%%  MTBF/(MTBF+MTTR)=%.4f%%\n",
             up/period*100, mtbf/(mtbf+mttr)*100}' durations.txt

Expected output

downtime=115 min  MTBF=14362 min (239.4 h)
availability=99.7338%  MTBF/(MTBF+MTTR)=99.7338%

The period is 30 days (43,200 minutes). Total downtime was 115 minutes across three failures, so uptime was 43,085 minutes and MTBF is about 14,362 minutes (roughly 10 days). The two availability figures match, which confirms the formula. Against a 99.9% SLO (43.2 minutes allowed), this month missed its target.

Lab 5: what if detection were faster?

Suppose better alerting cut every detection time in half and everything else stayed the same. Each incident then ends sooner by the minutes saved.

awk -v cut=0.5 '{d+=$2; r+=$3}
  END{down=r-cut*d
      printf "downtime now=%d min; with %.0f%% faster detection=%.1f min\n",
             r, cut*100, down
      printf "availability now=%.4f%%  after=%.4f%%\n",
             (43200-r)/43200*100, (43200-down)/43200*100}' durations.txt

Expected output

downtime now=115 min; with 50% faster detection=106.0 min
availability now=99.7338%  after=99.7546%

Halving detection time saves 9 minutes in this sample. It is a modest gain, which is a useful lesson: detection is one lever, and the bigger gains here would come from shorter repair times and fewer incidents. The model is simplified because real recovery time can also depend on how quickly the problem was noticed.

Lab 6 (worksheet): start your own incident log

Record at least these columns for every incident, and fill in real times from your monitoring and ticket history:

7. Common mistakes

  • Using MTTR without saying what it means
  • Reporting only the mean, not the median or the spread
  • Drawing conclusions from three incidents
  • Using the numbers to compare or blame people
  • Treating restoration as the end, and skipping the postmortem and follow-up actions
  • Measuring detection from when the alert fired instead of from when users were affected

8. Quick check: test yourself

  • Define MTTD, MTTR, and MTBF in one line each.
  • Why is the median sometimes better than the mean for incident durations?
  • If MTBF is 200 hours and MTTR is 1 hour, what is availability?
  • Name two ways to reduce MTTD and two to reduce MTTR.
  • Why should these measures not be used to rank engineers?

Suggested answers: time to notice, time to restore, and average operating time between failures; one long incident can skew the mean while the median shows the typical case; 200 / 201 = 99.50%; symptom-based and burn-rate alerts, and runbooks and one-step rollback; it encourages hiding incidents rather than learning from them.

Glossary

The first ten days at a glance

Your turn: How does your team define MTTR today, and where does the incident clock start?

Coming next, Day 11: metrics, logs, and traces. This starts Phase 3, Observability and Monitoring.

Follow @leadwithsree for the daily posts in this 30-day journey.

Read next →SRE in Practice: DAY 1 OF 30 What is SRE?⏱ 6 min

🛍️ Srilakshmi T's shelf

Courses, cohorts and more from the author
Srilakshmi T
Written by Srilakshmi T
Cloud and Linux SRE Engineer
🏅 Contributor writer

Cloud & Linux SRE Engineer with 5 years of IT experience, including Linux/Unix system administration and cloud platforms (AWS & Azure). Strong expertise in production support, reliability engineering, monitoring, automation, patching, and incident management across large-scale e

Visit my ByteStoreFreelancer - Cloud Engineer
Open to workHire me →

Get Srilakshmi T's next blog by email

One email a week when they publish. Add the weekly Best of ByteBlog too. No spam, unsubscribe with one click.

We email you once to confirm. By subscribing you agree to the privacy policy.

💬 Comments (0)

Top · Newest

Sign in to comment

More from Srilakshmi T

🛍️✨ From the author · ByteStoreCareer gap analysis and Mentoring - Basic Planโ‚น299View offer →