SRE in Practice | Day 10: MTTR, MTTD, and MTBF
Measure incidents without fooling yourself. Calculate MTTD, MTTR, and MTBF from an incident log, compare mean and median, and connect them to availability. Six hands-on labs.

✍️ Translate this blog into Hindi, Tamil or Telugu
What you will learn
- The incident timeline and the four measures built on it
- Why MTTR must be defined carefully, and why averages can mislead
- How MTBF and MTTR connect to availability
- Six hands-on labs, including a calculation from real timestamps
Safety first: All commands are read-only or touch only files under /tmp/sre-lab that you create. Run them on a system you are authorized to use. Never run cleanup or deletion commands on real data until you have reviewed a dry run, and do not paste logs containing credentials or customer data into public posts.
1. The life of an incident
Every incident has a timeline. The measures we cover today are just time intervals along it:

2. The four measures

A word of warning: The "R" in MTTR is ambiguous. It is used for repair, recovery, response, restore, and resolve, and these can differ a lot. Pick one definition, write it down, and measure it the same way every time. In this guide MTTR means start to service restored.
How they connect to availability
Availability = MTBF / (MTBF + MTTR)
Availability improves if failures get rarer (higher MTBF) or if each failure is shorter (lower MTTR). Lab 4 checks this formula against real numbers.
3. Limits of these measures
- Averages are fragile. One very long incident can distort the mean. Look at the median and the spread too.
- Small samples. With three incidents in a month, the numbers are anecdotes, not statistics.
- Incidents differ. A database failure and a typo in a config file are not comparable.
- Start times are estimates. Many incidents are only noticed afterwards, so the "start" is reconstructed from logs.
- They are lagging. They tell you what happened, not what is happening now. SLOs and burn rates show current user impact.
- Do not use them to rank people. That encourages hiding or reshaping incidents instead of learning from them.
4. How to improve each measure

5. A production-style example
This is an illustrative scenario. Over a month a team has three incidents. The third was found only after 11 minutes, because the main alert watched server CPU and not user errors. The postmortem adds an error-rate alert on the user-facing endpoint. The lab quantifies what a faster detection time would have saved.
6. Hands-on labs
Safety notes: All commands are read-only or touch only files under /tmp/sre-lab that you create. Run them on a system you are authorized to use. Never run cleanup or deletion commands on real data until you have reviewed a dry run, and do not paste logs containing credentials or customer data into public posts. The date commands in Lab 1 use GNU date, which is standard on Linux. On macOS install coreutils and use gdate instead.
Setup: create an incident log
mkdir -p /tmp/sre-lab && cd /tmp/sre-labcat > incidents.csv <<'EOF'
id,started,detected,resolved
INC-1,2026-10-03T02:10:00,2026-10-03T02:14:00,2026-10-03T02:40:00
INC-2,2026-10-11T14:00:00,2026-10-11T14:03:00,2026-10-11T14:25:00
INC-3,2026-10-19T09:30:00,2026-10-19T09:41:00,2026-10-19T10:30:00
EOFHere "resolved" is the time service was restored. Times are in one time zone, so any offset cancels out in the differences.
Lab 1: turn timestamps into durations
while IFS=, read id started detected resolved; do
[ "$id" = "id" ] && continue
ts=$(date -d "$started" +%s); td=$(date -d "$detected" +%s)
tr=$(date -d "$resolved" +%s)
echo "$id $(( (td-ts)/60 )) $(( (tr-ts)/60 ))"
done < incidents.csv > durations.txt
cat durations.txtExpected output
INC-1 4 30
INC-2 3 25
INC-3 11 60The columns are the incident id, minutes from start to detection, and minutes from start to restoration.
Lab 2: MTTD and MTTR
awk '{d+=$2; r+=$3}
END{printf "incidents=%d MTTD=%.1f min MTTR(mean)=%.1f min\n",
NR, d/NR, r/NR}' durations.txtExpected output
incidents=3 MTTD=6.0 min MTTR(mean)=38.3 minLab 3: why the median matters
awk '{print $3}' durations.txt | sort -n | awk '{a[NR]=$1}
END{m=(NR%2) ? a[(NR+1)/2] : (a[NR/2]+a[NR/2+1])/2
printf "MTTR(median)=%.1f min\n", m}'Expected output
MTTR(median)=30.0 minThe mean (38.3 min) is pulled up by the single 60-minute incident, while the median (30.0 min) shows a typical incident. If the third incident had lasted 600 minutes, the mean would jump but the median would not. Report both.
Lab 4: MTBF, and check the availability formula
🛍️✨ From the author · ByteStoreLinux & Cloud Administration Consultation | AWS & Azure Infrastructure Supportโน499View offer →awk -v period=43200 '{down+=$3; r+=$3; n++}
END{up=period-down; mtbf=up/n; mttr=r/n
printf "downtime=%d min MTBF=%.0f min (%.1f h)\n", down, mtbf, mtbf/60
printf "availability=%.4f%% MTBF/(MTBF+MTTR)=%.4f%%\n",
up/period*100, mtbf/(mtbf+mttr)*100}' durations.txtExpected output
downtime=115 min MTBF=14362 min (239.4 h)
availability=99.7338% MTBF/(MTBF+MTTR)=99.7338%The period is 30 days (43,200 minutes). Total downtime was 115 minutes across three failures, so uptime was 43,085 minutes and MTBF is about 14,362 minutes (roughly 10 days). The two availability figures match, which confirms the formula. Against a 99.9% SLO (43.2 minutes allowed), this month missed its target.
Lab 5: what if detection were faster?
Suppose better alerting cut every detection time in half and everything else stayed the same. Each incident then ends sooner by the minutes saved.
awk -v cut=0.5 '{d+=$2; r+=$3}
END{down=r-cut*d
printf "downtime now=%d min; with %.0f%% faster detection=%.1f min\n",
r, cut*100, down
printf "availability now=%.4f%% after=%.4f%%\n",
(43200-r)/43200*100, (43200-down)/43200*100}' durations.txtExpected output
downtime now=115 min; with 50% faster detection=106.0 min
availability now=99.7338% after=99.7546%Halving detection time saves 9 minutes in this sample. It is a modest gain, which is a useful lesson: detection is one lever, and the bigger gains here would come from shorter repair times and fewer incidents. The model is simplified because real recovery time can also depend on how quickly the problem was noticed.
Lab 6 (worksheet): start your own incident log
Record at least these columns for every incident, and fill in real times from your monitoring and ticket history:

7. Common mistakes
- Using MTTR without saying what it means
- Reporting only the mean, not the median or the spread
- Drawing conclusions from three incidents
- Using the numbers to compare or blame people
- Treating restoration as the end, and skipping the postmortem and follow-up actions
- Measuring detection from when the alert fired instead of from when users were affected
8. Quick check: test yourself
- Define MTTD, MTTR, and MTBF in one line each.
- Why is the median sometimes better than the mean for incident durations?
- If MTBF is 200 hours and MTTR is 1 hour, what is availability?
- Name two ways to reduce MTTD and two to reduce MTTR.
- Why should these measures not be used to rank engineers?
Suggested answers: time to notice, time to restore, and average operating time between failures; one long incident can skew the mean while the median shows the typical case; 200 / 201 = 99.50%; symptom-based and burn-rate alerts, and runbooks and one-step rollback; it encourages hiding incidents rather than learning from them.
Glossary

The first ten days at a glance

Your turn: How does your team define MTTR today, and where does the incident clock start?
Coming next, Day 11: metrics, logs, and traces. This starts Phase 3, Observability and Monitoring.
Follow @leadwithsree for the daily posts in this 30-day journey.
🛍️ Srilakshmi T's shelf
Courses, cohorts and more from the authorGet Srilakshmi T's next blog by email
One email a week when they publish. Add the weekly Best of ByteBlog too. No spam, unsubscribe with one click.
We email you once to confirm. By subscribing you agree to the privacy policy.
More from Srilakshmi T

SRE in Practice: DAY 1 OF 30 What is SRE?
Why a running server does not always mean a working service. Concept, production example,and a safe hands-on lab.
#aws
SRE in Practice | Day 3: The SRE Mindset
Look beyond "Is the server up?" and ask whether the service works reliably for the people who depend on it. Concept, infrastructure example, and a hands-on lab.

SRE in Practice | Day 2: SRE vs DevOps
Two complementary perspectives on delivering and running reliable software. Concept, production example, and a hands-on release-mapping lab.

SRE in Practice | Day 9: Error Budgets
Turn the SLO into a budget you can spend. Learn burn rates, burn-rate alerts, and error budget policies. Six hands-on labs with verified expected results.

💬 Comments (0)
Top · NewestSign in to comment