SRE in Practice — Day 4 of 30 Toil and Automation

What you will learn

• What toil is, and what it is not
• How to measure toil and calculate automation payback
• How to write safe automation: exit codes, dry runs, and idempotency
• Five hands-on labs with expected output, all using a throwaway folder

Safety first

All commands are read-only or touch only files under /tmp/sre-lab that you create. Run them on a system you are authorized to use. Never run cleanup or deletion commands on real data until you have reviewed a dry run, and do not paste logs containing credentials or customer data into public posts.

1. What is toil?

In SRE, toil is operational work that is manual, repetitive, automatable, tactical, and that leaves the service no better afterwards. Its cost grows in step with the service: twice as many servers or users means twice as much toil.

Good to know: Not all unpleasant work is toil. Engineering projects, on-call judgement, and writing a postmortem are not toil even when they are hard. Google's SRE guidance suggests keeping toil to no more than about half of an SRE's time, so the rest goes to engineering that reduces future work.

2. Why automation is an SRE habit

Automation is how SRE turns one person's repeated effort into a permanent improvement. It also reduces human error, makes actions consistent and auditable, and frees time for work that improves reliability. The goal is not to automate everything. The goal is to automate the work that is repetitive, well understood, and worth the effort.

The automation ladder

3. A production-style example

This is an illustrative scenario. A small team supports 40 Linux VMs on Azure and AWS. A week of their operational work includes the four tasks below. The lab turns this into numbers.

4. The automation ROI calculation

A simple rule:

Payback time = hours to automate ÷ hours saved per week

For account creation: 15 min × 10 runs = 150 min (2.5 h) per week. Automating it takes 6 hours, so payback is 6 / 2.5 = 2.4 weeks. After that, the saving is yours every week. Remember to include the cost of maintaining the automation.

5. Hands-on labs

Safety notes

All commands are read-only or touch only files under /tmp/sre-lab that you create. Run them on a system you are authorized to use. Never run cleanup or deletion commands on real data until you have reviewed a dry run, and do not paste logs containing credentials or customer data into public posts.

Setup

mkdir -p /tmp/sre-lab && cd /tmp/sre-lab

Lab 1: Build a toil inventory and rank by payback

Create the inventory as a CSV, then calculate weekly toil and payback for each task.

cat > toil.csv <<'EOF'
task,minutes_per_run,runs_per_week,automation_hours
restart_stuck_service,10,6,4
rotate_logs_manually,20,3,3
create_user_accounts,15,10,6
check_certificate_expiry,5,10,2
EOF

awk -F, 'NR>1{w=$2*$3
  printf "%5.1f weeks %-26s %4d min/week\n", $4*60/w, $1, w}' toil.csv | sort -n

awk -F, 'NR>1{t+=$2*$3}
  END{printf "TOTAL %d min/week = %.1f h/week = %.1f%% of a 40h week\n",
      t, t/60, t/60/40*100}' toil.csv

Expected output

  2.4 weeks check_certificate_expiry      50 min/week
  2.4 weeks create_user_accounts         150 min/week
  3.0 weeks rotate_logs_manually          60 min/week
  4.0 weeks restart_stuck_service         60 min/week
TOTAL 320 min/week = 5.3 h/week = 13.3% of a 40h week

The two tasks with the shortest payback are the best first candidates. Rows with equal payback may appear in either order. Try replacing the sample rows with your own team's real tasks.

Lab 2: Automate a check, with an exit code

A good operational script reports clearly and returns a meaningful exit code, so other tools can act on it.

cat > check_disk.sh <<'EOF'
#!/usr/bin/env bash
# usage: ./check_disk.sh <mount> <threshold-percent>
mount="${1:-/}"; limit="${2:-80}"
used=$(df -P "$mount" | awk 'NR==2 {gsub("%","",$5); print $5}')
if [ "$used" -ge "$limit" ]; then
   echo "WARN: $mount is ${used}% used (threshold ${limit}%)"; exit 1
fi
echo "OK: $mount is ${used}% used (threshold ${limit}%)"
EOF

chmod +x check_disk.sh

./check_disk.sh / 99; echo "exit=$?"
./check_disk.sh / 1; echo "exit=$?"  # forces the warning path

Expected output

OK: / is 49% used (threshold 99%)
exit=0
WARN: / is 49% used (threshold 1%)
exit=1

Your usage percentage will differ. Setting the threshold to 1% on purpose proves the warning path works, which is how you test any alert-style script.

Lab 3: Safe cleanup with a dry run first

Never delete first and check later. List what would be removed, review it, then delete. This lab uses files you create in a throwaway folder.

mkdir -p /tmp/sre-lab/logs && cd /tmp/sre-lab/logs
for d in 1 5 10 20 30; do touch -d "$d days ago" app-$d.log; done

find . -name '*.log' -mtime +7 -print   # DRY RUN: what would be removed?
find . -name '*.log' -mtime +7 -delete  # only after reviewing the list
ls

Expected output

./app-10.log
./app-20.log
./app-30.log
app-1.log
app-5.log

The three files older than 7 days are listed first (in any order), then removed, leaving the two recent files. The same dry-run habit applies to any bulk action in production.

Lab 4: Make a change idempotent

An idempotent action can run many times and leave the same result. This is what makes automation safe to re-run.

cd /tmp/sre-lab
for run in 1 2; do
  grep -qxF "max_connections=100" app.conf 2>/dev/null \
    || echo "max_connections=100" >> app.conf
done
cat app.conf
wc -l < app.conf

Expected output

max_connections=100
1

The setting is added once even though the loop ran twice, because the command checks before it writes. Configuration-management and IaC tools such as Ansible and Terraform are built on this idea.

Lab 5 (read and discuss): Schedule it

A script becomes level 3 on the ladder when it runs without a person. A typical cron entry runs the check every 15 minutes and keeps a log. Do not install schedules on shared or production systems without approval.

*/15 * * * * /opt/sre/check_disk.sh / 80 >> /var/log/check_disk.log 2>&1

Cron runs with a minimal environment, so use full paths. A check that only writes to a log is still easy to ignore, so a real version would alert someone when the exit code is non-zero. Systemd timers are a modern alternative with better logging.

6. Common mistakes

• Automating a process nobody understands, which speeds up mistakes
• Scripts that delete or change things with no dry run or confirmation
• Automation with no logging or alerting, so failures stay silent
• Treating a one-off fix as done, when the cause still produces the toil
• Never maintaining automation, until it quietly breaks
• Calling all hard or boring work toil, and automating the wrong things

7. Quick check: Test yourself

1. Name the six traits of toil and give one example from your own work.
2. Is writing a postmortem toil? Why or why not?
3. Using the lab numbers, why is account creation a better first candidate than log rotation?
4. What does idempotent mean, and why does it matter for automation?
5. Which task in your week would reach payback in under a month if automated?

Suggested answers: manual, repetitive, automatable, tactical, no enduring value, scales with growth; a postmortem is engineering work that improves the service, so it is not toil; account creation pays back in 2.4 weeks against 3.0; an idempotent action leaves the same result however many times it runs; check your own inventory.

Glossary

The first ten days at a glance

Your turn: Which repetitive task takes the most of your week, and what would it take to automate it?

Coming next, Day 5: the Four Golden Signals and reliability principles.

Follow Srilakshmi T for the daily posts in this 30-day journey.