SRE in Practice | Day 3: The SRE Mindset

What you will learn

• Why a server can be UP while the service is failing its users
• The difference between an infrastructure check and an SRE-minded check
• Why SRE alerts on symptoms users feel, and uses causes to diagnose
• How logs, metrics, traces, and dependency status fit together in an investigation
• The five habits of the SRE mindset, with a practical example for each
• A safe, read-only lab that measures success rate and latency from the user's side

Safety first: Everything in the lab is read-only and lightweight. Run it only on systems you are authorized to test, and do not restart services or change configuration in production without approval.

1. A server can be UP and still failing your users

Dashboards show the virtual machine running. The web server process is active. Yet users are getting errors. This is where the SRE mindset begins.

The mindset is a change in the question. Instead of stopping at "Is the infrastructure running?", an SRE asks "Is the service working reliably for the people who depend on it?"

Core idea: SRE applies software-engineering practices to operating services, and measures reliability from the user's perspective. It brings together monitoring, automation, incident response, and continuous improvement.

2. A practical infrastructure example

This is an illustrative scenario. An application runs on a Linux VM in Azure or AWS.

• The VM is running
• The web server process is active
• But the application cannot connect to its database, so users receive errors

A basic server check looks healthy. An SRE-minded check asks different questions:

The second column needs application health alongside infrastructure signals: logs, metrics, traces, and dependency status.

3. Symptoms versus causes

A symptom is something users feel: failed requests, slow responses, errors on a page. A cause is something inside the system: a stopped database, a full disk, a saturated connection pool, a bad deployment.

SRE teams generally alert on symptoms, because a symptom means users are affected right now. Cause-level signals remain valuable for diagnosis, and some (such as a disk predicted to fill within hours) deserve alerts too because they predict a symptom before users feel it.

4. Using the signals together

No single signal explains an incident. Each answers a different question. In our scenario they combine like this:

On cloud platforms, tools such as Azure Monitor (with Application Insights and Log Analytics) and Amazon CloudWatch help correlate application and infrastructure signals in one place. Which metrics exist depends on how your environment is instrumented.

5. Hands-on lab

Safety notes: All commands are read-only and light. Run them on a system you are authorized to test. Do not restart services, change configuration, or paste logs containing credentials or customer data into public posts.

Step 1: check an endpoint

Replace the URL with a health endpoint of an application you are authorized to test.

curl -i --max-time 5 http://localhost/health

Important: A successful response tells you that one endpoint answered one request. It does not prove that every user journey or dependency is healthy. A shallow health endpoint may return 200 whenever the process is alive, even if the database is down. A deeper check that also tests critical dependencies gives a more honest picture, but should itself be lightweight and safe.

Step 2: measure success rate and latency from the user's side

This sends 20 light requests, half a second apart, and summarises the results.

for i in $(seq 1 20); do
  curl -s -o /dev/null --max-time 5 \
    -w "%{http_code} %{time_total}\n" http://localhost/health
  sleep 0.5
done | awk '{t++; if ($1 ~ /^2/) ok++; sum+=$2; if ($2>max) max=$2}
END {printf "requests=%d success=%d success_rate=%.1f%% avg=%.3fs max=%.3fs\n",
     t, ok+0, (ok+0)/t*100, sum/t, max}'

Expected on a healthy system (numbers will differ): requests=20 success=20 success_rate=100.0% avg=0.012s max=0.030s. If the application is failing, the success rate drops and failed requests show up as non-2xx codes (a failed connection prints 000).

Keep it honest: Twenty probes is a smoke test, not a reliability measurement. Real SLIs are calculated from far more traffic over a period of time. We build them properly on Days 6-10.

Step 3: look one layer down (read-only)

If the checks fail, work down the layers. Replace myapp, the host, and the port with your own.

systemctl status myapp --no-pager       # is the service running?
journalctl -u myapp -n 50 --no-pager    # recent application logs
nc -zv db-host 5432                     # can we reach the database port?

Look in the logs for connection errors to the database. A reachable port still does not prove the application's credentials or capacity are fine. Service names, log locations, and tools vary by system, and some commands need sudo or may not be installed.

Optional: failure rate in Application Insights

If your application sends telemetry to a workspace-based Application Insights resource, this Log Analytics query shows failed requests over time. Table and column names differ for classic resources.

AppRequests
| where TimeGenerated > ago(30m)
| summarize total = count(),
            failed = countif(Success == false)
            by bin(TimeGenerated, 5m)
| extend failure_rate_pct = 100.0 * failed / total
| order by TimeGenerated asc

6. The SRE mindset in practice

Five habits turn the mindset into daily work:

7. Common mistakes

• Treating a green host dashboard as proof the service works
• Trusting a shallow health endpoint that never tests dependencies
• Alerting on every cause-level metric, which creates noisy pages nobody trusts
• Measuring only the home page while login or checkout is broken
• Fixing the incident but skipping the review, so it happens again
• Judging reliability from a handful of manual checks instead of real user traffic

8. Quick check: test yourself

• A VM shows running and CPU is normal, but users see errors. Name three questions an SRE-minded check asks.
• Give one symptom and one cause for the database outage scenario.
• Why might /health return 200 while users are failing?
• Which signal would show where in a request path the failure happens?
• Have you seen a healthy-looking server while the application was failing? What signal revealed the real issue?

Suggested answers: are requests succeeding, is latency acceptable, are users seeing errors, are dependencies healthy; a symptom is the rising 5xx rate and a cause is the unreachable database; a shallow health endpoint may only check that the process is alive; traces show where in the request path time or failure occurs.

Glossary

The 30-day roadmap at a glance

Your turn: Have you ever encountered a healthy-looking server while the application was failing? What signal helped you find the real issue? Share it in the comments on the LinkedIn post.

Follow @leadwithsree for more from the 30 Days of SRE Learning Journey.