systemd Services: Run Your App Forever on Linux
What you build: a small web app run as a systemd service that restarts itself after crashes, a kill -9 and a full reboot, on a free e2-micro VM. Every command tested.

✍️ Translate this blog into Hindi, Tamil or Telugu
In this lab you turn a small web app into a systemd service: a program Linux starts at boot, watches all the time and restarts by itself when it crashes. You crash it on purpose, break its unit file with a typo, reboot the whole server and change a setting without touching the main file, and each time you see exactly what systemd does.
Who it is for: beginners who can open a Linux terminal and want their app to keep running after they close it. Time: about 1 hour. Cost: free. An e2-micro VM in us-east1 is in Google Cloud's Always Free tier (one per account), and we delete it at the end.
Every command below was run for real by the CareerByteCode Team on a fresh Google Cloud project, and every output and screenshot is the real result we got. Only process ids (PIDs) and times will differ on your machine.
🧪⭐ Recommended by CareerByteCode · ByteLabs project bundleCloud and DevOps Realtime Starterโน499Build the projects →Part A: systemd in plain English
When Linux boots, the very first program it starts is systemd (process number 1). systemd then starts everything else: SSH, the network, web servers and your app. It also keeps an eye on each of them for as long as the server runs.

Words you will use:
- Service: a program that runs in the background with no terminal, such as a web server. Real-life example: the watchman of a building, who is on duty even when nobody is looking.
- Unit file: a small text file (ending in
.service) that tells systemd how to run one service: which command, as which user, and what to do if it stops. Real-life example: the duty sheet you give the watchman. - systemctl: the command you use to talk to systemd:
start,stop,restart,status,enable. - enable: start the service at every boot. start: start it now.
enable --nowdoes both. - Restart policy:
Restart=on-failuremeans "if the app dies with an error, start it again". - journald and journalctl: systemd collects everything a service prints into the journal, and
journalctl -u <name>reads it. - daemon-reload: tells systemd to read the unit files again after you change one.
Why not just run python3 app.py & in the terminal? Because that app stops when your SSH session ends, does not come back after a crash, does not start after a reboot, and its output is lost. Step 1 shows this for real.
Before you start
- A Google Cloud account (the free trial works) and a project.
- Open Cloud Shell (the
>_icon at the top right of the Google Cloud console) and set your project:
export PROJECT_ID="your-project-id"
gcloud config set project $PROJECT_ID
gcloud services enable compute.googleapis.comPart B: the lab
Step 1: a free e2-micro VM
Why: a small Linux server (Debian 12) to practise on. --no-service-account --no-scopes gives the VM no Google Cloud permissions, because this lab needs none.
gcloud compute instances create hello-lab --zone=us-east1-b --machine-type=e2-micro \
--image-family=debian-12 --image-project=debian-cloud --boot-disk-size=10GB --boot-disk-type=pd-standard \
--no-service-account --no-scopes 2>&1 | tail -2
until gcloud compute ssh hello-lab --zone=us-east1-b --tunnel-through-iap --command="echo ssh ok" 2>/dev/null | grep -q ok; do sleep 10; done; echo "SSH works"NAME ZONE MACHINE_TYPE PREEMPTIBLE INTERNAL_IP EXTERNAL_IP STATUS
hello-lab us-east1-b e2-micro 10.142.0.3 EXTERNAL_IP RUNNING
SSH worksConnect to it. In the console, Compute Engine > VM instances > SSH next to hello-lab opens a terminal in your browser, or from Cloud Shell:
gcloud compute ssh hello-lab --zone=us-east1-bEverything from Step 2 runs inside the VM, in a folder ~/lab265. Create the two files of the lab there: app.py (the app) and hello.service (its unit file). Both are shown in full below, so you can paste them with nano app.py and nano hello.service.
# A tiny web app for the systemd lab: answers on port 8080, and /crash makes it die (exit code 1) on purpose.
import http.server, os, time
START = time.strftime('%H:%M:%S')
class Handler(http.server.BaseHTTPRequestHandler):
def do_GET(self):
if self.path == '/crash':
print('crash requested, exiting with code 1', flush=True)
os._exit(1)
body = f'hello from pid {os.getpid()}, started {START}, greeting={os.environ.get("GREETING", "hi")}\n'.encode()
self.send_response(200); self.end_headers(); self.wfile.write(body)
def log_message(self, fmt, *args):
print('request', self.path, flush=True)
print('app starting on port 8080', flush=True)
http.server.HTTPServer(('0.0.0.0', 8080), Handler).serve_forever()The app answers on port 8080 with its PID, the time it started and a greeting read from the GREETING setting. The address /crash makes it exit with code 1 on purpose, so we can test what happens when an app crashes.
Step 2: run the app by hand (and see why that is not enough)
Why: first prove the app works, then prove its weakness. & runs it in the background of your terminal, curl asks it for the page, and kill %1 stops that background job, just like closing the terminal would.
# Run the app by hand first: it works, but only while this terminal is open
cat app.py | head -3
python3 app.py > manual.log 2>&1 &
sleep 2; curl -s localhost:8080/
kill %1; sleep 1; curl -s localhost:8080/ || echo "curl exit $?: nothing listens after the terminal job stops"# A tiny web app for the systemd lab: answers on port 8080, and /crash makes it die (exit code 1) on purpose.
import http.server, os, time
START = time.strftime('%H:%M:%S')
hello from pid 1064, started 19:56:53, greeting=hi
curl exit 7: nothing listens after the terminal job stops
curl exit 7 means "could not connect". Nobody restarted the app. That is the job systemd will do for us.
Step 3: install the app and its unit file
Why: a service should run as its own user with no login (useradd --system), from a fixed folder (/opt/hello), with its unit file in /etc/systemd/system/. Here is the unit file:
[Unit]
Description=Hello demo app (systemd lab)
After=network-online.target
Wants=network-online.target
StartLimitIntervalSec=60
StartLimitBurst=5
[Service]
User=hello
WorkingDirectory=/opt/hello
Environment=GREETING=namaste
ExecStart=/usr/bin/python3 /opt/hello/app.py
Restart=on-failure
RestartSec=3
[Install]
WantedBy=multi-user.targetWhat each line means:
| Line | Meaning |
|---|---|
Description= | The name you see in systemctl status |
After= / Wants=network-online.target | Start only once the network is up |
StartLimitIntervalSec=60, StartLimitBurst=5 | At most 5 starts in 60 seconds, so a broken app cannot restart forever (tested in Step 9) |
User=hello | Run as the user hello, not as root |
WorkingDirectory= | The folder the app runs in |
Environment=GREETING=namaste | A setting the app reads |
ExecStart= | The exact command, with full paths |
Restart=on-failure, RestartSec=3 | If it dies with an error, wait 3 seconds and start it again |
WantedBy=multi-user.target | enable hooks it into normal boot |
Now install and start it. daemon-reload makes systemd read the new file, and enable --now starts it now and at every boot.
# Install the app and its unit file, then start it with systemd
sudo useradd --system --no-create-home --shell /usr/sbin/nologin hello
sudo mkdir -p /opt/hello && sudo cp app.py /opt/hello/
sudo cp hello.service /etc/systemd/system/hello.service
cat /etc/systemd/system/hello.service
sudo systemctl daemon-reload
sudo systemctl enable --now hello
systemctl status hello --no-pager | head -9[Unit]
Description=Hello demo app (systemd lab)
After=network-online.target
Wants=network-online.target
StartLimitIntervalSec=60
StartLimitBurst=5
...
Tasks: 1 (limit: 1071)
Memory: 2.2M
CPU: 9ms
CGroup: /system.slice/hello.service
โโ1155 /usr/bin/python3 /opt/hello/app.py
Active: active (running) and a Main PID mean systemd started it and is watching it. The Created symlink line is what enable does: it links the unit into the boot target.
Step 4: check it with curl and journalctl
Why: "running" is not proof. Ask the app itself, check which user it runs as, and read its logs from the journal.
# Is it working? curl it and read its logs with journalctl
curl -s localhost:8080/
ps -o user,pid,cmd -C python3
journalctl -u hello --no-pager -n 5 -o cathello from pid 1155, started 19:57:08, greeting=namaste
USER PID CMD
hello 1155 /usr/bin/python3 /opt/hello/app.py
Started hello.service - Hello demo app (systemd lab).
app starting on port 8080
request /The app now runs as the user hello, its greeting comes from the unit file (namaste), and everything it printed is in the journal. -o cat prints just the messages, and -n 5 the last 5 lines.
Break it: crash the app
We call /crash, so the app dies in the middle of the request, just like an app that hits a bug. Then we watch systemd.
# Break it: crash the app on purpose, then watch systemd restart it
systemctl show hello -p MainPID -p NRestarts
curl -s localhost:8080/crash; echo "curl exit $? (the app died mid-request)"
sleep 1; systemctl is-active hello
sleep 4; systemctl is-active hello
curl -s localhost:8080/
systemctl show hello -p MainPID -p NRestarts
journalctl -u hello --no-pager -n 8 -o short | cut -c17- | sed 's/^[^ ]* //'MainPID=1155
NRestarts=0
curl exit 52 (the app died mid-request)
activating
active
hello from pid 1233, started 19:57:32, greeting=namaste
MainPID=1233
NRestarts=1
python3[1155]: crash requested, exiting with code 1
systemd[1]: hello.service: Main process exited, code=exited, status=1/FAILURE
systemd[1]: hello.service: Failed with result 'exit-code'.
systemd[1]: hello.service: Scheduled restart job, restart counter is at 1.
systemd[1]: Stopped hello.service - Hello demo app (systemd lab).
systemd[1]: Started hello.service - Hello demo app (systemd lab).
python3[1233]: app starting on port 8080
python3[1233]: request /
Read it from the top:
curl exit 52means the server closed the connection without an answer: the app died.- One second later the state is
activating: systemd is waiting the 3 seconds ofRestartSec. - Then it is
activeagain, with a new PID (1233 instead of 1155) andNRestarts=1. - The journal tells the whole story:
status=1/FAILURE,Scheduled restart job, restart counter is at 1,Started.
You did nothing. That is the point of a service.
A kill -9 and a clean stop
Why: kill -9 (the hard kill) is a crash too, so systemd restarts the app. But systemctl stop is you asking it to stop, so systemd leaves it stopped. That is why Restart=on-failure is the usual choice: it restarts after errors, never against your wish.
# kill -9 is a crash too, but systemctl stop is a clean stop: no restart
sudo kill -9 $(systemctl show hello -p MainPID --value)
sleep 5; systemctl is-active hello; systemctl show hello -p NRestarts
sudo systemctl stop hello; sleep 5
systemctl is-active hello
sudo systemctl start hello; systemctl is-active helloactive
NRestarts=2
inactive
activeThe restart counter went up to 2 after kill -9, and after systemctl stop the state stayed inactive until we started it again.
Break it: a typo in the unit file
The most common real mistake: ExecStart=/usr/bin/python ... instead of python3. Debian 12 has no /usr/bin/python. We also make the second most common mistake: we restart before running daemon-reload.
# Break it: a typo in ExecStart (python instead of python3) and a forgotten daemon-reload
sudo sed -i 's|/usr/bin/python3 |/usr/bin/python |' /etc/systemd/system/hello.service
sudo systemctl restart hello
sudo systemctl daemon-reload
sudo systemctl restart hello; sleep 2
systemctl status hello --no-pager | head -8
journalctl -u hello --no-pager -n 4 -o catWarning: The unit file, source configuration file or drop-ins of hello.service changed on disk. Run 'systemctl daemon-reload' to reload units.
โ hello.service - Hello demo app (systemd lab)
Loaded: loaded (/etc/systemd/system/hello.service; enabled; preset: enabled)
Active: activating (auto-restart) (Result: exit-code) since Wed 2026-11-18 19:58:04 UTC; 2s ago
Process: 1348 ExecStart=/usr/bin/python /opt/hello/app.py (code=exited, status=203/EXEC)
Main PID: 1348 (code=exited, status=203/EXEC)
CPU: 1ms
hello.service: Failed to locate executable /usr/bin/python: No such file or directory
hello.service: Failed at step EXEC spawning /usr/bin/python: No such file or directory
hello.service: Main process exited, code=exited, status=203/EXEC
hello.service: Failed with result 'exit-code'.
Two lessons:
- The first
restartprintedWarning: The unit file ... changed on disk. Run 'systemctl daemon-reload'. Without daemon-reload systemd still uses the old copy it read earlier, so that restart worked with the old, correct line. - After daemon-reload, the start fails with
status=203/EXEC. 203/EXEC always means "the program in ExecStart could not be run": a wrong path, a missing file or a missing execute permission. The journal says it plainly:Failed to locate executable /usr/bin/python.
Fix it: correct the path, check, reload, restart
Why: systemd-analyze verify checks a unit file before you use it (it prints nothing when the file is fine). reset-failed clears the failed state and its start counter.
# Fix it: correct the path, reload, restart, check
sudo sed -i 's|/usr/bin/python |/usr/bin/python3 |' /etc/systemd/system/hello.service
sudo systemd-analyze verify /etc/systemd/system/hello.service && echo "unit file OK"
sudo systemctl daemon-reload
sudo systemctl reset-failed hello
sudo systemctl restart hello; sleep 2
systemctl is-active hello; curl -s localhost:8080/unit file OK
active
hello from pid 1422, started 19:58:18, greeting=namasteTip: find the real path of a program with command -v python3 and always use full paths in ExecStart.
Step 5: reboot the whole VM
Why: the real test of "run forever". From Cloud Shell (not inside the VM), reboot the VM, wait until SSH works again, then check the app without starting anything by hand.
# Reboot the whole VM (run from Cloud Shell), then check the app came back on its own
gcloud compute ssh hello-lab --zone=us-east1-b --tunnel-through-iap --command="sudo systemctl reboot" 2>/dev/null; echo "reboot sent"
sleep 30
until gcloud compute ssh hello-lab --zone=us-east1-b --tunnel-through-iap --command="true" 2>/dev/null; do sleep 10; done
gcloud compute ssh hello-lab --zone=us-east1-b --tunnel-through-iap --command="uptime -p; systemctl is-enabled hello; systemctl is-active hello; curl -s localhost:8080/" 2>/dev/nullreboot sent
up 0 minutes
enabled
active
hello from pid 425, started 19:58:51, greeting=namaste
up 0 minutes proves the server really rebooted, enabled and active prove systemd started the app at boot, and the app answers with a new PID and a start time after the reboot.
Step 6: change a setting with a drop-in override
Why: never edit a unit file that a package installed, because an update can overwrite it. Put your change in a drop-in file: /etc/systemd/system/hello.service.d/override.conf. systemd reads the main file first and then the drop-in, and the later value wins. (sudo systemctl edit hello creates the same file in an editor.)
# Change a setting without editing the main file: a drop-in override
sudo mkdir -p /etc/systemd/system/hello.service.d
printf '[Service]\nEnvironment=GREETING=vanakkam\n' | sudo tee /etc/systemd/system/hello.service.d/override.conf
sudo systemctl daemon-reload && sudo systemctl restart hello; sleep 2
curl -s localhost:8080/
systemctl cat hello --no-pager | grep -E "^#|GREETING"[Service]
Environment=GREETING=vanakkam
hello from pid 1009, started 19:59:34, greeting=vanakkam
# /etc/systemd/system/hello.service
Environment=GREETING=namaste
# /etc/systemd/system/hello.service.d/override.conf
Environment=GREETING=vanakkam
systemctl cat hello shows the unit as systemd sees it: the main file, then the drop-in. The app now greets with vanakkam.
Step 7: the restart limit
Why: if an app crashes the moment it starts (a bad config, a missing database), restarting it forever only fills the logs. Our unit allows 5 starts in 60 seconds. We crash the app 6 times in a row.
# Crash it 6 times in a row: StartLimitBurst=5 stops the restart loop
for i in 1 2 3 4 5 6; do curl -s localhost:8080/crash > /dev/null; sleep 4; done
systemctl is-active hello
systemctl status hello --no-pager | grep -E "Active|start-limit" | head -3
sudo systemctl reset-failed hello && sudo systemctl start hello && sleep 2 && curl -s localhost:8080/failed
Active: failed (Result: exit-code) since Wed 2026-11-18 20:00:05 UTC; 5s ago
hello from pid 1082, started 20:00:10, greeting=vanakkamAfter the 5 automatic restarts systemd gives up and marks the service failed. You fix the real cause, then run systemctl reset-failed and start it again, as the last line shows.
Step 8: remove the service
Why: clean up inside the VM. disable --now stops it and removes the boot link. After deleting the files, daemon-reload makes systemd forget the unit.
# Remove the service from the VM (before deleting the VM itself)
sudo systemctl disable --now hello
sudo rm -rf /etc/systemd/system/hello.service /etc/systemd/system/hello.service.d /opt/hello
sudo systemctl daemon-reload
systemctl list-unit-files hello.service --no-pager | tail -1Removed "/etc/systemd/system/multi-user.target.wants/hello.service".
0 unit files listed.0 unit files listed. proves systemd no longer knows the unit. (systemctl list-unit-files exits with code 1 when it finds nothing, which is why this step's exit code is 1. Here that is the result we want.)
POC: prove it works
- An app that died by itself (exit code 1) and by
kill -9came back on its own, with a new PID each time (NRestarts=1, then2). - A clean
systemctl stopwas respected: no restart. - A typo in
ExecStartwas found fromstatus=203/EXECand the journal, then fixed. - After a full VM reboot the app was
enabledandactiveand answered without any manual step. - A drop-in override changed a setting without editing the main unit file.
- The restart limit stopped a crash loop after 5 restarts.

Quiz task: check what you learned
- Your app runs from
/home/ana/app/server.jswith Node.js. Write theExecStartline. *(Answer:ExecStart=/usr/bin/node /home/ana/app/server.js. Find the Node path withcommand -v node.)* - You edited the unit file but nothing changed. What did you forget? *(Answer:
sudo systemctl daemon-reload, then restart.)* systemctl statusshowsstatus=203/EXEC. What do you check first? *(Answer: the path inExecStartand that the file exists and can be run.)*- How do you see the last 20 log lines of the service? *(Answer:
journalctl -u hello -n 20.)* - You want the app to restart even after a clean exit with code 0. Which setting? *(Answer:
Restart=always.)*
Clean up (delete everything)
Why: the e2-micro is free for one VM per account, and its disk counts too. Exit the VM (exit), then in Cloud Shell:
gcloud compute instances delete hello-lab --zone=us-east1-b --quiet 2>&1 | tail -1
echo "hello-lab VMs left: $(gcloud compute instances list --filter="name=hello-lab" --format="value(name)" | wc -l)"
echo "hello-lab disks left: $(gcloud compute disks list --filter="name~^hello-lab" --format="value(name)" | wc -l)"Deleted [https://www.googleapis.com/compute/v1/projects/your-project-id/zones/us-east1-b/instances/hello-lab].
hello-lab VMs left: 0
hello-lab disks left: 0
FAQ
**What is the difference between start and enable?** start runs the service now, once. enable makes it start at every boot. Use enable --now for both.
**Restart=on-failure or Restart=always?** on-failure restarts after a crash, a kill or an error exit code, but not after a clean exit. always restarts even after a clean exit. Web apps usually use on-failure. Use always for programs that should never stop.
Where do I put my own unit files? In /etc/systemd/system/. Files from packages live in /usr/lib/systemd/system/. Do not edit those; use a drop-in instead.
Can I run a service without root? Yes. User= runs it as another user (as in this lab). Also, systemctl --user services run in a user's own session without sudo.
How do I pass secrets to a service? Not in the unit file, because anyone on the server can read it with systemctl cat. Use EnvironmentFile=/etc/hello/env with that file at chmod 600, or a secret store like Secret Manager.
Is this how Docker and Kubernetes work too? The idea is the same: a supervisor that restarts what crashes. On one Linux server, systemd does it. In Kubernetes the kubelet does it for containers, with restartPolicy.
Troubleshooting
| You see | Why | Fix |
|---|---|---|
status=203/EXEC | The program in ExecStart is missing or not executable | Use the full path (command -v python3), check ls -l |
Warning: The unit file ... changed on disk | You edited the file but systemd has the old copy | sudo systemctl daemon-reload, then restart |
status=200/CHDIR | WorkingDirectory does not exist | Create the folder or fix the path |
status=217/USER | The user in User= does not exist | sudo useradd --system ... or fix the name |
Start request repeated too quickly | The start limit was hit | Fix the crash, sudo systemctl reset-failed <name>, start |
| App works by hand but not as a service | Different user, folder or environment | Check User=, WorkingDirectory=, Environment= and read journalctl -u <name> |
What next
- New to the Linux terminal? Start with Linux for Cloud Beginners: 50 Commands on a Free VM.
- Put a web server in front of your service: Nginx for Beginners: Web Server, Reverse Proxy and HTTPS.
- Watch the server it runs on: Monitor Linux Servers with Node Exporter and Ops Agent.
- Automate the setup: Bash Scripting for DevOps: Automate Boring Tasks.
- More hands-on posts: ByteBlog. Practice in ByteLabs and see the courses.
Prepared by the CareerByteCode Team from our tested lab, with AI help for the write-up.
⭐ Keep learning with CareerByteCode
Hand-picked to go deeper on this topicGet CareerByteCode Team's next blog by email
One email a week when they publish. Add the weekly Best of ByteBlog too. No spam, unsubscribe with one click.
We email you once to confirm. By subscribing you agree to the privacy policy.
Related blogs
CareerByteCode TeamCron Jobs and Log Rotation on Linux
What you build: a disk report job run by cron every minute with its log rotated by logrotate, on a free VM, plus 3 real mistakes: relative paths, the % sign and an insecure log folder.
#beginners
CareerByteCode TeamLinux Users, Groups and sudo Explained
What you build: a shared Linux server for 3 people on a free VM: users, a developers group, a sudo rule that allows one job only, visudo catching a typo, the sudo log, and an account locked and remove
#beginners
CareerByteCode TeamFrom Windows to Linux: A Gentle Start for Beginners
What you build: everyday Windows tasks (files, search, zip, Task Manager, services, apps, network, admin) done the Linux way on a free VM, plus the 3 classic surprises and their fixes.
#beginners
CareerByteCode TeamLinux for Cloud Beginners: 50 Commands on a Free VM
What you build: a free e2-micro Linux VM where you practise 50 everyday commands for files, text, permissions, processes, network and packages, in your own Google Cloud account. Every step tested.
💬 Comments (0)
Top · NewestSign in to comment