CareerByteCode
ByteBlogby CareerByteCode

systemd Services: Run Your App Forever on Linux

What you build: a small web app run as a systemd service that restarts itself after crashes, a kill -9 and a full reboot, on a free e2-micro VM. Every command tested.

systemd Services: Run Your App Forever on Linux

✍️ Translate this blog into Hindi, Tamil or Telugu

In this lab you turn a small web app into a systemd service: a program Linux starts at boot, watches all the time and restarts by itself when it crashes. You crash it on purpose, break its unit file with a typo, reboot the whole server and change a setting without touching the main file, and each time you see exactly what systemd does.

Who it is for: beginners who can open a Linux terminal and want their app to keep running after they close it. Time: about 1 hour. Cost: free. An e2-micro VM in us-east1 is in Google Cloud's Always Free tier (one per account), and we delete it at the end.

Every command below was run for real by the CareerByteCode Team on a fresh Google Cloud project, and every output and screenshot is the real result we got. Only process ids (PIDs) and times will differ on your machine.

🧪⭐ Recommended by CareerByteCode · ByteLabs project bundleCloud and DevOps Realtime Starterโ‚น499Build the projects →

Part A: systemd in plain English

When Linux boots, the very first program it starts is systemd (process number 1). systemd then starts everything else: SSH, the network, web servers and your app. It also keeps an eye on each of them for as long as the server runs.

systemd in one picture
You write a unit file, systemctl hands it to systemd, systemd starts and watches the app, and journald keeps its logs

Words you will use:

  • Service: a program that runs in the background with no terminal, such as a web server. Real-life example: the watchman of a building, who is on duty even when nobody is looking.
  • Unit file: a small text file (ending in .service) that tells systemd how to run one service: which command, as which user, and what to do if it stops. Real-life example: the duty sheet you give the watchman.
  • systemctl: the command you use to talk to systemd: start, stop, restart, status, enable.
  • enable: start the service at every boot. start: start it now. enable --now does both.
  • Restart policy: Restart=on-failure means "if the app dies with an error, start it again".
  • journald and journalctl: systemd collects everything a service prints into the journal, and journalctl -u <name> reads it.
  • daemon-reload: tells systemd to read the unit files again after you change one.

Why not just run python3 app.py & in the terminal? Because that app stops when your SSH session ends, does not come back after a crash, does not start after a reboot, and its output is lost. Step 1 shows this for real.

Before you start

  1. A Google Cloud account (the free trial works) and a project.
  2. Open Cloud Shell (the >_ icon at the top right of the Google Cloud console) and set your project:
export PROJECT_ID="your-project-id"
gcloud config set project $PROJECT_ID
gcloud services enable compute.googleapis.com

Part B: the lab

Step 1: a free e2-micro VM

Why: a small Linux server (Debian 12) to practise on. --no-service-account --no-scopes gives the VM no Google Cloud permissions, because this lab needs none.

gcloud compute instances create hello-lab --zone=us-east1-b --machine-type=e2-micro \
  --image-family=debian-12 --image-project=debian-cloud --boot-disk-size=10GB --boot-disk-type=pd-standard \
  --no-service-account --no-scopes 2>&1 | tail -2
until gcloud compute ssh hello-lab --zone=us-east1-b --tunnel-through-iap --command="echo ssh ok" 2>/dev/null | grep -q ok; do sleep 10; done; echo "SSH works"
NAME       ZONE        MACHINE_TYPE  PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP  STATUS
hello-lab  us-east1-b  e2-micro                   10.142.0.3   EXTERNAL_IP  RUNNING
SSH works

Connect to it. In the console, Compute Engine > VM instances > SSH next to hello-lab opens a terminal in your browser, or from Cloud Shell:

gcloud compute ssh hello-lab --zone=us-east1-b

Everything from Step 2 runs inside the VM, in a folder ~/lab265. Create the two files of the lab there: app.py (the app) and hello.service (its unit file). Both are shown in full below, so you can paste them with nano app.py and nano hello.service.

# A tiny web app for the systemd lab: answers on port 8080, and /crash makes it die (exit code 1) on purpose.
import http.server, os, time
START = time.strftime('%H:%M:%S')
class Handler(http.server.BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path == '/crash':
            print('crash requested, exiting with code 1', flush=True)
            os._exit(1)
        body = f'hello from pid {os.getpid()}, started {START}, greeting={os.environ.get("GREETING", "hi")}\n'.encode()
        self.send_response(200); self.end_headers(); self.wfile.write(body)
    def log_message(self, fmt, *args):
        print('request', self.path, flush=True)
print('app starting on port 8080', flush=True)
http.server.HTTPServer(('0.0.0.0', 8080), Handler).serve_forever()

The app answers on port 8080 with its PID, the time it started and a greeting read from the GREETING setting. The address /crash makes it exit with code 1 on purpose, so we can test what happens when an app crashes.

Step 2: run the app by hand (and see why that is not enough)

Why: first prove the app works, then prove its weakness. & runs it in the background of your terminal, curl asks it for the page, and kill %1 stops that background job, just like closing the terminal would.

# Run the app by hand first: it works, but only while this terminal is open
cat app.py | head -3
python3 app.py > manual.log 2>&1 &
sleep 2; curl -s localhost:8080/
kill %1; sleep 1; curl -s localhost:8080/ || echo "curl exit $?: nothing listens after the terminal job stops"
# A tiny web app for the systemd lab: answers on port 8080, and /crash makes it die (exit code 1) on purpose.
import http.server, os, time
START = time.strftime('%H:%M:%S')
hello from pid 1064, started 19:56:53, greeting=hi
curl exit 7: nothing listens after the terminal job stops
Run by hand
The app answers while the terminal job runs; after kill, curl exit 7 means nothing listens any more

curl exit 7 means "could not connect". Nobody restarted the app. That is the job systemd will do for us.

Step 3: install the app and its unit file

Why: a service should run as its own user with no login (useradd --system), from a fixed folder (/opt/hello), with its unit file in /etc/systemd/system/. Here is the unit file:

[Unit]
Description=Hello demo app (systemd lab)
After=network-online.target
Wants=network-online.target
StartLimitIntervalSec=60
StartLimitBurst=5

[Service]
User=hello
WorkingDirectory=/opt/hello
Environment=GREETING=namaste
ExecStart=/usr/bin/python3 /opt/hello/app.py
Restart=on-failure
RestartSec=3

[Install]
WantedBy=multi-user.target

What each line means:

LineMeaning
Description=The name you see in systemctl status
After= / Wants=network-online.targetStart only once the network is up
StartLimitIntervalSec=60, StartLimitBurst=5At most 5 starts in 60 seconds, so a broken app cannot restart forever (tested in Step 9)
User=helloRun as the user hello, not as root
WorkingDirectory=The folder the app runs in
Environment=GREETING=namasteA setting the app reads
ExecStart=The exact command, with full paths
Restart=on-failure, RestartSec=3If it dies with an error, wait 3 seconds and start it again
WantedBy=multi-user.targetenable hooks it into normal boot

Now install and start it. daemon-reload makes systemd read the new file, and enable --now starts it now and at every boot.

# Install the app and its unit file, then start it with systemd
sudo useradd --system --no-create-home --shell /usr/sbin/nologin hello
sudo mkdir -p /opt/hello && sudo cp app.py /opt/hello/
sudo cp hello.service /etc/systemd/system/hello.service
cat /etc/systemd/system/hello.service
sudo systemctl daemon-reload
sudo systemctl enable --now hello
systemctl status hello --no-pager | head -9
[Unit]
Description=Hello demo app (systemd lab)
After=network-online.target
Wants=network-online.target
StartLimitIntervalSec=60
StartLimitBurst=5
...
      Tasks: 1 (limit: 1071)
     Memory: 2.2M
        CPU: 9ms
     CGroup: /system.slice/hello.service
             โ””โ”€1155 /usr/bin/python3 /opt/hello/app.py
Unit installed
enable --now creates the boot link and the service is active (running) as PID 1155

Active: active (running) and a Main PID mean systemd started it and is watching it. The Created symlink line is what enable does: it links the unit into the boot target.

Step 4: check it with curl and journalctl

Why: "running" is not proof. Ask the app itself, check which user it runs as, and read its logs from the journal.

# Is it working? curl it and read its logs with journalctl
curl -s localhost:8080/
ps -o user,pid,cmd -C python3
journalctl -u hello --no-pager -n 5 -o cat
hello from pid 1155, started 19:57:08, greeting=namaste
USER         PID CMD
hello       1155 /usr/bin/python3 /opt/hello/app.py
Started hello.service - Hello demo app (systemd lab).
app starting on port 8080
request /

The app now runs as the user hello, its greeting comes from the unit file (namaste), and everything it printed is in the journal. -o cat prints just the messages, and -n 5 the last 5 lines.

Break it: crash the app

We call /crash, so the app dies in the middle of the request, just like an app that hits a bug. Then we watch systemd.

# Break it: crash the app on purpose, then watch systemd restart it
systemctl show hello -p MainPID -p NRestarts
curl -s localhost:8080/crash; echo "curl exit $? (the app died mid-request)"
sleep 1; systemctl is-active hello
sleep 4; systemctl is-active hello
curl -s localhost:8080/
systemctl show hello -p MainPID -p NRestarts
journalctl -u hello --no-pager -n 8 -o short | cut -c17- | sed 's/^[^ ]* //'
MainPID=1155
NRestarts=0
curl exit 52 (the app died mid-request)
activating
active
hello from pid 1233, started 19:57:32, greeting=namaste
MainPID=1233
NRestarts=1
python3[1155]: crash requested, exiting with code 1
systemd[1]: hello.service: Main process exited, code=exited, status=1/FAILURE
systemd[1]: hello.service: Failed with result 'exit-code'.
systemd[1]: hello.service: Scheduled restart job, restart counter is at 1.
systemd[1]: Stopped hello.service - Hello demo app (systemd lab).
systemd[1]: Started hello.service - Hello demo app (systemd lab).
python3[1233]: app starting on port 8080
python3[1233]: request /
Crash and restart
The app dies with exit code 1, systemd schedules a restart, and 3 seconds later a new PID answers: NRestarts=1

Read it from the top:

  • curl exit 52 means the server closed the connection without an answer: the app died.
  • One second later the state is activating: systemd is waiting the 3 seconds of RestartSec.
  • Then it is active again, with a new PID (1233 instead of 1155) and NRestarts=1.
  • The journal tells the whole story: status=1/FAILURE, Scheduled restart job, restart counter is at 1, Started.

You did nothing. That is the point of a service.

A kill -9 and a clean stop

Why: kill -9 (the hard kill) is a crash too, so systemd restarts the app. But systemctl stop is you asking it to stop, so systemd leaves it stopped. That is why Restart=on-failure is the usual choice: it restarts after errors, never against your wish.

# kill -9 is a crash too, but systemctl stop is a clean stop: no restart
sudo kill -9 $(systemctl show hello -p MainPID --value)
sleep 5; systemctl is-active hello; systemctl show hello -p NRestarts
sudo systemctl stop hello; sleep 5
systemctl is-active hello
sudo systemctl start hello; systemctl is-active hello
active
NRestarts=2
inactive
active

The restart counter went up to 2 after kill -9, and after systemctl stop the state stayed inactive until we started it again.

Break it: a typo in the unit file

The most common real mistake: ExecStart=/usr/bin/python ... instead of python3. Debian 12 has no /usr/bin/python. We also make the second most common mistake: we restart before running daemon-reload.

# Break it: a typo in ExecStart (python instead of python3) and a forgotten daemon-reload
sudo sed -i 's|/usr/bin/python3 |/usr/bin/python |' /etc/systemd/system/hello.service
sudo systemctl restart hello
sudo systemctl daemon-reload
sudo systemctl restart hello; sleep 2
systemctl status hello --no-pager | head -8
journalctl -u hello --no-pager -n 4 -o cat
Warning: The unit file, source configuration file or drop-ins of hello.service changed on disk. Run 'systemctl daemon-reload' to reload units.
โ— hello.service - Hello demo app (systemd lab)
     Loaded: loaded (/etc/systemd/system/hello.service; enabled; preset: enabled)
     Active: activating (auto-restart) (Result: exit-code) since Wed 2026-11-18 19:58:04 UTC; 2s ago
    Process: 1348 ExecStart=/usr/bin/python /opt/hello/app.py (code=exited, status=203/EXEC)
   Main PID: 1348 (code=exited, status=203/EXEC)
        CPU: 1ms
hello.service: Failed to locate executable /usr/bin/python: No such file or directory
hello.service: Failed at step EXEC spawning /usr/bin/python: No such file or directory
hello.service: Main process exited, code=exited, status=203/EXEC
hello.service: Failed with result 'exit-code'.
Bad unit file
systemd warns the file changed on disk; after daemon-reload the start fails with status=203/EXEC: the executable does not exist

Two lessons:

  1. The first restart printed Warning: The unit file ... changed on disk. Run 'systemctl daemon-reload'. Without daemon-reload systemd still uses the old copy it read earlier, so that restart worked with the old, correct line.
  2. After daemon-reload, the start fails with status=203/EXEC. 203/EXEC always means "the program in ExecStart could not be run": a wrong path, a missing file or a missing execute permission. The journal says it plainly: Failed to locate executable /usr/bin/python.

Fix it: correct the path, check, reload, restart

Why: systemd-analyze verify checks a unit file before you use it (it prints nothing when the file is fine). reset-failed clears the failed state and its start counter.

# Fix it: correct the path, reload, restart, check
sudo sed -i 's|/usr/bin/python |/usr/bin/python3 |' /etc/systemd/system/hello.service
sudo systemd-analyze verify /etc/systemd/system/hello.service && echo "unit file OK"
sudo systemctl daemon-reload
sudo systemctl reset-failed hello
sudo systemctl restart hello; sleep 2
systemctl is-active hello; curl -s localhost:8080/
unit file OK
active
hello from pid 1422, started 19:58:18, greeting=namaste

Tip: find the real path of a program with command -v python3 and always use full paths in ExecStart.

🎓⭐ Recommended by CareerByteCode · Live trainingCloud & DevOps Live Cohort Internshipโ‚น2,360See the training →

Step 5: reboot the whole VM

Why: the real test of "run forever". From Cloud Shell (not inside the VM), reboot the VM, wait until SSH works again, then check the app without starting anything by hand.

# Reboot the whole VM (run from Cloud Shell), then check the app came back on its own
gcloud compute ssh hello-lab --zone=us-east1-b --tunnel-through-iap --command="sudo systemctl reboot" 2>/dev/null; echo "reboot sent"
sleep 30
until gcloud compute ssh hello-lab --zone=us-east1-b --tunnel-through-iap --command="true" 2>/dev/null; do sleep 10; done
gcloud compute ssh hello-lab --zone=us-east1-b --tunnel-through-iap --command="uptime -p; systemctl is-enabled hello; systemctl is-active hello; curl -s localhost:8080/" 2>/dev/null
reboot sent
up 0 minutes
enabled
active
hello from pid 425, started 19:58:51, greeting=namaste
After reboot
After the reboot: up 0 minutes, enabled, active, and the app answers with a new PID

up 0 minutes proves the server really rebooted, enabled and active prove systemd started the app at boot, and the app answers with a new PID and a start time after the reboot.

Step 6: change a setting with a drop-in override

Why: never edit a unit file that a package installed, because an update can overwrite it. Put your change in a drop-in file: /etc/systemd/system/hello.service.d/override.conf. systemd reads the main file first and then the drop-in, and the later value wins. (sudo systemctl edit hello creates the same file in an editor.)

# Change a setting without editing the main file: a drop-in override
sudo mkdir -p /etc/systemd/system/hello.service.d
printf '[Service]\nEnvironment=GREETING=vanakkam\n' | sudo tee /etc/systemd/system/hello.service.d/override.conf
sudo systemctl daemon-reload && sudo systemctl restart hello; sleep 2
curl -s localhost:8080/
systemctl cat hello --no-pager | grep -E "^#|GREETING"
[Service]
Environment=GREETING=vanakkam
hello from pid 1009, started 19:59:34, greeting=vanakkam
# /etc/systemd/system/hello.service
Environment=GREETING=namaste
# /etc/systemd/system/hello.service.d/override.conf
Environment=GREETING=vanakkam
Drop-in override
The override file sets GREETING=vanakkam; systemctl cat shows both files and the app uses the new value

systemctl cat hello shows the unit as systemd sees it: the main file, then the drop-in. The app now greets with vanakkam.

Step 7: the restart limit

Why: if an app crashes the moment it starts (a bad config, a missing database), restarting it forever only fills the logs. Our unit allows 5 starts in 60 seconds. We crash the app 6 times in a row.

# Crash it 6 times in a row: StartLimitBurst=5 stops the restart loop
for i in 1 2 3 4 5 6; do curl -s localhost:8080/crash > /dev/null; sleep 4; done
systemctl is-active hello
systemctl status hello --no-pager | grep -E "Active|start-limit" | head -3
sudo systemctl reset-failed hello && sudo systemctl start hello && sleep 2 && curl -s localhost:8080/
failed
     Active: failed (Result: exit-code) since Wed 2026-11-18 20:00:05 UTC; 5s ago
hello from pid 1082, started 20:00:10, greeting=vanakkam

After the 5 automatic restarts systemd gives up and marks the service failed. You fix the real cause, then run systemctl reset-failed and start it again, as the last line shows.

Step 8: remove the service

Why: clean up inside the VM. disable --now stops it and removes the boot link. After deleting the files, daemon-reload makes systemd forget the unit.

# Remove the service from the VM (before deleting the VM itself)
sudo systemctl disable --now hello
sudo rm -rf /etc/systemd/system/hello.service /etc/systemd/system/hello.service.d /opt/hello
sudo systemctl daemon-reload
systemctl list-unit-files hello.service --no-pager | tail -1
Removed "/etc/systemd/system/multi-user.target.wants/hello.service".
0 unit files listed.

0 unit files listed. proves systemd no longer knows the unit. (systemctl list-unit-files exits with code 1 when it finds nothing, which is why this step's exit code is 1. Here that is the result we want.)

POC: prove it works

  • An app that died by itself (exit code 1) and by kill -9 came back on its own, with a new PID each time (NRestarts=1, then 2).
  • A clean systemctl stop was respected: no restart.
  • A typo in ExecStart was found from status=203/EXEC and the journal, then fixed.
  • After a full VM reboot the app was enabled and active and answered without any manual step.
  • A drop-in override changed a setting without editing the main unit file.
  • The restart limit stopped a crash loop after 5 restarts.
Quiz and POC
Step 7: after 6 crashes in a row systemd marks the service failed; reset-failed and start bring it back

Quiz task: check what you learned

  1. Your app runs from /home/ana/app/server.js with Node.js. Write the ExecStart line. *(Answer: ExecStart=/usr/bin/node /home/ana/app/server.js. Find the Node path with command -v node.)*
  2. You edited the unit file but nothing changed. What did you forget? *(Answer: sudo systemctl daemon-reload, then restart.)*
  3. systemctl status shows status=203/EXEC. What do you check first? *(Answer: the path in ExecStart and that the file exists and can be run.)*
  4. How do you see the last 20 log lines of the service? *(Answer: journalctl -u hello -n 20.)*
  5. You want the app to restart even after a clean exit with code 0. Which setting? *(Answer: Restart=always.)*

Clean up (delete everything)

Why: the e2-micro is free for one VM per account, and its disk counts too. Exit the VM (exit), then in Cloud Shell:

gcloud compute instances delete hello-lab --zone=us-east1-b --quiet 2>&1 | tail -1
echo "hello-lab VMs left: $(gcloud compute instances list --filter="name=hello-lab" --format="value(name)" | wc -l)"
echo "hello-lab disks left: $(gcloud compute disks list --filter="name~^hello-lab" --format="value(name)" | wc -l)"
Deleted [https://www.googleapis.com/compute/v1/projects/your-project-id/zones/us-east1-b/instances/hello-lab].
hello-lab VMs left: 0
hello-lab disks left: 0
Clean up
The hello-lab VM and its disk are deleted: 0 left

FAQ

**What is the difference between start and enable?** start runs the service now, once. enable makes it start at every boot. Use enable --now for both.

**Restart=on-failure or Restart=always?** on-failure restarts after a crash, a kill or an error exit code, but not after a clean exit. always restarts even after a clean exit. Web apps usually use on-failure. Use always for programs that should never stop.

Where do I put my own unit files? In /etc/systemd/system/. Files from packages live in /usr/lib/systemd/system/. Do not edit those; use a drop-in instead.

Can I run a service without root? Yes. User= runs it as another user (as in this lab). Also, systemctl --user services run in a user's own session without sudo.

How do I pass secrets to a service? Not in the unit file, because anyone on the server can read it with systemctl cat. Use EnvironmentFile=/etc/hello/env with that file at chmod 600, or a secret store like Secret Manager.

Is this how Docker and Kubernetes work too? The idea is the same: a supervisor that restarts what crashes. On one Linux server, systemd does it. In Kubernetes the kubelet does it for containers, with restartPolicy.

Troubleshooting

You seeWhyFix
status=203/EXECThe program in ExecStart is missing or not executableUse the full path (command -v python3), check ls -l
Warning: The unit file ... changed on diskYou edited the file but systemd has the old copysudo systemctl daemon-reload, then restart
status=200/CHDIRWorkingDirectory does not existCreate the folder or fix the path
status=217/USERThe user in User= does not existsudo useradd --system ... or fix the name
Start request repeated too quicklyThe start limit was hitFix the crash, sudo systemctl reset-failed <name>, start
App works by hand but not as a serviceDifferent user, folder or environmentCheck User=, WorkingDirectory=, Environment= and read journalctl -u <name>

What next

Prepared by the CareerByteCode Team from our tested lab, with AI help for the write-up.

Read next →Cron Jobs and Log Rotation on Linux⏱ 13 min

⭐ Keep learning with CareerByteCode

Hand-picked to go deeper on this topic
News, guides and updates from the CareerByteCode team
🏅 Expert writer

The CareerByteCode team writes about what is new on careerbytecode.in: courses, labs, exams, events and the community, which is free to join for influencers and content creators in tech.

Get CareerByteCode Team's next blog by email

One email a week when they publish. Add the weekly Best of ByteBlog too. No spam, unsubscribe with one click.

We email you once to confirm. By subscribing you agree to the privacy policy.

💬 Comments (0)

Top · Newest

Sign in to comment

Related blogs

🧪⭐ Recommended by CareerByteCode · ByteLabs project bundleCloud and DevOps Realtime Starterโ‚น499Build the projects →