CareerByteCode
ByteBlogby CareerByteCode

Monitor Linux Servers with Node Exporter and Ops Agent

What you build: CPU, memory and disk alerts for VMs, in your own Google Cloud account. Every step, command and screenshot explained for beginners.

Monitor Linux Servers with Node Exporter and Ops Agent

✍️ Translate this blog into Hindi, Tamil or Telugu

In this lab you set up monitoring for a real Linux server on Google Cloud: you create a small e2-micro VM, install Google's Ops Agent and the open-source Prometheus Node Exporter, read the server's memory and CPU from Cloud Monitoring, create an alert policy, and then push the CPU to the limit to see the numbers climb. You find the busy process, stop it, and finish by deleting everything.

Who it is for: beginners who run (or will run) Linux servers and want to know what is happening on them before users complain. Time: about 1 hour. Cost: free tier or a few cents (one e2-micro VM in us-east1 is in the Always Free tier; agent metrics stay far inside the free monthly amount for one VM).

Every command below was run for real by the CareerByteCode Team on a fresh Google Cloud project, and every output and screenshot is the real result we got (ids and user names are blurred). We ran the commands from a terminal with the Google Cloud CLI; Cloud Shell in your browser has the same tools, so you can copy and paste each command there.

🧪⭐ Recommended by CareerByteCode · ByteLabs project bundleCloud and DevOps Realtime Starterโ‚น499Build the projects →

Part A: server monitoring in plain English

Two ways to watch a Linux server
Hypervisor metrics see the VM from outside; agents inside see memory, disk and processes; Cloud Monitoring stores and alerts
  • Metric: a number measured again and again, like "CPU busy percent" or "memory used percent".
  • Hypervisor metrics: what Google sees from outside the VM (CPU, disk and network traffic). They work with no install, but they cannot see memory or disk space inside the VM.
  • Ops Agent: Google's agent you install inside the VM. It sends memory, disk space, CPU per state, processes and system logs to Cloud Monitoring and Cloud Logging.
  • Node Exporter: the standard Prometheus agent for Linux. It shows hundreds of metrics as plain text on port 9100 (/metrics), for any Prometheus-style tool to read ("scrape").
  • Alert policy: a rule in Cloud Monitoring like "CPU above 40 percent for 1 minute". When it is true, Monitoring opens an incident and can notify you (email, chat, phone).

Real-life example: a car. The speed camera on the road (hypervisor metrics) sees how fast you drive. The dashboard inside the car (the agent) also shows fuel, engine temperature and oil. The warning light (the alert) tells you before the engine fails.

Why both agents? Ops Agent is the easy way into Google Cloud Monitoring. Node Exporter is what most Prometheus and Grafana setups use, on any cloud. Learning both makes you ready for either.

Before you start

  1. A Google Cloud account (the free trial works) and a project with billing linked.
  2. Open Cloud Shell (the >_ icon at the top of the Google Cloud console).
  3. Save your project id (put your own id between the quotes):
export PROJECT_ID="your-project-id"
  1. The first time you run gcloud compute ssh, gcloud creates an SSH key for you: press Enter at the passphrase questions (or choose a passphrase).

Part B: the lab

Step 1: a small network with SSH only through IAP

Why: the server gets its own network, and SSH is allowed only from Google's IAP range (35.235.240.0/20), never from the whole internet.

gcloud config set project $PROJECT_ID
gcloud services enable compute.googleapis.com monitoring.googleapis.com logging.googleapis.com iap.googleapis.com
gcloud compute networks create mon-vpc --subnet-mode=custom 2>&1 | tail -1
gcloud compute networks subnets create mon-subnet --network=mon-vpc --region=us-east1 --range=10.20.0.0/24 2>&1 | tail -1
gcloud compute firewall-rules create mon-allow-ssh-iap --network=mon-vpc --direction=INGRESS --action=ALLOW --rules=tcp:22 --source-ranges=35.235.240.0/20 2>&1 | tail -1
Updated property [core/project].
Operation "operations/acat.p2-PROJECT_NUMBER-55544272-f377-4af4-8a1d-fc89cf779533" finished successfully.

mon-subnet  us-east1  mon-vpc  10.20.0.0/24  IPV4_ONLY
mon-allow-ssh-iap  mon-vpc  INGRESS    1000      tcp:22        False

Step 2: create the Linux server

Why: e2-micro is the smallest machine type and is in the free tier in us-east1. Debian 12 is a common server Linux. The VM keeps a public IP so it can download the agents.

gcloud compute instances create mon-vm --zone=us-east1-b --machine-type=e2-micro \
  --subnet=mon-subnet --image-family=debian-12 --image-project=debian-cloud \
  --scopes=https://www.googleapis.com/auth/cloud-platform 2>&1 | tail -2
gcloud compute instances list --format="table(name,machineType.basename(),networkInterfaces[0].networkIP,status)"
NAME    ZONE        MACHINE_TYPE  PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP    STATUS
mon-vm  us-east1-b  e2-micro                   10.20.0.2    EXTERNAL_IP  RUNNING
NAME    MACHINE_TYPE  NETWORK_IP  STATUS
mon-vm  e2-micro      10.20.0.2   RUNNING
Network and VM
Steps 1 and 2: mon-vpc, the IAP SSH rule and mon-vm running (public IP blurred)

Step 3: install the Google Cloud Ops Agent

Why: Google's script adds the agent's package source and installs it. systemctl is-active checks its two parts: the metrics collector and the log collector.

gcloud compute ssh mon-vm --zone=us-east1-b --tunnel-through-iap --strict-host-key-checking=no --command="
curl -sSO https://dl.google.com/cloudagents/add-google-cloud-ops-agent-repo.sh &&
sudo bash add-google-cloud-ops-agent-repo.sh --also-install > /tmp/ops-install.log 2>&1;
tail -2 /tmp/ops-install.log;
systemctl is-active google-cloud-ops-agent-opentelemetry-collector google-cloud-ops-agent-fluent-bit" 2>&1 | grep -v "^Store key\|^Waiting"
Created symlink /etc/systemd/system/multi-user.target.wants/google-cloud-ops-agent.service โ†’ /lib/systemd/system/google-cloud-ops-agent.service.
google-cloud-ops-agent  installation succeeded.
active
active
Ops Agent installed
Step 3: installation succeeded, both parts active

Step 4: install Node Exporter as a service

Why: we download the official release (version 1.8.2) from GitHub, copy the program to /usr/local/bin, create a user with no login rights for it, and write a small systemd service file so it starts on every boot and restarts if it stops.

gcloud compute ssh mon-vm --zone=us-east1-b --tunnel-through-iap --strict-host-key-checking=no --command="
cd /tmp && curl -sSLO https://github.com/prometheus/node_exporter/releases/download/v1.8.2/node_exporter-1.8.2.linux-amd64.tar.gz &&
tar xzf node_exporter-1.8.2.linux-amd64.tar.gz &&
sudo cp node_exporter-1.8.2.linux-amd64/node_exporter /usr/local/bin/ &&
sudo useradd --system --no-create-home --shell /usr/sbin/nologin node_exporter 2>/dev/null;
printf \"[Unit]\nDescription=Node Exporter\nAfter=network.target\n[Service]\nUser=node_exporter\nExecStart=/usr/local/bin/node_exporter\nRestart=always\n[Install]\nWantedBy=multi-user.target\n\" | sudo tee /etc/systemd/system/node_exporter.service > /dev/null &&
sudo systemctl daemon-reload && sudo systemctl enable --now node_exporter 2>/dev/null &&
sleep 2 && systemctl is-active node_exporter && /usr/local/bin/node_exporter --version 2>&1 | head -1" 2>&1 | grep -v "^Store key\|^Waiting"
active
node_exporter, version 1.8.2 (branch: HEAD, revision: f1e0e8360aa60b6cb5e5cc1560bed348fc2c1895)
Node Exporter installed
Step 4: node_exporter 1.8.2 is active

Step 5: read raw metrics from Node Exporter

Why: to see exactly what a Prometheus server would collect. We pick four lines: the 1-minute load, available memory, total memory and free disk space on /.

gcloud compute ssh mon-vm --zone=us-east1-b --tunnel-through-iap --strict-host-key-checking=no --command="
curl -s localhost:9100/metrics | grep -E \"^node_load1 |^node_memory_MemAvailable_bytes |^node_memory_MemTotal_bytes |^node_filesystem_avail_bytes.*mountpoint=\\\"/\\\"\" ;
echo; echo metrics exposed: \$(curl -s localhost:9100/metrics | grep -vc \"^#\")" 2>&1 | grep -v "^Store key\|^Waiting"
node_filesystem_avail_bytes{device="/dev/sda1",device_error="",fstype="ext4",mountpoint="/"} 6.89815552e+09
node_load1 0.86
node_memory_MemAvailable_bytes 5.49670912e+08
node_memory_MemTotal_bytes 1.017065472e+09

metrics exposed: 478
Node Exporter metrics
Step 5: load 0.86, about 550 MB of 1 GB memory available, about 6.9 GB disk free, 478 metrics in total

How to read them: node_memory_MemAvailable_bytes 5.49670912e+08 is about 550 MB available out of 1.017065472e+09 (about 1 GB) in total. node_load1 0.86 means about 0.86 processes were waiting for or using the CPU over the last minute.

Step 6: read the server's metrics from Cloud Monitoring

Why: the Ops Agent already sends data to Cloud Monitoring. We read it with the Monitoring API. First save this small helper that prints the last 5 points of each series:

cat > show_metric.py <<'EOF'
import json, sys
d = json.load(sys.stdin)
for s in d.get("timeSeries", []):
    lab = s["metric"].get("labels", {})
    name = s["metric"]["type"].split("/", 1)[1] + (" " + ",".join(k + "=" + v for k, v in lab.items()) if lab else "")
    pts = sorted(s["points"], key=lambda p: p["interval"]["endTime"])[-5:]
    vals = ["%s %.1f" % (p["interval"]["endTime"][11:16], (lambda v: v * 100 if v <= 1.0 and "utilization" in name else v)(float(list(p["value"].values())[0]))) for p in pts]
    print(name + ":  " + "  |  ".join(vals))
EOF

Then ask for the memory used percent (from the agent) and the CPU utilization (from the hypervisor) of this VM:

ID=$(gcloud compute instances describe mon-vm --zone=us-east1-b --format="value(id)")
for M in "agent.googleapis.com/memory/percent_used\" AND metric.labels.state=\"used" "compute.googleapis.com/instance/cpu/utilization"; do
curl -s -G -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  "https://monitoring.googleapis.com/v3/projects/$PROJECT_ID/timeSeries" \
  --data-urlencode "filter=metric.type=\"$M\" AND resource.labels.instance_id=\"$ID\"" \
  --data-urlencode "interval.startTime=$(date -u -d "-10 min" +%Y-%m-%dT%H:%M:%SZ)" \
  --data-urlencode "interval.endTime=$(date -u +%Y-%m-%dT%H:%M:%SZ)" | python3 show_metric.py
done
memory/percent_used state=used:  23:26 35.1
Memory in Cloud Monitoring
Step 6: the Ops Agent reports memory 35.1 percent used; the CPU series had not arrived yet

Only memory came back on our first read, at 35.1 percent used. That is normal: the hypervisor CPU series for a new VM appears a few minutes later. Memory is a metric only the agent can give you.

Step 7: create an alert, CPU above 40 percent for 1 minute

Why: you cannot watch graphs all day. The policy file says: for this VM's compute.googleapis.com/instance/cpu/utilization, average each minute, and if the value is above 0.4 (40 percent) for 60 seconds, open an incident. The VM id is filled in by the command inside $(...).

cat > cpu-policy.json <<EOF
{
  "displayName": "mon-vm CPU above 40 percent",
  "combiner": "OR",
  "conditions": [{
    "displayName": "CPU utilization > 40% for 1 min",
    "conditionThreshold": {
      "filter": "resource.type=\"gce_instance\" AND resource.labels.instance_id=\"$(gcloud compute instances describe mon-vm --zone=us-east1-b --format="value(id)")\" AND metric.type=\"compute.googleapis.com/instance/cpu/utilization\"",
      "comparison": "COMPARISON_GT",
      "thresholdValue": 0.4,
      "duration": "60s",
      "aggregations": [{"alignmentPeriod": "60s", "perSeriesAligner": "ALIGN_MEAN"}]
    }
  }]
}
EOF
gcloud monitoring policies create --policy-from-file=cpu-policy.json --format="value(displayName,enabled)"
gcloud monitoring policies list --format="table(displayName,enabled,conditions[0].displayName)"
Created alert policy [projects/your-project-id/alertPolicies/3307443294420353742].
mon-vm CPU above 40 percent	True
DISPLAY_NAME                 ENABLED  CONDITIONS_DISPLAY_NAME
mon-vm CPU above 40 percent  True     CPU utilization > 40% for 1 min
Alert policy
Step 7: the policy is created and enabled (ids blurred)

We first tried 80 percent, then lowered it to 40 so the policy fires sooner in this lab. In real life pick a level that means "users will feel it", often 80 to 90 percent for several minutes. To get an email, add a notification channel in Monitoring > Alerting > Edit notification channels and attach it to the policy.

📝⭐ Recommended by CareerByteCode · Certification examGoogle Analytics Certification (GA4) PracticeTake the examTake the exam →

Break it: keep both CPUs busy

yes > /dev/null is a tiny program that does nothing but use CPU. We start two copies (the VM has 2 vCPUs) for 6 minutes, in the background, then look at top:

gcloud compute ssh mon-vm --zone=us-east1-b --tunnel-through-iap --strict-host-key-checking=no --command="
nohup timeout 360 sh -c \"yes > /dev/null & yes > /dev/null & wait\" > /dev/null 2>&1 &
sleep 20; echo load1: \$(cut -d\" \" -f1 /proc/loadavg); top -bn1 | head -12 | tail -6" 2>&1 | grep -v "^Store key\|^Waiting"
load1: 2.04
    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   2708 you  20   0    2912    868    784 R 100.0   0.1   0:20.49 yes
   2707 you  20   0    2912    884    796 R  92.0   0.1   0:20.43 yes
   2380 root      20   0 2154880 174940 107944 S   4.0  17.6   0:08.03 otelops+
      1 root      20   0  102596  12012   8656 S   0.0   1.2   0:02.60 systemd
      2 root      20   0       0      0      0 S   0.0   0.0   0:00.00 kthreadd
CPU load
Two yes processes use about 100 percent and 92 percent CPU; load 2.04

POC: Cloud Monitoring sees the CPU climb

Wait 3 to 4 minutes (hypervisor CPU data arrives with a delay), then read both CPU series:

ID=$(gcloud compute instances describe mon-vm --zone=us-east1-b --format="value(id)")
for M in "compute.googleapis.com/instance/cpu/utilization" "agent.googleapis.com/cpu/utilization\" AND metric.labels.cpu_state=\"user"; do
curl -s -G -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  "https://monitoring.googleapis.com/v3/projects/$PROJECT_ID/timeSeries" \
  --data-urlencode "filter=metric.type=\"$M\" AND resource.labels.instance_id=\"$ID\"" \
  --data-urlencode "interval.startTime=$(date -u -d "-12 min" +%Y-%m-%dT%H:%M:%SZ)" \
  --data-urlencode "interval.endTime=$(date -u +%Y-%m-%dT%H:%M:%SZ)" | python3 show_metric.py
done
instance/cpu/utilization instance_name=mon-vm:  23:31 1.0  |  23:32 1.1  |  23:33 99.0  |  23:34 99.3  |  23:35 98.3
cpu/utilization cpu_number=,cpu_state=user:  23:33 49.8  |  23:34 50.7  |  23:35 48.7  |  23:36 51.7  |  23:37 50.2
CPU in Cloud Monitoring
The hypervisor CPU jumps from 1 percent to 99 percent; the agent shows about 50 percent in user state

Proof: the hypervisor CPU utilization went from about 1 percent to 99 percent at 23:33 and stayed there, far above the 40 percent line of the policy for more than a minute, so the condition is true. The agent's cpu_state=user series shows about 50 percent: it measures each state (user, system, idle ...) of the guest separately, which is why the numbers differ. We could not open the console during this run, so we proved the condition from the metric itself; in your project the open incident appears in Monitoring > Alerting.

Fix it: find the busy process and stop it

On a real server you do not know what is busy. ps sorted by CPU shows it at once. (If 6 minutes have passed since Step 8, start the load again with the Step 8 command first, as we did.)

gcloud compute ssh mon-vm --zone=us-east1-b --tunnel-through-iap --strict-host-key-checking=no --command="
echo before:; ps -eo pid,comm,%cpu --sort=-%cpu | head -4;
sudo pkill -x yes; sleep 15;
echo after:; ps -eo pid,comm,%cpu --sort=-%cpu | head -3; echo load1: \$(cut -d\" \" -f1 /proc/loadavg)" 2>&1 | grep -v "^Store key\|^Waiting"
before:
    PID COMMAND         %CPU
   2941 yes             98.2
   2940 yes             98.0
   2954 ps              50.0
after:
    PID COMMAND         %CPU
   2380 otelopscol       2.3
   2671 kworker/0:0-eve  0.8
load1: 0.67
Find and stop
Before: two yes processes at about 98 percent; after pkill: only the agent at 2.3 percent

The two yes processes are gone and the load drops. Within a few minutes the CPU metric falls below 40 percent and Monitoring closes the incident by itself.

Clean up (delete everything)

for P in $(gcloud monitoring policies list --filter="displayName:mon-vm" --format="value(name)"); do gcloud monitoring policies delete $P --quiet 2>&1 | tail -1; done
gcloud compute instances delete mon-vm --zone=us-east1-b --quiet 2>&1 | tail -1
gcloud compute firewall-rules delete mon-allow-ssh-iap --quiet 2>&1 | tail -1
gcloud compute networks subnets delete mon-subnet --region=us-east1 --quiet 2>&1 | tail -1
gcloud compute networks delete mon-vpc --quiet 2>&1 | tail -1
echo "VMs: $(gcloud compute instances list --format="value(name)" | wc -l), alert policies: $(gcloud monitoring policies list --format="value(name)" | wc -l), mon-vpc networks: $(gcloud compute networks list --filter="name=mon-vpc" --format="value(name)" | wc -l)"
Deleted policy [3307443294420353742].
Deleted [https://www.googleapis.com/compute/v1/projects/your-project-id/zones/us-east1-b/instances/mon-vm].
Deleted [https://www.googleapis.com/compute/v1/projects/your-project-id/global/firewalls/mon-allow-ssh-iap].
Deleted [https://www.googleapis.com/compute/v1/projects/your-project-id/regions/us-east1/subnetworks/mon-subnet].
Deleted [https://www.googleapis.com/compute/v1/projects/your-project-id/global/networks/mon-vpc].
VMs: 0, alert policies: 0, mon-vpc networks: 0
Clean up
0 VMs, 0 alert policies, 0 mon-vpc networks

FAQ

Do I need both agents? No. For Google Cloud Monitoring alone, Ops Agent is enough. Use Node Exporter when you (or your company) run Prometheus and Grafana.

Why can Google not see memory without an agent? Memory use is inside the guest operating system. The hypervisor only sees how much memory the VM was given, not how the Linux kernel uses it.

Is port 9100 open to the internet? No. Our firewall allows only SSH from IAP. Never expose Node Exporter publicly; let only your Prometheus server reach it.

Why 40 percent here and not 80? To make the lab's alert fire quickly. Choose thresholds by asking "at what level will users notice?", and add a duration so short spikes do not wake you up.

How do I add disk space alerts? Create a policy on agent.googleapis.com/disk/percent_used (from the Ops Agent) with a threshold like 85 percent.

Can I collect Node Exporter metrics into Cloud Monitoring? Yes. The Ops Agent can scrape Prometheus endpoints with a prometheus receiver in its configuration, and Google Cloud Managed Service for Prometheus can too.

Troubleshooting

You seeWhyFix
No memory or disk metricsThe Ops Agent is not installed or not activesystemctl is-active google-cloud-ops-agent-opentelemetry-collector on the VM, reinstall with Step 3
CPU series is emptyA new VM's hypervisor metrics arrive after a few minutesWait 3 to 5 minutes and read again
curl: (7) Failed to connect to localhost port 9100Node Exporter is not runningsudo systemctl status node_exporter, check the service file
SSH hangs or cannot connectNo firewall rule for the IAP rangeRun Step 1's firewall command
Ops Agent install fails on downloadThe VM has no internet accessGive it a public IP or a Cloud NAT
The alert never opensThreshold too high for this machine, or the duration not reachedCheck the metric values (Step 9), lower the threshold or run the load longer

What next

Prepared by the CareerByteCode Team from our tested lab, with AI help for the write-up.

Read next →Cloud Logging Hands-On: Find Any Error in Seconds⏱ 11 min

⭐ Keep learning with CareerByteCode

Hand-picked to go deeper on this topic
News, guides and updates from the CareerByteCode team
🏅 Contributor writer

The CareerByteCode team writes about what is new on careerbytecode.in: courses, labs, exams, events and the community, which is free to join for influencers and content creators in tech.

Get CareerByteCode Team's next blog by email

One email a week when they publish. Add the weekly Best of ByteBlog too. No spam, unsubscribe with one click.

We email you once to confirm. By subscribing you agree to the privacy policy.

💬 Comments (0)

Top · Newest

Sign in to comment

Related blogs

🧪⭐ Recommended by CareerByteCode · ByteLabs project bundleCloud and DevOps Realtime Starterโ‚น499Build the projects →