Google Cloud Managed Prometheus: No Servers to Run
What you build: Prometheus metrics from a GKE app in Google Cloud Managed Service for Prometheus with one PodMonitoring, PromQL queries and a silent typo found. Every step tested.

✍️ Translate this blog into Hindi, Tamil or Telugu
Running your own Prometheus means looking after a server: disks fill up, memory grows with every new metric, upgrades break, and when the cluster has a bad day, so does your monitoring. Google Cloud Managed Service for Prometheus (people call it GMP) keeps the part you know (Prometheus metrics and PromQL) and removes the server: collectors on your GKE nodes scrape your apps, and Google stores the data in Monarch, the same system behind Cloud Monitoring. In this lab you scrape an app with one small object, query it with PromQL from the command line, break the scrape with a one-letter typo and find it.
Who it is for: anyone who runs apps on GKE and wants metrics without running Prometheus. If Prometheus itself is new to you, read Prometheus and Grafana on Kubernetes from Scratch first. Time: about 45 minutes. Cost: a small GKE cluster for about 10 minutes plus a few cents of ingested samples, covered by the free trial credit.
Every command below was run for real by the CareerByteCode Team, and every output is the real result we got. We typed the commands in a Cloud Shell-like Linux VM (Debian with gcloud, kubectl, curl and python3); Cloud Shell in your browser has the same tools.
Part A: managed Prometheus in plain English
With your own Prometheus, one server scrapes every app, stores everything on its disk and answers queries. With GMP, the work is split: a collector on each node scrapes the pods on that node and sends the samples to Google; Google stores them and answers PromQL through an API. You never size a disk.

Words you will use:
- Collector: the managed scraper that runs on every GKE node (in the namespace
gmp-system). Google installs and upgrades it. - PodMonitoring: a Kubernetes object that says "scrape pods with these labels, on this named port, this often". It replaces the scrape config you would write for your own Prometheus.
- Monarch: Google's time-series database. It keeps GMP samples for 24 months, with no disk for you to manage.
- PromQL: the Prometheus query language. GMP accepts it through a Prometheus-compatible API, so Grafana and scripts work unchanged.
- Named port: in the pod spec, a container port can have a name (
metrics). PodMonitoring refers to the name, not the number. - Real-life example: your own Prometheus is a generator in your backyard. GMP is the electricity grid: you plug in and pay for what you use.
Before you start
- A Google Cloud project with billing linked (the free trial works) and Cloud Shell open.
- One small GKE cluster. Managed collection is on by default for new GKE clusters, so nothing extra is needed. We created it once for this batch of Kubernetes labs:
gcloud container clusters create lab-cluster \
--zone=us-east1-b --num-nodes=2 --machine-type=e2-standard-4 \
--disk-type=pd-standard --disk-size=50 --release-channel=regular \
--no-enable-autoupgrade --enable-dataplane-v2Part B: the lab
Step 1: check that managed collection is on
Why: before adding anything, confirm the collectors run. True means managed collection is enabled; the pods are one collector per node plus the GMP operator.
# Managed Service for Prometheus is ON by default on new GKE clusters: its collectors already run on every node
gcloud container clusters get-credentials lab-cluster --zone=us-east1-b > /dev/null 2>&1
gcloud container clusters describe lab-cluster --zone=us-east1-b --format="value(monitoringConfig.managedPrometheusConfig.enabled)"
kubectl -n gmp-system get pods --no-headers | awk '{print $1, $3}' | sed -E 's/-[a-z0-9]{5}$//' | sort | uniq -cTrue
1 collector-87f9n Running
1 collector-jzw2v Running
1 gmp-operator-5798f976cc-fhlhc RunningStep 2: an app with Prometheus metrics
Why: we need something to scrape. Google's own example app (from the prometheus-engine project) publishes metrics on a port named metrics: a counter of requests by status code and a histogram of random numbers. We peek at its /metrics page through a port-forward:
# An example app that exposes Prometheus metrics on a port named "metrics" (Google's own example)
kubectl create namespace gmp-test > /dev/null
kubectl -n gmp-test apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/prometheus-engine/v0.15.3/examples/example-app.yaml > /dev/null && kubectl -n gmp-test rollout status deploy/prom-example --timeout=120s
POD=$(kubectl -n gmp-test get pod -l app.kubernetes.io/name=prom-example -o jsonpath='{.items[0].metadata.name}')
kubectl -n gmp-test port-forward $POD 1234:metrics > /dev/null 2>&1 & sleep 3; curl -s localhost:1234/metrics | grep -E "^# (HELP|TYPE)|^[a-z_]+\{" | grep -vE "^go_|^process_|^# (HELP|TYPE) (go|process)" | head -8; kill %1Waiting for deployment "prom-example" rollout to finish: 0 out of 3 new replicas have been updated...
Waiting for deployment "prom-example" rollout to finish: 0 of 3 updated replicas are available...
Waiting for deployment "prom-example" rollout to finish: 1 of 3 updated replicas are available...
Waiting for deployment "prom-example" rollout to finish: 2 of 3 updated replicas are available...
deployment "prom-example" successfully rolled out
# HELP example_random_numbers A histogram of normally distributed random numbers.
# TYPE example_random_numbers histogram
example_random_numbers_bucket{le="0"} 0
example_random_numbers_bucket{le="0.1"} 2.163269e+06
example_random_numbers_bucket{le="0.2"} 4.307048e+06
example_random_numbers_bucket{le="0.30000000000000004"} 6.405985e+06
example_random_numbers_bucket{le="0.4"} 8.444725e+06
example_random_numbers_bucket{le="0.5"} 1.0401686e+07
Step 3: one PodMonitoring object
Why: this is the whole "scrape config". It selects the pods labelled app.kubernetes.io/name: prom-example and scrapes their port metrics every 30 seconds. Save it as podmonitoring.yaml:
# PodMonitoring: tell Managed Prometheus which pods to scrape, on which named port, and how often
apiVersion: monitoring.googleapis.com/v1
kind: PodMonitoring
metadata:
name: example-app
namespace: gmp-test
spec:
selector:
matchLabels:
app.kubernetes.io/name: prom-example
endpoints:
- port: metrics
interval: 30s# Tell Managed Prometheus to scrape it: one PodMonitoring object (no Prometheus server to install)
cat podmonitoring.yaml | grep -v "^#"
kubectl apply -f podmonitoring.yaml && sleep 100
kubectl -n gmp-test get podmonitoring example-app -o jsonpath='{range .status.conditions[*]}{.type}={.status}{"\n"}{end}'apiVersion: monitoring.googleapis.com/v1
kind: PodMonitoring
metadata:
name: example-app
namespace: gmp-test
spec:
selector:
matchLabels:
app.kubernetes.io/name: prom-example
endpoints:
- port: metrics
interval: 30s
podmonitoring.monitoring.googleapis.com/example-app created
ConfigurationCreateSuccess=TrueConfigurationCreateSuccess=True means the operator accepted the object and gave the collectors their new job. We waited 100 seconds so a few scrapes reach Monarch.
Step 4: query with PromQL
Why: the data is in Google's store now, so we ask it through the Prometheus-compatible API at monitoring.googleapis.com. promq.sh signs in with your gcloud login (no key, no password) and prints each result. Save it as promq.sh:
#!/bin/bash
# promq.sh "<PromQL>": run a PromQL query against Managed Service for Prometheus (the Cloud Monitoring Prometheus API)
# and print each series' labels and value. Signs in with your gcloud login.
curl -s -H "Authorization: Bearer $(gcloud auth print-access-token)" \
"https://monitoring.googleapis.com/v1/projects/$(gcloud config get-value project 2>/dev/null)/location/global/prometheus/api/v1/query" \
--data-urlencode "query=$1" | python3 -c '
import json, sys
d = json.load(sys.stdin)
if d.get("status") != "success": print("error:", d.get("error", d)); sys.exit(1)
r = d["data"]["result"]
if not r: print("(no data)")
for s in r:
m = {k: v for k, v in s["metric"].items() if k in ("__name__", "pod", "code", "job", "namespace", "le")}
print(m, "=", s["value"][1])'# Query it with PromQL through the Cloud Monitoring API: is it up, and how many requests per second?
chmod +x promq.sh
./promq.sh 'up{namespace="gmp-test"}'
./promq.sh 'sum by (code) (rate(example_requests_total{namespace="gmp-test"}[5m]))' | head -4
./promq.sh 'count({namespace="gmp-test"})' || echo "(open-source Prometheus allows this; Managed Prometheus needs an exact metric name)"
./promq.sh 'count(example_random_numbers_bucket{namespace="gmp-test"})'{'__name__': 'up', 'job': 'example-app', 'namespace': 'gmp-test', 'pod': 'prom-example-55b9d8f5b9-2b4wv'} = 1
{'__name__': 'up', 'job': 'example-app', 'namespace': 'gmp-test', 'pod': 'prom-example-55b9d8f5b9-62sfp'} = 1
{'__name__': 'up', 'job': 'example-app', 'namespace': 'gmp-test', 'pod': 'prom-example-55b9d8f5b9-h2752'} = 1
{'code': '200'} = 0.2066666666666667
{'code': '500'} = 0.06666666666666667
error: invalid parameter "query": substitute queries failed: series selector must specify an exact metric name: invalid argument
(open-source Prometheus allows this; Managed Prometheus needs an exact metric name)
{} = 186Three things to notice:
up = 1for all three pods: the collectors scrape them.sum by (code) (rate(...))works exactly as in open-source Prometheus: about 0.21 requests per second with code 200 and 0.07 with code 500.count({namespace="gmp-test"})is refused: GMP needs an exact metric name in every selector. In open-source Prometheus this query is allowed. With a metric name (count(example_random_numbers_bucket{...})) it works: 186 series.

Break it: a one-letter typo
We create a second PodMonitoring whose port is metric instead of metrics, the kind of typo everyone makes, and wait 100 seconds:
# Break it: a PodMonitoring that points at a port name that does not exist (a very common typo)
sed 's/name: example-app/name: example-app-typo/; s/port: metrics/port: metric/' podmonitoring.yaml | kubectl apply -f - && sleep 100
kubectl -n gmp-test get podmonitoring example-app-typo -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.message}{"\n"}{end}' | cut -c1-200
./promq.sh 'up{namespace="gmp-test", job="example-app-typo"}'podmonitoring.monitoring.googleapis.com/example-app-typo created
ConfigurationCreateSuccess=True
(no data)The operator says ConfigurationCreateSuccess=True again: the object is valid YAML and a valid PodMonitoring, so nothing complains. But up for that job returns no data: no pod has a port named metric, so nothing is scraped. A silent failure like this is why you always check up (or the target status) after adding a scrape.
Fix it
Delete the broken object (in real life: correct the port name). The working PodMonitoring keeps scraping all three pods:
# Fix it: delete the broken object (the correct one keeps scraping); PodMonitoring status and up tell you which works
kubectl -n gmp-test delete podmonitoring example-app-typo
./promq.sh 'up{namespace="gmp-test"}'podmonitoring.monitoring.googleapis.com "example-app-typo" deleted from gmp-test namespace
{'__name__': 'up', 'job': 'example-app', 'namespace': 'gmp-test', 'pod': 'prom-example-55b9d8f5b9-2b4wv'} = 1
{'__name__': 'up', 'job': 'example-app', 'namespace': 'gmp-test', 'pod': 'prom-example-55b9d8f5b9-62sfp'} = 1
{'__name__': 'up', 'job': 'example-app', 'namespace': 'gmp-test', 'pod': 'prom-example-55b9d8f5b9-h2752'} = 1
POC: prove it works
- Managed collection was on, with one collector per node (Step 1).
- One PodMonitoring made the collectors scrape the app, and
up = 1for all three pods (Steps 3 and 4). - PromQL through the Google API answered rate and count queries; a query without a metric name was refused (Step 4).
- A typo in the port name was accepted but scraped nothing, and
upshowed it (Break it).
Clean up (delete everything)
Why: GMP charges for the samples it ingests (your app's metrics; GKE's own system metrics are free). Deleting the namespace removes the app and its PodMonitoring, so ingestion stops. Managed collection itself stays part of the cluster.
# Clean up: the example app and its PodMonitoring (Managed Prometheus itself stays part of the cluster)
kubectl delete namespace gmp-test --wait=true 2>&1 | tail -1
echo "PodMonitorings left: $(kubectl get podmonitoring -A --no-headers 2>/dev/null | wc -l)"namespace "gmp-test" deleted
PodMonitorings left: 0
When you have finished all your Kubernetes labs, delete the cluster:
gcloud container clusters delete lab-cluster --zone=us-east1-b --quietFAQ
Can Grafana read GMP? Yes. Grafana talks to GMP's Prometheus-compatible API through a small data source syncer that refreshes the login token. We did not set up Grafana in this lab.
What about alerts? GMP can evaluate Prometheus rules with Rules and ClusterRules objects and send them to an Alertmanager, or you can alert in Cloud Monitoring with PromQL. Not tested here.
Is it really free? No. You pay per million samples ingested. Scrape every 30 or 60 seconds rather than every 5, and avoid labels with endless values (user ids, full URLs), and the bill stays small.
PodMonitoring or ClusterPodMonitoring? PodMonitoring works in one namespace; ClusterPodMonitoring selects pods in every namespace. Teams usually own their own PodMonitorings.
Can I keep my own Prometheus? Yes. GMP also has a "self-deployed" mode where you run Prometheus with Google's image and it sends the data to Monarch. This lab uses managed collection only.
Troubleshooting
| You see | Why | Fix |
|---|---|---|
up returns no data | The PodMonitoring selects no pods, or the port name is wrong | Compare the labels and the port name in the pod spec with the PodMonitoring |
series selector must specify an exact metric name | GMP needs a metric name in every selector | Add the name, for example count(my_metric{namespace="x"}) |
PERMISSION_DENIED from the API | Your login cannot read monitoring data | You need the Monitoring Viewer role on the project |
No gmp-system pods | Managed collection is off on this cluster | gcloud container clusters update lab-cluster --enable-managed-prometheus --zone=us-east1-b (not needed in our run) |
| Data appears only after minutes | Scrape interval plus the time to reach Monarch | Wait two scrape intervals before you judge |
What next
- Turn the same metrics into an alert people act on: MLOps for Beginners: From Notebook to Production shows monitoring a live service end to end.
- The self-managed way, for comparison: Prometheus and Grafana on Kubernetes from Scratch.
- More hands-on posts: ByteBlog. Practice in ByteLabs and see the courses.
Prepared by the CareerByteCode Team from our tested lab, with AI help for the write-up.
⭐ Keep learning with CareerByteCode
Hand-picked to go deeper on this topicGet CareerByteCode Team's next blog by email
One email a week when they publish. Add the weekly Best of ByteBlog too. No spam, unsubscribe with one click.
We email you once to confirm. By subscribing you agree to the privacy policy.
Related blogs
CareerByteCode TeamAlertmanager: Route Alerts to Email, Slack and On-Call
What you build: Alertmanager on GKE with routing, grouping, silences and inhibition tested, a case typo that misroutes a page, and a hot-reload fix. Every step tested.
#google-cloud
CareerByteCode TeamObservability Interview Questions with Hands-On Proof
What you build: 15 observability interview answers, each proved with a real command on GKE with Prometheus: rate, p99, cardinality, SLOs, alerts. Every step tested.
#google-cloud
CareerByteCode TeamLogs to BigQuery: Analyse a Month of Logs with SQL
What you build: a log sink from Cloud Logging to BigQuery and four SQL reports on a month of web traffic, errors and slow pages, in your own Google Cloud account. Every step tested.
#google-cloud
CareerByteCode TeamOpenTelemetry for Beginners: Traces, Metrics and Logs
What you build: an instrumented app sending traces to Cloud Trace, in your own Google Cloud account. Every step, command and screenshot explained for intermediates.
💬 Comments (0)
Top · NewestSign in to comment