OpenTelemetry for Beginners: Traces, Metrics and Logs
What you build: an instrumented app sending traces to Cloud Trace, in your own Google Cloud account. Every step, command and screenshot explained for intermediates.

✍️ Translate this blog into Hindi, Tamil or Telugu
In this lab you add OpenTelemetry to a small Python web app, run it on Cloud Run, and see all three kinds of telemetry in Google Cloud: traces in Cloud Trace (each request as a timeline), a metric in Cloud Monitoring (a checkout counter) and logs in Cloud Logging that carry the trace id. Then the payment step gets slow, and you find the cause from the trace in seconds instead of guessing. You finish by deleting everything.
Who it is for: developers and ops beginners who have heard "observability" and want to see it working (intermediate Python helps, but every line is explained). Time: about 1.5 hours. Cost: free tier (Cloud Trace gives 2.5 million spans a month free, custom metrics and logs stay inside their free amounts, and Cloud Run handles a few dozen requests).
Every command below was run for real by the CareerByteCode Team on a fresh Google Cloud project, and every output and screenshot is the real result we got. We ran the commands from a terminal with the Google Cloud CLI; Cloud Shell in your browser has the same tools, so you can copy and paste each command there.
🧪⭐ Recommended by CareerByteCode · ByteLabs project bundleCloud and DevOps Realtime StarterβΉ499Build the projects →Part A: OpenTelemetry in plain English
OpenTelemetry (OTel for short) is a free, open standard and set of libraries for describing what your app is doing. You write the "what happened" code once; exporters send it to the tool of your choice (here Google Cloud; tomorrow maybe another tool, without changing your code).

The three signals:
- Trace: the full story of one request. It is made of spans: one span per piece of work, with a start time, an end time and attributes. Spans nest:
checkoutcontainscheck_stockandcharge_card. - Metric: a number measured over time, like "checkouts handled". A counter only goes up.
- Log: a line of text about one event. If it carries the trace id, you can jump from the log line to the full trace.
Real-life example: a parcel tracking number. Every scan at every hub (a span) is stamped with the same tracking number (the trace id), so you see exactly which hub held the parcel up. The counter at the post office (the metric) says how many parcels moved today, and the note "parcel damaged" (the log) carries the tracking number too.
Before you start
- A Google Cloud account (the free trial works) and a project with billing linked.
- Open Cloud Shell (the
>_icon at the top of the Google Cloud console). - Save your project id (put your own id between the quotes):
export PROJECT_ID="your-project-id"- Run every step from your home folder (
cd ~). Steps that work on the app start withcd otel-shopthemselves.
Part B: the lab
Step 1: turn on the APIs
Why: Cloud Trace receives spans, Cloud Monitoring receives the metric, Cloud Run runs the app, and Cloud Build plus Artifact Registry build it from source.
gcloud config set project $PROJECT_ID
gcloud services enable cloudtrace.googleapis.com monitoring.googleapis.com logging.googleapis.com run.googleapis.com cloudbuild.googleapis.com artifactregistry.googleapis.com
gcloud services list --enabled --filter="name:(cloudtrace.googleapis.com OR monitoring.googleapis.com)" --format="value(name)"Updated property [core/project].
Operation "operations/acat.p2-PROJECT_NUMBER-72a46ac2-6641-4182-972e-d9afad8d9599" finished successfully.
projects/PROJECT_NUMBER/services/cloudtrace.googleapis.com
projects/PROJECT_NUMBER/services/monitoring.googleapis.comStep 2: write an app with OpenTelemetry inside
Why: a tiny shop with one page, /checkout. Read main.py in four parts:
- Setup: a
TracerProviderwith aBatchSpanProcessorthat sends spans to Cloud Trace in batches; aMeterProviderthat reads the counter every 15 seconds and sends it to Cloud Monitoring.add_unique_identifier=Truegives each running copy of the app its own time series, so two copies never overwrite each other's points. - Spans:
with tracer.start_as_current_span("checkout"):starts a span and ends it when the block ends. The two inner spans become its children automatically.pay.set_attribute(...)adds a searchable detail. - Metric:
checkouts.add(1, {"result": "ok"})adds 1 to the counter with a label. - Logs:
log()prints one JSON line withlogging.googleapis.com/traceandspanId. Cloud Run reads these special fields and links the log line to the trace.
?slow=1 makes charge_card wait 1.5 seconds instead of 0.12; we use it to "break" the app later. requirements.txt pins the OpenTelemetry packages, and .python-version asks the builder for Python 3.13.
mkdir -p otel-shop && cd otel-shop
cat > requirements.txt <<EOF
opentelemetry-sdk==1.27.0
opentelemetry-exporter-gcp-trace==1.7.0
opentelemetry-exporter-gcp-monitoring==1.7.0a0
EOF
cat > main.py <<EOF
import json, logging, os, random, time
from http.server import BaseHTTPRequestHandler, HTTPServer
from urllib.parse import urlparse, parse_qs
from opentelemetry import trace, metrics
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.exporter.cloud_trace import CloudTraceSpanExporter
from opentelemetry.exporter.cloud_monitoring import CloudMonitoringMetricsExporter
logging.basicConfig(level=logging.INFO) # shows any export error in the Cloud Run logs
PROJECT = os.environ.get("GOOGLE_CLOUD_PROJECT", "")
resource = Resource.create({"service.name": "otel-shop"})
tp = TracerProvider(resource=resource)
tp.add_span_processor(BatchSpanProcessor(CloudTraceSpanExporter(project_id=PROJECT)))
trace.set_tracer_provider(tp)
reader = PeriodicExportingMetricReader(CloudMonitoringMetricsExporter(project_id=PROJECT, add_unique_identifier=True), export_interval_millis=15000)
metrics.set_meter_provider(MeterProvider(resource=resource, metric_readers=[reader]))
tracer = trace.get_tracer("otel-shop")
checkouts = metrics.get_meter("otel-shop").create_counter("checkout_count", description="Checkouts handled")
def log(message, span):
ctx = span.get_span_context()
print(json.dumps({"severity": "INFO", "message": message,
"logging.googleapis.com/trace": "projects/" + PROJECT + "/traces/" + format(ctx.trace_id, "032x"),
"logging.googleapis.com/spanId": format(ctx.span_id, "016x")}), flush=True)
class Handler(BaseHTTPRequestHandler):
def do_GET(self):
url = urlparse(self.path)
if url.path != "/checkout":
self.send_response(404); self.end_headers(); return
slow = "slow" in parse_qs(url.query)
with tracer.start_as_current_span("checkout") as span:
with tracer.start_as_current_span("check_stock"):
time.sleep(0.05)
with tracer.start_as_current_span("charge_card") as pay:
pay.set_attribute("payment.gateway", "demo-gateway")
time.sleep(1.5 if slow else 0.12)
checkouts.add(1, {"result": "ok"})
log("checkout done" + (" (slow gateway)" if slow else ""), span)
self.send_response(200); self.send_header("Content-Type", "text/plain"); self.end_headers()
self.wfile.write(b"checkout ok\n")
HTTPServer(("", int(os.environ.get("PORT", "8080"))), Handler).serve_forever()
EOF
cat > Procfile <<EOF
web: python main.py
EOF
echo "3.13" > .python-version
ls -1A.python-version
Procfile
main.py
requirements.txt
Step 3: deploy it to Cloud Run
Why: --source=. builds the container from the folder for you. --max-instances=1 keeps this lab to one copy of the app. GOOGLE_CLOUD_PROJECT tells the exporters which project to write to.
cd otel-shop
gcloud run deploy otel-shop --source=. --region=us-east1 --allow-unauthenticated \
--max-instances=1 --set-env-vars=GOOGLE_CLOUD_PROJECT=$PROJECT_ID 2>&1 | tail -3Done.
Service [otel-shop] revision [otel-shop-00002-7r9] has been deployed and is serving 100 percent of traffic.
Service URL: https://otel-shop-PROJECT_NUMBER.us-east1.run.appStep 4: send five checkouts
URL=$(gcloud run services describe otel-shop --region=us-east1 --format="value(status.url)")
for i in 1 2 3 4 5; do curl -s -w " %{time_total}s\n" $URL/checkout; donecheckout ok
0.341210s
checkout ok
0.348185s
checkout ok
0.341743s
checkout ok
0.335710s
checkout ok
0.350720s
Step 5: read the traces from Cloud Trace
Why: in the console you would open Trace explorer; here we call the Cloud Trace API and print each trace as a list of spans with their duration. First save this small helper that turns the API's answer into a readable list:
cat > show_traces.py <<'EOF'
import json, sys
from datetime import datetime
def t(s): return datetime.fromisoformat(s.replace("Z", "+00:00").split(".")[0] + "+00:00" if "." in s else s.replace("Z", "+00:00"))
def ms(sp):
a = sp["startTime"]; b = sp["endTime"]
fa = float("0." + a.split(".")[1].rstrip("Z")) if "." in a else 0.0
fb = float("0." + b.split(".")[1].rstrip("Z")) if "." in b else 0.0
return round(((t(b) - t(a)).total_seconds() + fb - fa) * 1000)
data = json.load(sys.stdin)
for tr in data.get("traces", []):
spans = [s for s in tr["spans"] if s["name"] in ("checkout", "check_stock", "charge_card")]
if not spans: continue
print("trace", tr["traceId"][:12] + "...")
for s in sorted(spans, key=lambda s: s["startTime"]):
print(" %-12s %5d ms" % (s["name"], ms(s)))
EOFNow ask for the 3 newest traces whose root span is checkout:
curl -s -G -H "Authorization: Bearer $(gcloud auth print-access-token)" \
"https://cloudtrace.googleapis.com/v1/projects/$PROJECT_ID/traces" \
--data-urlencode "view=COMPLETE" --data-urlencode "pageSize=3" \
--data-urlencode "filter=root:checkout" | python3 show_traces.pytrace 04e41ad82f4b...
checkout 172 ms
check_stock 51 ms
charge_card 121 ms
trace 14ddc09b50b0...
checkout 172 ms
check_stock 51 ms
charge_card 120 ms
trace 190ee135c51d...
checkout 172 ms
check_stock 51 ms
charge_card 121 ms
Each trace tells the same story: checkout took about 172 ms, of which check_stock was 51 ms and charge_card 121 ms. That matches the sleeps in the code.
Step 6: logs that point to their trace
Why: when a user reports a problem, you usually start from a log line. Because our log lines carry the trace id, every line points to its full trace.
gcloud logging read "resource.type=cloud_run_revision AND resource.labels.service_name=otel-shop AND textPayload=\"checkout done\"" \
--freshness=30m --limit=3 --format="table(timestamp.date(%H:%M:%S),textPayload,trace.basename(),spanId)"TIMESTAMP TEXT_PAYLOAD TRACE SPAN_ID
23:05:18 checkout done 65d745b2c4b576278cb7751c6dc3decf 0fd4315e75acd2b4
23:05:18 checkout done 250628579d6d11cc723de49ce0a3fb34 e0864d4cf0edefa8
23:05:18 checkout done a86ed88fd0b91ff8a21bd3934c143d8c 77a803df2538df53
Break it: the payment gateway gets slow
Users say checkout "feels slow". We call the page with ?slow=1, which is what a slow payment provider would look like:
URL=$(gcloud run services describe otel-shop --region=us-east1 --format="value(status.url)")
for i in 1 2 3; do curl -s -w " %{time_total}s\n" "$URL/checkout?slow=1"; donecheckout ok
1.750340s
checkout ok
1.720725s
checkout ok
1.723516s📝⭐ Recommended by CareerByteCode · Certification examGoogle Analytics Certification (GA4) PracticeTake the examTake the exam →
Every checkout now takes about 1.7 seconds instead of 0.35. But which part is slow: the stock check, the card payment, the network, Cloud Run?
Find the cause with the trace, not by guessing
Ask Cloud Trace only for checkouts slower than 1,000 ms (latency:1000ms), and find the matching log line:
sleep 20
curl -s -G -H "Authorization: Bearer $(gcloud auth print-access-token)" \
"https://cloudtrace.googleapis.com/v1/projects/$PROJECT_ID/traces" \
--data-urlencode "view=COMPLETE" --data-urlencode "pageSize=2" \
--data-urlencode "filter=root:checkout latency:1000ms" | python3 show_traces.py
gcloud logging read "resource.type=cloud_run_revision AND resource.labels.service_name=otel-shop AND textPayload:slow" \
--freshness=10m --limit=2 --format="table(timestamp.date(%H:%M:%S),textPayload,trace.basename())"trace 18397e3725f0...
checkout 1552 ms
check_stock 50 ms
charge_card 1501 ms
trace 81a3a2073147...
checkout 1551 ms
check_stock 51 ms
charge_card 1500 ms
TIMESTAMP TEXT_PAYLOAD TRACE
23:06:53 EXTERNAL_IP - - [17/Nov/2026 23:06:53] "GET /checkout?slow=1 HTTP/1.1" 200 -
23:06:53 checkout done (slow gateway) f0631c496f0875acb78b7662e1a17095
Answer in seconds: check_stock is still 50 ms; charge_card takes 1,500 ms. The problem is the payment gateway, not your database, not Cloud Run. The log line "checkout done (slow gateway)" carries the trace id of the slow request, so a support engineer can open the same trace from the log.
POC: the metric counted every checkout
curl -s -G -H "Authorization: Bearer $(gcloud auth print-access-token)" \
"https://monitoring.googleapis.com/v3/projects/$PROJECT_ID/timeSeries" \
--data-urlencode "filter=metric.type=\"workload.googleapis.com/checkout_count\"" \
--data-urlencode "interval.startTime=$(date -u -d "-2 min" +%Y-%m-%dT%H:%M:%SZ)" \
--data-urlencode "interval.endTime=$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
| python3 -c "import sys,json; d=json.load(sys.stdin); [print(\"metric:\", s[\"metric\"][\"type\"], \"| result:\", s[\"metric\"][\"labels\"][\"result\"], \"| checkouts counted:\", s[\"points\"][0][\"value\"][\"int64Value\"]) for s in d.get(\"timeSeries\",[])]"metric: workload.googleapis.com/checkout_count | result: ok | checkouts counted: 8
Proof: the counter says 8: the 5 normal checkouts and the 3 slow ones. The same number is a graph in Monitoring > Metrics explorer (search for checkout_count), ready for a dashboard or an alert.
Two real lessons from our first try, so you do not repeat them:
- Without
add_unique_identifier=True, two copies of the app wrote to the same time series and Cloud Monitoring refused points with "Points must be written in order". - Always check the app's own logs for exporter errors (
logging.basicConfig(level=logging.INFO)makes them visible). If traces do not show up, the answer is usually there.
Clean up (delete everything)
Why: remove the service, the images that --source created and the source bucket. Trace, metric and log data expire on their own (traces after 30 days).
gcloud run services delete otel-shop --region=us-east1 --quiet 2>&1 | tail -1
gcloud artifacts repositories delete cloud-run-source-deploy --location=us-east1 --quiet 2>&1 | tail -1
for B in $(gcloud storage buckets list --format="value(name)"); do gcloud storage rm --recursive gs://$B --quiet > /dev/null 2>&1; done
echo "services: $(gcloud run services list --format="value(name)" | wc -l), repositories: $(gcloud artifacts repositories list --location=us-east1 --format="value(name)" 2>/dev/null | wc -l), buckets: $(gcloud storage buckets list --format="value(name)" | wc -l)"Deleted service [otel-shop].
Deleted repository [cloud-run-source-deploy].
services: 0, repositories: 0, buckets: 0
FAQ
Do I need to change my code to switch tools? Only the exporter setup lines. The spans, attributes and counters stay the same. That is the main promise of OpenTelemetry.
What is the difference between a trace and a log? A log is one event. A trace is the whole journey of one request, with timing for every step. Use both, linked by the trace id.
Is there automatic instrumentation? Yes. OpenTelemetry has instrumentation packages for Flask, Django, requests, database drivers and more, which create spans without code changes. We wrote spans by hand so you can see what they are.
What does BatchSpanProcessor do? It collects finished spans in memory and sends them in groups every few seconds, so your requests are not slowed down by sending telemetry.
Why did the first request in Step 4 not take longer? On our second deployment the instance was already warm. A cold start (the first request to a new instance) can take several seconds.
Where do I see this in the console? Trace explorer for traces, Metrics explorer for the counter, Logs Explorer for the log lines (click a trace id to open the trace).
Troubleshooting
| You see | Why | Fix |
|---|---|---|
| No traces in Cloud Trace | The exporter is failing or the API is off | Check the Cloud Run logs for errors; run Step 1 again |
Points must be written in order in the logs | Two app copies write to the same metric series | CloudMonitoringMetricsExporter(..., add_unique_identifier=True) |
Build fails with failed to resolve version matching | The Python version in .python-version is not offered by the builder | Use a version from the list in the build log (we used 3.13) |
| Log lines have no trace link | The JSON line misses logging.googleapis.com/trace | Print the trace field as in log(), with the full projects/.../traces/... value |
| The metric shows nothing | Metrics are sent every 15 seconds and need a minute to appear | Wait 1 to 2 minutes and read again |
Permission denied writing traces or metrics | The service account lacks Cloud Trace Agent or Monitoring Metric Writer | Give those roles to the Cloud Run service account |
What next
- Learn to search and route the logs: "Cloud Logging Hands-On: Find Any Error in Seconds", in this series on ByteBlog.
- Our tested Kubernetes lab: Zero to Kubernetes in 2.5 hours.
- More hands-on posts on ByteBlog, guided projects in ByteLabs and the full courses.
Prepared by the CareerByteCode Team from our tested lab, with AI help for the write-up.
⭐ Keep learning with CareerByteCode
Hand-picked to go deeper on this topicGet CareerByteCode Team's next blog by email
One email a week when they publish. Add the weekly Best of ByteBlog too. No spam, unsubscribe with one click.
We email you once to confirm. By subscribing you agree to the privacy policy.
More from CareerByteCode Team
CareerByteCode TeamLogs to BigQuery: Analyse a Month of Logs with SQL
What you build: a log sink from Cloud Logging to BigQuery and four SQL reports on a month of web traffic, errors and slow pages, in your own Google Cloud account. Every step tested.
#beginners
CareerByteCode TeamMonitor Linux Servers with Node Exporter and Ops Agent
What you build: CPU, memory and disk alerts for VMs, in your own Google Cloud account. Every step, command and screenshot explained for beginners.
#beginners
CareerByteCode TeamCloud Logging Hands-On: Find Any Error in Seconds
What you build: log queries, log-based metrics and sinks, in your own Google Cloud account. Every step, command and screenshot explained for beginners.
#bigquery
CareerByteCode TeamBigQuery ML: Train a Model with Just SQL
What you build: a churn prediction model written in SQL with BigQuery ML, scored and used for predictions, in your own Google Cloud account. Every step tested.
Readers of this blog also read

AI Humanoid Robotics for Paddy Cultivation Robotics-as-a-Service Pay Per Acre
Why Agriculture? Food is one most essential needs of humanity, making agriculture a critical and high value industry with ever green demand. As the global...
#combine-technologiesDonβt Learn Every Technology β Learn How to Solve Problems
Perspective towards Technology is Matter than how hard we try to learn.
💬 Comments (0)
Top · NewestSign in to comment