CareerByteCode
ByteBlogby CareerByteCode

OpenTelemetry for Beginners: Traces, Metrics and Logs

What you build: an instrumented app sending traces to Cloud Trace, in your own Google Cloud account. Every step, command and screenshot explained for intermediates.

OpenTelemetry for Beginners: Traces, Metrics and Logs

✍️ Translate this blog into Hindi, Tamil or Telugu

In this lab you add OpenTelemetry to a small Python web app, run it on Cloud Run, and see all three kinds of telemetry in Google Cloud: traces in Cloud Trace (each request as a timeline), a metric in Cloud Monitoring (a checkout counter) and logs in Cloud Logging that carry the trace id. Then the payment step gets slow, and you find the cause from the trace in seconds instead of guessing. You finish by deleting everything.

Who it is for: developers and ops beginners who have heard "observability" and want to see it working (intermediate Python helps, but every line is explained). Time: about 1.5 hours. Cost: free tier (Cloud Trace gives 2.5 million spans a month free, custom metrics and logs stay inside their free amounts, and Cloud Run handles a few dozen requests).

Every command below was run for real by the CareerByteCode Team on a fresh Google Cloud project, and every output and screenshot is the real result we got. We ran the commands from a terminal with the Google Cloud CLI; Cloud Shell in your browser has the same tools, so you can copy and paste each command there.

🧪⭐ Recommended by CareerByteCode · ByteLabs project bundleCloud and DevOps Realtime Starterβ‚Ή499Build the projects →

Part A: OpenTelemetry in plain English

OpenTelemetry (OTel for short) is a free, open standard and set of libraries for describing what your app is doing. You write the "what happened" code once; exporters send it to the tool of your choice (here Google Cloud; tomorrow maybe another tool, without changing your code).

OpenTelemetry flow
Spans and a counter leave the app through two exporters; log lines carry the trace id

The three signals:

  • Trace: the full story of one request. It is made of spans: one span per piece of work, with a start time, an end time and attributes. Spans nest: checkout contains check_stock and charge_card.
  • Metric: a number measured over time, like "checkouts handled". A counter only goes up.
  • Log: a line of text about one event. If it carries the trace id, you can jump from the log line to the full trace.

Real-life example: a parcel tracking number. Every scan at every hub (a span) is stamped with the same tracking number (the trace id), so you see exactly which hub held the parcel up. The counter at the post office (the metric) says how many parcels moved today, and the note "parcel damaged" (the log) carries the tracking number too.

Before you start

  1. A Google Cloud account (the free trial works) and a project with billing linked.
  2. Open Cloud Shell (the >_ icon at the top of the Google Cloud console).
  3. Save your project id (put your own id between the quotes):
export PROJECT_ID="your-project-id"
  1. Run every step from your home folder (cd ~). Steps that work on the app start with cd otel-shop themselves.

Part B: the lab

Step 1: turn on the APIs

Why: Cloud Trace receives spans, Cloud Monitoring receives the metric, Cloud Run runs the app, and Cloud Build plus Artifact Registry build it from source.

gcloud config set project $PROJECT_ID
gcloud services enable cloudtrace.googleapis.com monitoring.googleapis.com logging.googleapis.com run.googleapis.com cloudbuild.googleapis.com artifactregistry.googleapis.com
gcloud services list --enabled --filter="name:(cloudtrace.googleapis.com OR monitoring.googleapis.com)" --format="value(name)"
Updated property [core/project].
Operation "operations/acat.p2-PROJECT_NUMBER-72a46ac2-6641-4182-972e-d9afad8d9599" finished successfully.
projects/PROJECT_NUMBER/services/cloudtrace.googleapis.com
projects/PROJECT_NUMBER/services/monitoring.googleapis.com

Step 2: write an app with OpenTelemetry inside

Why: a tiny shop with one page, /checkout. Read main.py in four parts:

  • Setup: a TracerProvider with a BatchSpanProcessor that sends spans to Cloud Trace in batches; a MeterProvider that reads the counter every 15 seconds and sends it to Cloud Monitoring. add_unique_identifier=True gives each running copy of the app its own time series, so two copies never overwrite each other's points.
  • Spans: with tracer.start_as_current_span("checkout"): starts a span and ends it when the block ends. The two inner spans become its children automatically. pay.set_attribute(...) adds a searchable detail.
  • Metric: checkouts.add(1, {"result": "ok"}) adds 1 to the counter with a label.
  • Logs: log() prints one JSON line with logging.googleapis.com/trace and spanId. Cloud Run reads these special fields and links the log line to the trace.

?slow=1 makes charge_card wait 1.5 seconds instead of 0.12; we use it to "break" the app later. requirements.txt pins the OpenTelemetry packages, and .python-version asks the builder for Python 3.13.

mkdir -p otel-shop && cd otel-shop
cat > requirements.txt <<EOF
opentelemetry-sdk==1.27.0
opentelemetry-exporter-gcp-trace==1.7.0
opentelemetry-exporter-gcp-monitoring==1.7.0a0
EOF
cat > main.py <<EOF
import json, logging, os, random, time
from http.server import BaseHTTPRequestHandler, HTTPServer
from urllib.parse import urlparse, parse_qs
from opentelemetry import trace, metrics
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.exporter.cloud_trace import CloudTraceSpanExporter
from opentelemetry.exporter.cloud_monitoring import CloudMonitoringMetricsExporter

logging.basicConfig(level=logging.INFO)   # shows any export error in the Cloud Run logs
PROJECT = os.environ.get("GOOGLE_CLOUD_PROJECT", "")
resource = Resource.create({"service.name": "otel-shop"})
tp = TracerProvider(resource=resource)
tp.add_span_processor(BatchSpanProcessor(CloudTraceSpanExporter(project_id=PROJECT)))
trace.set_tracer_provider(tp)
reader = PeriodicExportingMetricReader(CloudMonitoringMetricsExporter(project_id=PROJECT, add_unique_identifier=True), export_interval_millis=15000)
metrics.set_meter_provider(MeterProvider(resource=resource, metric_readers=[reader]))
tracer = trace.get_tracer("otel-shop")
checkouts = metrics.get_meter("otel-shop").create_counter("checkout_count", description="Checkouts handled")

def log(message, span):
    ctx = span.get_span_context()
    print(json.dumps({"severity": "INFO", "message": message,
        "logging.googleapis.com/trace": "projects/" + PROJECT + "/traces/" + format(ctx.trace_id, "032x"),
        "logging.googleapis.com/spanId": format(ctx.span_id, "016x")}), flush=True)

class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        url = urlparse(self.path)
        if url.path != "/checkout":
            self.send_response(404); self.end_headers(); return
        slow = "slow" in parse_qs(url.query)
        with tracer.start_as_current_span("checkout") as span:
            with tracer.start_as_current_span("check_stock"):
                time.sleep(0.05)
            with tracer.start_as_current_span("charge_card") as pay:
                pay.set_attribute("payment.gateway", "demo-gateway")
                time.sleep(1.5 if slow else 0.12)
            checkouts.add(1, {"result": "ok"})
            log("checkout done" + (" (slow gateway)" if slow else ""), span)
        self.send_response(200); self.send_header("Content-Type", "text/plain"); self.end_headers()
        self.wfile.write(b"checkout ok\n")

HTTPServer(("", int(os.environ.get("PORT", "8080"))), Handler).serve_forever()
EOF
cat > Procfile <<EOF
web: python main.py
EOF
echo "3.13" > .python-version
ls -1A
.python-version
Procfile
main.py
requirements.txt
The APIs and the app
Steps 1 and 2: the APIs are on and the app files are written

Step 3: deploy it to Cloud Run

Why: --source=. builds the container from the folder for you. --max-instances=1 keeps this lab to one copy of the app. GOOGLE_CLOUD_PROJECT tells the exporters which project to write to.

cd otel-shop
gcloud run deploy otel-shop --source=. --region=us-east1 --allow-unauthenticated \
  --max-instances=1 --set-env-vars=GOOGLE_CLOUD_PROJECT=$PROJECT_ID 2>&1 | tail -3
Done.
Service [otel-shop] revision [otel-shop-00002-7r9] has been deployed and is serving 100 percent of traffic.
Service URL: https://otel-shop-PROJECT_NUMBER.us-east1.run.app

Step 4: send five checkouts

URL=$(gcloud run services describe otel-shop --region=us-east1 --format="value(status.url)")
for i in 1 2 3 4 5; do curl -s -w "  %{time_total}s\n" $URL/checkout; done
checkout ok
  0.341210s
checkout ok
  0.348185s
checkout ok
  0.341743s
checkout ok
  0.335710s
checkout ok
  0.350720s
Deploy and call
Steps 3 and 4: the service is live and answers five checkouts in about 0.35 s each (ids blurred)

Step 5: read the traces from Cloud Trace

Why: in the console you would open Trace explorer; here we call the Cloud Trace API and print each trace as a list of spans with their duration. First save this small helper that turns the API's answer into a readable list:

cat > show_traces.py <<'EOF'
import json, sys
from datetime import datetime
def t(s): return datetime.fromisoformat(s.replace("Z", "+00:00").split(".")[0] + "+00:00" if "." in s else s.replace("Z", "+00:00"))
def ms(sp):
    a = sp["startTime"]; b = sp["endTime"]
    fa = float("0." + a.split(".")[1].rstrip("Z")) if "." in a else 0.0
    fb = float("0." + b.split(".")[1].rstrip("Z")) if "." in b else 0.0
    return round(((t(b) - t(a)).total_seconds() + fb - fa) * 1000)
data = json.load(sys.stdin)
for tr in data.get("traces", []):
    spans = [s for s in tr["spans"] if s["name"] in ("checkout", "check_stock", "charge_card")]
    if not spans: continue
    print("trace", tr["traceId"][:12] + "...")
    for s in sorted(spans, key=lambda s: s["startTime"]):
        print("   %-12s %5d ms" % (s["name"], ms(s)))
EOF

Now ask for the 3 newest traces whose root span is checkout:

curl -s -G -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  "https://cloudtrace.googleapis.com/v1/projects/$PROJECT_ID/traces" \
  --data-urlencode "view=COMPLETE" --data-urlencode "pageSize=3" \
  --data-urlencode "filter=root:checkout" | python3 show_traces.py
trace 04e41ad82f4b...
   checkout       172 ms
   check_stock     51 ms
   charge_card    121 ms
trace 14ddc09b50b0...
   checkout       172 ms
   check_stock     51 ms
   charge_card    120 ms
trace 190ee135c51d...
   checkout       172 ms
   check_stock     51 ms
   charge_card    121 ms
Traces
Step 5: each checkout is 172 ms: check_stock 51 ms + charge_card 121 ms

Each trace tells the same story: checkout took about 172 ms, of which check_stock was 51 ms and charge_card 121 ms. That matches the sleeps in the code.

Step 6: logs that point to their trace

Why: when a user reports a problem, you usually start from a log line. Because our log lines carry the trace id, every line points to its full trace.

gcloud logging read "resource.type=cloud_run_revision AND resource.labels.service_name=otel-shop AND textPayload=\"checkout done\"" \
  --freshness=30m --limit=3 --format="table(timestamp.date(%H:%M:%S),textPayload,trace.basename(),spanId)"
TIMESTAMP  TEXT_PAYLOAD   TRACE                             SPAN_ID
23:05:18   checkout done  65d745b2c4b576278cb7751c6dc3decf  0fd4315e75acd2b4
23:05:18   checkout done  250628579d6d11cc723de49ce0a3fb34  e0864d4cf0edefa8
23:05:18   checkout done  a86ed88fd0b91ff8a21bd3934c143d8c  77a803df2538df53
Logs with trace ids
Step 6: each 'checkout done' line has its trace id and span id

Break it: the payment gateway gets slow

Users say checkout "feels slow". We call the page with ?slow=1, which is what a slow payment provider would look like:

URL=$(gcloud run services describe otel-shop --region=us-east1 --format="value(status.url)")
for i in 1 2 3; do curl -s -w "  %{time_total}s\n" "$URL/checkout?slow=1"; done
checkout ok
  1.750340s
checkout ok
  1.720725s
checkout ok
  1.723516s
📝⭐ Recommended by CareerByteCode · Certification examGoogle Analytics Certification (GA4) PracticeTake the examTake the exam →
Slow checkouts
Break it: three checkouts with ?slow=1 take about 1.7 s each

Every checkout now takes about 1.7 seconds instead of 0.35. But which part is slow: the stock check, the card payment, the network, Cloud Run?

Find the cause with the trace, not by guessing

Ask Cloud Trace only for checkouts slower than 1,000 ms (latency:1000ms), and find the matching log line:

sleep 20
curl -s -G -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  "https://cloudtrace.googleapis.com/v1/projects/$PROJECT_ID/traces" \
  --data-urlencode "view=COMPLETE" --data-urlencode "pageSize=2" \
  --data-urlencode "filter=root:checkout latency:1000ms" | python3 show_traces.py
gcloud logging read "resource.type=cloud_run_revision AND resource.labels.service_name=otel-shop AND textPayload:slow" \
  --freshness=10m --limit=2 --format="table(timestamp.date(%H:%M:%S),textPayload,trace.basename())"
trace 18397e3725f0...
   checkout      1552 ms
   check_stock     50 ms
   charge_card   1501 ms
trace 81a3a2073147...
   checkout      1551 ms
   check_stock     51 ms
   charge_card   1500 ms
TIMESTAMP  TEXT_PAYLOAD                                                                      TRACE
23:06:53   EXTERNAL_IP - - [17/Nov/2026 23:06:53] "GET /checkout?slow=1 HTTP/1.1" 200 -
23:06:53   checkout done (slow gateway)                                                      f0631c496f0875acb78b7662e1a17095
Find the cause
The slow traces: charge_card 1500 ms, check_stock still 50 ms; the log line says slow gateway

Answer in seconds: check_stock is still 50 ms; charge_card takes 1,500 ms. The problem is the payment gateway, not your database, not Cloud Run. The log line "checkout done (slow gateway)" carries the trace id of the slow request, so a support engineer can open the same trace from the log.

POC: the metric counted every checkout

curl -s -G -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  "https://monitoring.googleapis.com/v3/projects/$PROJECT_ID/timeSeries" \
  --data-urlencode "filter=metric.type=\"workload.googleapis.com/checkout_count\"" \
  --data-urlencode "interval.startTime=$(date -u -d "-2 min" +%Y-%m-%dT%H:%M:%SZ)" \
  --data-urlencode "interval.endTime=$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
  | python3 -c "import sys,json; d=json.load(sys.stdin); [print(\"metric:\", s[\"metric\"][\"type\"], \"| result:\", s[\"metric\"][\"labels\"][\"result\"], \"| checkouts counted:\", s[\"points\"][0][\"value\"][\"int64Value\"]) for s in d.get(\"timeSeries\",[])]"
metric: workload.googleapis.com/checkout_count | result: ok | checkouts counted: 8
Metric
Cloud Monitoring holds workload.googleapis.com/checkout_count = 8 (5 normal + 3 slow)

Proof: the counter says 8: the 5 normal checkouts and the 3 slow ones. The same number is a graph in Monitoring > Metrics explorer (search for checkout_count), ready for a dashboard or an alert.

Two real lessons from our first try, so you do not repeat them:

  • Without add_unique_identifier=True, two copies of the app wrote to the same time series and Cloud Monitoring refused points with "Points must be written in order".
  • Always check the app's own logs for exporter errors (logging.basicConfig(level=logging.INFO) makes them visible). If traces do not show up, the answer is usually there.

Clean up (delete everything)

Why: remove the service, the images that --source created and the source bucket. Trace, metric and log data expire on their own (traces after 30 days).

gcloud run services delete otel-shop --region=us-east1 --quiet 2>&1 | tail -1
gcloud artifacts repositories delete cloud-run-source-deploy --location=us-east1 --quiet 2>&1 | tail -1
for B in $(gcloud storage buckets list --format="value(name)"); do gcloud storage rm --recursive gs://$B --quiet > /dev/null 2>&1; done
echo "services: $(gcloud run services list --format="value(name)" | wc -l), repositories: $(gcloud artifacts repositories list --location=us-east1 --format="value(name)" 2>/dev/null | wc -l), buckets: $(gcloud storage buckets list --format="value(name)" | wc -l)"
Deleted service [otel-shop].
Deleted repository [cloud-run-source-deploy].
services: 0, repositories: 0, buckets: 0
Clean up
0 services, 0 repositories, 0 buckets left

FAQ

Do I need to change my code to switch tools? Only the exporter setup lines. The spans, attributes and counters stay the same. That is the main promise of OpenTelemetry.

What is the difference between a trace and a log? A log is one event. A trace is the whole journey of one request, with timing for every step. Use both, linked by the trace id.

Is there automatic instrumentation? Yes. OpenTelemetry has instrumentation packages for Flask, Django, requests, database drivers and more, which create spans without code changes. We wrote spans by hand so you can see what they are.

What does BatchSpanProcessor do? It collects finished spans in memory and sends them in groups every few seconds, so your requests are not slowed down by sending telemetry.

Why did the first request in Step 4 not take longer? On our second deployment the instance was already warm. A cold start (the first request to a new instance) can take several seconds.

Where do I see this in the console? Trace explorer for traces, Metrics explorer for the counter, Logs Explorer for the log lines (click a trace id to open the trace).

Troubleshooting

You seeWhyFix
No traces in Cloud TraceThe exporter is failing or the API is offCheck the Cloud Run logs for errors; run Step 1 again
Points must be written in order in the logsTwo app copies write to the same metric seriesCloudMonitoringMetricsExporter(..., add_unique_identifier=True)
Build fails with failed to resolve version matchingThe Python version in .python-version is not offered by the builderUse a version from the list in the build log (we used 3.13)
Log lines have no trace linkThe JSON line misses logging.googleapis.com/tracePrint the trace field as in log(), with the full projects/.../traces/... value
The metric shows nothingMetrics are sent every 15 seconds and need a minute to appearWait 1 to 2 minutes and read again
Permission denied writing traces or metricsThe service account lacks Cloud Trace Agent or Monitoring Metric WriterGive those roles to the Cloud Run service account

What next

Prepared by the CareerByteCode Team from our tested lab, with AI help for the write-up.

Read next →Logs to BigQuery: Analyse a Month of Logs with SQL⏱ 13 min

⭐ Keep learning with CareerByteCode

Hand-picked to go deeper on this topic
News, guides and updates from the CareerByteCode team
🏅 Contributor writer

The CareerByteCode team writes about what is new on careerbytecode.in: courses, labs, exams, events and the community, which is free to join for influencers and content creators in tech.

Get CareerByteCode Team's next blog by email

One email a week when they publish. Add the weekly Best of ByteBlog too. No spam, unsubscribe with one click.

We email you once to confirm. By subscribing you agree to the privacy policy.

💬 Comments (0)

Top · Newest

Sign in to comment

More from CareerByteCode Team

Readers of this blog also read

🧪⭐ Recommended by CareerByteCode · ByteLabs project bundleCloud and DevOps Realtime Starterβ‚Ή499Build the projects →