MLflow on Google Cloud: Track Every Experiment
What you build: an MLflow tracking server on Google Cloud with models in Cloud Storage, five tracked runs, a registered champion and the missing-URI mistake. Every step tested.

✍️ Translate this blog into Hindi, Tamil or Telugu
"Which settings gave us that good model last month?" If the answer is a spreadsheet, a notebook cell or someone's memory, you need experiment tracking. MLflow is the open-source standard: every training run logs its parameters, metrics and model file to one place, you compare runs side by side, register the best one under a name, and load it by that name in production. In this lab you run an MLflow tracking server on a Google Cloud VM with model files in Cloud Storage, track five experiments, register and load the champion, and reproduce the classic "my runs disappeared" mistake.
Who it is for: data scientists and ML engineers who train more than one model and want a record they can trust. Time: about 45 minutes. Cost: one VM for about 15 minutes and a few MB in Cloud Storage, covered by the free trial credit.
Every command below was run for real by the CareerByteCode Team, and every output is the real result we got. The VM was created with gcloud from a Cloud Shell-like Linux machine; MLflow ran on the VM over SSH.
Part A: MLflow in plain English

- Run: one training attempt. MLflow records its parameters (settings), metrics (results), tags and artifacts (files, such as the model).
- Experiment: a group of runs for one problem (here
wine-classifier). - Tracking server: the service runs are logged to. Ours keeps run history in a small SQLite database and model files in Cloud Storage.
- Model registry: named models with versions (
wine-classifierversion 1) and aliases (champion), so apps load "the champion", not a file path. - MLFLOW_TRACKING_URI: tells a script where the server is. Forget it, and MLflow logs to the local folder instead.
- Real-life example: a lab notebook where every experiment is written down automatically, with the sample kept in a labelled freezer drawer (Cloud Storage), and a sticker on the drawer that holds the current best result (the champion alias).
Before you start
A Google Cloud project with billing linked (the free trial works). The MLflow server and the ML libraries need memory: on a 2 GB VM our first run hung, so we use an e2-standard-2 (8 GB).
Part B: the lab
Step 1: a VM for MLflow
# A VM for the lab with enough memory for the Python ML libraries (e2-standard-2: 2 vCPUs, 8 GB)
gcloud compute instances create mlflow-vm --zone=us-east1-b --machine-type=e2-standard-2 --image-family=debian-12 --image-project=debian-cloud --boot-disk-size=20GB --service-account=lab-shell@$PROJECT_ID.iam.gserviceaccount.com --scopes=cloud-platform > /dev/null 2>&1 && echo "mlflow-vm created"
until gcloud compute ssh mlflow-vm --zone=us-east1-b --tunnel-through-iap --command="true" > /dev/null 2>&1; do sleep 10; done; echo "SSH works"mlflow-vm created
SSH worksStep 2: MLflow and an artifact bucket
# MLflow 3 and scikit-learn in a Python virtual environment, plus a bucket for the model files (artifacts)
sudo apt-get update -qq > /dev/null && sudo apt-get install -y -qq python3-venv > /dev/null 2>&1; python3 -m venv .venv && . .venv/bin/activate && pip install -q "mlflow==3.*" "scikit-learn==1.*" "google-cloud-storage==2.*" 2>&1 | grep -v notice | tail -1; mlflow --version
echo "mlflow-$(echo -n $PROJECT_ID | md5sum | cut -c1-8)" > .bucket
gcloud storage buckets create gs://$(cat .bucket) --location=us-east1 --uniform-bucket-level-access --public-access-prevention > /dev/null && echo "artifact bucket created"mlflow, version 3.18.0
Creating gs://mlflow-2c18be0d/...
artifact bucket createdStep 3: the tracking server
Why: one server for the whole team. Run history goes to SQLite (enough for a lab; use Cloud SQL for a team), model files go to the bucket. It listens only on the VM itself (127.0.0.1); reach it from your laptop with an SSH tunnel.
# The tracking server: run history in a small SQLite database, model files in Cloud Storage. Port 5000, this VM only
. .venv/bin/activate
nohup mlflow server --backend-store-uri sqlite:///mlflow.db --artifacts-destination gs://$(cat .bucket)/artifacts --host 127.0.0.1 --port 5000 --workers 1 > server.log 2>&1 &
for i in $(seq 1 90); do curl -s localhost:5000/health > /dev/null && break; sleep 2; done
H=$(curl -s localhost:5000/health); echo "server health: $H"; [ -n "$H" ] || { tail -15 server.log; exit 1; }server health: OKStep 4: five experiments
Why: two model types with different settings, on scikit-learn's wine data set (three grape cultivars from 13 chemical measurements). train.py logs parameters, a data-version tag, accuracy and F1, and the model itself. Save it as train.py:
# train.py <model> [params...]: train one model on the wine quality data (sklearn's built-in wine dataset),
# and log everything to MLflow: parameters, metrics, the model file and a tag with the data version.
import sys
import mlflow
from sklearn.datasets import load_wine
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, f1_score
from sklearn.model_selection import train_test_split
kind = sys.argv[1]
params = dict(a.split("=") for a in sys.argv[2:])
X, y = load_wine(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42, stratify=y)
if kind == "forest":
model = RandomForestClassifier(n_estimators=int(params.get("trees", 50)), max_depth=int(params.get("depth", 3)), random_state=42)
else:
model = LogisticRegression(C=float(params.get("C", 1.0)), max_iter=int(params.get("iters", 50)))
mlflow.set_experiment("wine-classifier")
with mlflow.start_run(run_name=f"{kind} {' '.join(sys.argv[2:])}") as run:
mlflow.log_params({"model": kind, **params})
mlflow.set_tag("data_version", "sklearn-wine-v1")
model.fit(X_train, y_train)
pred = model.predict(X_test)
acc, f1 = accuracy_score(y_test, pred), f1_score(y_test, pred, average="macro")
mlflow.log_metrics({"accuracy": acc, "f1_macro": f1})
# MLflow 3 saves scikit-learn models safely with skops; tree models must be trusted on purpose
mlflow.sklearn.log_model(model, name="model", input_example=X_test[:2], skops_trusted_types=["sklearn.tree._tree.Tree"])
print(f"run {run.info.run_id[:8]} {kind:<8} {str(params):<28} accuracy {acc:.3f} f1 {f1:.3f}")One line deserves attention: MLflow 3 saves scikit-learn models with skops, a safe format that refuses to load unknown types. Tree models (random forests) must be trusted on purpose with skops_trusted_types; without it, our first forest runs failed when saving the model.
# Five experiments: two model types, different settings. Each run logs params, metrics and the model
. .venv/bin/activate; export MLFLOW_TRACKING_URI=http://127.0.0.1:5000 PYTHONWARNINGS=ignore
python3 train.py logistic C=0.1 iters=50 2>/dev/null
python3 train.py logistic C=1.0 iters=500 2>/dev/null
python3 train.py forest trees=10 depth=2 2>/dev/null
python3 train.py forest trees=100 depth=3 2>/dev/null
python3 train.py forest trees=200 depth=8 2>/dev/nullrun 41994824 logistic {'C': '0.1', 'iters': '50'} accuracy 0.981 f1 0.981
๐ View run logistic C=0.1 iters=50 at: http://127.0.0.1:5000/#/experiments/1/runs/41994824f0734abb9534a8c4fb78a3e6
๐งช View experiment at: http://127.0.0.1:5000/#/experiments/1
run d7d51669 logistic {'C': '1.0', 'iters': '500'} accuracy 0.981 f1 0.981
๐ View run logistic C=1.0 iters=500 at: http://127.0.0.1:5000/#/experiments/1/runs/d7d51669d72b4a9f8f3cdbfc8ce917f4
๐งช View experiment at: http://127.0.0.1:5000/#/experiments/1
run fb530ab6 forest {'trees': '10', 'depth': '2'} accuracy 0.944 f1 0.946
๐ View run forest trees=10 depth=2 at: http://127.0.0.1:5000/#/experiments/1/runs/fb530ab6e3774de0ae6534f93582ae1b
๐งช View experiment at: http://127.0.0.1:5000/#/experiments/1
run 04ecd89e forest {'trees': '100', 'depth': '3'} accuracy 1.000 f1 1.000
๐ View run forest trees=100 depth=3 at: http://127.0.0.1:5000/#/experiments/1/runs/04ecd89eca79478199af37ae72daf9fb
๐งช View experiment at: http://127.0.0.1:5000/#/experiments/1
run 6478f5db forest {'trees': '200', 'depth': '8'} accuracy 1.000 f1 1.000
๐ View run forest trees=200 depth=8 at: http://127.0.0.1:5000/#/experiments/1/runs/6478f5db294d497f9a0dcd0b8e44657d
๐งช View experiment at: http://127.0.0.1:5000/#/experiments/1Step 5: compare the runs
# Compare: ask MLflow for the runs, best F1 first (the same table the MLflow web page shows)
. .venv/bin/activate; export MLFLOW_TRACKING_URI=http://127.0.0.1:5000
python3 -c "
import mlflow
df = mlflow.search_runs(experiment_names=['wine-classifier'], order_by=['metrics.f1_macro DESC'])
print(df[['tags.mlflow.runName', 'metrics.accuracy', 'metrics.f1_macro']].round(3).to_string(index=False))"
gcloud storage ls -r gs://$(cat .bucket)/artifacts | grep -c "MLmodel" | sed 's/$/ model folders stored in Cloud Storage/' tags.mlflow.runName metrics.accuracy metrics.f1_macro
forest trees=200 depth=8 1.000 1.000
forest trees=100 depth=3 1.000 1.000
logistic C=1.0 iters=500 0.981 0.981
logistic C=0.1 iters=50 0.981 0.981
forest trees=10 depth=2 0.944 0.946
5 model folders stored in Cloud StorageThe deep forests reached 100 percent on the test split (this data set is small and easy); all five models are stored in Cloud Storage. In real projects, look at the gap between settings, not just the top line.

Step 6: register the champion
# Register the best run as version 1 of a named model and give it the alias "champion"
. .venv/bin/activate; export MLFLOW_TRACKING_URI=http://127.0.0.1:5000
python3 -c "
import mlflow
from mlflow import MlflowClient
best = mlflow.search_runs(experiment_names=['wine-classifier'], order_by=['metrics.f1_macro DESC'], max_results=1).iloc[0]
mv = mlflow.register_model(f'runs:/{best.run_id}/model', 'wine-classifier')
MlflowClient().set_registered_model_alias('wine-classifier', 'champion', mv.version)
print('registered wine-classifier version', mv.version, 'from run', best.run_id[:8], 'as champion')" 2>&1 | grep -v "^Successfully\|Created version"2026/11/19 10:47:41 WARNING mlflow.tracking._model_registry.fluent: Run with id 6478f5db294d497f9a0dcd0b8e44657d has no artifacts at artifact path 'model', registering model based on models:/m-27ddda785cf049d7b4beee5487dc7f1c instead
2026/11/19 10:47:41 INFO mlflow.store.model_registry.abstract_store: Waiting up to 300 seconds for model version to finish creation. Model name: wine-classifier, version 1
registered wine-classifier version 1 from run 6478f5db as championThe best run became wine-classifier version 1 with the alias champion. The warning is MLflow 3 telling us that models are now stored as their own logged model objects (models:/m-...) rather than inside the run's folder; registration used that, which is the new normal.
Step 7: load the champion like an app would
# Use it like a serving app would: load "the champion", not a file path, and predict two wines
. .venv/bin/activate; export MLFLOW_TRACKING_URI=http://127.0.0.1:5000
python3 -c "
import mlflow
from sklearn.datasets import load_wine
m = mlflow.pyfunc.load_model('models:/wine-classifier@champion')
print('predicted classes:', [int(x) for x in m.predict(load_wine().data[[0, 100]])], '(true: [0, 1])')" 2>/dev/nullpredicted classes: [0, 1] (true: [0, 1])The app asks for models:/wine-classifier@champion. When you promote a better model, you move the alias; the app code does not change.
Break it: the runs that "disappeared"
A teammate trains in a new terminal and forgets to set MLFLOW_TRACKING_URI:
# Break it: a teammate trains in a new terminal and forgets MLFLOW_TRACKING_URI. The run "disappears"
. .venv/bin/activate; unset MLFLOW_TRACKING_URI; export PYTHONWARNINGS=ignore
mkdir -p ~/teammate && (cd ~/teammate && python3 ~/lab243/train.py forest trees=300 depth=10 2>/dev/null)
echo "runs on the server: $(curl -s -X POST localhost:5000/api/2.0/mlflow/runs/search -H 'Content-Type: application/json' -d '{"experiment_ids": ["1"], "max_results": 100}' | python3 -c 'import json, sys; print(len(json.load(sys.stdin).get("runs", [])))')"
echo "what appeared in the teammate folder instead: $(ls ~/teammate | tr '\n' ' ')"run 3d8fd4a1 forest {'trees': '300', 'depth': '10'} accuracy 1.000 f1 1.000
runs on the server: 5
what appeared in the teammate folder instead: mlflow.db mlrunsThe run trained fine, but the server still has 5 runs: MLflow silently wrote this one to mlflow.db and mlruns in the teammate's own folder. Nobody else will ever see it. This is the most common MLflow support question.
Fix it: set the tracking URI for everyone
Put it where every shell, job and CI pipeline picks it up (a profile line here; environment settings in CI):
# Fix it: set the tracking URI for everyone (a profile line, a .env file, or the CI settings) and run again
echo 'export MLFLOW_TRACKING_URI=http://127.0.0.1:5000' >> ~/.bashrc; . .venv/bin/activate; export MLFLOW_TRACKING_URI=http://127.0.0.1:5000 PYTHONWARNINGS=ignore
python3 train.py forest trees=300 depth=10 2>/dev/null
echo "runs on the server: $(curl -s -X POST localhost:5000/api/2.0/mlflow/runs/search -H 'Content-Type: application/json' -d '{"experiment_ids": ["1"], "max_results": 100}' | python3 -c 'import json, sys; print(len(json.load(sys.stdin).get("runs", [])))')"run 57551cfb forest {'trees': '300', 'depth': '10'} accuracy 1.000 f1 1.000
๐ View run forest trees=300 depth=10 at: http://127.0.0.1:5000/#/experiments/1/runs/57551cfbd93e4512938e9cba61ab60d4
๐งช View experiment at: http://127.0.0.1:5000/#/experiments/1
runs on the server: 6The same run now lands on the server: 6 runs.

POC: prove it works
- Five runs logged with parameters, metrics and models; models in Cloud Storage (Steps 4 and 5).
- The best run registered as
wine-classifierwith thechampionalias and loaded by that name (Steps 6 and 7). - A missing tracking URI sent a run to a local folder; setting it fixed it (Break it, Fix it).
Clean up (delete everything)
# Clean up: delete the VM (the server, its database and the local files go with it) and the artifact bucket
gcloud compute instances delete mlflow-vm --zone=us-east1-b --quiet 2>&1 | tail -1
gcloud storage rm -r gs://mlflow-$(echo -n $PROJECT_ID | md5sum | cut -c1-8) --quiet > /dev/null 2>&1; echo "left: $(gcloud compute instances list --filter='name=mlflow-vm' --format='value(name)' | wc -l) VMs, $(gcloud storage ls 2>/dev/null | grep -c mlflow-) buckets"Deleted [https://www.googleapis.com/compute/v1/projects/your-project-id/zones/us-east1-b/instances/mlflow-vm].
left: 0 VMs, 0 bucketsFAQ
SQLite or a real database? SQLite is fine for one person. For a team, use PostgreSQL or MySQL (for example Cloud SQL) as the backend store, so several people can log at once.
How do I open the MLflow web page? From your laptop: gcloud compute ssh mlflow-vm --tunnel-through-iap -- -L 5000:localhost:5000, then open http://localhost:5000. We used the API in this lab.
MLflow or Vertex AI Experiments? Vertex AI Experiments is Google's managed tracking, integrated with Vertex AI training. MLflow runs anywhere and is the most widely used open-source option. Both answer "what did each run do?".
**Why trust sklearn.tree._tree.Tree?** Because we created the model ourselves. skops blocks types it cannot verify, so a model file from someone else cannot run harmful code when loaded. Only trust types you have reviewed.
What should I log? Every parameter you change, the data version (a tag or a hash), the metrics you decide with, and the model. If you cannot reproduce a run from its record, the record is incomplete.
Troubleshooting
| You see | Why | Fix |
|---|---|---|
| Runs are missing on the server | The script used no tracking URI and logged locally | Set MLFLOW_TRACKING_URI everywhere you train |
The saved sklearn model references untrusted types | MLflow 3 uses skops; tree models need trusting | Pass skops_trusted_types=["sklearn.tree._tree.Tree"] to log_model |
The server never answers /health | Not started, or out of memory | Read server.log; use a VM with enough memory |
| Training hangs | The tracking server is unreachable and MLflow keeps retrying | Check the server first; then train |
| Model files fail to upload | The VM's account cannot write to the bucket | Grant it Storage Object Admin on the bucket |
What next
- Put the model into a full loop: the MLOps resume project in this series.
- Take a model all the way to production: MLOps for Beginners: From Notebook to Production.
- More hands-on posts: ByteBlog. Practice in ByteLabs and see the courses.
Prepared by the CareerByteCode Team from our tested lab, with AI help for the write-up.
⭐ Keep learning with CareerByteCode
Hand-picked to go deeper on this topicGet CareerByteCode Team's next blog by email
One email a week when they publish. Add the weekly Best of ByteBlog too. No spam, unsubscribe with one click.
We email you once to confirm. By subscribing you agree to the privacy policy.
Related blogs
CareerByteCode TeamMLOps Project for Your Resume on Google Cloud
What you build: an MLOps loop on Google Cloud: quality gate, model registry, batch scoring to BigQuery, drift checks and a retrain that must earn its place. Every step tested.
#ci
CareerByteCode TeamCI/CD for Machine Learning with Cloud Build
What you build: a Cloud Build pipeline that retrains a churn model whenever new data lands in a bucket, checks it with a quality gate and redeploys on Cloud Run; broken data is stopped automatically.
#feature
CareerByteCode TeamFeature Store Explained with a Real Example
What you build: Feast with real customer data: a leaky training set that scores 94 percent, the point-in-time truth, online serving and TTL. Every step tested.
#google-cloud
CareerByteCode TeamMLOps Interview Questions Answered with Labs
What you build: 6 MLOps interview answers proved with real scikit-learn experiments: leakage, reproducibility, imbalance, skew, versioning, latency. Every step tested.
💬 Comments (0)
Top · NewestSign in to comment