CareerByteCode
ByteBlogby CareerByteCode

MLflow on Google Cloud: Track Every Experiment

What you build: an MLflow tracking server on Google Cloud with models in Cloud Storage, five tracked runs, a registered champion and the missing-URI mistake. Every step tested.

MLflow on Google Cloud: Track Every Experiment

✍️ Translate this blog into Hindi, Tamil or Telugu

"Which settings gave us that good model last month?" If the answer is a spreadsheet, a notebook cell or someone's memory, you need experiment tracking. MLflow is the open-source standard: every training run logs its parameters, metrics and model file to one place, you compare runs side by side, register the best one under a name, and load it by that name in production. In this lab you run an MLflow tracking server on a Google Cloud VM with model files in Cloud Storage, track five experiments, register and load the champion, and reproduce the classic "my runs disappeared" mistake.

Who it is for: data scientists and ML engineers who train more than one model and want a record they can trust. Time: about 45 minutes. Cost: one VM for about 15 minutes and a few MB in Cloud Storage, covered by the free trial credit.

Every command below was run for real by the CareerByteCode Team, and every output is the real result we got. The VM was created with gcloud from a Cloud Shell-like Linux machine; MLflow ran on the VM over SSH.

🧪⭐ Recommended by CareerByteCode · ByteLabs project bundleCloud and DevOps Realtime Starterโ‚น499Build the projects →

Part A: MLflow in plain English

MLflow on Google Cloud
Training scripts log to the tracking server; run history goes to SQLite, model files to Cloud Storage; the registry names the champion
  • Run: one training attempt. MLflow records its parameters (settings), metrics (results), tags and artifacts (files, such as the model).
  • Experiment: a group of runs for one problem (here wine-classifier).
  • Tracking server: the service runs are logged to. Ours keeps run history in a small SQLite database and model files in Cloud Storage.
  • Model registry: named models with versions (wine-classifier version 1) and aliases (champion), so apps load "the champion", not a file path.
  • MLFLOW_TRACKING_URI: tells a script where the server is. Forget it, and MLflow logs to the local folder instead.
  • Real-life example: a lab notebook where every experiment is written down automatically, with the sample kept in a labelled freezer drawer (Cloud Storage), and a sticker on the drawer that holds the current best result (the champion alias).

Before you start

A Google Cloud project with billing linked (the free trial works). The MLflow server and the ML libraries need memory: on a 2 GB VM our first run hung, so we use an e2-standard-2 (8 GB).

Part B: the lab

Step 1: a VM for MLflow

# A VM for the lab with enough memory for the Python ML libraries (e2-standard-2: 2 vCPUs, 8 GB)
gcloud compute instances create mlflow-vm --zone=us-east1-b --machine-type=e2-standard-2 --image-family=debian-12 --image-project=debian-cloud --boot-disk-size=20GB --service-account=lab-shell@$PROJECT_ID.iam.gserviceaccount.com --scopes=cloud-platform > /dev/null 2>&1 && echo "mlflow-vm created"
until gcloud compute ssh mlflow-vm --zone=us-east1-b --tunnel-through-iap --command="true" > /dev/null 2>&1; do sleep 10; done; echo "SSH works"
mlflow-vm created
SSH works

Step 2: MLflow and an artifact bucket

# MLflow 3 and scikit-learn in a Python virtual environment, plus a bucket for the model files (artifacts)
sudo apt-get update -qq > /dev/null && sudo apt-get install -y -qq python3-venv > /dev/null 2>&1; python3 -m venv .venv && . .venv/bin/activate && pip install -q "mlflow==3.*" "scikit-learn==1.*" "google-cloud-storage==2.*" 2>&1 | grep -v notice | tail -1; mlflow --version
echo "mlflow-$(echo -n $PROJECT_ID | md5sum | cut -c1-8)" > .bucket
gcloud storage buckets create gs://$(cat .bucket) --location=us-east1 --uniform-bucket-level-access --public-access-prevention > /dev/null && echo "artifact bucket created"
mlflow, version 3.18.0
Creating gs://mlflow-2c18be0d/...
artifact bucket created

Step 3: the tracking server

Why: one server for the whole team. Run history goes to SQLite (enough for a lab; use Cloud SQL for a team), model files go to the bucket. It listens only on the VM itself (127.0.0.1); reach it from your laptop with an SSH tunnel.

# The tracking server: run history in a small SQLite database, model files in Cloud Storage. Port 5000, this VM only
. .venv/bin/activate
nohup mlflow server --backend-store-uri sqlite:///mlflow.db --artifacts-destination gs://$(cat .bucket)/artifacts --host 127.0.0.1 --port 5000 --workers 1 > server.log 2>&1 &
for i in $(seq 1 90); do curl -s localhost:5000/health > /dev/null && break; sleep 2; done
H=$(curl -s localhost:5000/health); echo "server health: $H"; [ -n "$H" ] || { tail -15 server.log; exit 1; }
server health: OK

Step 4: five experiments

Why: two model types with different settings, on scikit-learn's wine data set (three grape cultivars from 13 chemical measurements). train.py logs parameters, a data-version tag, accuracy and F1, and the model itself. Save it as train.py:

# train.py <model> [params...]: train one model on the wine quality data (sklearn's built-in wine dataset),
# and log everything to MLflow: parameters, metrics, the model file and a tag with the data version.
import sys
import mlflow
from sklearn.datasets import load_wine
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, f1_score
from sklearn.model_selection import train_test_split

kind = sys.argv[1]
params = dict(a.split("=") for a in sys.argv[2:])
X, y = load_wine(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42, stratify=y)
if kind == "forest":
    model = RandomForestClassifier(n_estimators=int(params.get("trees", 50)), max_depth=int(params.get("depth", 3)), random_state=42)
else:
    model = LogisticRegression(C=float(params.get("C", 1.0)), max_iter=int(params.get("iters", 50)))

mlflow.set_experiment("wine-classifier")
with mlflow.start_run(run_name=f"{kind} {' '.join(sys.argv[2:])}") as run:
    mlflow.log_params({"model": kind, **params})
    mlflow.set_tag("data_version", "sklearn-wine-v1")
    model.fit(X_train, y_train)
    pred = model.predict(X_test)
    acc, f1 = accuracy_score(y_test, pred), f1_score(y_test, pred, average="macro")
    mlflow.log_metrics({"accuracy": acc, "f1_macro": f1})
    # MLflow 3 saves scikit-learn models safely with skops; tree models must be trusted on purpose
    mlflow.sklearn.log_model(model, name="model", input_example=X_test[:2], skops_trusted_types=["sklearn.tree._tree.Tree"])
    print(f"run {run.info.run_id[:8]} {kind:<8} {str(params):<28} accuracy {acc:.3f} f1 {f1:.3f}")

One line deserves attention: MLflow 3 saves scikit-learn models with skops, a safe format that refuses to load unknown types. Tree models (random forests) must be trusted on purpose with skops_trusted_types; without it, our first forest runs failed when saving the model.

# Five experiments: two model types, different settings. Each run logs params, metrics and the model
. .venv/bin/activate; export MLFLOW_TRACKING_URI=http://127.0.0.1:5000 PYTHONWARNINGS=ignore
python3 train.py logistic C=0.1 iters=50 2>/dev/null
python3 train.py logistic C=1.0 iters=500 2>/dev/null
python3 train.py forest trees=10 depth=2 2>/dev/null
python3 train.py forest trees=100 depth=3 2>/dev/null
python3 train.py forest trees=200 depth=8 2>/dev/null
run 41994824 logistic {'C': '0.1', 'iters': '50'}  accuracy 0.981 f1 0.981
๐Ÿƒ View run logistic C=0.1 iters=50 at: http://127.0.0.1:5000/#/experiments/1/runs/41994824f0734abb9534a8c4fb78a3e6
๐Ÿงช View experiment at: http://127.0.0.1:5000/#/experiments/1
run d7d51669 logistic {'C': '1.0', 'iters': '500'} accuracy 0.981 f1 0.981
๐Ÿƒ View run logistic C=1.0 iters=500 at: http://127.0.0.1:5000/#/experiments/1/runs/d7d51669d72b4a9f8f3cdbfc8ce917f4
๐Ÿงช View experiment at: http://127.0.0.1:5000/#/experiments/1
run fb530ab6 forest   {'trees': '10', 'depth': '2'} accuracy 0.944 f1 0.946
๐Ÿƒ View run forest trees=10 depth=2 at: http://127.0.0.1:5000/#/experiments/1/runs/fb530ab6e3774de0ae6534f93582ae1b
๐Ÿงช View experiment at: http://127.0.0.1:5000/#/experiments/1
run 04ecd89e forest   {'trees': '100', 'depth': '3'} accuracy 1.000 f1 1.000
๐Ÿƒ View run forest trees=100 depth=3 at: http://127.0.0.1:5000/#/experiments/1/runs/04ecd89eca79478199af37ae72daf9fb
๐Ÿงช View experiment at: http://127.0.0.1:5000/#/experiments/1
run 6478f5db forest   {'trees': '200', 'depth': '8'} accuracy 1.000 f1 1.000
๐Ÿƒ View run forest trees=200 depth=8 at: http://127.0.0.1:5000/#/experiments/1/runs/6478f5db294d497f9a0dcd0b8e44657d
๐Ÿงช View experiment at: http://127.0.0.1:5000/#/experiments/1

Step 5: compare the runs

# Compare: ask MLflow for the runs, best F1 first (the same table the MLflow web page shows)
. .venv/bin/activate; export MLFLOW_TRACKING_URI=http://127.0.0.1:5000
python3 -c "
import mlflow
df = mlflow.search_runs(experiment_names=['wine-classifier'], order_by=['metrics.f1_macro DESC'])
print(df[['tags.mlflow.runName', 'metrics.accuracy', 'metrics.f1_macro']].round(3).to_string(index=False))"
gcloud storage ls -r gs://$(cat .bucket)/artifacts | grep -c "MLmodel" | sed 's/$/ model folders stored in Cloud Storage/'
     tags.mlflow.runName  metrics.accuracy  metrics.f1_macro
forest trees=200 depth=8             1.000             1.000
forest trees=100 depth=3             1.000             1.000
logistic C=1.0 iters=500             0.981             0.981
 logistic C=0.1 iters=50             0.981             0.981
 forest trees=10 depth=2             0.944             0.946
5 model folders stored in Cloud Storage

The deep forests reached 100 percent on the test split (this data set is small and easy); all five models are stored in Cloud Storage. In real projects, look at the gap between settings, not just the top line.

Five tracked runs
Five runs with parameters and metrics, sorted by F1, and five model folders in Cloud Storage

Step 6: register the champion

# Register the best run as version 1 of a named model and give it the alias "champion"
. .venv/bin/activate; export MLFLOW_TRACKING_URI=http://127.0.0.1:5000
python3 -c "
import mlflow
from mlflow import MlflowClient
best = mlflow.search_runs(experiment_names=['wine-classifier'], order_by=['metrics.f1_macro DESC'], max_results=1).iloc[0]
mv = mlflow.register_model(f'runs:/{best.run_id}/model', 'wine-classifier')
MlflowClient().set_registered_model_alias('wine-classifier', 'champion', mv.version)
print('registered wine-classifier version', mv.version, 'from run', best.run_id[:8], 'as champion')" 2>&1 | grep -v "^Successfully\|Created version"
2026/11/19 10:47:41 WARNING mlflow.tracking._model_registry.fluent: Run with id 6478f5db294d497f9a0dcd0b8e44657d has no artifacts at artifact path 'model', registering model based on models:/m-27ddda785cf049d7b4beee5487dc7f1c instead
2026/11/19 10:47:41 INFO mlflow.store.model_registry.abstract_store: Waiting up to 300 seconds for model version to finish creation. Model name: wine-classifier, version 1
registered wine-classifier version 1 from run 6478f5db as champion

The best run became wine-classifier version 1 with the alias champion. The warning is MLflow 3 telling us that models are now stored as their own logged model objects (models:/m-...) rather than inside the run's folder; registration used that, which is the new normal.

Step 7: load the champion like an app would

# Use it like a serving app would: load "the champion", not a file path, and predict two wines
. .venv/bin/activate; export MLFLOW_TRACKING_URI=http://127.0.0.1:5000
python3 -c "
import mlflow
from sklearn.datasets import load_wine
m = mlflow.pyfunc.load_model('models:/wine-classifier@champion')
print('predicted classes:', [int(x) for x in m.predict(load_wine().data[[0, 100]])], '(true: [0, 1])')" 2>/dev/null
predicted classes: [0, 1] (true: [0, 1])

The app asks for models:/wine-classifier@champion. When you promote a better model, you move the alias; the app code does not change.

Break it: the runs that "disappeared"

A teammate trains in a new terminal and forgets to set MLFLOW_TRACKING_URI:

📝⭐ Recommended by CareerByteCode · Certification examGoogle Analytics Certification (GA4) PracticeTake the examTake the exam →
# Break it: a teammate trains in a new terminal and forgets MLFLOW_TRACKING_URI. The run "disappears"
. .venv/bin/activate; unset MLFLOW_TRACKING_URI; export PYTHONWARNINGS=ignore
mkdir -p ~/teammate && (cd ~/teammate && python3 ~/lab243/train.py forest trees=300 depth=10 2>/dev/null)
echo "runs on the server: $(curl -s -X POST localhost:5000/api/2.0/mlflow/runs/search -H 'Content-Type: application/json' -d '{"experiment_ids": ["1"], "max_results": 100}' | python3 -c 'import json, sys; print(len(json.load(sys.stdin).get("runs", [])))')"
echo "what appeared in the teammate folder instead: $(ls ~/teammate | tr '\n' ' ')"
run 3d8fd4a1 forest   {'trees': '300', 'depth': '10'} accuracy 1.000 f1 1.000
runs on the server: 5
what appeared in the teammate folder instead: mlflow.db mlruns

The run trained fine, but the server still has 5 runs: MLflow silently wrote this one to mlflow.db and mlruns in the teammate's own folder. Nobody else will ever see it. This is the most common MLflow support question.

Fix it: set the tracking URI for everyone

Put it where every shell, job and CI pipeline picks it up (a profile line here; environment settings in CI):

# Fix it: set the tracking URI for everyone (a profile line, a .env file, or the CI settings) and run again
echo 'export MLFLOW_TRACKING_URI=http://127.0.0.1:5000' >> ~/.bashrc; . .venv/bin/activate; export MLFLOW_TRACKING_URI=http://127.0.0.1:5000 PYTHONWARNINGS=ignore
python3 train.py forest trees=300 depth=10 2>/dev/null
echo "runs on the server: $(curl -s -X POST localhost:5000/api/2.0/mlflow/runs/search -H 'Content-Type: application/json' -d '{"experiment_ids": ["1"], "max_results": 100}' | python3 -c 'import json, sys; print(len(json.load(sys.stdin).get("runs", [])))')"
run 57551cfb forest   {'trees': '300', 'depth': '10'} accuracy 1.000 f1 1.000
๐Ÿƒ View run forest trees=300 depth=10 at: http://127.0.0.1:5000/#/experiments/1/runs/57551cfbd93e4512938e9cba61ab60d4
๐Ÿงช View experiment at: http://127.0.0.1:5000/#/experiments/1
runs on the server: 6

The same run now lands on the server: 6 runs.

Champion, break and fix
The champion loaded by alias predicts correctly; a run without the tracking URI went to a local folder; with it, the server has 6 runs

POC: prove it works

  • Five runs logged with parameters, metrics and models; models in Cloud Storage (Steps 4 and 5).
  • The best run registered as wine-classifier with the champion alias and loaded by that name (Steps 6 and 7).
  • A missing tracking URI sent a run to a local folder; setting it fixed it (Break it, Fix it).

Clean up (delete everything)

# Clean up: delete the VM (the server, its database and the local files go with it) and the artifact bucket
gcloud compute instances delete mlflow-vm --zone=us-east1-b --quiet 2>&1 | tail -1
gcloud storage rm -r gs://mlflow-$(echo -n $PROJECT_ID | md5sum | cut -c1-8) --quiet > /dev/null 2>&1; echo "left: $(gcloud compute instances list --filter='name=mlflow-vm' --format='value(name)' | wc -l) VMs, $(gcloud storage ls 2>/dev/null | grep -c mlflow-) buckets"
Deleted [https://www.googleapis.com/compute/v1/projects/your-project-id/zones/us-east1-b/instances/mlflow-vm].
left: 0 VMs, 0 buckets

FAQ

SQLite or a real database? SQLite is fine for one person. For a team, use PostgreSQL or MySQL (for example Cloud SQL) as the backend store, so several people can log at once.

How do I open the MLflow web page? From your laptop: gcloud compute ssh mlflow-vm --tunnel-through-iap -- -L 5000:localhost:5000, then open http://localhost:5000. We used the API in this lab.

MLflow or Vertex AI Experiments? Vertex AI Experiments is Google's managed tracking, integrated with Vertex AI training. MLflow runs anywhere and is the most widely used open-source option. Both answer "what did each run do?".

**Why trust sklearn.tree._tree.Tree?** Because we created the model ourselves. skops blocks types it cannot verify, so a model file from someone else cannot run harmful code when loaded. Only trust types you have reviewed.

What should I log? Every parameter you change, the data version (a tag or a hash), the metrics you decide with, and the model. If you cannot reproduce a run from its record, the record is incomplete.

Troubleshooting

You seeWhyFix
Runs are missing on the serverThe script used no tracking URI and logged locallySet MLFLOW_TRACKING_URI everywhere you train
The saved sklearn model references untrusted typesMLflow 3 uses skops; tree models need trustingPass skops_trusted_types=["sklearn.tree._tree.Tree"] to log_model
The server never answers /healthNot started, or out of memoryRead server.log; use a VM with enough memory
Training hangsThe tracking server is unreachable and MLflow keeps retryingCheck the server first; then train
Model files fail to uploadThe VM's account cannot write to the bucketGrant it Storage Object Admin on the bucket

What next

Prepared by the CareerByteCode Team from our tested lab, with AI help for the write-up.

Read next →MLOps Project for Your Resume on Google Cloud⏱ 12 min

⭐ Keep learning with CareerByteCode

Hand-picked to go deeper on this topic
News, guides and updates from the CareerByteCode team
🏅 Expert writer

The CareerByteCode team writes about what is new on careerbytecode.in: courses, labs, exams, events and the community, which is free to join for influencers and content creators in tech.

Get CareerByteCode Team's next blog by email

One email a week when they publish. Add the weekly Best of ByteBlog too. No spam, unsubscribe with one click.

We email you once to confirm. By subscribing you agree to the privacy policy.

💬 Comments (0)

Top · Newest

Sign in to comment

Related blogs

🧪⭐ Recommended by CareerByteCode · ByteLabs project bundleCloud and DevOps Realtime Starterโ‚น499Build the projects →