CareerByteCode
ByteBlogby CareerByteCode

Build a RAG Chatbot on Your Own PDFs with Vertex AI

What you build: a chatbot that answers from your own PDFs with sources, using Vertex AI embeddings and Gemini (no API key), deployed on Cloud Run, in your own Google Cloud account. Every step tested.

Build a RAG Chatbot on Your Own PDFs with Vertex AI

✍️ Translate this blog into Hindi, Tamil or Telugu

In this lab you build a RAG chatbot: a small Python app that answers questions from your own PDF files and shows which page each answer came from. It uses two Vertex AI models: an embedding model (text-embedding-005) to find the right pages, and Gemini (gemini-3.5-flash-lite) to write the answer from those pages only. Then you deploy it on Cloud Run with its own service account and sign-in required. There is no API key anywhere: everything signs in with your Google identity.

Who it is for: developers and cloud learners with basic Python who want to understand how "chat with your documents" really works, without a framework hiding the steps. Time: about 1.5 to 2 hours. Cost: a few paise. This whole lab made about 12 Gemini calls and 10 embedding calls (well under USD 0.10 at list prices) plus a few seconds of Cloud Run time, which the free trial credit covers. Delete everything at the end.

Every command below was run for real by the CareerByteCode Team on a fresh Google Cloud project, and every output and screenshot is the real result we got. We typed the commands in a Cloud Shell-like Linux VM (Debian with gcloud, Python 3.11, curl and jq); Cloud Shell in your browser has the same tools ready.

🎓⭐ Recommended by CareerByteCode · Live trainingAI Engineering and Agentic AI Programโ‚น35,400See the training →

Part A: RAG in plain English

A large language model (LLM) like Gemini knows a lot about the world, but it has never read your company's PDFs. Ask it about your leave policy and it can only guess, or say it does not know. RAG (retrieval-augmented generation) fixes that in two moves: first retrieve the few pieces of your documents that match the question, then generate an answer with those pieces pasted into the prompt.

How the RAG chatbot answers
PDFs become chunks, chunks become vectors, the question finds the closest chunks, Gemini answers from them with sources

Words you will use:

  • Chunk: a small piece of a document (here about 300 characters, one or two rules). Real-life example: cutting a long book into index cards, one idea per card.
  • Embedding: a list of numbers (a vector) that captures the meaning of a text. Texts with similar meaning get similar numbers, even with different words. text-embedding-005 gives 768 numbers per text.
  • Similarity: how close two vectors point, from 0 (unrelated) to 1 (same meaning). We use cosine similarity. Real-life example: two people pointing in almost the same direction on a map.
  • Index: all chunks with their vectors, saved in one file (index.json). Big systems use a vector database (Vertex AI Vector Search, AlloyDB, Firestore vector search); for a few hundred chunks a file is enough.
  • Grounding: telling the model to answer only from the given text and to say "I could not find that" otherwise. It is what makes answers checkable.
  • Service account: a Google identity for a program. The deployed app signs in to Vertex AI as rag-bot, which holds only the Vertex AI User role.

Before you start

  1. A Google Cloud account (the free trial works) and a project with billing linked.
  2. Open Cloud Shell (the >_ icon at the top right of the Google Cloud console).
  3. Set your project and turn on the APIs this lab uses. Replace the text in quotes with your own project id:
export PROJECT_ID="your-project-id"
gcloud config set project $PROJECT_ID
gcloud services enable aiplatform.googleapis.com run.googleapis.com cloudbuild.googleapis.com artifactregistry.googleapis.com

In Cloud Shell you are already signed in, so the Python SDK finds your login by itself (this is called Application Default Credentials). On your own laptop, run gcloud auth application-default login once instead. Make a folder for the lab and work inside it:

mkdir -p ~/rag-lab && cd ~/rag-lab
export GOOGLE_CLOUD_PROJECT=$PROJECT_ID

Part B: the lab

Step 1: Python and the Google Gen AI SDK

Why: a virtual environment keeps this lab's Python packages apart from everything else. google-genai is Google's SDK for Gemini and the embedding models; pypdf reads PDFs; fpdf2 makes our sample PDFs; numpy does the vector maths.

# a Python virtual environment with the Google Gen AI SDK (no API key: it uses your gcloud sign-in)
python3 -m venv .venv && source .venv/bin/activate
pip install -q google-genai==1.* pypdf==5.* fpdf2==2.* numpy==2.* flask==3.1.* 2>&1 | grep -v "^\[notice\]" | tail -2
python -c "import google.genai, pypdf, numpy; print('google-genai', google.genai.__version__, '| pypdf', pypdf.__version__, '| numpy', numpy.__version__)"
gcloud services list --enabled --filter="name:aiplatform.googleapis.com" --format="value(name)"
google-genai 1.75.0 | pypdf 5.9.0 | numpy 2.4.6
projects/PROJECT_NUMBER/services/aiplatform.googleapis.com

Step 2: three sample PDFs

Why: you need documents to chat with. We make three short policy PDFs for a made-up company, Nimbus Tech (leave, travel, IT security; 2 pages each). With your own PDFs, just copy them into the folder and skip this step. Save this as make_pdfs.py (the full file has three documents; the shape is the same for each):

from fpdf import FPDF
DOCS = {
    "leave-policy.pdf": [
        ("Nimbus Tech leave policy (2026)", [
            "Every full-time employee gets 18 days of paid annual leave per calendar year. ...",
            "Up to 6 unused days can be carried over to the next year. ...",
        ]),
        ("Asking for leave", ["For leave longer than 10 working days in a row, the department head must also approve.", "..."]),
    ],
    # "travel-policy.pdf": [...], "it-security.pdf": [...]
}
for name, pages in DOCS.items():
    pdf = FPDF()
    for heading, paras in pages:
        pdf.add_page(); pdf.set_font("Helvetica", "B", 16); pdf.multi_cell(0, 10, heading); pdf.ln(4)
        pdf.set_font("Helvetica", size=12)
        for p in paras:
            pdf.multi_cell(0, 7, p); pdf.ln(3)
    pdf.output(name)
    print("made", name, "with", len(pages), "pages")
# make 3 sample PDFs (a made-up company's HR, travel and IT rules), then read one back as text
source .venv/bin/activate
python make_pdfs.py
ls -l *.pdf | awk '{print $5, $9}'
python -c "from pypdf import PdfReader; print(PdfReader('travel-policy.pdf').pages[0].extract_text()[:260])"
made leave-policy.pdf with 2 pages
made travel-policy.pdf with 2 pages
made it-security.pdf with 2 pages
2031 it-security.pdf
2021 leave-policy.pdf
1975 travel-policy.pdf
Nimbus Tech travel policy
Book all work travel through the travel desk. Flights are economy class for trips shorter than 6 hours.
The daily food allowance is 1,200 rupees in metro cities and 800 rupees in other cities. No receipts
are needed for the food allow
Setup and PDFs
Steps 1 and 2: the SDK is ready and three PDFs are made; pypdf reads the text back

The last lines prove the text can be read back out of the PDF. Scanned PDFs (photos of paper) have no text layer; for those you would add OCR first, for example with Document AI.

Step 3: chunk the PDFs and embed every chunk

Why: this is the "retrieval" half. rag.py holds the shared code. Note the two clients: embeddings on us-central1 and Gemini on the global endpoint, both with vertexai=True, which means "sign in with Google, no key". Save it as rag.py:

# The RAG engine: embed text with a Vertex AI embedding model, find the closest chunks, ask Gemini with them.
# No API key: google-genai signs in with your gcloud login (or the Cloud Run service account).
import os
import numpy as np
from google import genai
from google.genai import types

PROJECT = os.environ["GOOGLE_CLOUD_PROJECT"]
EMBED_MODEL = os.environ.get("EMBED_MODEL", "text-embedding-005")
EMBED_DIM = int(os.environ.get("EMBED_DIM", "768"))
CHAT_MODEL = os.environ.get("CHAT_MODEL", "gemini-3.5-flash-lite")
embedder = genai.Client(vertexai=True, project=PROJECT, location="us-central1")
chat = genai.Client(vertexai=True, project=PROJECT, location="global")


def embed(texts, task):
    r = embedder.models.embed_content(model=EMBED_MODEL, contents=texts,
        config=types.EmbedContentConfig(task_type=task, output_dimensionality=EMBED_DIM))
    return [e.values for e in r.embeddings]


def search(index, question, k=3):
    q = np.array(embed([question], "RETRIEVAL_QUERY")[0])
    m = np.array([c["vec"] for c in index["chunks"]])
    scores = (m @ q) / (np.linalg.norm(m, axis=1) * np.linalg.norm(q))
    best = np.argsort(-scores)[:k]
    return [(index["chunks"][i], float(scores[i])) for i in best]


RULES = ("Answer the question using ONLY the context below. If the context does not contain the answer, reply exactly: "
         "I could not find that in the documents. Keep it to 2 sentences and end with the sources in square brackets, "
         "like [leave-policy.pdf p1]. Use plain punctuation, no long dashes.")


def answer(index, question):
    hits = search(index, question)
    context = "\n\n".join(f"[{c['source']} p{c['page']}] {c['text']}" for c, _ in hits)
    r = chat.models.generate_content(model=CHAT_MODEL,
        contents=f"Context:\n{context}\n\nQuestion: {question}",
        config=types.GenerateContentConfig(system_instruction=RULES, temperature=0.1, max_output_tokens=200))
    u = r.usage_metadata
    return {"answer": r.text.strip(), "sources": [f"{c['source']} p{c['page']} ({s:.2f})" for c, s in hits],
            "tokens": {"in": u.prompt_token_count, "out": u.candidates_token_count}}

ingest.py reads each page, cuts it into chunks of about 300 characters at sentence ends, embeds all chunks in one call with task_type="RETRIEVAL_DOCUMENT", and saves index.json:

# Reads every PDF in this folder, splits the text into chunks, embeds them and saves index.json.
import glob, json, sys
from pypdf import PdfReader
import rag

out = sys.argv[1] if len(sys.argv) > 1 else "index.json"
chunks = []
for path in sorted(glob.glob("*.pdf")):
    for n, page in enumerate(PdfReader(path).pages, start=1):
        text = " ".join(page.extract_text().split())
        # chunks of about 300 characters, cut at sentence ends, so one chunk = one or two rules
        part = ""
        for sentence in text.replace(". ", ".\n").split("\n"):
            if len(part) + len(sentence) > 300 and part:
                chunks.append({"source": path, "page": n, "text": part.strip()})
                part = ""
            part += sentence + " "
        if part.strip():
            chunks.append({"source": path, "page": n, "text": part.strip()})
vectors = rag.embed([c["text"] for c in chunks], "RETRIEVAL_DOCUMENT")
for c, v in zip(chunks, vectors):
    c["vec"] = v
json.dump({"model": rag.EMBED_MODEL, "dim": rag.EMBED_DIM, "chunks": chunks}, open(out, "w"))
print(f"{len(chunks)} chunks from {len(set(c['source'] for c in chunks))} PDFs, {len(vectors[0])} numbers per vector, saved {out}")
print("first chunk:", chunks[0]["source"], "p" + str(chunks[0]["page"]), "->", chunks[0]["text"][:90], "...")
print("its vector starts:", [round(x, 4) for x in vectors[0][:5]])
# split the PDFs into chunks and turn every chunk into a vector with text-embedding-005 (one API call for all chunks)
source .venv/bin/activate
export GOOGLE_CLOUD_PROJECT=$PROJECT_ID
python ingest.py
ls -l index.json | awk '{print "index.json", $5, "bytes"}'
12 chunks from 3 PDFs, 768 numbers per vector, saved index.json
first chunk: it-security.pdf p1 -> Nimbus Tech IT security rules Lock your laptop screen whenever you leave your desk. Laptop ...
its vector starts: [-0.0418, -0.0119, -0.0391, 0.0153, -0.0335]
index.json 206845 bytes
Embeddings
12 chunks from 3 PDFs, 768 numbers per vector, saved in a 207 KB index file

Twelve chunks, each now a list of 768 numbers. You never read these numbers yourself; what matters is that chunks about "leave" sit close together and far from chunks about "passwords".

Step 4: ask questions and get answers with sources

Why: this is the "generation" half. For each question, search() embeds the question (with task_type="RETRIEVAL_QUERY"), scores it against all 12 chunks and keeps the top 3. answer() pastes those 3 chunks into the prompt with the rules: answer only from the context, 2 sentences, cite the source. Save ask.py:

# python ask.py "question" [index file] : prints the answer, the sources (with similarity) and the tokens used
import json, sys
import rag

index = json.load(open(sys.argv[2] if len(sys.argv) > 2 else "index.json"))
r = rag.answer(index, sys.argv[1])
print("Q:", sys.argv[1])
print("A:", r["answer"])
print("   sources:", "; ".join(r["sources"]), "| tokens", r["tokens"])
# ask questions: the closest 3 chunks go to Gemini with the question; the answer cites its sources
source .venv/bin/activate
export GOOGLE_CLOUD_PROJECT=$PROJECT_ID
python ask.py "How many days of paid leave do I get, and how many can I carry over?"
python ask.py "What is the food allowance in Chennai?"
python ask.py "I lost my laptop. What should I do?"
Q: How many days of paid leave do I get, and how many can I carry over?
A: You get 18 days of paid annual leave per calendar year, and up to 6 unused days can be carried over to the next year [leave-policy.pdf p1]. Any days above 6 are lost on 31 December [leave-policy.pdf p1].
   sources: leave-policy.pdf p1 (0.66); leave-policy.pdf p2 (0.66); leave-policy.pdf p2 (0.63) | tokens {'in': 263, 'out': 59}
Q: What is the food allowance in Chennai?
A: The daily food allowance is 1,200 rupees in metro cities and 800 rupees in other cities. No receipts are needed for this allowance. [travel-policy.pdf p1]
   sources: travel-policy.pdf p1 (0.64); travel-policy.pdf p1 (0.62); travel-policy.pdf p2 (0.58) | tokens {'in': 240, 'out': 42}
Q: I lost my laptop. What should I do?
A: Report a lost or stolen laptop by calling the IT helpdesk on extension 4040 within 1 hour [it-security.pdf p2].
   sources: it-security.pdf p2 (0.61); it-security.pdf p1 (0.50); it-security.pdf p2 (0.48) | tokens {'in': 222, 'out': 32}
Answers with sources
Correct answers from the PDFs, each with its page and the similarity of the top 3 chunks

Look closely at the Chennai answer: Gemini gives both numbers (metro and other cities) instead of picking one. The PDF says "metro cities" but never says which cities are metros, so the model cannot know Chennai's rate from the documents. Grounding stopped it from guessing; the real fix is in the document (add the list of metro cities), not in the prompt. (In an earlier run of the same lab, Gemini even said in words that the documents do not name Chennai: answers can vary a little from run to run.) Each question used about 220 to 260 input tokens and 25 to 60 output tokens, because only 3 small chunks are sent, not the whole PDFs.

Step 5: a question the PDFs do not answer

Why: a good RAG app must say "I don't know" instead of inventing. We also ask Gemini the leave question without RAG, to see the difference.

# a question the PDFs do not answer, then the same company question to Gemini WITHOUT the documents
source .venv/bin/activate
export GOOGLE_CLOUD_PROJECT=$PROJECT_ID
python ask.py "Does Nimbus Tech give free lunch in the office?"
python - <<'PY'
import rag
r = rag.chat.models.generate_content(model=rag.CHAT_MODEL, contents="How many days of paid annual leave do Nimbus Tech employees get? Answer in 1 sentence.")
print("Without RAG:", r.text.strip())
PY
Q: Does Nimbus Tech give free lunch in the office?
A: I could not find that in the documents. 
[travel-policy.pdf p1, leave-policy.pdf p1, it-security.pdf p1]
   sources: travel-policy.pdf p1 (0.75); leave-policy.pdf p1 (0.71); it-security.pdf p1 (0.62) | tokens {'in': 296, 'out': 36}
Without RAG: I am sorry, but I do not have access to specific information regarding the employee benefits or paid annual leave policies of Nimbus Tech.
Guard rails
Not in the PDFs: the bot says so. Without RAG, Gemini cannot know company rules

The bot refused to guess about free lunch (it still listed the nearest pages as sources; you could hide sources when the answer is "not found"). Without the documents, Gemini honestly says it has no access to Nimbus Tech's policies. With RAG, the same question got "18 days, 6 can be carried over" with the page number.

Break it and fix it: vectors of different sizes

A teammate sets EMBED_DIM=256 "to save space", but the index was built with 768 numbers per vector. Now the question vector and the chunk vectors do not match.

# Break it: a teammate sets EMBED_DIM=256 "to save space", but the index was built with 768 numbers per vector
source .venv/bin/activate
export GOOGLE_CLOUD_PROJECT=$PROJECT_ID
EMBED_DIM=256 python ask.py "What is the hotel limit in a metro city?" 2>&1 | tail -3
    scores = (m @ q) / (np.linalg.norm(m, axis=1) * np.linalg.norm(q))
              ~~^~~
ValueError: matmul: Input operand 1 has a mismatch in its core dimension 0, with gufunc signature (n?,k),(k,m?)->(n?,m?) (size 256 is different from 768)
Break it
numpy refuses: size 256 is different from 768

The maths simply cannot compare a 256-number vector with 768-number vectors, so numpy stops with size 256 is different from 768. A worse version of this bug gives no error: if someone switches to a different embedding model with the same size, the numbers mean different things and the search quietly returns wrong chunks. Fix: questions and chunks must use the same model and the same size. That is why ingest.py saves both in the index:

# Fix it: questions and chunks must use the SAME embedding model and size. Check what the index was built with, then ask again
source .venv/bin/activate
export GOOGLE_CLOUD_PROJECT=$PROJECT_ID
python -c "import json; i = json.load(open('index.json')); print('index built with', i['model'], 'dim', i['dim'])"
EMBED_DIM=768 python ask.py "What is the hotel limit in a metro city?"
index built with text-embedding-005 dim 768
Q: What is the hotel limit in a metro city?
A: The hotel limit per night is 6,000 rupees in metro cities [travel-policy.pdf p1].
   sources: travel-policy.pdf p1 (0.72); travel-policy.pdf p2 (0.54); travel-policy.pdf p1 (0.51) | tokens {'in': 215, 'out': 25}
🎓⭐ Recommended by CareerByteCode · Live trainingLearn AI, Build 12+ AI Agentsโ‚น10,620See the training →
Fix it
The index says text-embedding-005, 768: use the same for questions and the answer comes back

Rule: if you change the embedding model or size, rebuild the whole index (run ingest.py again).

Step 6: deploy the chatbot to Cloud Run

Why: so other people and apps can use it. app.py is a small Flask web app with GET /ask?q=.... It runs as a new service account rag-bot that may only call Vertex AI (least privilege), and --no-allow-unauthenticated means callers must sign in. Save app.py, requirements.txt and a one-line Procfile (it tells Cloud Run how to start the app):

# The chatbot as a small web app for Cloud Run: GET /ask?q=your+question
import json
from flask import Flask, request, jsonify
import rag

app = Flask(__name__)
INDEX = json.load(open("index.json"))


@app.get("/")
def home():
    return f"RAG chatbot is running with {len(INDEX['chunks'])} chunks. Try /ask?q=your+question\n"


@app.get("/ask")
def ask():
    q = request.args.get("q", "").strip()
    if not q:
        return jsonify(error="add ?q=your question"), 400
    return jsonify(question=q, **rag.answer(INDEX, q))
# requirements.txt
flask==3.1.*
gunicorn==23.*
google-genai==1.*
numpy==2.*

# Procfile
web: gunicorn --bind :$PORT --workers 1 --threads 4 app:app

gcloud run deploy --source . sends the folder to Cloud Build, which builds a container image for you (no Dockerfile needed), stores it in Artifact Registry and starts it on Cloud Run. The first deploy asks to create the cloud-run-source-deploy repository; --quiet says yes.

# deploy the chatbot to Cloud Run with its own service account (Vertex AI User only), sign-in required
gcloud iam service-accounts create rag-bot --display-name="RAG chatbot on Cloud Run" 2>&1 | tail -1
until gcloud iam service-accounts describe rag-bot@$PROJECT_ID.iam.gserviceaccount.com > /dev/null 2>&1; do sleep 5; done
gcloud projects add-iam-policy-binding $PROJECT_ID --member="serviceAccount:rag-bot@$PROJECT_ID.iam.gserviceaccount.com" \
  --role=roles/aiplatform.user --condition=None --format="value(etag)" > /dev/null && echo "granted roles/aiplatform.user to rag-bot"
mkdir -p app && cp app.py rag.py index.json requirements.txt Procfile app/ && cd app
gcloud run deploy rag-bot --source . --region=us-east1 --no-allow-unauthenticated \
  --service-account=rag-bot@$PROJECT_ID.iam.gserviceaccount.com \
  --set-env-vars=GOOGLE_CLOUD_PROJECT=$PROJECT_ID --memory=512Mi --max-instances=1 --quiet 2>&1 | tail -4
Service account email: rag-bot@your-project-id.iam.gserviceaccount.com
Updated IAM policy for project [your-project-id].
granted roles/aiplatform.user to rag-bot
Done.
Service [rag-bot] revision [rag-bot-00001-hrk] has been deployed and is serving 100 percent of traffic.
Service URL: https://rag-bot-PROJECT_NUMBER.us-east1.run.app
Proxy locally with: gcloud run services proxy rag-bot --region us-east1 --project your-project-id

Step 7: call the deployed chatbot

Why: prove that strangers are refused and signed-in callers get answers. gcloud auth print-identity-token makes a short-lived token that says who you are.

# call the deployed chatbot: first without a token (refused), then with your identity token
URL=$(gcloud run services describe rag-bot --region=us-east1 --format="value(status.url)")
curl -s -o /dev/null -w "no token: HTTP %{http_code}\n" "$URL/ask?q=hello"
TOKEN=$(gcloud auth print-identity-token --audiences=$URL)
curl -s -H "Authorization: Bearer $TOKEN" "$URL/"
curl -s -G -H "Authorization: Bearer $TOKEN" "$URL/ask" --data-urlencode "q=Who must approve 12 working days of leave?" | jq .
no token: HTTP 403
RAG chatbot is running with 12 chunks. Try /ask?q=your+question
{
  "answer": "Leave longer than 10 working days in a row must be approved by both your manager and the department head [leave-policy.pdf p2].",
  "question": "Who must approve 12 working days of leave?",
  "sources": [
    "leave-policy.pdf p2 (0.68)",
    "leave-policy.pdf p2 (0.61)",
    "leave-policy.pdf p1 (0.59)"
  ],
  "tokens": {
    "in": 256,
    "out": 31
  }
}
Cloud Run
Deployed in about 75 seconds; no token gives 403; with a token the bot answers with sources

12 working days is "longer than 10 working days in a row", so both the manager and the department head approve. The bot combined two rules from the same page correctly.

POC: prove it works

  • Three PDFs became 12 chunks and a 768-number vector each, in one embedding call.
  • Questions were answered from the right page, with sources, in about 250 input tokens each.
  • A question not in the PDFs got "I could not find that in the documents", not a made-up answer.
  • A vector size mismatch broke the search; using the same model and size fixed it.
  • The chatbot runs on Cloud Run as a least-privilege service account, sign-in required, with no API key anywhere.

Clean up (delete everything)

Why: Cloud Run costs nothing while idle, but the image repository and the upload bucket keep small storage charges, and an unused service account is a security risk.

# delete the service, its image repository and source bucket, the service account and its role
gcloud run services delete rag-bot --region=us-east1 --quiet 2>&1 | tail -1
gcloud artifacts repositories delete cloud-run-source-deploy --location=us-east1 --quiet 2>&1 | tail -1
for b in $(gcloud storage buckets list --format="value(name)" | grep -E "run-sources|cloudbuild"); do gcloud storage rm -r gs://$b --quiet > /dev/null 2>&1 && echo "deleted bucket (source upload)"; done
gcloud projects remove-iam-policy-binding $PROJECT_ID --member="serviceAccount:rag-bot@$PROJECT_ID.iam.gserviceaccount.com" \
  --role=roles/aiplatform.user --condition=None --format="value(etag)" > /dev/null && echo "removed role"
gcloud iam service-accounts delete rag-bot@$PROJECT_ID.iam.gserviceaccount.com --quiet 2>&1 | tail -1
rm -rf app .venv index.json *.pdf
echo "Cloud Run services left: $(gcloud run services list --format="value(name)" | wc -l)"
echo "repositories left: $(gcloud artifacts repositories list --format="value(name)" 2>/dev/null | wc -l)"
echo "buckets left: $(gcloud storage buckets list --format="value(name)" | wc -l)"
Deleted service [rag-bot].
Deleted repository [cloud-run-source-deploy].
deleted bucket (source upload)
Updated IAM policy for project [your-project-id].
removed role
deleted service account [rag-bot@your-project-id.iam.gserviceaccount.com]
Cloud Run services left: 0
repositories left: 0
buckets left: 0
Clean up
0 Cloud Run services, 0 repositories, 0 buckets left

If you made a project only for this lab, you can also shut the whole project down: gcloud projects delete $PROJECT_ID.

FAQ

Why not just paste all my PDFs into the prompt? For 3 short files you could. For hundreds of pages it gets slow and costly (you pay for every input token on every question), and models answer worse when the right sentence is buried in a long prompt. RAG sends only the 3 best chunks.

Which chunk size is best? There is no single answer. Start with 300 to 1,000 characters with sentence ends, test real questions, and look at the sources: if the right rule is cut in half, make chunks bigger or overlap them.

Do I need a vector database? Not for a few thousand chunks; a file or a numpy array is fine. For large or changing document sets, use Vertex AI Vector Search, AlloyDB or Cloud SQL with pgvector, or Firestore vector search.

Is my data used to train Google's models? Google Cloud's terms for Vertex AI say customer data is not used to train its models without permission. Check the current terms for your use.

**Why gemini-3.5-flash-lite?** It is the smallest, cheapest current Gemini model and is good at "answer from this text" tasks. Use a bigger model when questions need more reasoning.

Can I use Vertex AI RAG Engine or Vertex AI Search instead? Yes, they do the chunking, embedding and search for you. Building it by hand once, as here, shows what those services do inside.

Troubleshooting

You seeWhyFix
DefaultCredentialsErrorThe SDK cannot find your loginIn Cloud Shell it is automatic; elsewhere run gcloud auth application-default login
403 PERMISSION_DENIED on Vertex AIAPI off or the account lacks the rolegcloud services enable aiplatform.googleapis.com; give the account roles/aiplatform.user
404 ... model not foundModel name or location wrongtext-embedding-005 on us-central1, Gemini on global, as in rag.py
size 256 is different from 768Question and chunks embedded with different sizes or modelsUse the same model and size; rebuild the index after any change
Answer says "could not find" but the PDF has itThe right chunk was not in the top 3Check sources; try bigger chunks or k=5
Cloud Run 403 ForbiddenNo identity token, or no Invoker roleSend Authorization: Bearer $(gcloud auth print-identity-token); grant roles/run.invoker to the caller

What next

Prepared by the CareerByteCode Team from our tested lab, with AI help for the write-up.

Read next →Vector Search with AlloyDB pgvector and Gemini⏱ 12 min

⭐ Keep learning with CareerByteCode

Hand-picked to go deeper on this topic
News, guides and updates from the CareerByteCode team
🏅 Expert writer

The CareerByteCode team writes about what is new on careerbytecode.in: courses, labs, exams, events and the community, which is free to join for influencers and content creators in tech.

Get CareerByteCode Team's next blog by email

One email a week when they publish. Add the weekly Best of ByteBlog too. No spam, unsubscribe with one click.

We email you once to confirm. By subscribing you agree to the privacy policy.

💬 Comments (0)

Top · Newest

Sign in to comment

Related blogs

🎓⭐ Recommended by CareerByteCode · Live trainingAI Engineering and Agentic AI Programโ‚น35,400See the training →