CareerByteCode
ByteBlogby CareerByteCode

OPA Gatekeeper and Kyverno: Policy as Code for Kubernetes

What you build: the same no-latest rule in OPA Gatekeeper (Rego) and Kyverno (YAML) on GKE: block, audit, mutate and compare. Every step tested.

OPA Gatekeeper and Kyverno: Policy as Code for Kubernetes

✍️ Translate this blog into Hindi, Tamil or Telugu

Anyone with access to your cluster can deploy an image tagged latest, a pod without limits, or a container that runs as root. Code review does not stop that: the cluster has to. In this lab you write the same rule ("no latest images") in the two most used policy tools for Kubernetes, OPA Gatekeeper and Kyverno, test both on a real GKE cluster, and compare how they block, audit, report and cost.

Who it is for: Kubernetes users and DevSecOps beginners who know pods and namespaces. Time: about 1 hour. Cost: a small GKE cluster for about 15 minutes, covered by the free trial credit. Delete it at the end.

Every command below was run for real by the CareerByteCode Team, and every output is the real result we got. We typed the commands in a Cloud Shell-like Linux VM (Debian with gcloud, kubectl and helm); Cloud Shell in your browser has the same tools.

🧪⭐ Recommended by CareerByteCode · ByteLabs project bundleCloud and DevOps Realtime Starterโ‚น499Build the projects →

Part A: policy as code in plain English

When you run kubectl apply, your request goes to the Kubernetes API server. Before it saves anything, the API server can ask admission webhooks: small services that answer "allow", "deny" or "allow, but change this first". A policy engine is such a webhook, plus a way to write the rules as files you keep in Git.

How an admission policy works
kubectl sends the pod to the API server, which asks the policy engine before saving it

Words you will use:

  • Admission controller / webhook: the checkpoint every new object passes. Real-life example: the security desk that checks your badge before you enter the building.
  • Validate: deny an object that breaks a rule. Mutate: change an object on the way in (for example, add a missing label).
  • Audit / background scan: check objects that already exist. A webhook only sees new requests, so a pod created before the rule passes unseen until the audit finds it.
  • OPA Gatekeeper: the Kubernetes version of Open Policy Agent. Rules are written in Rego, a small policy language. A ConstraintTemplate holds the Rego code; a Constraint says where to apply it.
  • Kyverno: a policy engine made only for Kubernetes. Rules are plain Kubernetes YAML (ClusterPolicy), with patterns such as "*:* & !*:latest".
  • **Why latest is bad**: nginx:latest means "whatever is newest today". Two nodes can run two different versions, and a rollback cannot go back to a known version.

Before you start

  1. A Google Cloud project with billing linked (the free trial works) and Cloud Shell open.
  2. One small GKE cluster. We created it once for this batch of Kubernetes labs:
gcloud services enable container.googleapis.com
gcloud container clusters create lab-cluster \
  --zone=us-east1-b --num-nodes=2 --machine-type=e2-standard-4 \
  --disk-type=pd-standard --disk-size=50 --release-channel=regular \
  --no-enable-autoupgrade --enable-dataplane-v2

Part B: the lab

Step 1: a namespace with an "old" pod

Why: in real clusters, bad objects already exist before you add a policy. We create a namespace policy-demo and one pod with nginx:latest before any rule, to see whether each tool finds it later.

# Connect to the cluster and make the namespace the policies will guard. One pod with :latest exists BEFORE any policy
gcloud container clusters get-credentials lab-cluster --zone=us-east1-b > /dev/null 2>&1
kubectl create namespace policy-demo > /dev/null && kubectl -n policy-demo run old-pod --image=nginx:latest > /dev/null && echo "old-pod (nginx:latest) created before any policy"
old-pod (nginx:latest) created before any policy

Step 2: install Gatekeeper and write the rule in Rego

Why: Gatekeeper splits a rule in two files. The ConstraintTemplate holds the Rego code: each violation block is one way to break the rule (no : in the image means no tag; ending in :latest means the latest tag). Save it as gk-template.yaml:

# Gatekeeper ConstraintTemplate: refuse containers whose image has no tag or the tag "latest" (policy written in Rego)
apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  name: k8snolatest
spec:
  crd:
    spec:
      names:
        kind: K8sNoLatest
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8snolatest
        violation[{"msg": msg}] {
          c := input.review.object.spec.containers[_]
          not contains(c.image, ":")
          msg := sprintf("image %v has no tag; use a fixed version", [c.image])
        }
        violation[{"msg": msg}] {
          c := input.review.object.spec.containers[_]
          endswith(c.image, ":latest")
          msg := sprintf("image %v uses the latest tag; use a fixed version", [c.image])
        }

The Constraint turns the template on, only for pods in policy-demo (so the lab cannot block anything else in the cluster). Save it as gk-constraint.yaml:

# Gatekeeper Constraint: apply the K8sNoLatest rule to pods in namespace policy-demo
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sNoLatest
metadata:
  name: no-latest-in-policy-demo
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
    namespaces: ["policy-demo"]

We install Gatekeeper v3.18.2 from its official file, then apply both:

# Install OPA Gatekeeper (an admission controller), then a ConstraintTemplate (Rego) and a Constraint
kubectl apply -f https://raw.githubusercontent.com/open-policy-agent/gatekeeper/v3.18.2/deploy/gatekeeper.yaml > /dev/null && kubectl -n gatekeeper-system rollout status deploy/gatekeeper-controller-manager --timeout=180s
kubectl apply -f gk-template.yaml && sleep 15 && kubectl apply -f gk-constraint.yaml && sleep 15
kubectl get constrainttemplates,k8snolatest --no-headers | awk '{print $1}'
Waiting for deployment "gatekeeper-controller-manager" rollout to finish: 0 out of 3 new replicas have been updated...
Waiting for deployment "gatekeeper-controller-manager" rollout to finish: 0 of 3 updated replicas are available...
Waiting for deployment "gatekeeper-controller-manager" rollout to finish: 1 of 3 updated replicas are available...
Waiting for deployment "gatekeeper-controller-manager" rollout to finish: 2 of 3 updated replicas are available...
deployment "gatekeeper-controller-manager" successfully rolled out
constrainttemplate.templates.gatekeeper.sh/k8snolatest created
k8snolatest.constraints.gatekeeper.sh/no-latest-in-policy-demo created
constrainttemplate.templates.gatekeeper.sh/k8snolatest
k8snolatest.constraints.gatekeeper.sh/no-latest-in-policy-demo
Gatekeeper installed
Steps 1 and 2: the old pod, Gatekeeper v3.18.2, the template and the constraint

Break it: deploy images without a fixed version

We try three pods: nginx:latest, nginx with no tag, and nginx:1.27.

# Test Gatekeeper: latest refused, no tag refused, a fixed version allowed
kubectl -n policy-demo run bad-latest --image=nginx:latest 2>&1 | cut -c1-200
kubectl -n policy-demo run bad-notag --image=nginx 2>&1 | cut -c1-200
kubectl -n policy-demo run good --image=nginx:1.27 && kubectl -n policy-demo delete pod good --wait=false > /dev/null
Error from server (Forbidden): admission webhook "validation.gatekeeper.sh" denied the request: [no-latest-in-policy-demo] image nginx:latest uses the latest tag; use a fixed version
Error from server (Forbidden): admission webhook "validation.gatekeeper.sh" denied the request: [no-latest-in-policy-demo] image nginx has no tag; use a fixed version
pod/good created

Gatekeeper refused both bad pods with our own message, and allowed the pinned one. The fix for a developer is simple: use a fixed version tag such as nginx:1.27.

Step 3: Gatekeeper's audit finds the old pod

Why: the webhook never saw old-pod, because it existed before the rule. Gatekeeper's audit checks existing objects every 60 seconds and writes what it finds into the Constraint's status:

# Gatekeeper's audit finds what was already running before the rule (old-pod)
sleep 60; kubectl get k8snolatest no-latest-in-policy-demo -o jsonpath='{"violations found by audit: "}{.status.totalViolations}{"\n"}{range .status.violations[*]}{.name}{": "}{.message}{"\n"}{end}'
violations found by audit: 1
old-pod: image nginx:latest uses the latest tag; use a fixed version
Gatekeeper blocks and audits
Break it: latest and no tag refused, the pinned pod allowed; the audit finds old-pod

Step 4: what Gatekeeper costs, then remove it

Why: a policy engine runs in your cluster and uses CPU and memory. We measure it with kubectl top, then remove Gatekeeper completely so Kyverno can be tested alone.

# What Gatekeeper costs to run, then remove it before installing Kyverno
sleep 20; kubectl top pods -n gatekeeper-system --no-headers | awk '{c+=$2; m+=$3; n++} END {print "gatekeeper:", n, "pods, cpu", c "m, memory", m "Mi"}'
kubectl delete -f gk-constraint.yaml > /dev/null; kubectl delete -f gk-template.yaml > /dev/null; kubectl delete -f https://raw.githubusercontent.com/open-policy-agent/gatekeeper/v3.18.2/deploy/gatekeeper.yaml > /dev/null 2>&1; echo "gatekeeper removed"
gatekeeper: 4 pods, cpu 130m, memory 256Mi
gatekeeper removed

Step 5: install Kyverno and write the same rule in YAML

Why: Kyverno rules are normal Kubernetes YAML. The validate rule uses a pattern: the image must match *:* (it has a tag) and must not match *:latest. The second policy is a mutate rule: +(team): unassigned means "add the label team: unassigned only if the pod has no team label". Save it as kyverno-policies.yaml:

# Kyverno: the same rule as plain YAML (validate), plus a mutate rule that adds a missing label
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: no-latest-tag
spec:
  validationFailureAction: Enforce
  background: true
  rules:
    - name: require-fixed-tag
      match:
        any:
          - resources:
              kinds: ["Pod"]
              namespaces: ["policy-demo"]
      validate:
        message: "image must use a fixed version tag (not latest, not empty)"
        pattern:
          spec:
            containers:
              - image: "*:* & !*:latest"
---
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: add-team-label
spec:
  rules:
    - name: default-team
      match:
        any:
          - resources:
              kinds: ["Pod"]
              namespaces: ["policy-demo"]
      mutate:
        patchStrategicMerge:
          metadata:
            labels:
              +(team): unassigned

We install Kyverno with its Helm chart (version 3.3.4), then apply the policies:

# Install Kyverno with Helm, then the same rule as plain YAML, and a mutate rule
helm repo add kyverno https://kyverno.github.io/kyverno/ > /dev/null 2>&1 && helm repo update kyverno > /dev/null 2>&1
helm install kyverno kyverno/kyverno -n kyverno --create-namespace --version 3.3.4 --wait --timeout 6m 2>&1 | grep -E "STATUS|Chart version"
kubectl apply -f kyverno-policies.yaml && sleep 20 && kubectl get clusterpolicy
STATUS: deployed
Chart version: 3.3.4
clusterpolicy.kyverno.io/no-latest-tag created
clusterpolicy.kyverno.io/add-team-label created
NAME             ADMISSION   BACKGROUND   READY   AGE   MESSAGE
add-team-label   true        true         True    20s   Ready
no-latest-tag    true        true         True    20s   Ready

Step 6: test Kyverno with the same three pods

# Test Kyverno: the same three pods, and the mutate rule adds a label nobody typed
kubectl -n policy-demo run bad-latest --image=nginx:latest 2>&1 | grep -E "denied|blocked|require-fixed-tag" | head -2 | cut -c1-200
kubectl -n policy-demo run bad-notag --image=nginx 2>&1 | grep -E "denied|blocked|require-fixed-tag" | head -1 | cut -c1-200
kubectl -n policy-demo run good --image=nginx:1.27 && kubectl -n policy-demo get pod good -o jsonpath='{"labels: "}{.metadata.labels}{"\n"}'
Error from server: admission webhook "validate.kyverno.svc-fail" denied the request: 
resource Pod/policy-demo/bad-latest was blocked due to the following policies 
Error from server: admission webhook "validate.kyverno.svc-fail" denied the request: 
pod/good created
labels: {"run":"good","team":"unassigned","topology.kubernetes.io/region":"us-east1","topology.kubernetes.io/zone":"us-east1-b"}
Kyverno blocks and mutates
Kyverno 3.3.4: both bad pods refused, the good pod gets team: unassigned

Both bad pods were refused by the webhook validate.kyverno.svc-fail (Kyverno prints the policy name and our message on the following lines; our filter kept only the first lines). The pinned pod was allowed, and Kyverno added team: unassigned, a label nobody typed. That is the mutate rule at work. Gatekeeper can also mutate, but with separate objects (Assign, AssignMetadata).

📝⭐ Recommended by CareerByteCode · Certification examCIPS Level 5 Advanced Diploma in Procurement and Supply PracticeTake the examTake the exam →

Step 7: Kyverno's reports find the old pod

Why: Kyverno's background scan writes a PolicyReport per object: how many rules it passed and failed. You can read them with kubectl, and tools such as Policy Reporter can show them in a web page.

# Kyverno's background scan writes policy reports, including old-pod
sleep 60; kubectl -n policy-demo get policyreport -o jsonpath='{range .items[*]}{.scope.name}{": pass="}{.summary.pass}{" fail="}{.summary.fail}{"\n"}{end}'
kubectl top pods -n kyverno --no-headers | awk '{c+=$2; m+=$3; n++} END {print "kyverno:", n, "pods, cpu", c "m, memory", m "Mi"}'
good: pass=2 fail=0
old-pod: pass=0 fail=1
kyverno: 4 pods, cpu 24m, memory 166Mi

old-pod fails one rule (its latest tag), and the new good pod passes both.

Step 8: the same rule, side by side

# Policy as code, side by side: how long is the same rule in each tool?
echo "Gatekeeper (Rego, template + constraint): $(grep -cvE '^\s*(#|$)' gk-template.yaml) + $(grep -cvE '^\s*(#|$)' gk-constraint.yaml) lines"
echo "Kyverno (YAML, validate rule only): $(sed -n '1,/^---/p' kyverno-policies.yaml | grep -cvE '^\s*(#|$|---)') lines"
Gatekeeper (Rego, template + constraint): 23 + 10 lines
Kyverno (YAML, validate rule only): 20 lines
Reports and footprint
PolicyReports, the footprint of each tool and the line count of the same rule

POC: prove it works

  • Both tools refused nginx:latest and nginx (no tag) and allowed nginx:1.27 (Break it, Step 6).
  • Both found the pod that existed before the rule: Gatekeeper's audit reported 1 violation and Kyverno's report showed fail=1 for old-pod (Steps 3 and 7).
  • Kyverno's mutate rule added team: unassigned to a pod without it (Step 6).

Which one should you choose?

OPA GatekeeperKyverno
Rule languageRego (a policy language)Kubernetes YAML with patterns
Our rule23 + 10 lines (template + constraint)20 lines
Message to the userFully custom (sprintf in Rego)Custom message field
Existing objectsAudit, results in the Constraint statusBackground scan, PolicyReport objects
Change objects (mutate)Yes, separate Assign objectsYes, in the same policy
Footprint in our run4 pods, about 130m CPU, 256 MiB4 pods, about 24m CPU, 166 MiB
Same rules outside KubernetesYes, OPA is used for APIs, Terraform and moreKubernetes only (plus its CLI)

A simple way to decide: if your team already writes Rego or wants one policy language for many systems, choose Gatekeeper. If you want rules that look like the YAML your team already writes, choose Kyverno. The footprint numbers are from one short run with a single rule; they grow with the number of objects and rules.

Clean up (delete everything)

Why: webhooks that stay behind can block deployments later. We remove Kyverno, its policies, and the namespaces, and make sure Gatekeeper is gone too.

# Clean up: Kyverno, its policies and the test namespace
kubectl delete -f kyverno-policies.yaml --ignore-not-found > /dev/null 2>&1; helm uninstall kyverno -n kyverno 2>&1 | tail -1
kubectl delete namespace kyverno policy-demo --ignore-not-found --wait=true 2>&1 | tail -2
kubectl delete -f https://raw.githubusercontent.com/open-policy-agent/gatekeeper/v3.18.2/deploy/gatekeeper.yaml --ignore-not-found > /dev/null 2>&1; echo "namespaces left: $(kubectl get ns --no-headers | grep -cE '^(kyverno|policy-demo|gatekeeper-system) ')"
namespace "kyverno" deleted
namespace "policy-demo" deleted
namespaces left: 0
Clean up
Clean up: 0 lab namespaces left

When you have finished all your Kubernetes labs, delete the cluster:

gcloud container clusters delete lab-cluster --zone=us-east1-b --quiet

FAQ

Can a policy engine break my cluster? Yes, if it blocks system pods or its webhook is down while set to "fail closed". Start with one namespace (as we did), use audit or Audit mode first, and exclude kube-system.

Audit mode or enforce mode? Start in audit (Gatekeeper enforcementAction: dryrun, Kyverno validationFailureAction: Audit), read the reports for a week, fix the findings, then switch to enforce.

Does GKE have its own policy tool? Yes, Policy Controller (part of GKE Enterprise) is built on Gatekeeper. We used the open-source projects so the lab works on any cluster.

Do I need to learn Rego? Only for Gatekeeper. The Gatekeeper library has many ready-made templates, so you often only write Constraints.

Can I test policies before the cluster? Yes: kyverno apply and gator (for Gatekeeper) test policies against YAML files in a CI pipeline. We did not test these here.

Troubleshooting

You seeWhyFix
The Constraint is not found after applying the templateGatekeeper needs a few seconds to create the new kindWait 10 to 15 seconds, then apply the Constraint
An old pod is not reportedThe audit runs every 60 secondsWait a minute and read the status again
All pods in the cluster are blockedThe rule matches every namespace, or the webhook is downLimit namespaces, exclude kube-system, check the engine's pods
kubectl top says metrics not availableThe metrics server needs a minute after new pods startWait and run it again
Pods still blocked after uninstallingThe webhook configuration was left behindDelete the ValidatingWebhookConfiguration of the tool

What next

Prepared by the CareerByteCode Team from our tested lab, with AI help for the write-up.

Read next →Cloud Security Interview Questions with Hands-On Labs⏱ 9 min

⭐ Keep learning with CareerByteCode

Hand-picked to go deeper on this topic
News, guides and updates from the CareerByteCode team
🏅 Expert writer

The CareerByteCode team writes about what is new on careerbytecode.in: courses, labs, exams, events and the community, which is free to join for influencers and content creators in tech.

Get CareerByteCode Team's next blog by email

One email a week when they publish. Add the weekly Best of ByteBlog too. No spam, unsubscribe with one click.

We email you once to confirm. By subscribing you agree to the privacy policy.

💬 Comments (0)

Top · Newest

Sign in to comment

Related blogs

🧪⭐ Recommended by CareerByteCode · ByteLabs project bundleCloud and DevOps Realtime Starterโ‚น499Build the projects →