CareerByteCode
ByteBlogby CareerByteCode

Helm Chart CI/CD: Lint, Test and Publish Charts Automatically

What you build: a Helm chart pipeline: lint, kubeconform, helm-unittest, then publish to Artifact Registry with immutable versions, plus a real failing test, a refused overwrite, and install/upgrade/r

Helm Chart CI/CD: Lint, Test and Publish Charts Automatically

✍️ Translate this blog into Hindi, Tamil or Telugu

A Helm chart is how most teams package an app for Kubernetes, and a chart is code: it can have bugs, break policies, and be published with the wrong version. In this lab you treat a chart like real software. You lint it, render it and validate the YAML against the Kubernetes schemas, write unit tests for it, and build a Cloud Build pipeline that tests and publishes it to Artifact Registry as an OCI artifact. You hit two real problems: a unit test that catches a policy break in Helm's own starter chart, and a publish that is refused because someone forgot to bump the version. Finally you install, upgrade and roll back the published chart on a GKE cluster.

Who it is for: DevOps engineers who know basic Kubernetes and have used helm install. Time: about 1 to 1.5 hours. Cost: a small GKE cluster for about an hour (a small compute cost; delete it after) plus free-tier Cloud Build minutes and a few KB in Artifact Registry.

Every command below was run for real by the CareerByteCode Team in a fresh Google Cloud project, and every output is the real result we got.

🧪⭐ Recommended by CareerByteCode · ByteLabs project bundleCloud and DevOps Realtime Starterโ‚น499Build the projects →

Part A: chart CI/CD in plain English

Chart pipeline in one picture
Lint, render and validate, unit-test, package, publish to an OCI registry with immutable versions, then install
  • Helm chart: templates (templates/*.yaml) plus default settings (values.yaml) and metadata (Chart.yaml, with the chart's version).
  • **helm lint**: checks the chart's structure and templates.
  • **helm template + kubeconform**: renders the chart to plain YAML, then validates every object against the official Kubernetes API schemas (wrong field names, wrong types).
  • helm-unittest: a plugin that renders templates with given values and checks the result (equal, isNotNull...). Real-life example: a checklist an inspector ticks before a building opens.
  • OCI registry: Helm 3 can store charts in a container registry like Artifact Registry (oci://...). With immutable tags, a published version can never be overwritten, so 0.1.0 always means the same chart.

Before you start

  1. A Google Cloud account with billing and a project. This lab uses a small GKE cluster (not free).
  2. Open Cloud Shell (the >_ icon at the top right of the console); it has helm, kubectl and gcloud. Turn on the APIs and create the cluster (about 5 minutes):
export PROJECT_ID="your-project-id"
gcloud config set project $PROJECT_ID
gcloud services enable container.googleapis.com cloudbuild.googleapis.com artifactregistry.googleapis.com
gcloud container clusters create lab-cluster --zone=us-east1-b --num-nodes=2 \
  --machine-type=e2-standard-4 --disk-type=pd-standard --disk-size=50
  1. Make a folder ~/lab117 with two files shown below: deployment_test.yaml (the unit tests) and cloudbuild.yaml (the pipeline).

Part B: the lab

Step 1: a chart and the tools

Why: helm create web makes Helm's standard starter chart (an nginx deployment, a service and friends). kubeconform and helm-unittest are the testing tools.

# A chart from Helm's own starter template, then the tools: kubeconform (checks YAML against Kubernetes schemas) and helm-unittest
gcloud container clusters get-credentials lab-cluster --zone=us-east1-b > /dev/null 2>&1; helm version --short
rm -rf web && helm create web > /dev/null && find web -type f | sort | head -12
curl -fsSL https://github.com/yannh/kubeconform/releases/download/v0.6.7/kubeconform-linux-amd64.tar.gz | sudo tar xz -C /usr/local/bin kubeconform && kubeconform -v
helm plugin install https://github.com/helm-unittest/helm-unittest --version v0.7.2 > /dev/null 2>&1; helm plugin list
v3.22.1+gd6c1a0e
web/.helmignore
web/Chart.yaml
web/templates/NOTES.txt
web/templates/_helpers.tpl
web/templates/deployment.yaml
web/templates/hpa.yaml
web/templates/httproute.yaml
web/templates/ingress.yaml
web/templates/service.yaml
web/templates/serviceaccount.yaml
web/templates/tests/test-connection.yaml
web/values.yaml
v0.6.7
NAME    	VERSION	DESCRIPTION                                                                         
unittest	0.7.2  	Unit test for helm chart in YAML with ease to keep your chart functional and robust.

Step 2: lint, render and validate

# Lint the chart, render it, and validate the rendered YAML against the Kubernetes API schemas
helm lint web
helm template demo web | kubeconform -summary -strict
==> Linting web
[INFO] Chart.yaml: icon is recommended

1 chart(s) linted, 0 chart(s) failed
Summary: 4 resources found parsing stdin - Valid: 4, Invalid: 0, Errors: 0, Skipped: 0

1 chart(s) linted, 0 chart(s) failed and kubeconform found all 4 rendered objects valid. Structure and schema are fine. But are the values right for our team?

Break it: a unit test catches a policy problem

Our team rule: every container must set CPU and memory requests (so the scheduler can place it and autoscaling works). The tests:

suite: deployment
templates:
  - deployment.yaml
tests:
  - it: uses the replica count from values
    set:
      replicaCount: 3
    asserts:
      - equal:
          path: spec.replicas
          value: 3
  - it: uses the image tag from values
    set:
      image.tag: "1.27"
    asserts:
      - equal:
          path: spec.template.spec.containers[0].image
          value: nginx:1.27
  - it: always sets CPU and memory requests
    asserts:
      - isNotNull:
          path: spec.template.spec.containers[0].resources.requests.cpu
      - isNotNull:
          path: spec.template.spec.containers[0].resources.requests.memory
# Break it: unit tests for the chart. The starter template sets no CPU or memory requests, and our policy needs them
mkdir -p web/tests && cp deployment_test.yaml web/tests/ && helm unittest web 2>&1 | grep -vE "^$|^ *$" | head -16
### Chart [ web ] web
 FAIL  deployment	web/tests/deployment_test.yaml
	- always sets CPU and memory requests
		- asserts[0] `isNotNull` fail
			Template:	web/templates/deployment.yaml
			DocumentIndex:	0
			Path:	spec.template.spec.containers[0].resources.requests.cpu expected to exists
		- asserts[1] `isNotNull` fail
			Template:	web/templates/deployment.yaml
			DocumentIndex:	0
			Path:	spec.template.spec.containers[0].resources.requests.memory expected to exists
Charts:      1 failed, 0 passed, 1 total
Test Suites: 1 failed, 0 passed, 1 total
Tests:       1 failed, 2 passed, 3 total
Snapshot:    0 passed, 0 total
Time:        14.559784ms

Two tests pass (replica count and image tag come from values), and one fails: Helm's own starter chart ships with resources: {}, so the rendered container has no requests. Lint and schema checks cannot catch this, because empty resources are valid Kubernetes; only a test that encodes your policy can.

Fix it: requests in values.yaml

# Fix it: give the container requests (and limits) in values.yaml, then lint and test again
python3 - <<'PY'
import re
p = "web/values.yaml"; s = open(p).read()
s = s.replace("resources: {}", "resources:\n  requests:\n    cpu: 20m\n    memory: 32Mi\n  limits:\n    memory: 64Mi", 1)
open(p, "w").write(s)
PY
grep -A5 "^resources:" web/values.yaml
helm lint web | tail -1 && helm unittest web 2>&1 | grep -E "PASS|FAIL|Tests:|Charts:"
resources:
  requests:
    cpu: 20m
    memory: 32Mi
  limits:
    memory: 64Mi
1 chart(s) linted, 0 chart(s) failed
 PASS  deployment	web/tests/deployment_test.yaml
Charts:      1 passed, 1 total
Tests:       3 passed, 3 total
Lint, test, fix
Lint and schema checks pass, a unit test fails on missing requests, and after the values fix 3 of 3 tests pass

Step 3: a registry with immutable versions, and a pipeline account

Why: --immutable-tags means a pushed version can never be replaced. The pipeline runs as its own service account with three roles only. New roles take a little time to work, so a tiny probe build retries until the account can run builds and write logs.

# A chart registry: Artifact Registry stores Helm charts as OCI artifacts. Immutable tags: a version can never be overwritten
gcloud artifacts repositories create charts --repository-format=docker --location=us-east1 --immutable-tags --description="Helm charts" 2>&1 | tail -1
gcloud iam service-accounts create chart-ci-1119 --display-name="Chart pipeline" 2>&1 | tail -1
until gcloud iam service-accounts describe chart-ci-1119@$PROJECT_ID.iam.gserviceaccount.com > /dev/null 2>&1; do sleep 5; done
for r in roles/artifactregistry.writer roles/logging.logWriter roles/storage.objectViewer; do
  until gcloud projects add-iam-policy-binding $PROJECT_ID --member="serviceAccount:chart-ci-1119@$PROJECT_ID.iam.gserviceaccount.com" --role=$r --condition=None --format="value(etag)" > /dev/null 2>&1; do sleep 5; done; echo "chart-ci-1119: $r"
done
printf 'steps:\n- name: bash\n  args: ["-c", "echo the pipeline account works"]\noptions:\n  logging: CLOUD_LOGGING_ONLY\n' > probe.yaml
for i in $(seq 1 20); do PID_=$(gcloud builds submit --no-source --config=probe.yaml --region=us-east1 --service-account=projects/$PROJECT_ID/serviceAccounts/chart-ci-1119@$PROJECT_ID.iam.gserviceaccount.com --async --format="value(id)" 2>/dev/null); sleep 30; gcloud builds log $PID_ --region=us-east1 2>/dev/null | grep -q "pipeline account works" && break; sleep 20; done; echo "probe build: $(gcloud builds describe $PID_ --region=us-east1 --format='value(status)') and its log is readable, after $i tries"
Created repository [charts].
Service account email: chart-ci-1119@your-project-id.iam.gserviceaccount.com
chart-ci-1119: roles/artifactregistry.writer
chart-ci-1119: roles/logging.logWriter
chart-ci-1119: roles/storage.objectViewer
probe build: SUCCESS and its log is readable, after 3 tries

Step 4: the chart pipeline publishes 0.1.0

Why: the pipeline (cloudbuild.yaml) runs lint and unit tests, then packages the chart and pushes it to oci://us-east1-docker.pkg.dev/<project>/charts. If a test fails, nothing is published.

# Chart pipeline: lint, render and validate, unit-test, then package and publish to Artifact Registry (OCI).
steps:
- id: lint-and-test
  name: alpine/helm:3.16.4
  entrypoint: sh
  args: ["-c", "apk add -q git > /dev/null && helm plugin install https://github.com/helm-unittest/helm-unittest --version v0.7.2 > /dev/null 2>&1; helm lint web && helm unittest web"]
- id: publish
  name: alpine/helm:3.16.4
  entrypoint: sh
  args: ["-c", "apk add -q curl > /dev/null && TOKEN=$$(curl -s -H 'Metadata-Flavor: Google' http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token | sed -E 's/.*\"access_token\":\"([^\"]+)\".*/\\1/') && echo $$TOKEN | helm registry login -u oauth2accesstoken --password-stdin us-east1-docker.pkg.dev > /dev/null && helm package web > /dev/null && helm push web-$$(grep '^version:' web/Chart.yaml | cut -d' ' -f2).tgz oci://us-east1-docker.pkg.dev/$PROJECT_ID/charts"]
options:
  logging: CLOUD_LOGGING_ONLY
# The chart pipeline (Cloud Build): lint, unit tests, package, push version 0.1.0
cat cloudbuild.yaml | grep -E "^- id|helm (lint|unittest|package|push)" | sed -E 's/^ +args: \["-c", "//' | cut -c1-120
ID=$(gcloud builds submit --config=cloudbuild.yaml --region=us-east1 --service-account=projects/$PROJECT_ID/serviceAccounts/chart-ci-1119@$PROJECT_ID.iam.gserviceaccount.com . --async --format="value(id)" 2>/dev/null)
for i in $(seq 1 60); do gcloud builds describe $ID --region=us-east1 --format="value(status)" | grep -qE "SUCCESS|FAILURE" && break; sleep 10; done; sleep 20
echo "build ${ID:0:8}: $(gcloud builds describe $ID --region=us-east1 --format='value(status)')"
gcloud builds log $ID --region=us-east1 2>/dev/null | grep -E "linted|failed|PASS|Tests:|Pushed|Digest" | sed -E 's/^Step #[0-9]+ - "[a-z-]+": //' | head -8
gcloud artifacts docker tags list us-east1-docker.pkg.dev/$PROJECT_ID/charts/web --format="value(tag)" 2>/dev/null
- id: lint-and-test
apk add -q git > /dev/null && helm plugin install https://github.com/helm-unittest/helm-unittest --version v0.7.2 > /dev
- id: publish
apk add -q curl > /dev/null && TOKEN=$$(curl -s -H 'Metadata-Flavor: Google' http://metadata.google.internal/computeMeta
build f89c33a0: SUCCESS
Digest: sha256:9b25e60ae264940b276e32866d37e3088e70c4e2d1784b964dc3f90346281a74
1 chart(s) linted, 0 chart(s) failed
 PASS  deployment	web/tests/deployment_test.yaml
Tests:       3 passed, 3 total
Pushed: us-east1-docker.pkg.dev/your-project-id/charts/web:0.1.0
Digest: sha256:8cdd14dc54cbc0af5dc828001ece5c1b1e4ff30ec3658ce439ec5fa2b2e63f20
0.1.0

The build passed lint and 3 tests, and pushed web:0.1.0 to the registry.

Break it: change the chart, forget the version

A teammate changes replicaCount from 1 to 2 but leaves version: 0.1.0. The pipeline runs.

# Break it: change the chart but forget to bump the version. The registry refuses to overwrite 0.1.0
sed -i 's/replicaCount: 1/replicaCount: 2/' web/values.yaml && grep "^version:" web/Chart.yaml
ID=$(gcloud builds submit --config=cloudbuild.yaml --region=us-east1 --service-account=projects/$PROJECT_ID/serviceAccounts/chart-ci-1119@$PROJECT_ID.iam.gserviceaccount.com . --async --format="value(id)" 2>/dev/null)
for i in $(seq 1 60); do gcloud builds describe $ID --region=us-east1 --format="value(status)" | grep -qE "SUCCESS|FAILURE" && break; sleep 10; done; sleep 20
echo "build ${ID:0:8}: $(gcloud builds describe $ID --region=us-east1 --format='value(status)')"
gcloud builds log $ID --region=us-east1 2>/dev/null | grep -iE "immutable|denied|error:|Pushed" | sed -E 's/^Step #[0-9]+ - "[a-z-]+": //' | cut -c1-200 | head -3
version: 0.1.0
build b5258efa: FAILURE
Error: failed commit on ref "manifest-sha256:8ca5506d393b315ff70d7d8c10a2ac9678cc15d8ff35e2f4aa8f18d52c4730b9": unexpected status from PUT request to https://us-east1-docker.pkg.dev/v2/your-project-id
ERROR: build step 1 "alpine/helm:3.16.4" failed: step exited with non-zero status: 1

The registry refused the push (failed commit on ref ... unexpected status from PUT request): version 0.1.0 already exists and tags are immutable. Without immutable tags this push would have silently replaced 0.1.0, and anyone installing "0.1.0" would get a different chart than yesterday.

Fix it: bump the version

# Fix it: bump the chart version (0.2.0) and run the pipeline again
sed -i 's/^version: 0.1.0/version: 0.2.0/' web/Chart.yaml && grep "^version:" web/Chart.yaml
ID=$(gcloud builds submit --config=cloudbuild.yaml --region=us-east1 --service-account=projects/$PROJECT_ID/serviceAccounts/chart-ci-1119@$PROJECT_ID.iam.gserviceaccount.com . --async --format="value(id)" 2>/dev/null)
for i in $(seq 1 60); do gcloud builds describe $ID --region=us-east1 --format="value(status)" | grep -qE "SUCCESS|FAILURE" && break; sleep 10; done; sleep 20
echo "build ${ID:0:8}: $(gcloud builds describe $ID --region=us-east1 --format='value(status)')"
gcloud artifacts docker tags list us-east1-docker.pkg.dev/$PROJECT_ID/charts/web --format="value(tag)" 2>/dev/null | tr '\n' ' '; echo
version: 0.2.0
build 0312b28b: SUCCESS
0.1.0 0.2.0
🎓⭐ Recommended by CareerByteCode · Live trainingCloud & DevOps Live Cohort Internshipโ‚น2,360See the training →
Publish and immutability
The pipeline publishes 0.1.0; an unchanged version is refused by the immutable registry; 0.2.0 is published

Both versions now live side by side: 0.1.0 and 0.2.0.

Step 5: install, upgrade and roll back from the registry

Why: this is how a team consumes the chart: install a pinned version, upgrade to a new one, and roll back with Helm's release history.

# Use the published chart: install 0.1.0 from the registry, upgrade to 0.2.0, then roll back
gcloud auth print-access-token | helm registry login -u oauth2accesstoken --password-stdin us-east1-docker.pkg.dev 2>&1 | tail -1
helm install shop oci://us-east1-docker.pkg.dev/$PROJECT_ID/charts/web --version 0.1.0 -n charts-demo --create-namespace --wait 2>&1 | grep -E "STATUS|REVISION"
helm upgrade shop oci://us-east1-docker.pkg.dev/$PROJECT_ID/charts/web --version 0.2.0 -n charts-demo --wait 2>&1 | grep -E "STATUS|REVISION"
echo "pods with 0.2.0: $(kubectl -n charts-demo get pods --no-headers | grep -c Running)"
helm rollback shop 1 -n charts-demo --wait 2>&1 | tail -1
helm history shop -n charts-demo | awk '{print $1, $2, $(NF-3), $(NF-2), $(NF-1), $NF}' | column -t | cut -c1-90
echo "pods after rollback: $(kubectl -n charts-demo get deploy -o jsonpath='{.items[0].spec.replicas}')"
Login Succeeded
STATUS: deployed
REVISION: 1
STATUS: deployed
REVISION: 2
pods with 0.2.0: 2
Rollback was a success! Happy Helming!
REVISION  UPDATED  CHART      APP       VERSION  DESCRIPTION
1         Thu      web-0.1.0  1.16.0    Install  complete
2         Thu      web-0.2.0  1.16.0    Upgrade  complete
3         Thu      1.16.0     Rollback  to       1
pods after rollback: 1
Install, upgrade, rollback
Installed 0.1.0 from the registry, upgraded to 0.2.0 (2 pods), rolled back to revision 1

Revision 1 is chart 0.1.0, revision 2 is 0.2.0 (2 pods), and the rollback created revision 3, back to 0.1.0's single replica.

POC: prove it works

  • helm lint and kubeconform passed the starter chart; a unit test still caught its missing resource requests.
  • After the fix, 3 of 3 tests passed in the pipeline, and web:0.1.0 was published to Artifact Registry.
  • Republishing 0.1.0 with changes was refused by the immutable registry; 0.2.0 was published.
  • The chart was installed from the registry, upgraded to 0.2.0 and rolled back.

Quiz task: check what you learned

  1. Which check finds a misspelled Kubernetes field: lint, kubeconform or unit tests? (Answer: kubeconform (schema validation).)
  2. Why did lint not catch the missing requests? (Answer: empty resources are valid; only a test that encodes your rule fails.)
  3. What happens when you push the same chart version to an immutable registry? (Answer: the push is refused.)
  4. How do you install a chart from an OCI registry? *(Answer: helm install <name> oci://<registry>/<chart> --version <v>.)*
  5. Which Helm command returns to the previous release? *(Answer: helm rollback <release> <revision>.)*

Clean up (delete everything)

# Clean up: the release, the chart registry, the pipeline account and its roles, the build bucket
helm uninstall shop -n charts-demo 2>&1 | tail -1; kubectl delete namespace charts-demo --ignore-not-found 2>&1 | tail -1
gcloud artifacts repositories delete charts --location=us-east1 --quiet 2>&1 | tail -1
for r in roles/artifactregistry.writer roles/logging.logWriter roles/storage.objectViewer; do for t in 1 2 3 4 5; do gcloud projects remove-iam-policy-binding $PROJECT_ID --member="serviceAccount:chart-ci-1119@$PROJECT_ID.iam.gserviceaccount.com" --role=$r --condition=None --format="value(etag)" > /dev/null 2>&1 && break; sleep 5; done; done; echo "roles removed"
gcloud iam service-accounts delete chart-ci-1119@$PROJECT_ID.iam.gserviceaccount.com --quiet 2>&1 | tail -1
for b in $(gcloud storage buckets list --format="value(name)" | grep -E "cloudbuild"); do gcloud storage rm -r gs://$b --quiet > /dev/null 2>&1 && echo "deleted the build bucket"; done
rm -rf web probe.yaml
release "shop" uninstalled
namespace "charts-demo" deleted
Deleted repository [charts].
roles removed
deleted service account [chart-ci-1119@your-project-id.iam.gserviceaccount.com]
deleted the build bucket
Clean up
Clean up: the release, the chart registry, the pipeline account and the build bucket removed

Then delete the cluster when you are done with all Kubernetes labs:

gcloud container clusters delete lab-cluster --zone=us-east1-b --quiet

FAQ

OCI registry or a classic Helm repository (index.yaml)? OCI registries are the modern default: one place for images and charts, the same access control and immutability. Classic repositories (for example on a bucket) still work everywhere.

What is chart-testing (ct)? A tool from the Helm project that lints and installs only the changed charts in a repository, in a real cluster. Teams with many charts add it to the pipeline; it needs a cluster or kind in CI.

Should the chart version follow the app version? They are separate: version is the chart's own version (change it on every chart change), appVersion is the app it deploys.

How do I sign charts? Push them to Artifact Registry and sign with Cosign or Helm provenance files; verify before install.

How do I stop someone deploying an untested chart? Only the pipeline account may push to the chart registry, and clusters install only from that registry.

Why does the pipeline install helm-unittest at run time? To keep the example short. In real pipelines, bake the plugin into your own builder image so builds are faster and repeatable.

Troubleshooting

You seeWhyFix
isNotNull fail ... resources.requests.cpuNo requests in valuesSet resources.requests in values.yaml
unexpected status from PUT request on pushThe version already exists in an immutable repositoryBump version in Chart.yaml
denied: Permission ... uploadArtifactsThe pipeline account lacks Artifact Registry Writer, or roles are not active yetGrant the role; wait (the probe loop)
Error: ... not found on installWrong OCI path or versiongcloud artifacts docker tags list ... to see versions
helm registry login failsExpired tokenPipe a fresh gcloud auth print-access-token into helm registry login -u oauth2accesstoken --password-stdin (Step 5)

What next

Prepared by the CareerByteCode Team from our tested lab, with AI help for the write-up.

Read next →Flux CD vs Argo CD: Hands-On GitOps Comparison⏱ 15 min

⭐ Keep learning with CareerByteCode

Hand-picked to go deeper on this topic
News, guides and updates from the CareerByteCode team
🏅 Expert writer

The CareerByteCode team writes about what is new on careerbytecode.in: courses, labs, exams, events and the community, which is free to join for influencers and content creators in tech.

Get CareerByteCode Team's next blog by email

One email a week when they publish. Add the weekly Best of ByteBlog too. No spam, unsubscribe with one click.

We email you once to confirm. By subscribing you agree to the privacy policy.

💬 Comments (0)

Top · Newest

Sign in to comment

Related blogs

🧪⭐ Recommended by CareerByteCode · ByteLabs project bundleCloud and DevOps Realtime Starterโ‚น499Build the projects →