subuilds.dev

Self-Host GitLab EE on K3s with Cloudflare Tunnel

· 14 min read
self-hosted-infra

The official GitLab Helm chart targets production: 8 vCPU, 30 GB RAM, multi-node, registry on its own subdomain, ports 80/443 wide open to the internet. For a homelab on a Mac Mini, none of that is realistic. You want one URL that works the same on LAN and WAN, zero inbound ports on the home router, and a setup that fits in 6 vCPU / 12 GB without turning the box into a paperweight.

This is the real working configuration: K3s on a Debian 13 ARM64 VM, Traefik + cloudflared + cert-manager, Pi-hole split-horizon DNS, and the four undocumented gotchas that will eat an afternoon if you don’t see them coming.

What You’ll Cover

  1. The architecture — what runs where, and which traffic takes which path
  2. DNS-01 cert issuance with cert-manager and a Cloudflare API token
  3. Split-horizon DNS on Pi-hole so LAN traffic doesn’t hairpin out to Cloudflare
  4. Cloudflared as an in-cluster Deployment — no port forwarding, ever
  5. A minimal values.yaml that survives a 12 GB VM
  6. ARM64 image overrides — for both the MinIO server and the mc client
  7. The hostname-prefix override the chart needs when your hostname isn’t gitlab.<domain>
  8. The GitLab Duo homepage 500 on fresh installs and how to skip it

Architecture

Diagram

Two request paths share the same hostname and the same Let’s Encrypt cert:

  • LAN client → Pi-hole returns the VM’s LAN IP → Traefik on the VM → GitLab. No internet round trip.
  • Remote client → Cloudflare public DNS → Cloudflare edge → outbound QUIC tunnel held open by the in-cluster cloudflared pod → Traefik → GitLab. The home router never accepts an inbound connection.

The single LE cert covers gitlab.<your-domain> and is issued by cert-manager via DNS-01 (Cloudflare API). No HTTP-01, no port 80 exposed, nothing to forward.

Why This Stack

A few choices that look arbitrary until you’ve tried the alternatives:

DecisionReason
UTM, not OrbStackOrbStack hardcodes a 192.168.139.x managed bridge; the VM can’t get a real LAN IP on 192.168.1.0/24. UTM supports true bridged networking, so the VM is just another box on the LAN.
K3s, not k0s or minikubeShips Traefik + local-path-provisioner + ServiceLB out of the box — three fewer things to install. PVCs land under /var/lib/rancher/k3s/storage/.
DNS-01, not HTTP-01Doesn’t require the VM to be reachable on port 80 from the internet. The Cloudflare API token only needs Zone:Read + DNS:Edit on the zone.
Cloudflare Tunnel, not port forwardNo inbound ports on the home router. The tunnel is an outbound QUIC connection from inside the cluster to Cloudflare’s edge — same direction as any other outbound flow.
MinIO bundled, registry offObject storage is required for artifacts/uploads/LFS. The Container Registry is optional — pull from Docker Hub or GHCR and skip a second hostname and Ingress.
Resource reality check

The official chart recommends 8 vCPU / 30 GB. A 6 vCPU / 12 GB VM is below the floor. Expect a slow first boot (10–20 min for migrations + image pulls), ~7–9 GB RAM at idle, and no HA. Fine for a lab, not for a team of 30.

Prerequisites

  • K3s cluster up and kubectl get nodes shows Ready
  • A domain on Cloudflare (this post uses example.com)
  • A Pi-hole at 192.168.1.2 you can add local DNS records to
  • At least 50 GB free under /var/lib/rancher/k3s/storage/

Install Helm and the GitLab chart:

curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
helm repo add gitlab https://charts.gitlab.io/
helm repo update

DNS-01 with cert-manager

Create a Cloudflare API token scoped to Zone:Read + DNS:Edit on your zone only — nothing else. Store it as a secret:

kubectl create namespace cert-manager
kubectl create secret generic cloudflare-api-token \
  --namespace cert-manager \
  --from-literal=api-token='<paste-token>'

Install cert-manager and apply a ClusterIssuer:

helm repo add jetstack https://charts.jetstack.io
helm install cert-manager jetstack/cert-manager \
  --namespace cert-manager --create-namespace \
  --set crds.enabled=true
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-cloudflare
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: you@example.com
    privateKeySecretRef:
      name: letsencrypt-cloudflare-account-key
    solvers:
      - dns01:
          cloudflare:
            apiTokenSecretRef:
              name: cloudflare-api-token
              key: api-token
        selector:
          dnsZones: ["example.com"]
Use the staging endpoint while debugging

Swap acme-v02 for acme-staging-v02 while you iterate. Let’s Encrypt’s production rate limits are low and a misconfigured token will burn through them fast. Switch back once issuance works end-to-end.

Pre-create the Certificate before installing GitLab so the secret exists when the Ingress comes up:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: gitlab-tls
  namespace: gitlab
spec:
  secretName: gitlab-tls
  issuerRef:
    name: letsencrypt-cloudflare
    kind: ClusterIssuer
  commonName: gitlab.example.com
  dnsNames:
    - gitlab.example.com
  privateKey:
    algorithm: ECDSA
    size: 256
  duration: 2160h    # 90d
  renewBefore: 360h  # 15d
kubectl create namespace gitlab
kubectl apply -f gitlab-tls-cert.yaml
kubectl -n gitlab get certificate gitlab-tls -w
# wait for READY=True (typically 30–120s)

Split-Horizon DNS on Pi-hole

This is the part most homelab guides skip. Without it, every LAN request to gitlab.example.com hairpins out through your ISP, into Cloudflare, back through the tunnel, into your VM — three hops away from a box that’s six feet from your desk.

The fix is one local DNS record. On the Pi-hole at 192.168.1.2, go to Settings → Local DNS Records → Add:

Domain:  gitlab.example.com
IP:      192.168.1.20    (your K3s VM's LAN IP)

Equivalent via dnsmasq config:

address=/gitlab.example.com/192.168.1.20

Reload with pihole restartdns. Verify from a LAN client:

dig @192.168.1.2 gitlab.example.com +short
# expected: 192.168.1.20   (NOT a Cloudflare anycast address)
LAN clients must actually use Pi-hole

If a device queries 1.1.1.1 directly (some VPN configs, DoH browsers, hard-coded clients), it bypasses Pi-hole and hairpins through Cloudflare. Set Pi-hole as primary DNS via your router’s DHCP option 6, and verify with scutil --dns on macOS or resolvectl status on Linux.

Cloudflared in the Cluster

Run the three one-time setup commands from any machine with cloudflared installed — a workstation is fine, the binary does not need to live on the K3s VM:

cloudflared tunnel login
cloudflared tunnel create gitlab-homelab
cloudflared tunnel route dns gitlab-homelab gitlab.example.com

The create command writes credentials to ~/.cloudflared/<uuid>.json. Load them into the cluster:

kubectl -n gitlab create secret generic cloudflared-credentials \
  --from-file=credentials.json=$HOME/.cloudflared/<tunnel-uuid>.json

Deploy cloudflared as a regular Kubernetes Deployment, pointing at the in-cluster Traefik service:

apiVersion: v1
kind: ConfigMap
metadata:
  name: cloudflared-config
  namespace: gitlab
data:
  config.yaml: |
    tunnel: <tunnel-uuid>
    credentials-file: /etc/cloudflared/creds/credentials.json
    ingress:
      - hostname: gitlab.example.com
        service: https://traefik.kube-system.svc.cluster.local:443
        originRequest:
          originServerName: gitlab.example.com   # MUST match a SAN on gitlab-tls
      - service: http_status:404
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cloudflared
  namespace: gitlab
spec:
  replicas: 1
  selector:
    matchLabels: { app: cloudflared }
  template:
    metadata:
      labels: { app: cloudflared }
    spec:
      containers:
        - name: cloudflared
          image: cloudflare/cloudflared:latest
          args: ["tunnel", "--config", "/etc/cloudflared/config/config.yaml", "run"]
          volumeMounts:
            - { name: config, mountPath: /etc/cloudflared/config, readOnly: true }
            - { name: creds, mountPath: /etc/cloudflared/creds, readOnly: true }
      volumes:
        - { name: config, configMap: { name: cloudflared-config } }
        - { name: creds, secret: { secretName: cloudflared-credentials } }
kubectl apply -f cloudflared.yaml
kubectl -n gitlab logs deploy/cloudflared
# expect: "Registered tunnel connection" lines, one per CF edge

In the Cloudflare dashboard, set the public hostname to proxied (orange cloud). Cloudflare terminates TLS at the edge with its own cert, then the tunnel re-encrypts to Traefik, which presents the LE cert internally. That’s why originServerName must match a SAN on gitlab-tls — otherwise Traefik’s SNI matching falls back and serves the wrong cert.

The Minimal values.yaml

Save this as gitlab-values.yaml. It strips everything optional and survives on the 12 GB VM:

global:
  edition: ee
  hosts:
    domain: example.com
    # gitlab:
    #   name: gitlabee.example.com   # uncomment if your prefix isn't `gitlab.`
    https: true
  ingress:
    configureCertmanager: false
    class: traefik
    tls:
      enabled: true
      secretName: gitlab-tls
  initialRootPassword:
    secret: gitlab-initial-root-password
    key: password
  kas:
    enabled: false

nginx-ingress:
  enabled: false

prometheus:
  install: false
gitlab-runner:
  install: false
installCertmanager: false

registry:
  enabled: false
gitlab:
  webservice:
    minReplicas: 1
    maxReplicas: 1
    resources:
      requests: { cpu: 400m, memory: 2.5Gi }
  sidekiq:
    minReplicas: 1
    maxReplicas: 1
    resources:
      requests: { cpu: 200m, memory: 1Gi }
  gitaly:
    persistence:
      storageClass: local-path
      size: 20Gi

postgresql:
  persistence: { storageClass: local-path, size: 8Gi }
redis:
  master:
    persistence: { storageClass: local-path, size: 5Gi }

minio:
  image: quay.io/minio/minio
  imageTag: RELEASE.2025-09-07T16-13-09Z
  imagePullPolicy: IfNotPresent
  minioMc:
    image: quay.io/minio/mc
    tag: RELEASE.2025-09-07T16-02-50Z
  ingress:
    enabled: false
  persistence: { storageClass: local-path, size: 20Gi }

Pre-create the root password secret so you don’t have to grep for it later:

kubectl create secret generic gitlab-initial-root-password \
  --namespace gitlab \
  --from-literal=password='ChangeMe-Strong-Password!'

Install:

helm install gitlab gitlab/gitlab \
  --namespace gitlab \
  --values gitlab-values.yaml \
  --timeout 600s

kubectl -n gitlab get pods -w

First install pulls several GB of images and runs DB migrations. On a 6 vCPU ARM64 VM, plan for 15–20 minutes before gitlab-webservice-default-* is 2/2 Running.

The Gotchas (Four That Cost an Afternoon)

Everything above works on x86 with the default hostname. The four issues below are the ones that hit on ARM64, with non-default hostnames, or on a fresh install.

ARM64 #1: The MinIO Server Image

The bundled MinIO subchart pins RELEASE.2017-12-28T01-21-00Z — an amd64-only image from 2017. On ARM64 the pod hits exec format error and never starts.

The fix is in the minio block above:

minio:
  image: quay.io/minio/minio
  imageTag: RELEASE.2025-09-07T16-13-09Z

quay.io/minio/minio is multi-arch. The chart-set MINIO_ACCESS_KEY / MINIO_SECRET_KEY env vars are still honored by current MinIO as deprecated aliases for MINIO_ROOT_USER / MINIO_ROOT_PASSWORD, so credentials keep working without further changes.

Do not also set extraEnv: MINIO_ROOT_USER

Pointing both MINIO_ROOT_USER and MINIO_ACCESS_KEY at the same secret looks harmless, but on some versions MinIO prefers ROOT_* while the bucket-create Job’s mc client still reads the legacy creds — producing 403 SignatureDoesNotMatch in a loop. Let the chart manage credentials end-to-end.

ARM64 #2: The MinIO mc Client Image

This one is not in the official chart docs. The chart pins a separate image for the mc client used by the gitlab-minio-create-buckets Job:

registry.gitlab.com/gitlab-org/cloud-native/mirror/images/minio/mc:RELEASE.2018-07-13T00-53-22Z

A 2018 amd64-only image. On ARM64 the Job loops in CrashLoopBackOff with a tell-tale signature:

  • Exit code 255
  • Pod starts and exits in the same second
  • kubectl logs … --previous returns “unable to retrieve container logs”

And because the buckets never get created, GitLab returns 500 on any page that touches object storage — which is most of them, including /dashboard/home. The webservice pods are 2/2 Running, the cert is valid, the tunnel is green on the Cloudflare dashboard. Everything looks fine. Nothing works.

The fix is the minioMc block in values.yaml above:

minio:
  minioMc:
    image: quay.io/minio/mc
    tag: RELEASE.2025-09-07T16-02-50Z

After a helm upgrade, force the Job to re-run with the new image:

kubectl -n gitlab delete job -l component=create-buckets
kubectl -n gitlab get pods -l component=create-buckets -w
# expect: Completed

Then bounce webservice + sidekiq so they pick up the now-existing buckets:

kubectl -n gitlab rollout restart \
  deploy/gitlab-webservice-default \
  deploy/gitlab-sidekiq-all-in-1-v2

Hostname Prefix: Override global.hosts.gitlab.name

The chart constructs the GitLab hostname as gitlab.<global.hosts.domain> by default. If you set domain: example.com and point the tunnel at gitlabee.example.com (or git.example.com, or anything that isn’t a gitlab. prefix), Traefik builds an Ingress for the default name and the tunnel returns 404 page not found on the prefix you actually use.

The Cloudflare dashboard is green. The cert is valid. The webservice is healthy. Traefik just has no Ingress matching the Host header.

global:
  hosts:
    domain: example.com
    gitlab:
      name: gitlabee.example.com   # the actual hostname you serve

Verify after helm upgrade:

kubectl -n gitlab get ingress
# HOSTS column must match the hostname in the cloudflared config AND the SAN on gitlab-tls

The same hostname must appear in three places: global.hosts.gitlab.name, the cloudflared ConfigMap’s hostname + originServerName, and the gitlab-tls Certificate’s dnsNames. Any drift between the three breaks at least one path.

The Duo Homepage 500 on Fresh Install

The first time anyone visits /dashboard/home on a freshly installed GitLab 17.x+, the controller lazily creates an internal bot user called duo_code_review_bot. On fresh installs that bot save can fail, and because rendering the homepage depends on the bot’s username, the whole page returns 500. /api/v4/user_counts 500s for the same reason.

The stack trace looks like this:

app/controllers/dashboard_controller.rb:50:in `home'
app/controllers/concerns/homepage_data.rb:21:in `homepage_app_data'
app/controllers/concerns/homepage_data.rb:66:in `duo_code_review_bot_username'
lib/users/internal.rb:110:in `duo_code_review_bot'
lib/users/internal.rb:241:in `create_unique_internal'
state_machines (0.100.4) lib/state_machines/machine.rb:569:in `save'

/dashboard/projects returns 200 — you’re not actually locked out, just out of the new home view.

The path of least resistance is to land users on the old dashboard:

Per-user: Top-right avatar → Preferences → HomepageYour Projects.

Instance-wide: Admin Area → Settings → Preferences → Navigation → Default dashboardYour Projects.

If you want to find the real root cause, reproduce it in the Rails console where the error prints cleanly:

kubectl -n gitlab exec -it deploy/gitlab-toolbox -- gitlab-rails runner '
  begin
    u = Users::Internal.duo_code_review_bot
    puts "OK: #{u.inspect}"
  rescue => e
    puts "FAILED: #{e.class}: #{e.message}"
    puts e.record.errors.full_messages if e.respond_to?(:record)
  end
'

Common culprits: stale row from a prior half-completed creation, email validation against an allowlist, or a Duo settings prerequisite. Once you have the validation error, fixing it is usually a one-liner.

Verify the End-to-End Path

From a LAN device using Pi-hole as its resolver:

dig gitlab.example.com +short
# → 192.168.1.20   (VM, not Cloudflare)

curl -vI https://gitlab.example.com 2>&1 | grep -E "(issuer|cf-ray|server:)"
# expected: issuer=Let's Encrypt, NO cf-ray header

From off-LAN (cellular):

dig gitlab.example.com +short
# → Cloudflare anycast

curl -vI https://gitlab.example.com 2>&1 | grep -E "(cf-ray|server:)"
# expected: cf-ray: <id>, server: cloudflare

Tunnel health:

kubectl -n gitlab logs deploy/cloudflared --tail=50
# expect: "Registered tunnel connection" lines, no repeating errors

Certificate:

kubectl -n gitlab get certificate gitlab-tls
# NAME         READY   SECRET       AGE
# gitlab-tls   True    gitlab-tls   3m

What You Have Now

ComponentWhat it does
UTM + Debian 13 VMReal LAN IP, ARM64 kernel, single-node K3s
K3s + TraefikSingle-binary Kubernetes with Ingress and storage included
cert-manager + DNS-01One LE cert covering your GitLab hostname, no port 80 exposure
cloudflared DeploymentOutbound QUIC tunnel — zero inbound ports on the home router
Pi-hole local DNSLAN clients hit the VM directly; no hairpin through Cloudflare
GitLab Helm chart (minimal)EE image, Postgres + Redis + Gitaly + MinIO bundled, no registry/KAS/Prometheus/Runner
Single hostnameSame URL, same cert, on LAN and WAN — no /etc/hosts workarounds

The VM idles around 7–9 GB RAM. First boot takes 15–20 minutes. Subsequent restarts are under a minute. Git over SSH works on the LAN (port 22 on the VM); on remote, use HTTPS with a Personal Access Token — Cloudflare Tunnel doesn’t expose plain SSH without cloudflared access tcp client-side.

Next Steps