Wildcard TLS certificates
Serve browser-trusted HTTPS from your klimax clusters — a real Let's Encrypt
certificate for *.lab.example.dev, valid on private
172.30.x.y addresses, with no custom CA to install and no certificate
warnings to click through.
This is the companion to Hostnames with ExternalDNS. Read that first — it sets up the DNS zone this guide issues a certificate for.
Why this is possible at all
A certificate authority will happily issue for a name that resolves to a private
address. What it will not do is reach that address to verify you control the
name. So the usual HTTP-01 challenge, where the CA connects to your
service on port 80, cannot work: Let's Encrypt has no route to
172.30.1.7.
DNS-01 sidesteps this entirely. Instead of proving
control by serving a file, you prove it by writing a TXT record into the zone. The CA
queries public DNS; nothing connects to your Mac. Since you already delegated a zone
and hold API credentials for it, you have everything the challenge needs.
| Self-signed / mkcert | Let's Encrypt via DNS-01 | |
|---|---|---|
| Browser trust | after installing a CA in every trust store | immediate, everywhere |
| Non-browser clients (Go, JVM, curl) | only after per-runtime trust config | out of the box |
| Works on a colleague's machine | no | yes |
| Wildcard across all lab services | possible | yes, one certificate |
| Renewal | manual | scriptable or automated |
The second row matters more than it looks. A JVM or Go client rejecting your self-signed CA is a classic afternoon lost to trust-store debugging — and it shows up exactly when you are demoing OIDC, mTLS or webhooks.
Prerequisites
- A delegated DNS zone and cloud credentials — see ExternalDNS, Steps 1–2.
- A running klimax cluster.
lego(brew install lego) for the one-shot path, orhelmfor the automated path.
$ export GCP_PROJECT=my-project
$ export DNS_ZONE_NAME=lab.example.dev
$ export DNS_ZONE_LITERAL=lab-example-dev
$ export CREDS_FILE="$HOME/.klimax/external-dns-credentials.json"
You can reuse the ExternalDNS service account —
roles/dns.admin already covers writing the
_acme-challenge TXT records.
Option A — one-shot certificate with lego
Best when you want a certificate now and do not need in-cluster automation.
$ GCE_PROJECT="${GCP_PROJECT}" \
GCE_SERVICE_ACCOUNT_FILE="${CREDS_FILE}" \
lego --email you@example.com \
--dns gcloud \
-d "*.${DNS_ZONE_NAME}" \
-a run
lego writes the TXT record, waits for propagation, completes the
challenge and cleans up after itself. The certificate and key land in
.lego/certificates/.
Request -d "*.${DNS_ZONE_NAME}" and
-d "${DNS_ZONE_NAME}" if you want the apex covered too. A wildcard
matches one label only: *.lab.example.dev covers
keycloak.lab.example.dev but neither lab.example.dev itself
nor a.b.lab.example.dev.
Load it into the cluster as a TLS secret — one certificate is valid in all of them:
$ for CTX in dev staging prod; do
kubectl --context "$CTX" -n kong create secret tls "wildcard-${DNS_ZONE_LITERAL}" \
--cert=".lego/certificates/_.${DNS_ZONE_NAME}.crt" \
--key=".lego/certificates/_.${DNS_ZONE_NAME}.key"
done
Renewing
Let's Encrypt certificates last 90 days. Re-run with renew and refresh
the secrets:
$ GCE_PROJECT="${GCP_PROJECT}" GCE_SERVICE_ACCOUNT_FILE="${CREDS_FILE}" \
lego --email you@example.com --dns gcloud -d "*.${DNS_ZONE_NAME}" renew
$ kubectl --context "${CTX}" -n kong create secret tls "wildcard-${DNS_ZONE_LITERAL}" \
--cert=".lego/certificates/_.${DNS_ZONE_NAME}.crt" \
--key=".lego/certificates/_.${DNS_ZONE_NAME}.key" \
--dry-run=client -o yaml | kubectl --context "${CTX}" apply -f -
For labs that outlive a certificate, Option B removes this chore.
Option B — automated renewal with cert-manager
Best for long-lived clusters, or when you would rather not think about expiry.
$ helm repo add jetstack https://charts.jetstack.io
$ helm repo update
$ helm --kube-context "${CTX}" upgrade -i cert-manager jetstack/cert-manager \
-n cert-manager --create-namespace --set crds.enabled=true
$ kubectl --context "${CTX}" -n cert-manager create secret generic clouddns-credentials \
--from-file=credentials.json="${CREDS_FILE}"
Create a ClusterIssuer using the DNS-01 solver:
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-dns
spec:
acme:
email: you@example.com
server: https://acme-v02.api.letsencrypt.org/directory
privateKeySecretRef:
name: letsencrypt-dns-account-key
solvers:
- dns01:
cloudDNS:
project: my-project
serviceAccountSecretRef:
name: clouddns-credentials
key: credentials.json
Then request the wildcard, and cert-manager keeps it renewed:
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: wildcard-lab
namespace: kong
spec:
secretName: wildcard-lab-example-dev
issuerRef:
name: letsencrypt-dns
kind: ClusterIssuer
dnsNames:
- "*.lab.example.dev"
- "lab.example.dev"
$ kubectl --context "${CTX}" -n kong get certificate wildcard-lab -w
Use the staging ACME endpoint while iterating. Let's Encrypt
enforces rate limits — notably 5 duplicate certificates per week — and a
misconfigured solver burns through them fast. Swap server: for
https://acme-staging-v02.api.letsencrypt.org/directory until issuance
succeeds, then switch back. Staging certificates are untrusted by design; that is
expected.
Using the certificate
Reference the secret from whatever terminates TLS. For a Kubernetes Ingress:
spec:
tls:
- hosts:
- "keycloak.lab.example.dev"
secretName: wildcard-lab-example-dev
For a Kong Gateway data plane, mount it as a secret volume and point the listener at it:
# values-gw-dp.yaml
proxy:
tls:
enabled: true
env:
ssl_cert: /etc/secrets/wildcard/tls.crt
ssl_cert_key: /etc/secrets/wildcard/tls.key
secretVolumes:
- wildcard-lab-example-dev
Then, from your Mac, with no trust-store changes at all:
$ curl -sS https://keycloak.lab.example.dev/
Troubleshooting
| Symptom | Likely cause |
|---|---|
| Challenge times out | TXT record not visible publicly. Check dig +short TXT _acme-challenge.lab.example.dev; usually incomplete NS delegation |
rateLimited from ACME | Too many issuance attempts — switch to the staging endpoint and wait out the window |
| Issued, but the browser still warns | The proxy is still serving the old certificate — restart or reload it so it re-reads the secret |
Works in curl, fails in a JVM or Go client | Almost always a leftover self-signed certificate still being served, not a trust problem. Verify with openssl s_client -connect keycloak.lab.example.dev:443 |
cert-manager stuck Pending | kubectl describe challenge -A shows the solver error; usually the credentials secret or project: is wrong |
Wildcard rejected for a.b.lab.example.dev | A wildcard covers one label — request a deeper wildcard or a dedicated name |