Secrets

Two things don’t live in Git: the Secrets the app needs at runtime, and (potentially) a pull credential for its private GHCR image. Both are created on the Ultron Infra node by hand and recreated on a rebuild.

flowchart LR
  Op([operator]) -->|kubectl create secret| S1[(app secrets<br/>kc, backup creds)]
  S1 --> Pods[app pods + Jobs]
  GHCR[GHCR package] -->|public?| Pull{pullable<br/>anonymously?}
  Pull -->|yes| Pods
  Pull -->|no| IPS[(imagePullSecret)]
  IPS --> Pods

Out-of-band secrets

Anything sensitive that isn’t auto-generated by an operator is created directly with kubectl, not committed. The convention is to stage values in a temp env-file and load it with --from-env-file, so no secret value ever appears in shell history or a manifest:

# Write the values to a temp file (NEVER commit this).
cat > /tmp/<app>-secret.env <<'EOF'
KC_API_CLIENT_SECRET=...
EOF

kubectl create secret generic <app>-kc \
  --namespace <app> \
  --from-env-file=/tmp/<app>-secret.env

rm -f /tmp/<app>-secret.env

The keys then map into the Rollout via secretKeyRef:

env:
  - name: KC_API_CLIENT_SECRET
    valueFrom:
      secretKeyRef:
        name: <app>-kc
        key: KC_API_CLIENT_SECRET

Typical out-of-band secrets for an app:

Secret	Holds	Consumed by
<app>-kc	auth client secret(s)	Rollout pods
<app>-pg-backup-creds	Object Storage S3 keys (ACCESS_KEY_ID, ACCESS_SECRET_KEY)	CNPG Cluster

Not auto-generated: <app>-pg-app (the DB_URL) is created by CNPG — see Database & migrations. Only the backup creds are by hand.

Keep a list of every by-hand secret in the platform runbook so a rebuild is “reinstall + resync + recreate these N secrets.” A future fast-follow is declarative secrets (Sealed Secrets / SOPS) so the rebuild is 100% from Git.

Oracle Object Storage key gotcha

For <app>-pg-backup-creds, the Customer Secret Key pair is easy to swap: the access key is clean hex, the secret key contains +/=. Swapping them puts a / into the SigV4 credential and breaks header parsing — backups fail silently. Keep ACCESS_KEY_ID = the hex value.

GHCR: public package vs imagePullSecret

The cluster has to pull ghcr.io/<owner>/<app>. Two options:

1. Make the GHCR package public (simplest). The image pulls anonymously — no pull secret needed. Set the package’s visibility to Public in GitHub. Good for images with no embedded secrets (these are static binaries on distroless, so nothing sensitive is baked in).

2. Keep the package private + add an imagePullSecret. Create a docker-registry secret with a GHCR PAT (scope read:packages) and reference it from the pod spec:

   kubectl create secret docker-registry ghcr-pull \
     --namespace <app> \
     --docker-server=ghcr.io \
     --docker-username=<owner> \
     --docker-password=<ghcr-pat-read-packages>

   spec:
     template:
       spec:
         imagePullSecrets:
           - name: ghcr-pull

This platform takes option 1 — the GHCR package is public, so there’s no pull secret to manage. Choose option 2 only if the image must stay private.

Done

That’s the full playbook. Back to the recipe or the Architecture overview.