Secrets

On Ultron Infra, Git is the source of truth for everything except secret values. Manifests reference Secrets by name, but the values themselves never land in the repo. There are three kinds of secret on the cluster, and one rule that governs all of them.

Three kinds

flowchart TD
  subgraph InGit[in Git: names only]
    Ref[manifests reference<br/>secrets by name]
  end
  CNPG[CNPG operator] -->|auto-generates| G1[(&lt;app&gt;-pg-app<br/>keycloak-test-pg-app)]
  Hand[you, via kubectl] -->|out of band| G2[(&lt;app&gt;-kc<br/>&lt;app&gt;-pg-backup-creds<br/>keycloak-pg-backup-creds)]
  Ref -.secretKeyRef.-> G1
  Ref -.secretKeyRef.-> G2

1. Operator-generated (CNPG)

When you declare a CNPG Cluster, the operator mints the DB credentials and writes the <cluster>-app Secret. You never create these and never see the password — you just reference the generated key:

- name: DB_URL
  valueFrom:
    secretKeyRef:
      name: <app>-pg-app    # created by CNPG, not by you
      key: uri

keycloak-test-pg-app works the same way for the Keycloak database. See Operators.

2. Out-of-band (kubectl-created)

The genuinely secret values that no operator can generate are created by hand with kubectl and are not in Git. These must be recreated on a rebuild:

Secret	Namespace	What it holds
<app>-kc	<app>	Keycloak <app>-api client secret (KC_API_CLIENT_SECRET)
<app>-pg-backup-creds	<app>	Oracle Object Storage S3 keys for API DB backups
keycloak-pg-backup-creds	keycloak	Oracle Object Storage S3 keys for Keycloak DB backups

Workloads consume them by reference — for example the API’s Keycloak client secret:

- name: KC_API_CLIENT_SECRET
  valueFrom:
    secretKeyRef:
      name: <app>-kc
      key: KC_API_CLIENT_SECRET

For the backup credentials, the access key is clean hex; the secret key contains +/= — don’t swap them, or a / ends up in the SigV4 credential and breaks request signing.

The rule: no secret values in Git

Manifests carry names and references, never values. This keeps the private gitops repo safe to read and means a rebuild is “resync from Git + recreate the handful of out-of-band secrets.” The Keycloak realm config is the other non-Git item — it lives in the Keycloak database, which is itself backed up.

Future direction

Recreating secrets by hand is the one manual step left in disaster recovery. The planned fix is declarative encrypted secrets — Sealed Secrets or SOPS — so the (encrypted) values can live in Git and rebuild becomes 100% from Git, with no out-of-band step at all.