sharang
6be727d404
3 Orca clusters (breakpilot-edge / breakpilot-control / breakpilot-app), 1 plane each. Single-VM core with Keycloak co-tenant on vm-edge. App cluster gets prod + stage VMs. Manifests reorganized into clusters/<name>/services/; validator now enforces per-cluster node whitelist. Multi-VM rollout gated on legal entity.
86 lines
3.9 KiB
Markdown
86 lines
3.9 KiB
Markdown
# Clusters
|
|
|
|
Each subdirectory under `clusters/` is the manifest set for one Orca cluster.
|
|
At migration time each will become its own Gitea repository under
|
|
`platform/` so the cluster boundary doubles as a repository boundary.
|
|
|
|
| Subdir | Future repo | VMs | Plane |
|
|
|---|---|---|---|
|
|
| [`breakpilot-edge/`](./breakpilot-edge/) | `platform/breakpilot-edge` | `vm-edge` (1) | Identity + Infra |
|
|
| [`breakpilot-control/`](./breakpilot-control/) | `platform/breakpilot-control` | `vm-control` (1) | Control |
|
|
| [`breakpilot-app/`](./breakpilot-app/) | `platform/breakpilot-app` | `vm-app-prod` + `vm-app-stage` (2) | App (was "Data") |
|
|
|
|
## Why one cluster per plane
|
|
|
|
- **Independent failure domain.** A runaway reconcile loop in `breakpilot-app`
|
|
cannot restart Keycloak. An ERPNext migration on `breakpilot-control` does
|
|
not stall portal logins.
|
|
- **Independent backup policy.** Each cluster has its own `[backup]` block
|
|
and its own S3 bucket so RPO targets in `INFRASTRUCTURE.md §6` can be set
|
|
per plane without cross-talk.
|
|
- **Independent IaC.** Each future repo has its own CI, CODEOWNERS, and
|
|
reconcile cadence. Touching identity does not gate a portal hotfix.
|
|
- **Same Orca everywhere.** Per-node ingress (every Orca node binds 80/443
|
|
with its own ACME) means each cluster's services get A records pointing
|
|
at their own VM. No central proxy SPOF.
|
|
|
|
## Why this is still a single repo today
|
|
|
|
We hold the three-cluster design in this one staging repo until the legal
|
|
entity is established and we can sign SysEleven / Hetzner business
|
|
contracts to provision actual VMs. Until then, everything runs on the
|
|
existing single VM under `orca-infra` and these manifests are
|
|
design-only — `make validate` keeps them honest, `make apply` is a no-op
|
|
for now.
|
|
|
|
When the entity lands the migration is mechanical: split each subdir into
|
|
its own Gitea repo, point its `cluster.toml` at a real VM, run
|
|
`orca deploy`.
|
|
|
|
## VM topology after migration
|
|
|
|
```
|
|
breakpilot-edge cluster → vm-edge (Keycloak + Gitea + Infisical + PowerDNS + Orca-Proxy)
|
|
breakpilot-control cluster → vm-control (Portal + Tenant-Registry + ERPNext + MariaDB + Stalwart)
|
|
breakpilot-app cluster → vm-app-prod (CERTifAI + compliance-* + Mongo + MinIO + Qdrant + LiteLLM)
|
|
\→ vm-app-stage (slim parallel set for staging)
|
|
```
|
|
|
|
`vm-app-stage` lives in the same Orca cluster as `vm-app-prod` and points
|
|
at PROD `breakpilot-edge` Keycloak + PROD `breakpilot-control`
|
|
tenant-registry, isolated by `tenant.kind = "stage"` per
|
|
`INFRASTRUCTURE.md §5`. No duplicated identity, no duplicated control plane.
|
|
|
|
## Identity is part of the core (single VM)
|
|
|
|
The May 18 `INFRASTRUCTURE.md` had Identity (Keycloak) co-tenant with Infra
|
|
on `vm-edge`, with JVM heap pinned at 1.5 GB so it cannot starve PowerDNS
|
|
/ Infisical. That decision stands — Identity lives in `breakpilot-edge` on
|
|
the same VM as the rest of infra. We will revisit splitting Keycloak onto
|
|
its own VM only if the JVM vs. infra fight actually shows up in alerts.
|
|
|
|
## Repurposing the existing Keycloak
|
|
|
|
`benjamin_boenisch/certifai` already runs a Keycloak instance. Plan is to
|
|
**repurpose that instance into `breakpilot-edge/services/keycloak.toml`**
|
|
rather than stand up a fresh one — same realm exports, same client
|
|
configurations, same user database. Migration is a hostname swap and a
|
|
realm export/import, not a re-bootstrap.
|
|
|
|
## Adding a new service
|
|
|
|
Pick the plane, then drop the manifest into the right `services/` dir:
|
|
|
|
```bash
|
|
clusters/breakpilot-<plane>/services/<service>.toml
|
|
# breakpilot-app extras:
|
|
clusters/breakpilot-app/services/{prod,stage}/<service>.toml
|
|
```
|
|
|
|
`scripts/validate.sh` enforces:
|
|
- `placement.node` must match one of the cluster's declared VMs
|
|
(`vm-edge` / `vm-control` / `vm-app-prod` / `vm-app-stage`)
|
|
- `resources.memory` is mandatory (§8 rule 5)
|
|
- no nested `depends_on` / `extra_ports` / `cmd` / `mounts` under
|
|
`[service.placement|resources|env|volume]`
|