docs(impl-plan): reflect 2026-06-30 cluster split decision #6

Merged
sharang merged 1 commits from refactor/cluster-split-impl-plan into main 2026-06-30 21:16:11 +00:00
+88 -54
View File
@@ -19,13 +19,21 @@ This is the build plan for an AI coding agent (Claude Code, executing PRs agains
## 1. Cross-cutting conventions (apply to every PR in every repo) ## 1. Cross-cutting conventions (apply to every PR in every repo)
### 1.1 Repo strategy ### 1.1 Repo strategy
Polyrepo under a new Gitea org `gitea.meghsakha.com/platform/`. One repo per deployable unit. Existing product repos stay where they are. Polyrepo under the Gitea org `gitea.meghsakha.com/platform/`. One repo per deployable unit. Existing product repos stay where they are.
**Repos to create:** **Infrastructure repos.** The IaC is split by Orca cluster — one cluster per plane, one repo per cluster — per the 2026-06-30 cluster-split decision (see [Plane model memory](../../../../home/sharang/.claude/projects/-home-sharang-workspace/memory/project_breakpilot_plane_model.md) and `INFRASTRUCTURE.md §6`). Until the legal entity is established and SysEleven / Hetzner business contracts can be signed, the cluster manifests live as design-only staging directories under `platform/orca-platform/clusters/`. At migration time each `clusters/<name>/` subdir becomes its own repo:
| Repo | Purpose | Created in |
|---|---|---|
| `platform/orca-platform` | IaC staging — cluster.toml templates, per-cluster service manifests under `clusters/breakpilot-{edge,control,app}/`, overlays, DNS zones, dev-compose | M1.1 |
| `platform/breakpilot-edge` | Identity + Infra cluster (vm-edge): Keycloak, Gitea, Infisical, PowerDNS, Orca-Proxy | M1.2 (split from `orca-platform/clusters/breakpilot-edge/`) |
| `platform/breakpilot-control` | Control cluster (vm-control): portal, tenant-registry, ERPNext, MariaDB, Stalwart, Frappe HD | M1.2 (split from `orca-platform/clusters/breakpilot-control/`) |
| `platform/breakpilot-app` | App cluster (vm-app-prod + vm-app-stage): CERTifAI, compliance-*, Mongo, MinIO, Qdrant, LiteLLM, pg-app | M1.2 (split from `orca-platform/clusters/breakpilot-app/`) |
**Product / business repos:**
| Repo | Purpose | Created in | | Repo | Purpose | Created in |
|---|---|---| |---|---|---|
| `platform/orca-platform` | IaC for VMs, Orca manifests, DNS, TLS, backups | M1.1 |
| `platform/tenant-registry` | Go service: tenant glue, audit, API keys | M4.1 | | `platform/tenant-registry` | Go service: tenant glue, audit, API keys | M4.1 |
| `platform/portal` | Next.js 15: customer area + backstage | M5.1 | | `platform/portal` | Next.js 15: customer area + backstage | M5.1 |
| `platform/docs` | Architecture, integration spec, this plan, runbooks | M0.1 | | `platform/docs` | Architecture, integration spec, this plan, runbooks | M0.1 |
@@ -110,13 +118,21 @@ Per language defaults:
- **Database migrations** are forward-only and run as an init container before the service starts. Migrations that delete columns require two releases (1: stop writing, 2: drop). - **Database migrations** are forward-only and run as an init container before the service starts. Migrations that delete columns require two releases (1: stop writing, 2: drop).
### 1.8 Environments ### 1.8 Environments
Three Orca clusters, all on the same hardware until volume justifies separation: Three Orca clusters, **one per plane** (not one per env). Stage and prod live in the same `breakpilot-app` cluster on different VMs:
| Env | Cluster name | Purpose | Data | Auto-deploy? | | Cluster | VMs | Plane | Envs hosted |
|---|---|---|---|---| |---|---|---|---|
| dev | local | Developer machine, docker-compose | fixtures | n/a | | `breakpilot-edge` | `vm-edge` (1) | Identity + Infra | shared by stage + prod (single Keycloak realm, `tenant.kind` differentiates) |
| stage | `orca-stage` | Pre-prod validation | seeded demo + synthetic customers | yes (on merge to main) | | `breakpilot-control` | `vm-control` (1) | Control | shared by stage + prod (single tenant-registry, `tenant.kind` differentiates) |
| prod | `orca-prod` | Live customer traffic | real | tag + gate | | `breakpilot-app` | `vm-app-prod`, `vm-app-stage` (2) | App | prod on `vm-app-prod`, stage on `vm-app-stage` |
| Env | Where it runs | Data | Auto-deploy? |
|---|---|---|---|
| dev | Developer machine, docker-compose (`platform/orca-platform/dev/`) | fixtures | n/a |
| stage | `vm-app-stage` in `breakpilot-app`; calls prod KC + prod tenant-registry under `tenant.kind = "stage"` | seeded demo + synthetic customers | yes (on merge to main) |
| prod | `vm-app-prod` in `breakpilot-app` + `vm-control` + `vm-edge` | real | tag + gate |
Identity and Control are NOT duplicated for stage. Stage workloads authenticate against the prod Keycloak realm with `tenant.kind = "stage"` and read tenant config from the prod `tenant-registry` (read-only for stage tenants) per `INFRASTRUCTURE.md §5`.
Domain pattern: Domain pattern:
- dev: `*.localhost` (mkcert) - dev: `*.localhost` (mkcert)
@@ -157,6 +173,7 @@ Every place where a future flag would gate behaviour MUST flow through a single
- **Tests:** n/a - **Tests:** n/a
- **Gate:** standard - **Gate:** standard
- **Effort:** S - **Effort:** S
- **Note:** the three cluster repos (`platform/breakpilot-edge`, `platform/breakpilot-control`, `platform/breakpilot-app`) are NOT created at M0.1 — their service manifests live as design-only directories under `platform/orca-platform/clusters/` until M1.2 splits them.
### M0.2 — CI templates + branch protection ### M0.2 — CI templates + branch protection
- **Depends on:** M0.1 - **Depends on:** M0.1
@@ -169,7 +186,7 @@ Every place where a future flag would gate behaviour MUST flow through a single
### M0.3 — Self-hosted DNS + wildcard TLS ### M0.3 — Self-hosted DNS + wildcard TLS
- **Depends on:** M1.2 (vm-edge must exist before PowerDNS lands) - **Depends on:** M1.2 (vm-edge must exist before PowerDNS lands)
- **Repos:** `platform/orca-platform` - **Repos:** `platform/breakpilot-edge`
- **Deliverables:** - **Deliverables:**
- **PowerDNS Authoritative** on `vm-edge` (Orca-managed). PostgreSQL backend on same VM (small; ~100 records). - **PowerDNS Authoritative** on `vm-edge` (Orca-managed). PostgreSQL backend on same VM (small; ~100 records).
- At the registrar (Benjamin's account): set `ns1.breakpilot.com` and `ns2.breakpilot.com` glue records pointing at vm-edge public IP; delegate the domain to those NS. - At the registrar (Benjamin's account): set `ns1.breakpilot.com` and `ns2.breakpilot.com` glue records pointing at vm-edge public IP; delegate the domain to those NS.
@@ -182,53 +199,68 @@ Every place where a future flag would gate behaviour MUST flow through a single
- **Gate:** standard + manual DNS-delegation check by both founders (irreversible from registrar side without 2448h propagation) - **Gate:** standard + manual DNS-delegation check by both founders (irreversible from registrar side without 2448h propagation)
- **Effort:** M (was S — registrar delegation + PowerDNS adds setup time vs. Cloudflare) - **Effort:** M (was S — registrar delegation + PowerDNS adds setup time vs. Cloudflare)
### M1.1 — `orca-platform` repo (IaC) ### M1.1 — `orca-platform` repo (IaC staging)
- **Depends on:** M0.1, M0.2 - **Depends on:** M0.1, M0.2
- **Repos:** `platform/orca-platform` - **Repos:** `platform/orca-platform`
- **Deliverables:** directory layout per `INFRASTRUCTURE.md`; one Orca manifest per VM × service; per-env overlays (`overlays/dev`, `overlays/stage`, `overlays/prod`); a `Makefile` with `make plan` / `make apply` per env. - **Deliverables:** per-cluster manifest layout under `clusters/breakpilot-{edge,control,app}/services/`; per-env overlays (`overlays/dev`, `overlays/stage`, `overlays/prod`); `cluster.toml.tmpl` template parameterized by cluster; a `Makefile` with `make plan` / `make apply` per env; `scripts/validate.sh` enforces a per-cluster `placement.node` whitelist (`breakpilot-edge → vm-edge`, `breakpilot-control → vm-control`, `breakpilot-app → {vm-app-prod, vm-app-stage}`).
- **Acceptance:** `make plan ENV=stage` produces a no-op diff once applied. - **Acceptance:** `make plan ENV=stage` and `make plan ENV=prod` both resolve cleanly; `make validate` passes the per-cluster node check on all manifests.
- **Tests:** `orca validate` runs in CI; PRs that break a manifest fail. - **Tests:** `make validate` runs in CI; PRs that break a manifest or cross-place a service into the wrong cluster's node fail.
- **Gate:** standard - **Gate:** standard
- **Effort:** M - **Effort:** M
- **Status (2026-06-30):** layout landed; cluster split refactor merged via PR #6 (`6be727d4`). Manifests under `clusters/breakpilot-*/services/` are stubs awaiting per-milestone fill-in. `make apply` is a no-op until M1.2.
### M1.2 — Provision VMs (locked topology) ### M1.2 — Provision VMs + split cluster repos (locked topology)
- **Depends on:** M1.1 (Orca manifest layout) - **Depends on:** M1.1 (manifest layout exists); **legal entity established** (gating SysEleven / Hetzner business contracts; no commercial activity before this).
- **Repos:** `platform/orca-platform` - **Repos:** `platform/orca-platform` (source of split), `platform/breakpilot-edge`, `platform/breakpilot-control`, `platform/breakpilot-app` (created in this milestone).
- **Deliverables:** the **4 VMs** from `INFRASTRUCTURE.md §1` provisioned on SysEleven (DUS2): - **Deliverables:** **4 VMs across 3 clusters**, one cluster per plane (per the 2026-06-30 cluster-split decision):
- **stage** (m2.small, public IP) — runs app-plane code only, calls prod KC + Stalwart
- **vm-edge** (m2.small, public IP) — Identity + Infra planes (orca-proxy, PowerDNS, Keycloak, pg-keycloak, Infisical, pg-infisical, Gitea) | Cluster | VM | Flavor | Plane | Services |
- **vm-control** (m2.medium) — Control plane (portal, tenant-registry, ERPNext, Frappe HD, MariaDB, Stalwart) |---|---|---|---|---|
- **vm-data** (m2.medium) — Data plane (CERTifAI, MongoDB, LiteLLM, compliance ×3, pg-app, Qdrant, MinIO) | `breakpilot-edge` | `vm-edge` | m2.small, public IP | Identity + Infra | orca-proxy, PowerDNS, Keycloak (repurposed from CERTifAI), pg-keycloak, Infisical, pg-infisical, redis-infisical, Gitea |
- Private network 10.0.0.0/16 between all four. Public ingress only via vm-edge (and stage's own IP for tester access). | `breakpilot-control` | `vm-control` | m2.medium | Control | portal, tenant-registry, ERPNext, Frappe HD, MariaDB, redis-erpnext, Stalwart |
- SSH disabled; only `orca exec` for shell access. | `breakpilot-app` | `vm-app-prod` | m2.medium | App (prod) | CERTifAI, compliance ×3, MongoDB, LiteLLM, pg-app, Qdrant, MinIO |
- **Acceptance:** every VM reachable from Orca control plane; private-network connectivity verified; resource limits per service set in manifest per `INFRASTRUCTURE.md §6` co-tenant notes. | `breakpilot-app` | `vm-app-stage` | m2.small, public IP | App (stage) | slim parallel set for staging; calls prod KC + prod tenant-registry under `tenant.kind = "stage"` |
- **Tests:** cold-start sequence from `INFRASTRUCTURE.md §10 Scenario F` runs successfully on stage VMs.
- **Gate:** standard + manual sign-off (touches infra spend and 36M commitment decision) - **Split the cluster repos:** for each cluster, lift `platform/orca-platform/clusters/breakpilot-<name>/` into its own Gitea repo `platform/breakpilot-<name>` with the §1.2 scaffolding + its own `cluster.toml` + its own `services/` tree. Empty out (or stub) the original `clusters/breakpilot-<name>/` in `orca-platform` and leave a pointer README.
- **Effort:** M - **Repurpose the existing Keycloak.** The CERTifAI Keycloak realm export + user database moves to `vm-edge` rather than standing up a fresh instance — a hostname swap and realm export/import, not a re-bootstrap.
- **Cost impact:** see COST_PLAN.md §3. Initial run: ~€552/mo On-Demand, dropping to ~€310/mo after 36M-upfront commit in Month 4. - **Private network** 10.0.0.0/16 between all four VMs.
- **Public ingress** per Orca's per-node-ingress model: every VM binds 80/443 and runs its own Let's Encrypt ACME; A records for a service point at the agent IP that runs the container, not at a master.
- **SSH disabled**; only `orca exec` for shell access.
- **Acceptance:**
- Every VM is reachable from its cluster's Orca master; private-network connectivity verified.
- Resource limits per service set in manifest per `INFRASTRUCTURE.md §6` co-tenant notes.
- `make validate` (in `orca-platform`) and the new per-cluster repo CIs all pass.
- Cold-start sequence from `INFRASTRUCTURE.md §10 Scenario F` runs successfully on `vm-app-stage`.
- **Tests:** cold-start drill on `vm-app-stage`; cross-cluster reachability matrix; KC realm-import smoke (stage tenant logs in).
- **Gate:** standard + manual sign-off (touches infra spend and 36M commitment decision) + legal-entity confirmation before any contract signing.
- **Effort:** L (was M — adds the 3-repo split + KC realm migration on top of provisioning).
- **Cost impact:** see COST_PLAN.md §3. Initial run: ~€552/mo On-Demand, dropping to ~€310/mo after 36M-upfront commit in Month 4. `vm-app-stage` is the cheapest VM in the topology so adding it doesn't materially shift the bill.
**Pre-M1.2 reality (today, single-VM era):** until this milestone lands, everything runs on the existing single VM (`46.225.100.82`) via `~/workspace/orca-infra`. M7.x work (tenant-registry-aware compliance-agent, per-tenant MCP) and other product work is meaningful on the single-VM setup and ports cleanly into `breakpilot-app/services/prod/` at migration time.
### M1.3 — Backups, monitoring, on-call ### M1.3 — Backups, monitoring, on-call
- **Depends on:** M1.2 - **Depends on:** M1.2
- **Repos:** `platform/orca-platform` - **Repos:** `platform/breakpilot-edge`, `platform/breakpilot-control`, `platform/breakpilot-app` (each cluster owns its own `[backup]` block + restore script per `INFRASTRUCTURE.md §6`)
- **Deliverables:** backup cron per VM per `INFRASTRUCTURE.md §3` (Postgres pg_dump, MinIO bucket replication); SigNoz OTel collector running on every VM; alert routing to `oncall@breakpilot.com`; restore runbook in `platform/docs/runbooks/restore.md`. - **Deliverables:** backup cron per VM per `INFRASTRUCTURE.md §3` (Postgres pg_dump on `pg-keycloak` / `pg-app` / `mariadb`, MinIO bucket replication); SigNoz OTel collector running on every VM; alert routing to `oncall@breakpilot.com`; restore runbook in `platform/docs/runbooks/restore.md`. Each cluster repo has its own S3 bucket per `INFRASTRUCTURE.md §6` so a runaway backup on `breakpilot-app` cannot fill the bucket `breakpilot-edge` writes to.
- **Acceptance:** restore drill on stage succeeds (script in `platform/orca-platform/scripts/restore-drill.sh`); SigNoz shows traces from a synthetic request. - **Acceptance:** restore drill on `vm-app-stage` succeeds (script in `breakpilot-app/scripts/restore-drill.sh`); SigNoz shows traces from a synthetic request through `vm-app-stage``vm-control``vm-edge`.
- **Tests:** disaster-recovery exercise per failure scenario in `INFRASTRUCTURE.md §10` — at least Scenarios A, B, F validated on stage. - **Tests:** disaster-recovery exercise per failure scenario in `INFRASTRUCTURE.md §10` — at least Scenarios A, B, F validated on stage.
- **Gate:** standard + manual sign-off - **Gate:** standard + manual sign-off
- **Effort:** L - **Effort:** L
### M2.1 — Keycloak deployment ### M2.1 — Keycloak deployment (repurpose from CERTifAI)
- **Depends on:** M1.2, M1.3 - **Depends on:** M1.2, M1.3
- **Repos:** `platform/orca-platform` - **Repos:** `platform/breakpilot-edge`
- **Deliverables:** Keycloak 26 on `vm-identity`, Postgres backing store on `vm-control`, exposed at `auth.breakpilot.com` and `auth.stage.breakpilot.com`. Realm import file in `orca-platform/keycloak/realm-export.json` (committed, source-of-truth). - **Deliverables:** Keycloak 26 on **`vm-edge`** (co-tenant with the rest of infra; JVM heap pinned `-Xmx1500m` so it cannot starve PowerDNS / Infisical, per `INFRASTRUCTURE.md §6`). Postgres backing store (`pg-keycloak`) also on `vm-edge`. Exposed at `auth.breakpilot.com`; stage tenants authenticate against the same instance under `tenant.kind = "stage"` (no separate stage realm). **The CERTifAI Keycloak instance is repurposed here** — its realm export + user database moves to `vm-edge` rather than standing up a fresh instance. Realm import file in `breakpilot-edge/services/keycloak/realm-export.json` (committed, source-of-truth).
- **Acceptance:** master admin login works; realm `breakpilot-prod` exists in both envs. - **Acceptance:** master admin login works; realm `breakpilot-prod` exists; the realm carries the migrated CERTifAI users; a stage tenant test login succeeds with `tenant.kind = "stage"` in the JWT.
- **Tests:** automated realm-state diff in CI (`kcadm` against checked-in export). - **Tests:** automated realm-state diff in CI (`kcadm` against checked-in export); side-by-side comparison of pre-migration and post-migration realm exports flags any silent drift.
- **Gate:** standard + security checklist - **Gate:** standard + security checklist + manual sign-off on the realm-import diff (irreversible without restoring from KC backup).
- **Effort:** M - **Effort:** M
- **Note:** the escape hatch — if the JVM/PowerDNS fight shows up in alerts after launch — is to flip Keycloak's `placement.node` to a new `vm-edge-identity` VM inside the same `breakpilot-edge` cluster. No schema migration, just a container move.
### M2.2 — Realm configuration: roles + protocol mappers + Organizations ### M2.2 — Realm configuration: roles + protocol mappers + Organizations
- **Depends on:** M2.1 - **Depends on:** M2.1
- **Repos:** `platform/orca-platform` (realm config) - **Repos:** `platform/breakpilot-edge` (realm config)
- **Deliverables:** Organizations feature enabled; realm roles `BREAKPILOT_ADMIN`, `SUPPORT_ENGINEER`, `SALES_REP`; org roles `IT_ADMIN`, `CXO`, `FINANCE`, `LEGAL`, `USER`; protocol mapper that calls Tenant Registry at token issuance for `products`, `plan`, `tenant_status` claims; SALES_REP guardrail policy (token only issuable with `org_id = demo`). - **Deliverables:** Organizations feature enabled; realm roles `BREAKPILOT_ADMIN`, `SUPPORT_ENGINEER`, `SALES_REP`; org roles `IT_ADMIN`, `CXO`, `FINANCE`, `LEGAL`, `USER`; protocol mapper that calls Tenant Registry at token issuance for `products`, `plan`, `tenant_status`, `tenant_kind` claims; SALES_REP guardrail policy (token only issuable with `org_id = demo`).
- **Acceptance:** a test user gets the expected JWT claims; a SALES_REP user cannot get a JWT for a non-demo org (verified by integration test). - **Acceptance:** a test user gets the expected JWT claims; a SALES_REP user cannot get a JWT for a non-demo org (verified by integration test).
- **Tests:** Keycloak integration suite in `platform/tenant-registry/test/keycloak_test.go`. - **Tests:** Keycloak integration suite in `platform/tenant-registry/test/keycloak_test.go`.
- **Gate:** standard + security checklist - **Gate:** standard + security checklist
@@ -236,16 +268,17 @@ Every place where a future flag would gate behaviour MUST flow through a single
### M3.1 — Infisical ### M3.1 — Infisical
- **Depends on:** M1.2 - **Depends on:** M1.2
- **Repos:** `platform/orca-platform` - **Repos:** `platform/breakpilot-edge`
- **Deliverables:** Infisical on `vm-secrets`, machine identity per service, secret paths laid out per `PRODUCT_INTEGRATION_SPEC.md §9.4`. - **Deliverables:** Infisical on **`vm-edge`** (co-tenant with Keycloak, PowerDNS, Gitea), machine identity per service, secret paths laid out per `PRODUCT_INTEGRATION_SPEC.md §9.4`.
- **Acceptance:** a stub service can read its secrets at startup; rotating a secret in Infisical UI is picked up on next pod start. - **Acceptance:** a stub service can read its secrets at startup; rotating a secret in Infisical UI is picked up on next pod start.
- **Tests:** smoke test container reads secrets. - **Tests:** smoke test container reads secrets.
- **Gate:** standard + security checklist - **Gate:** standard + security checklist
- **Effort:** S - **Effort:** S
- **Bootstrap exception (per `INFRASTRUCTURE.md §8 rule 3`):** Keycloak's `KC_DB_URL` lives in Orca env (not Infisical) because both run on `vm-edge` and we'd otherwise have a circular bootstrap dependency.
### M3.2 — Stalwart transactional email ### M3.2 — Stalwart transactional email
- **Depends on:** M0.3 (needs DNS records under our control), M3.1 - **Depends on:** M0.3 (needs DNS records under our control), M3.1
- **Repos:** `platform/orca-platform` - **Repos:** `platform/breakpilot-control`
- **Deliverables:** - **Deliverables:**
- **Stalwart** on `vm-control` (Orca-managed); reachable at `mail.breakpilot.com`. - **Stalwart** on `vm-control` (Orca-managed); reachable at `mail.breakpilot.com`.
- DNS records added to the zone in M0.3: `mail` A record, MX → mail, SPF (`v=spf1 mx -all`), DKIM (Stalwart-generated public key), DMARC (`p=quarantine; rua=mailto:dmarc@breakpilot.com`), reverse DNS (PTR) configured at the cloud provider for the vm-control public IP — coordinate with vm-edge since outbound mail must egress from a host with a clean PTR. - DNS records added to the zone in M0.3: `mail` A record, MX → mail, SPF (`v=spf1 mx -all`), DKIM (Stalwart-generated public key), DMARC (`p=quarantine; rua=mailto:dmarc@breakpilot.com`), reverse DNS (PTR) configured at the cloud provider for the vm-control public IP — coordinate with vm-edge since outbound mail must egress from a host with a clean PTR.
@@ -259,9 +292,10 @@ Every place where a future flag would gate behaviour MUST flow through a single
- **Effort:** L (deliverability tuning is the long tail) - **Effort:** L (deliverability tuning is the long tail)
**Phase 0 exit criteria:** **Phase 0 exit criteria:**
- Stage cluster boots cold from cron-driven nightly stop/start using only `INFRASTRUCTURE.md §5` ordering. - `vm-app-stage` boots cold from cron-driven nightly stop/start using only `INFRASTRUCTURE.md §5` ordering; `breakpilot-edge` and `breakpilot-control` clusters remain warm throughout (no nightly cycling for identity / control).
- A synthetic HTTPS request to `https://hello.stage.breakpilot.com` reaches a stub container. - A synthetic HTTPS request to `https://hello.stage.breakpilot.com` reaches a stub container on `vm-app-stage`.
- Restore drill on stage Postgres succeeds end-to-end. - Restore drill on `vm-app-prod` Postgres (`pg-app`) and `vm-control` Postgres (`pg-keycloak`, `pg-mariadb`) both succeed end-to-end.
- All three cluster repos (`platform/breakpilot-edge`, `platform/breakpilot-control`, `platform/breakpilot-app`) have green CI on main and a successful `make validate`.
--- ---
@@ -390,7 +424,7 @@ Every place where a future flag would gate behaviour MUST flow through a single
### M8.1 — ERPNext deployment ### M8.1 — ERPNext deployment
- **Depends on:** M1.2, M2.1 - **Depends on:** M1.2, M2.1
- **Repos:** `platform/orca-platform` - **Repos:** `platform/breakpilot-control`
- **Deliverables:** Frappe + ERPNext on `vm-control` (separate Postgres database from tenant_registry — see `INFRASTRUCTURE.md` RISK-1); reached at `erp.breakpilot.com`; Keycloak OIDC; IP-restricted at Orca-Proxy. - **Deliverables:** Frappe + ERPNext on `vm-control` (separate Postgres database from tenant_registry — see `INFRASTRUCTURE.md` RISK-1); reached at `erp.breakpilot.com`; Keycloak OIDC; IP-restricted at Orca-Proxy.
- **Acceptance:** us login works; a Customer record can be created manually. - **Acceptance:** us login works; a Customer record can be created manually.
- **Tests:** smoke test for OIDC; backup of Frappe filestore validated. - **Tests:** smoke test for OIDC; backup of Frappe filestore validated.
@@ -399,7 +433,7 @@ Every place where a future flag would gate behaviour MUST flow through a single
### M8.2 — ERPNext customization ### M8.2 — ERPNext customization
- **Depends on:** M8.1 - **Depends on:** M8.1
- **Repos:** `platform/orca-platform/erpnext-app/` - **Repos:** `platform/breakpilot-control` (`erpnext-app/` subdir under the cluster repo)
- **Deliverables:** custom Frappe app with: `tenant_id` field on `Customer`; `sales_owner` field on `Lead`; server scripts for the Sales Order → Tenant Registry webhook; `Cancel` workflow that calls Tenant Registry `/cancel`. - **Deliverables:** custom Frappe app with: `tenant_id` field on `Customer`; `sales_owner` field on `Lead`; server scripts for the Sales Order → Tenant Registry webhook; `Cancel` workflow that calls Tenant Registry `/cancel`.
- **Acceptance:** submitting a Sales Order in ERPNext triggers a tenant activation in stage Tenant Registry. - **Acceptance:** submitting a Sales Order in ERPNext triggers a tenant activation in stage Tenant Registry.
- **Tests:** server-script unit tests (Frappe test harness); integration test exercises the full webhook. - **Tests:** server-script unit tests (Frappe test harness); integration test exercises the full webhook.
@@ -424,7 +458,7 @@ Every place where a future flag would gate behaviour MUST flow through a single
### M9.1 — Frappe Helpdesk ### M9.1 — Frappe Helpdesk
- **Depends on:** M8.1 - **Depends on:** M8.1
- **Repos:** `platform/orca-platform` - **Repos:** `platform/breakpilot-control`
- **Deliverables:** Frappe HD on the same Frappe bench; customer portal embedded at `/[slug]/support/`. - **Deliverables:** Frappe HD on the same Frappe bench; customer portal embedded at `/[slug]/support/`.
- **Acceptance:** a customer user can submit a ticket; we receive it. - **Acceptance:** a customer user can submit a ticket; we receive it.
- **Tests:** Playwright spec for ticket submission. - **Tests:** Playwright spec for ticket submission.
@@ -433,7 +467,7 @@ Every place where a future flag would gate behaviour MUST flow through a single
### M9.2 — HD → Gitea escalation ### M9.2 — HD → Gitea escalation
- **Depends on:** M9.1 - **Depends on:** M9.1
- **Repos:** `platform/orca-platform/erpnext-app/` - **Repos:** `platform/breakpilot-control` (`erpnext-app/` subdir)
- **Deliverables:** server script that on a `Ticket: Escalate to Engineering` action creates a Gitea issue in the matching repo via Gitea REST API; reverse webhook from Gitea on issue close marks ticket resolved. - **Deliverables:** server script that on a `Ticket: Escalate to Engineering` action creates a Gitea issue in the matching repo via Gitea REST API; reverse webhook from Gitea on issue close marks ticket resolved.
- **Acceptance:** the round-trip works for a test ticket on stage. - **Acceptance:** the round-trip works for a test ticket on stage.
- **Tests:** integration test against stage Gitea. - **Tests:** integration test against stage Gitea.
@@ -598,7 +632,7 @@ These ship only when a paying customer requires them.
### M18.1 — Custom domains ### M18.1 — Custom domains
- **Depends on:** M0.3, M10.1 - **Depends on:** M0.3, M10.1
- **Repos:** `platform/orca-platform`, `platform/portal` - **Repos:** `platform/breakpilot-edge` (Orca-Proxy ACME), `platform/portal` (UI)
- **Deliverables:** ACME on-demand TLS in Orca-Proxy; portal UI for customer to add domain; CNAME verification. - **Deliverables:** ACME on-demand TLS in Orca-Proxy; portal UI for customer to add domain; CNAME verification.
- **Acceptance:** `compliance.acme.com` resolves and renders the Acme portal. - **Acceptance:** `compliance.acme.com` resolves and renders the Acme portal.
- **Tests:** integration test with a synthetic domain. - **Tests:** integration test with a synthetic domain.
@@ -607,7 +641,7 @@ These ship only when a paying customer requires them.
### M18.2 — Physical data isolation ### M18.2 — Physical data isolation
- **Depends on:** M4.1, M6.1, M7.1 - **Depends on:** M4.1, M6.1, M7.1
- **Repos:** all data-plane products + `tenant-registry` - **Repos:** all app-plane products + `platform/tenant-registry`
- **Deliverables:** option per tenant for a dedicated Postgres / Mongo schema or database; provisioning automation; migration path from logical → physical. - **Deliverables:** option per tenant for a dedicated Postgres / Mongo schema or database; provisioning automation; migration path from logical → physical.
- **Acceptance:** an enterprise tenant runs on a dedicated schema; cross-tenant queries are physically impossible. - **Acceptance:** an enterprise tenant runs on a dedicated schema; cross-tenant queries are physically impossible.
- **Tests:** isolation enforcement test. - **Tests:** isolation enforcement test.
@@ -645,7 +679,7 @@ When starting work, the first sequence of PRs should be:
2. **PR-2 to PR-7** (M0.1 continued): Bootstrap each of the other five repos with §1.2 scaffolding. Land in parallel. 2. **PR-2 to PR-7** (M0.1 continued): Bootstrap each of the other five repos with §1.2 scaffolding. Land in parallel.
3. **PR-8** (M0.2): CI templates + branch protection per repo. 3. **PR-8** (M0.2): CI templates + branch protection per repo.
4. **PR-9** (M1.1): `orca-platform` directory layout + first stub manifest. 4. **PR-9** (M1.1): `orca-platform` directory layout + first stub manifest.
5. **PR-10** (M1.2): VM provisioning (vm-edge, vm-identity, vm-secrets, vm-control first — DNS and Keycloak depend on these). 5. **PR-10** (M1.2): VM provisioning + cluster-repo split. Per the 2026-06-30 decision: 4 VMs across 3 clusters (`vm-edge` in `breakpilot-edge`, `vm-control` in `breakpilot-control`, `vm-app-prod` + `vm-app-stage` in `breakpilot-app`). Spin up `vm-edge` first since DNS, Keycloak, and Infisical all live there. **Gated on legal entity** (cannot sign SysEleven / Hetzner business contracts before).
6. **PR-11** (M0.3): PowerDNS on vm-edge + zone file + registrar NS delegation + wildcard TLS via Let's Encrypt DNS-01. 6. **PR-11** (M0.3): PowerDNS on vm-edge + zone file + registrar NS delegation + wildcard TLS via Let's Encrypt DNS-01.
After PR-11, the dependency graph fans out and parallel work begins. After PR-11, the dependency graph fans out and parallel work begins.
@@ -690,7 +724,7 @@ That's 18 milestones. With one full-time agent and standard human review pacing,
**Parallelism opportunities:** **Parallelism opportunities:**
- M6.x and M7.x can run fully in parallel (different repos, different stacks). - M6.x and M7.x can run fully in parallel (different repos, different stacks).
- M8.x is independent of all data-plane work once M2.2 is done. - M8.x is independent of all app-plane work once M2.2 is done.
- M15.x can begin as soon as M10.1 lands. - M15.x can begin as soon as M10.1 lands.
--- ---