feat(migrate): onboarding backfill (repositories + dast_targets -> onboarded_targets) #141

2026-07-10T17:50:12Z

sharang commented

2026-07-10 17:50:12 +00:00

Closes #132. Folds the legacy repositories + dast_targets collections into the unified onboarded_targets, preserving _id so every downstream record keyed by that hex id (findings, sbom, scan_runs, graph, dast_, pentest_) keeps resolving and existing webhook URLs keep working.

Mapping (pure, unit-tested — runs in CI)

repo_to_target: TrackedRepository → OnboardedTarget with a GitRepo artifact (branch, last-scanned watermark, auth); tracker config → scan_config.issue_tracker; webhook/schedule/counts carried over. Type defaults to BackendService (unconfirmed — the classifier refines later).
dast_to_artifact / dast_to_standalone_target: DastTarget → a LiveUrl artifact (crawl config + auth), or a standalone target (keeping the DAST _id) when repo-less.
fold_dast_into_target: append a LiveUrl to a repo-derived target (idempotent), promoting BackendService → WebApp for a browser app.

Orchestration

backfill_onboarded_targets(db, dry_run): id-preserving, idempotent (skips targets that already exist by _id, writes a schema_migrations marker). Folded DAST targets also relink dast_scan_runs/pentest_sessions history (old dast id → unified id), logged to onboarding_migration_log.
revert(db): replays the relink log, drops onboarded_targets + the log, clears the marker. Legacy collections are never deleted, so revert restores the pre-migration state.
CLI: compliance-agent migrate onboarding [--all | --tenant <id>] [--dry-run] [--revert] (--all via DatabasePool::list_tenant_ids).

Testing

5 mapper unit tests (CI).
1 integration test against real Mongo (fold + relink + idempotency + revert, end to end). Not run in CI (which is --lib only); verified locally against the running Mongo — passes.

Rollout

Additive — the legacy collections and code paths are untouched. Once the unified pipeline (#133) reads onboarded_targets, run migrate onboarding --all (optionally --dry-run first).

Refs #118.

Closes #132. Folds the legacy `repositories` + `dast_targets` collections into the unified `onboarded_targets`, **preserving `_id`** so every downstream record keyed by that hex id (findings, sbom, scan_runs, graph, dast_*, pentest_*) keeps resolving and existing webhook URLs keep working. ## Mapping (pure, unit-tested — runs in CI) - `repo_to_target`: `TrackedRepository` → `OnboardedTarget` with a `GitRepo` artifact (branch, last-scanned watermark, auth); tracker config → `scan_config.issue_tracker`; webhook/schedule/counts carried over. Type defaults to `BackendService` (unconfirmed — the classifier refines later). - `dast_to_artifact` / `dast_to_standalone_target`: `DastTarget` → a `LiveUrl` artifact (crawl config + auth), or a standalone target (keeping the DAST `_id`) when repo-less. - `fold_dast_into_target`: append a `LiveUrl` to a repo-derived target (idempotent), promoting `BackendService` → `WebApp` for a browser app. ## Orchestration - `backfill_onboarded_targets(db, dry_run)`: id-preserving, **idempotent** (skips targets that already exist by `_id`, writes a `schema_migrations` marker). Folded DAST targets also **relink** `dast_scan_runs`/`pentest_sessions` history (old dast id → unified id), logged to `onboarding_migration_log`. - `revert(db)`: replays the relink log, drops `onboarded_targets` + the log, clears the marker. Legacy collections are never deleted, so revert restores the pre-migration state. - CLI: `compliance-agent migrate onboarding [--all | --tenant <id>] [--dry-run] [--revert]` (`--all` via `DatabasePool::list_tenant_ids`). ## Testing - 5 mapper unit tests (CI). - 1 integration test against real Mongo (fold + relink + idempotency + revert, end to end). Not run in CI (which is `--lib` only); verified locally against the running Mongo — **passes**. ## Rollout Additive — the legacy collections and code paths are untouched. Once the unified pipeline (#133) reads `onboarded_targets`, run `migrate onboarding --all` (optionally `--dry-run` first). Refs #118.

sharang added this to the Onboarding v1 milestone 2026-07-10 17:50:12 +00:00

sharang added the enhancement onboarding labels 2026-07-10 17:50:12 +00:00

sharang added 2 commits 2026-07-10 17:50:13 +00:00

feat(migrate): id-preserving mapping core for onboarding backfill (#132 part 1) fa3a6b71fb

Pure transforms folding legacy records into the unified model, preserving _id so
every downstream collection keyed by that hex id keeps resolving:
- repo_to_target: TrackedRepository -> OnboardedTarget (GitRepo artifact with
  branch/watermark/auth; tracker config -> scan_config; webhook/schedule/counts).
- dast_to_artifact / dast_to_standalone_target: DastTarget -> LiveUrl artifact
  (crawl config + auth), or a standalone target when repo-less.
- fold_dast_into_target: append a LiveUrl to a repo-derived target (idempotent),
  promoting BackendService -> WebApp for a browser app.

5 unit tests. The per-tenant DB orchestration (idempotent backfill + revert) and
the `migrate onboarding` CLI subcommand are the next increment.

Refs #132.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

feat(migrate): per-tenant onboarding backfill + revert + CLI (#132 part 2)

CI / Check (pull_request) Successful in 8m15s

Details

CI / Detect Changes (pull_request) Has been skipped

Details

CI / Deploy Agent (pull_request) Has been skipped

Details

CI / Deploy Dashboard (pull_request) Has been skipped

Details

CI / Deploy Docs (pull_request) Has been skipped

Details

CI / Deploy MCP (pull_request) Has been skipped

Details

0d83859bcf

Wire the id-preserving mappers into a runnable, idempotent, reversible migration:
- backfill_onboarded_targets(db, dry_run): repositories -> onboarded_targets;
  dast_targets fold into the linked repo target (append LiveUrl, promote type,
  relink dast_scan_runs/pentest_sessions history) or migrate standalone. Skips
  existing (by _id), writes a schema_migrations marker; dry-run computes the
  report without writing. Legacy collections are never deleted.
- revert(db): replay the relink log, drop onboarded_targets + the log, clear the
  marker — restores the pre-migration state.
- CLI: `compliance-agent migrate onboarding [--all | --tenant <id>] [--dry-run]
  [--revert]`, with DatabasePool::list_tenant_ids for --all.
- database.rs: collection_named<T> accessor + list_tenant_ids helper.

Integration test (real Mongo, local-only — CI is --lib) covers fold + relink +
idempotency + revert end to end. 5 mapper unit tests run in CI.

Closes #132.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

sharang merged commit cf4afdda1b into main

2026-07-12 20:11:17 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: sharang/compliance-scanner-agent#141