Commit Graph

14 Commits

Author SHA1 Message Date
Sharang Parnerkar
482e8574ad refactor(backend/db): split repository.py + isms_repository.py per-aggregate
Phase 1 Step 5 of PHASE1_RUNBOOK.md.

compliance/db/repository.py (1547 LOC) decomposed into seven sibling
per-aggregate repository modules:

  regulation_repository.py     (268) — Regulation + Requirement
  control_repository.py        (291) — Control + ControlMapping
  evidence_repository.py       (143)
  risk_repository.py           (148)
  audit_export_repository.py   (110)
  service_module_repository.py (247)
  audit_session_repository.py  (478) — AuditSession + AuditSignOff

compliance/db/isms_repository.py (838 LOC) decomposed into two
sub-aggregate modules mirroring the models split:

  isms_governance_repository.py (354) — Scope, Policy, Objective, SoA
  isms_audit_repository.py      (499) — Finding, CAPA, Review, Internal Audit,
                                         Trail, Readiness

Both original files become thin re-export shims (37 and 25 LOC
respectively) so every existing import continues to work unchanged.
New code SHOULD import from the aggregate module directly.

All new sibling files under the 500-line hard cap; largest is
isms_audit_repository.py at 499 (on the edge; when Phase 1 Step 4
router->service extraction lands, the audit_session repo may split
further if growth exceeds 500).

Verified:
  - 173/173 pytest compliance/tests/ tests/contracts/ pass
  - OpenAPI 360 paths / 484 operations unchanged
  - All repo files under 500 LOC

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 18:08:39 +02:00
Sharang Parnerkar
3320ef94fc refactor: phase 0 guardrails + phase 1 step 2 (models.py split)
Squash of branch refactor/phase0-guardrails-and-models-split — 4 commits,
81 files, 173/173 pytest green, OpenAPI contract preserved (360 paths /
484 operations).

## Phase 0 — Architecture guardrails

Three defense-in-depth layers to keep the architecture rules enforced
regardless of who opens Claude Code in this repo:

  1. .claude/settings.json PreToolUse hook on Write/Edit blocks any file
     that would exceed the 500-line hard cap. Auto-loads in every Claude
     session in this repo.
  2. scripts/githooks/pre-commit (install via scripts/install-hooks.sh)
     enforces the LOC cap locally, freezes migrations/ without
     [migration-approved], and protects guardrail files without
     [guardrail-change].
  3. .gitea/workflows/ci.yaml gains loc-budget + guardrail-integrity +
     sbom-scan (syft+grype) jobs, adds mypy --strict for the new Python
     packages (compliance/{services,repositories,domain,schemas}), and
     tsc --noEmit for admin-compliance + developer-portal.

Per-language conventions documented in AGENTS.python.md, AGENTS.go.md,
AGENTS.typescript.md at the repo root — layering, tooling, and explicit
"what you may NOT do" lists. Root CLAUDE.md is prepended with the six
non-negotiable rules. Each of the 10 services gets a README.md.

scripts/check-loc.sh enforces soft 300 / hard 500 and surfaces the
current baseline of 205 hard + 161 soft violations so Phases 1-4 can
drain it incrementally. CI gates only CHANGED files in PRs so the
legacy baseline does not block unrelated work.

## Deprecation sweep

47 files. Pydantic V1 regex= -> pattern= (2 sites), class Config ->
ConfigDict in source_policy_router.py (schemas.py intentionally skipped;
it is the Phase 1 Step 3 split target). datetime.utcnow() ->
datetime.now(timezone.utc) everywhere including SQLAlchemy default=
callables. All DB columns already declare timezone=True, so this is a
latent-bug fix at the Python side, not a schema change.

DeprecationWarning count dropped from 158 to 35.

## Phase 1 Step 1 — Contract test harness

tests/contracts/test_openapi_baseline.py diffs the live FastAPI /openapi.json
against tests/contracts/openapi.baseline.json on every test run. Fails on
removed paths, removed status codes, or new required request body fields.
Regenerate only via tests/contracts/regenerate_baseline.py after a
consumer-updated contract change. This is the safety harness for all
subsequent refactor commits.

## Phase 1 Step 2 — models.py split (1466 -> 85 LOC shim)

compliance/db/models.py is decomposed into seven sibling aggregate modules
following the existing repo pattern (dsr_models.py, vvt_models.py, ...):

  regulation_models.py       (134) — Regulation, Requirement
  control_models.py          (279) — Control, Mapping, Evidence, Risk
  ai_system_models.py        (141) — AISystem, AuditExport
  service_module_models.py   (176) — ServiceModule, ModuleRegulation, ModuleRisk
  audit_session_models.py    (177) — AuditSession, AuditSignOff
  isms_governance_models.py  (323) — ISMSScope, Context, Policy, Objective, SoA
  isms_audit_models.py       (468) — Finding, CAPA, MgmtReview, InternalAudit,
                                     AuditTrail, Readiness

models.py becomes an 85-line re-export shim in dependency order so
existing imports continue to work unchanged. Schema is byte-identical:
__tablename__, column definitions, relationship strings, back_populates,
cascade directives all preserved.

All new sibling files are under the 500-line hard cap; largest is
isms_audit_models.py at 468. No file in compliance/db/ now exceeds
the hard cap.

## Phase 1 Step 3 — infrastructure only

backend-compliance/compliance/{schemas,domain,repositories}/ packages
are created as landing zones with docstrings. compliance/domain/
exports DomainError / NotFoundError / ConflictError / ValidationError /
PermissionError — the base classes services will use to raise
domain-level errors instead of HTTPException.

PHASE1_RUNBOOK.md at backend-compliance/PHASE1_RUNBOOK.md documents
the nine-step execution plan for Phase 1: snapshot baseline,
characterization tests, split models.py (this commit), split schemas.py
(next), extract services, extract repositories, mypy --strict, coverage.

## Verification

  backend-compliance/.venv-phase1: uv python install 3.12 + pip -r requirements.txt
  PYTHONPATH=. pytest compliance/tests/ tests/contracts/
  -> 173 passed, 0 failed, 35 warnings, OpenAPI 360/484 unchanged

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 13:18:29 +02:00
Benjamin Admin
95fcba34cd fix(quality): Ruff/CVE/TS-Fixes, 104 neue Tests, Complexity-Refactoring
Some checks failed
CI / go-lint (push) Has been skipped
CI / python-lint (push) Has been skipped
CI / nodejs-lint (push) Has been skipped
CI / test-go-ai-compliance (push) Failing after 30s
CI / test-python-backend-compliance (push) Successful in 30s
CI / test-python-document-crawler (push) Successful in 21s
CI / test-python-dsms-gateway (push) Successful in 17s
- Ruff: 144 auto-fixes (unused imports, == None → is None), F821/F811/F841 manuell
- CVEs: python-multipart>=0.0.22, weasyprint>=68.0, pillow>=12.1.1, npm audit fix (0 vulns)
- TS: 5 tote Drafting-Engine-Dateien entfernt, allowed-facts/sanitizer/StepHeader/context fixes
- Tests: +104 (ISMS 58, Evidence 18, VVT 14, Generation 14) → 1449 passed
- Refactoring: collect_ci_evidence (F→A), row_to_response (E→A), extract_requirements (E→A)
- Dead Code: pca-platform, 7 Go-Handler, dsr_api.py, duplicate Schemas entfernt

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-07 19:00:33 +01:00
Benjamin Admin
1e84df9769 feat(sdk): Multi-Tenancy, Versionierung, Change-Requests, Dokumentengenerierung (Phase 1-6)
All checks were successful
CI / go-lint (push) Has been skipped
CI / python-lint (push) Has been skipped
CI / nodejs-lint (push) Has been skipped
CI / test-go-ai-compliance (push) Successful in 32s
CI / test-python-backend-compliance (push) Successful in 30s
CI / test-python-document-crawler (push) Successful in 21s
CI / test-python-dsms-gateway (push) Successful in 18s
6-Phasen-Implementation fuer cloud-faehiges, mandantenfaehiges Compliance SDK:

Phase 1: Multi-Tenancy Fix
- Shared tenant_utils.py Dependency (UUID-Validierung, kein "default" mehr)
- VVT tenant_id Column + tenant-scoped Queries
- DSFA/Vendor DEFAULT_TENANT_ID von "default" auf UUID migriert
- Migration 035

Phase 2: Stammdaten-Erweiterung
- Company Profile um JSONB-Felder erweitert (processing_systems, ai_systems, technical_contacts)
- Regulierungs-Flags (NIS2, AI Act, ISO 27001)
- GET /template-context Endpoint
- Migration 036

Phase 3: Dokument-Versionierung
- 5 Versions-Tabellen (DSFA, VVT, TOM, Loeschfristen, Obligations)
- Shared versioning_utils.py Helper
- /{id}/versions Endpoints auf allen 5 Dokumenttypen
- Migration 037

Phase 4: Change-Request System
- Zentrale CR-Inbox mit CRUD + Accept/Reject/Edit Workflow
- Regelbasierte CR-Engine (VVT DPIA → DSFA CR, Datenkategorien → Loeschfristen CR)
- Audit-Trail
- Migration 038

Phase 5: Dokumentengenerierung
- 5 Template-Generatoren (DSFA, VVT, TOM, Loeschfristen, Obligations)
- Preview + Apply Endpoints (erzeugt CRs, keine direkten Dokumente)

Phase 6: Frontend-Integration
- Change-Request Inbox Page mit Stats, Filtern, Modals
- VersionHistory Timeline-Komponente
- SDKSidebar CR-Badge (60s Polling)
- Company Profile: 2 neue Wizard-Steps + "Dokumente generieren" CTA

Docs: 5 neue MkDocs-Seiten, CLAUDE.md aktualisiert
Tests: 97 neue Tests (alle bestanden)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-07 14:12:34 +01:00
Benjamin Admin
3593a4ff78 feat(tom): TOM-Backend in Python erstellen, Frontend von In-Memory auf DB migrieren
All checks were successful
CI / go-lint (push) Has been skipped
CI / python-lint (push) Has been skipped
CI / nodejs-lint (push) Has been skipped
CI / test-go-ai-compliance (push) Successful in 33s
CI / test-python-backend-compliance (push) Successful in 31s
CI / test-python-document-crawler (push) Successful in 22s
CI / test-python-dsms-gateway (push) Successful in 15s
- Migration 034: compliance_tom_state + compliance_tom_measures Tabellen
- Python Routes: State CRUD, Measures CRUD, Bulk-Upsert, Stats, CSV/JSON-Export
- Frontend-Proxy: In-Memory Storage durch Proxy zu backend-compliance ersetzt
- Go TOM-Handler als DEPRECATED markiert (Source of Truth ist jetzt Python)
- 44 Tests (alle bestanden)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-06 17:35:44 +01:00
Benjamin Admin
4cbfea5c1d feat(vvt): Go-Features nach Python portieren (Source of Truth)
Review-Daten (last_reviewed_at, next_review_at), created_by, DSFA-Link,
CSV-Export mit Semikolon-Trennung, overdue_review_count in Stats.
Go-VVT-Handler als DEPRECATED markiert. 32 Tests bestanden.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-06 17:14:38 +01:00
Benjamin Admin
b7c1a5da1a feat: Consent-Service Module nach Compliance migriert (DSR, E-Mail-Templates, Legal Docs, Banner)
All checks were successful
CI / go-lint (push) Has been skipped
CI / python-lint (push) Has been skipped
CI / nodejs-lint (push) Has been skipped
CI / test-go-ai-compliance (push) Successful in 36s
CI / test-python-backend-compliance (push) Successful in 31s
CI / test-python-document-crawler (push) Successful in 23s
CI / test-python-dsms-gateway (push) Successful in 18s
5-Phasen-Migration: Go consent-service Proxies durch native Python/FastAPI ersetzt.

Phase 1 — DSR (Betroffenenrechte): 6 Tabellen, 30 Endpoints, Frontend-API umgestellt
Phase 2 — E-Mail-Templates: 5 Tabellen, 20 Endpoints, neues Frontend, SDK_STEPS erweitert
Phase 3 — Legal Documents Extension: User Consents, Audit Log, Cookie-Kategorien
Phase 4 — Banner Consent: Device-Consents, Site-Configs, Kategorien, Vendors
Phase 5 — Cleanup: DSR-Proxy aus main.py entfernt, Frontend-URLs aktualisiert

148 neue Tests (50 + 47 + 26 + 25), alle bestanden.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-05 00:36:24 +01:00
Benjamin Admin
393eab6acd feat: Package 4 Nachbesserungen — History-Tracking, Pagination, Frontend-Fixes
All checks were successful
CI / go-lint (push) Has been skipped
CI / python-lint (push) Has been skipped
CI / nodejs-lint (push) Has been skipped
CI / test-go-ai-compliance (push) Successful in 36s
CI / test-python-backend-compliance (push) Successful in 31s
CI / test-python-document-crawler (push) Successful in 23s
CI / test-python-dsms-gateway (push) Successful in 18s
Backend:
- Migration 009: compliance_einwilligungen_consent_history Tabelle
- EinwilligungenConsentHistoryDB Modell (consent_id, action, version, ip, ua, source)
- _record_history() Helper: automatisch bei POST /consents (granted) + PUT /revoke (revoked)
- GET /consents/{id}/history Endpoint (vor revoke platziert für korrektes Routing)
- GET /consents: history-Array pro Eintrag (inline Sub-Query)
- 5 neue Tests (TestConsentHistoryTracking) — 32/32 bestanden

Frontend:
- consent/route.ts: limit+offset aus Frontend-Request weitergeleitet, total-Feld ergänzt
- Neuer Proxy consent/[id]/history/route.ts für GET /consents/{id}/history
- page.tsx: globalStats state + loadStats() (Backend /consents/stats für globale Zahlen)
- page.tsx: Stats-Kacheln auf globalStats umgestellt (nicht mehr page-relativ)
- page.tsx: history-Mapper: created_at→timestamp, consent_version→version
- page.tsx: loadStats() bei Mount + nach Revoke

Dokumentation:
- Developer Portal: neue API-Docs-Seite /api/einwilligungen (Consent + Legal Docs + Cookie Banner)
- developer-portal/app/api/page.tsx: Consent Management Abschnitt
- MkDocs: History-Endpoint, Pagination-Abschnitt, History-Tracking Abschnitt
- Deploy-Skript: scripts/apply_consent_history_migration.sh

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-03 11:54:25 +01:00
Benjamin Admin
113ecdfa77 feat: Package 4 Rechtliche Texte — DB-Persistenz fuer Legal Documents, Einwilligungen und Cookie Banner
All checks were successful
CI / go-lint (push) Has been skipped
CI / python-lint (push) Has been skipped
CI / nodejs-lint (push) Has been skipped
CI / test-go-ai-compliance (push) Successful in 46s
CI / test-python-backend-compliance (push) Successful in 32s
CI / test-python-document-crawler (push) Successful in 22s
CI / test-python-dsms-gateway (push) Successful in 17s
- Migration 007: compliance_legal_documents, _versions, _approvals (Approval-Workflow)
- Migration 008: compliance_einwilligungen_catalog, _company, _cookies, _consents
- Backend: legal_document_routes.py (11 Endpoints + draft→review→approved→published Workflow)
- Backend: einwilligungen_routes.py (10 Endpoints inkl. Stats, Pagination, Revoke)
- Frontend: /api/admin/consent/[[...path]] Catch-All-Proxy fuer Legal Documents
- Frontend: catalog/consent/cookie-banner routes von In-Memory auf DB-Proxy umgestellt
- Frontend: einwilligungen/page.tsx + cookie-banner/page.tsx laden jetzt via API (kein Mock)
- Tests: 44/44 pass (test_legal_document_routes.py + test_einwilligungen_routes.py)
- Deploy-Scripts: apply_legal_docs_migration.sh + apply_einwilligungen_migration.sh

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-03 08:25:13 +01:00
Benjamin Admin
34fc8dc654 feat: 6 Dokumentations-Module auf 100% — VVT Backend, Filter, PDF-Export
Phase 1 — VVT Backend (localStorage → API):
- migrations/006_vvt.sql: Neue Tabellen (vvt_organization, vvt_activities, vvt_audit_log)
- compliance/db/vvt_models.py: SQLAlchemy-Models für alle VVT-Tabellen
- compliance/api/vvt_routes.py: Vollständiger CRUD-Router (10 Endpoints)
- compliance/api/__init__.py: VVT-Router registriert
- compliance/api/schemas.py: VVT Pydantic-Schemas ergänzt
- app/(sdk)/sdk/vvt/page.tsx: API-Client + camelCase↔snake_case Mapping,
  localStorage durch persistente DB-Calls ersetzt (POST/PUT/DELETE/GET)
- tests/test_vvt_routes.py: 18 Tests (alle grün)

Phase 3 — Document Generator PDF-Export:
- document-generator/page.tsx: "Als PDF exportieren"-Button funktioniert jetzt
  via window.print() + Print-Window mit korrektem HTML
- Fallback-Banner wenn Template-Service (breakpilot-core) nicht erreichbar

Phase 4 — Source Policy erweiterte Filter:
- SourcesTab.tsx: source_type-Filter (Rechtlich / Leitlinien / Vorlagen / etc.)
- PIIRulesTab.tsx: category-Filter (E-Mail / Telefon / IBAN / etc.)
- source_policy_router.py: Backend-Endpoints unterstützen jetzt source_type
  und category als Query-Parameter
- requirements.txt: reportlab==4.2.5 ergänzt (fehlende Audit-PDF-Dependency)

Phase 2 — Training (Migration-Skripte):
- scripts/apply_training_migrations.sh: SSH-Skript für Mac Mini
- scripts/apply_vvt_migration.sh: Vollständiges Deploy-Skript für VVT

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-02 17:14:58 +01:00
Benjamin Admin
d48ebc5211 feat: 7 Analyse-Module auf 100% — Backend-Endpoints, DB-Model, Frontend-Persistenz
Alle 7 Analyse-Module (Requirements → Report) von ~80% auf 100% gebracht:
- Modul 1 (Requirements): POST/DELETE Endpoints + Frontend-Anbindung + Rollback
- Modul 2 (Controls): Evidence-Linking UI mit Validity-Badge
- Modul 3 (Evidence): Pagination (Frontend + Backend)
- Modul 4 (Risk Matrix): Mitigation-UI, Residual Risk, Status-Workflow
- Modul 5 (AI Act): AISystemDB Model, 6 CRUD-Endpoints, Backend-Persistenz
- Modul 6 (Audit Checklist): PDF-Download + Session-History
- Modul 7 (Audit Report): Detail-Seite mit Checklist Sign-Off + Navigation

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-02 15:52:23 +01:00
Benjamin Admin
f7a0b11e41 fix: Vorbereitung-Module auf 100% — Feld-Fixes, Backend-Persistenz, Endpoints
- ScopeExportTab: 11 Feldnamen-Mismatches gegen ScopeDecision Interface korrigiert
  (level→determinedLevel, riskScore→risk_score, hardTriggers→triggeredHardTriggers,
  depthDescription→depth, effortEstimate→estimatedEffort, isMandatory→required,
  triggeredByHardTrigger→triggeredBy, effortDays→estimatedEffort)
- Company Profile: GET vom Backend beim Mount, snake_case→camelCase, SDK State Fallback
- Modules: Aktivierung/Deaktivierung ans Backend schreiben (activate/deactivate Endpoints)
- Obligations: Explizites Fehler-Banner statt stiller Fallback bei Backend-Fehler
- Source Policy: BlockedContentDB Model + GET /api/v1/admin/blocked-content Endpoint
- Import: Offline-Modus Label fuer Backend-Fallback

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 12:02:40 +01:00
Benjamin Admin
a228b3b528 feat: add RAG corpus versioning and source policy backend
All checks were successful
CI / go-lint (push) Has been skipped
CI / python-lint (push) Has been skipped
CI / nodejs-lint (push) Has been skipped
CI / test-go-ai-compliance (push) Successful in 34s
CI / test-python-backend-compliance (push) Successful in 32s
CI / test-python-document-crawler (push) Successful in 23s
CI / test-python-dsms-gateway (push) Successful in 18s
Part 1 — RAG Corpus Versioning:
- New DB table compliance_corpus_versions (migration 017)
- Go CorpusVersionStore with CRUD operations
- Assessment struct extended with corpus_version_id
- API endpoints: GET /rag/corpus-status, /rag/corpus-versions/:collection
- RAG routes (search, regulations) now registered in main.go
- Ingestion script registers corpus versions after each run
- Frontend staleness badge in SDK sidebar

Part 3 — Source Policy Backend:
- New FastAPI router with CRUD for allowed sources, PII rules,
  operations matrix, audit trail, stats, and compliance report
- SQLAlchemy models for all source policy tables (migration 001)
- Frontend API base corrected from edu-search:8088/8089 to
  backend-compliance:8002/api

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 07:58:08 +01:00
Benjamin Boenisch
4435e7ea0a Initial commit: breakpilot-compliance - Compliance SDK Platform
Services: Admin-Compliance, Backend-Compliance,
AI-Compliance-SDK, Consent-SDK, Developer-Portal,
PCA-Platform, DSMS

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-11 23:47:28 +01:00