Commit Graph

8 Commits

Author SHA1 Message Date
Sharang Parnerkar
ae008d7d25 refactor(backend/api): extract DSFA schemas + services (Step 4 — file 14 of 18)
- Create compliance/schemas/dsfa.py (161 LOC) — extract DSFACreate,
  DSFAUpdate, DSFAStatusUpdate, DSFASectionUpdate, DSFAApproveRequest
- Create compliance/services/dsfa_service.py (386 LOC) — CRUD + helpers
  + stats + audit-log + CSV export; uses domain errors
- Create compliance/services/dsfa_workflow_service.py (347 LOC) — status
  update, section update, submit-for-review, approve, export JSON, versions
- Rewrite compliance/api/dsfa_routes.py (339 LOC) as thin handlers with
  Depends + translate_domain_errors(); re-export legacy symbols via __all__
- Add [mypy-compliance.api.dsfa_routes] ignore_errors = False to mypy.ini
- Update tests: 422 -> 400 for domain ValidationError (6 assertions)
- Regenerate OpenAPI baseline (360 paths / 484 operations — unchanged)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-09 19:20:48 +02:00
Sharang Parnerkar
3320ef94fc refactor: phase 0 guardrails + phase 1 step 2 (models.py split)
Squash of branch refactor/phase0-guardrails-and-models-split — 4 commits,
81 files, 173/173 pytest green, OpenAPI contract preserved (360 paths /
484 operations).

## Phase 0 — Architecture guardrails

Three defense-in-depth layers to keep the architecture rules enforced
regardless of who opens Claude Code in this repo:

  1. .claude/settings.json PreToolUse hook on Write/Edit blocks any file
     that would exceed the 500-line hard cap. Auto-loads in every Claude
     session in this repo.
  2. scripts/githooks/pre-commit (install via scripts/install-hooks.sh)
     enforces the LOC cap locally, freezes migrations/ without
     [migration-approved], and protects guardrail files without
     [guardrail-change].
  3. .gitea/workflows/ci.yaml gains loc-budget + guardrail-integrity +
     sbom-scan (syft+grype) jobs, adds mypy --strict for the new Python
     packages (compliance/{services,repositories,domain,schemas}), and
     tsc --noEmit for admin-compliance + developer-portal.

Per-language conventions documented in AGENTS.python.md, AGENTS.go.md,
AGENTS.typescript.md at the repo root — layering, tooling, and explicit
"what you may NOT do" lists. Root CLAUDE.md is prepended with the six
non-negotiable rules. Each of the 10 services gets a README.md.

scripts/check-loc.sh enforces soft 300 / hard 500 and surfaces the
current baseline of 205 hard + 161 soft violations so Phases 1-4 can
drain it incrementally. CI gates only CHANGED files in PRs so the
legacy baseline does not block unrelated work.

## Deprecation sweep

47 files. Pydantic V1 regex= -> pattern= (2 sites), class Config ->
ConfigDict in source_policy_router.py (schemas.py intentionally skipped;
it is the Phase 1 Step 3 split target). datetime.utcnow() ->
datetime.now(timezone.utc) everywhere including SQLAlchemy default=
callables. All DB columns already declare timezone=True, so this is a
latent-bug fix at the Python side, not a schema change.

DeprecationWarning count dropped from 158 to 35.

## Phase 1 Step 1 — Contract test harness

tests/contracts/test_openapi_baseline.py diffs the live FastAPI /openapi.json
against tests/contracts/openapi.baseline.json on every test run. Fails on
removed paths, removed status codes, or new required request body fields.
Regenerate only via tests/contracts/regenerate_baseline.py after a
consumer-updated contract change. This is the safety harness for all
subsequent refactor commits.

## Phase 1 Step 2 — models.py split (1466 -> 85 LOC shim)

compliance/db/models.py is decomposed into seven sibling aggregate modules
following the existing repo pattern (dsr_models.py, vvt_models.py, ...):

  regulation_models.py       (134) — Regulation, Requirement
  control_models.py          (279) — Control, Mapping, Evidence, Risk
  ai_system_models.py        (141) — AISystem, AuditExport
  service_module_models.py   (176) — ServiceModule, ModuleRegulation, ModuleRisk
  audit_session_models.py    (177) — AuditSession, AuditSignOff
  isms_governance_models.py  (323) — ISMSScope, Context, Policy, Objective, SoA
  isms_audit_models.py       (468) — Finding, CAPA, MgmtReview, InternalAudit,
                                     AuditTrail, Readiness

models.py becomes an 85-line re-export shim in dependency order so
existing imports continue to work unchanged. Schema is byte-identical:
__tablename__, column definitions, relationship strings, back_populates,
cascade directives all preserved.

All new sibling files are under the 500-line hard cap; largest is
isms_audit_models.py at 468. No file in compliance/db/ now exceeds
the hard cap.

## Phase 1 Step 3 — infrastructure only

backend-compliance/compliance/{schemas,domain,repositories}/ packages
are created as landing zones with docstrings. compliance/domain/
exports DomainError / NotFoundError / ConflictError / ValidationError /
PermissionError — the base classes services will use to raise
domain-level errors instead of HTTPException.

PHASE1_RUNBOOK.md at backend-compliance/PHASE1_RUNBOOK.md documents
the nine-step execution plan for Phase 1: snapshot baseline,
characterization tests, split models.py (this commit), split schemas.py
(next), extract services, extract repositories, mypy --strict, coverage.

## Verification

  backend-compliance/.venv-phase1: uv python install 3.12 + pip -r requirements.txt
  PYTHONPATH=. pytest compliance/tests/ tests/contracts/
  -> 173 passed, 0 failed, 35 warnings, OpenAPI 360/484 unchanged

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 13:18:29 +02:00
Benjamin Admin
95fcba34cd fix(quality): Ruff/CVE/TS-Fixes, 104 neue Tests, Complexity-Refactoring
Some checks failed
CI / go-lint (push) Has been skipped
CI / python-lint (push) Has been skipped
CI / nodejs-lint (push) Has been skipped
CI / test-go-ai-compliance (push) Failing after 30s
CI / test-python-backend-compliance (push) Successful in 30s
CI / test-python-document-crawler (push) Successful in 21s
CI / test-python-dsms-gateway (push) Successful in 17s
- Ruff: 144 auto-fixes (unused imports, == None → is None), F821/F811/F841 manuell
- CVEs: python-multipart>=0.0.22, weasyprint>=68.0, pillow>=12.1.1, npm audit fix (0 vulns)
- TS: 5 tote Drafting-Engine-Dateien entfernt, allowed-facts/sanitizer/StepHeader/context fixes
- Tests: +104 (ISMS 58, Evidence 18, VVT 14, Generation 14) → 1449 passed
- Refactoring: collect_ci_evidence (F→A), row_to_response (E→A), extract_requirements (E→A)
- Dead Code: pca-platform, 7 Go-Handler, dsr_api.py, duplicate Schemas entfernt

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-07 19:00:33 +01:00
Benjamin Admin
1e84df9769 feat(sdk): Multi-Tenancy, Versionierung, Change-Requests, Dokumentengenerierung (Phase 1-6)
All checks were successful
CI / go-lint (push) Has been skipped
CI / python-lint (push) Has been skipped
CI / nodejs-lint (push) Has been skipped
CI / test-go-ai-compliance (push) Successful in 32s
CI / test-python-backend-compliance (push) Successful in 30s
CI / test-python-document-crawler (push) Successful in 21s
CI / test-python-dsms-gateway (push) Successful in 18s
6-Phasen-Implementation fuer cloud-faehiges, mandantenfaehiges Compliance SDK:

Phase 1: Multi-Tenancy Fix
- Shared tenant_utils.py Dependency (UUID-Validierung, kein "default" mehr)
- VVT tenant_id Column + tenant-scoped Queries
- DSFA/Vendor DEFAULT_TENANT_ID von "default" auf UUID migriert
- Migration 035

Phase 2: Stammdaten-Erweiterung
- Company Profile um JSONB-Felder erweitert (processing_systems, ai_systems, technical_contacts)
- Regulierungs-Flags (NIS2, AI Act, ISO 27001)
- GET /template-context Endpoint
- Migration 036

Phase 3: Dokument-Versionierung
- 5 Versions-Tabellen (DSFA, VVT, TOM, Loeschfristen, Obligations)
- Shared versioning_utils.py Helper
- /{id}/versions Endpoints auf allen 5 Dokumenttypen
- Migration 037

Phase 4: Change-Request System
- Zentrale CR-Inbox mit CRUD + Accept/Reject/Edit Workflow
- Regelbasierte CR-Engine (VVT DPIA → DSFA CR, Datenkategorien → Loeschfristen CR)
- Audit-Trail
- Migration 038

Phase 5: Dokumentengenerierung
- 5 Template-Generatoren (DSFA, VVT, TOM, Loeschfristen, Obligations)
- Preview + Apply Endpoints (erzeugt CRs, keine direkten Dokumente)

Phase 6: Frontend-Integration
- Change-Request Inbox Page mit Stats, Filtern, Modals
- VersionHistory Timeline-Komponente
- SDKSidebar CR-Badge (60s Polling)
- Company Profile: 2 neue Wizard-Steps + "Dokumente generieren" CTA

Docs: 5 neue MkDocs-Seiten, CLAUDE.md aktualisiert
Tests: 97 neue Tests (alle bestanden)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-07 14:12:34 +01:00
Benjamin Admin
6a940344c2 feat(dsfa): Go DSFA deprecated, URL-Fix, fehlende Endpoints + 145 Tests
All checks were successful
CI / go-lint (push) Has been skipped
CI / python-lint (push) Has been skipped
CI / nodejs-lint (push) Has been skipped
CI / test-go-ai-compliance (push) Successful in 34s
CI / test-python-backend-compliance (push) Successful in 30s
CI / test-python-document-crawler (push) Successful in 27s
CI / test-python-dsms-gateway (push) Successful in 18s
- Go: DEPRECATED-Kommentare an allen 6 DSFA-Handlern + Route-Block
- api.ts: URL-Fix /dsgvo/dsfas → /dsfa (Detail-Seite war komplett kaputt)
- Python: Section-Update, Workflow (submit/approve), Export (JSON+CSV), UCCA-Stubs
- Tests: 145/145 bestanden (Schema + Route-Integration mit TestClient+SQLite)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-06 19:41:12 +01:00
Benjamin Admin
789c215e5e feat: DSFA vollständiges DB-Schema + PDF-Ingest + Tests
All checks were successful
CI / go-lint (push) Has been skipped
CI / python-lint (push) Has been skipped
CI / nodejs-lint (push) Has been skipped
CI / test-go-ai-compliance (push) Successful in 36s
CI / test-python-backend-compliance (push) Successful in 37s
CI / test-python-document-crawler (push) Successful in 23s
CI / test-python-dsms-gateway (push) Successful in 22s
- Migration 030: alle fehlenden Spalten für compliance_dsfas (Sections 0-7)
  flat fields: processing_description, legal_basis, dpo_*, authority_*, ...
  JSONB arrays: risks, mitigations, wp248_criteria_met, ai_trigger_ids, ...
  JSONB objects: section_progress, threshold_analysis, review_schedule, metadata
- dsfa_routes.py: DSFACreate/DSFAUpdate erweitert (60+ neue Optional-Felder)
  _dsfa_to_response: alle neuen Felder mit safe _get() Helper
  PUT-Handler: vollständige JSONB_FIELDS-Liste (22 Felder)
- Tests: 101 (+49) Tests — TestAIUseCaseModules + TestDSFAFullSchema
- ingest-dsfa-bundesland.sh: KNOWN_PDF_URLS (15 direkte URLs), download_pdfs()
  find_pdf_for_state() Helper, PDF-first mit Text-Fallback in ingest_all()

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-05 10:03:09 +01:00
Benjamin Admin
05aa0ee2c6 fix: DSFA Router-Prefix und Proxy-Pfad korrigiert
All checks were successful
CI / go-lint (push) Has been skipped
CI / python-lint (push) Has been skipped
CI / nodejs-lint (push) Has been skipped
CI / test-go-ai-compliance (push) Successful in 35s
CI / test-python-backend-compliance (push) Successful in 32s
CI / test-python-document-crawler (push) Successful in 22s
CI / test-python-dsms-gateway (push) Successful in 21s
Router-Prefix /v1/dsfa → /dsfa (konsistent mit allen anderen Routes)
Proxy-Pfad /api/v1/dsfa → /api/compliance/dsfa

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-04 23:22:17 +01:00
Benjamin Admin
a694b9d9ea feat: DSFA Modul — Backend, Proxy, Frontend-Migration, Tests + Mock-Daten entfernt
All checks were successful
CI / go-lint (push) Has been skipped
CI / python-lint (push) Has been skipped
CI / nodejs-lint (push) Has been skipped
CI / test-go-ai-compliance (push) Successful in 38s
CI / test-python-backend-compliance (push) Successful in 38s
CI / test-python-document-crawler (push) Successful in 22s
CI / test-python-dsms-gateway (push) Successful in 19s
- Migration 024: compliance_dsfas + compliance_dsfa_audit_log Tabellen
- dsfa_routes.py: CRUD + stats + audit-log + PATCH status Endpoints
- Proxy: /api/sdk/v1/dsfa/[[...path]] → backend-compliance:8002/api/v1/dsfa
- dsfa/page.tsx: mockDSFAs entfernt → echte API (loadDSFAs, handleCreateDSFA, handleStatusChange, handleDeleteDSFA)
- GeneratorWizard: kontrollierte Inputs + onSubmit-Handler
- reporting/page.tsx: getMockReport() Fallback entfernt → Fehlerstate
- dsr/[requestId]/page.tsx: mockCommunications entfernt → leeres Array (TODO: Backend fehlt)
- 52 neue Tests (680 gesamt, alle grün)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-04 22:41:05 +01:00