14 Commits

Author	SHA1	Message	Date
Benjamin Admin	662327e8b4	feat(compliance-check): MC-Classification + Embedding + Vendor-Redundanz + Action-Recipes + Borlabs-Features ... Massiv-Update auf Basis BMW-Test-Iterationen (v1→v9): Core Compliance-Check - Sonnet check_type Klassifikation: text/process/review fuer alle 1874 MCs in compliance.doc_check_controls (script + Sidecar /data/mc_classification.db). rag_document_checker filtert auf check_type='text' fuer doc_check. Plus fits_doc_type-Audit (v2) + ui_only-Audit fuer DSA/E-Commerce-MCs in falscher doc_type-Schublade. - scope_requires-Filter: biometric/ai_decision/child_targeting MCs werden per business_profile gefiltert (FRT skipped fuer BMW etc.). - Embedding-Match (BGE-M3) als Phase-3 nach Regex-Match: Per-doc_type-Threshold-Override (impressum 0.50, dse/cookie 0.60), Short-Field-Rescue (15-Wort-Chunks) fuer Pflichtfelder im Impressum. Title+check_question als Embedding-Input fuer mehr Kontext. - Cookie-Text-Routing: consent-tester gibt cmp_cookie_text aus dem CMP-Reconstruct zurueck, Backend bevorzugt das gegen DOM-Extraction wenn richer (BMW 1824 vs 600 Worte). Vendor-Redundanz + EU-Alternativen + Cost-Saving - vendor_redundancy.analyze() — funktionale Kategorisierung der CMP-Vendors, Detektion von Mehrfach-Anbietern pro Kategorie, EU-Alternative-Lookup (Matomo, IONOS, HERE, Friendly Captcha, Smart AdServer, ...). - vendor_cost_estimator: Tier-Inferenz aus Cookie-Footprint (Cookie-Anzahl + Premium-Feature-Cookies + Third-Party-Quote → starter/professional/ enterprise/premier). - Self-Service-Werbung (Google/Meta/Pinterest/...) = 0 Lizenz-Kosten (nur Media-Spend, separat). DSP-Plattformen behalten enge Range. - Tier-aware Saving-Range: bei Enterprise/Premier nutzen wir den oberen 40-100%-Band der Listpreise, nicht starter→premier. - Multi-Function-Tools (Matomo Pro, SAP CX, IONOS Cloud, Userlike, Smart AdServer, HERE Maps, Vimeo Pro, LamaPoll) — ein Tool ersetzt mehrere Kategorien gleichzeitig. Cookie-Wissens-DB + Funktionale Klassifikation - cookie_knowledge_db: 50 kuratierte Top-Cookies (Google/Meta/Adobe/MS/...) mit vendor, exact_purpose, data_collected, IAB-TCF-IDs, reid_risk, schrems_ii_status, EuGH-Urteile, EU-Alternative. - cookie_function_classifier: pro Cookie funktionale Rolle (tracking_id, ad_pixel, session_id, ab_test, csrf, ...) + blocking_impact. Country-Inferenz aus Rechtsform - cookie_link_validator: Country-Field wird aus Vendor-Name abgeleitet (A/S=DK, GmbH=DE, Inc=US, B.V.=NL, ...) plus Vendor-Lookup-Table. Reduziert false-positive no_country-Flags bei eindeutig-EU-Vendors (Adform DK, Pinterest IE). Action-Recipes + Doc-Anchor-Locator - finding_action_recipes: pro Finding-Typ (no_cookies_listed, no_country, broken_opt_out, "Auftragsverarbeiter erwaehnen", "Art. 22 Profiling", ...) eine strukturierte Anweisung mit what/why/fix_text/where/example. Zum 1:1-Einfuegen in Kunden-Dokumente. - doc_anchor_locator: Embedding-basiert (BGE-M3 cosine) — sucht den passenden Absatz im existierenden Kundendokument fuer jeden Finding. Per-Run Thread-Local-Cache. Fallback: keyword-Match. - Email-Rendering integriert Recipe + Anchor pro Doc-Pruefungs-Fail + Vendor-Flag-Liste mit aufklappbarer Action-Liste. - Score-Erklaerung pro Vendor-Zeile (3/5-Untertitel + Tooltip). Migration-Pipeline (Compliance-Check -> Customer Banner/Documents) - migration_to_banner.py: Vendor-Liste -> CookieBannerConfig mit 4 Kategorien + Review-Flags. - migration_to_document.py: Vendor-Liste -> Cookie-Policy + VVT-Register + Privacy-Policy-Pre-Fills. - agent_migration_routes: 3 Preview-Endpoints (banner-preview, document-preview, summary). Persistierung der cmp_vendors in /data/compliance_audits.db check_payloads-Tabelle. Borlabs-Parity Cookie-Banner-Features - Consent-Historie im Banner: window.bpShowConsentHistory() + localStorage. - Content-Blocker: cookie-banner-content-blocker.ts — YouTube/Maps/Video Placeholder bis Einwilligung. - Google Consent Mode v2 erweitert: wait_for_update + region=EEA/CH/GB. - Consent-Log Export (CSV/JSON) per einwilligungen_export_routes. Bug-Fixes - canonical_control_routes: _jsonish-Helper fuer string-typed jsonb, similar-controls-Endpoint mit _has_embedding_col()-Cache (kein 500 mehr). - Control-Library Frontend: defensive .map-Coercer in 2 Detail-Views. - Embedding-Service-Batching (32er Batches statt 165 in einem Call). - KeyError 'control_id' in MC-Result-Aggregation (defensive .get). - Master-Controls-Klick-Through von /sdk/master-controls auf /sdk/control-library?control=<id> mit URL-Param-Auto-Open. - Dockerfile: /data pre-chowned auf appuser (Audit-DB-Schreibrecht). - Cookie-Text-Routing-Bug (cmp_reconstructed > DOM-extraction). - doc_type-aware MC-Filter (statt all-text-MCs). - Master-Contract-Dedup (60 BMW-Internal-Eintraege = 1 Adobe-Vertrag). - A3-v2-Audit hat 24 UI-Sprache-MCs als 'process' reklassifiziert. Tests - test_migration_mappers.py (9 Tests) - test_migration_endpoints.py (4 Tests) Skripte (one-shot) - classify_mc_check_type.py (v1) + _v2 (PK=control_id,doc_type) - audit_mc_doctype_fit.py (v1 fits) + _v2 (ui_only + scope_requires) BMW-Run-Bilanz v1 (broken) -> v9 (alle Fixes): DSE 7,5% -> 81-83% Impressum 4% -> 100% (6 echte MCs alle erfuellt) Cookie 0% -> 79-83% (CMP-Text-Routing + Embedding) Plus: 10 Konsolidierungs-Kategorien, geschaetzte Saving 200k-3M / Jahr Plus: Action-Recipes + Doc-Anchors fuer jeden Fail Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-18 18:30:08 +02:00
Benjamin Admin	fe6764df9a	fix: ensure JSONB array fields are always arrays in control API ... Backend: _ensure_list() converts null/string/malformed JSONB to [] for requirements, test_procedure, evidence, open_anchors, tags. Frontend: defensive Array.isArray() check on ControlDetail.tsx. Fixes: TypeError: A.requirements.map is not a function Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-05-02 21:18:10 +02:00
Sharang Parnerkar	e3a1822883	refactor(admin): split training, control-provenance, iace/verification, training/learner, ControlDetail ... All 5 files reduced below 500 LOC (hard cap) by extracting sub-components: - training/page.tsx: 780→278 LOC — imports existing _components/, adds BlocksSection - control-provenance/page.tsx: 739→145 LOC — extracts provenance-data.ts, ProvenanceHelpers, LicenseMatrix, SourceRegistry - iace/[projectId]/verification/page.tsx: 673→164 LOC — extracts VerificationForm, CompleteModal, SuggestEvidenceModal, VerificationTable - training/learner/page.tsx: 560→216 LOC — extracts AssignmentsList, ContentView, QuizView, CertificatesView - ControlDetail.tsx: 878→311 LOC — adds ControlSourceCitation, ControlTraceability, ControlRegulatorySection, ControlSimilarControls, ControlReviewActions siblings Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-04-17 12:26:39 +02:00
Benjamin Admin	db7c207464	feat: V1 Control Enrichment — Eigenentwicklung-Label, regulatorisches Matching & Vergleichsansicht ... 863 v1-Controls (manuell geschrieben, ohne Rechtsgrundlage) werden als "Eigenentwicklung" gekennzeichnet und automatisch mit regulatorischen Controls (DSGVO, NIS2, OWASP etc.) per Embedding-Similarity abgeglichen. Backend: - Migration 080: v1_control_matches Tabelle (Cross-Reference) - v1_enrichment.py: Batch-Matching via BGE-M3 + Qdrant (Threshold 0.75) - 3 neue API-Endpoints: enrich-v1-matches, v1-matches, v1-enrichment-stats - 6 Tests (dry-run, execution, matches, pagination, detection) Frontend: - Orange "Eigenentwicklung"-Badge statt grauem "v1" (wenn kein Source) - "Regulatorische Abdeckung"-Sektion im ControlDetail mit Match-Karten - Side-by-Side V1CompareView (Eigenentwicklung vs. regulatorisch gedeckt) - Prev/Next Navigation durch alle Matches Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-26 10:32:08 +01:00
Benjamin Admin	5e9cab6ab5	feat: evidence_type Feld (code/process/hybrid) fuer Controls ... Neues Feld auf canonical_controls klassifiziert, ob ein Control technisch im Source Code (code), organisatorisch via Dokumente (process) oder beides (hybrid) nachgewiesen wird. Inklusive Backfill-Endpoint, Frontend-Badge/Filter und MkDocs-Dokumentation. - Migration 079: evidence_type VARCHAR(20) + Index - Backend: Filter, Backfill-Endpoint mit Domain-Heuristik, CRUD - Frontend: EvidenceTypeBadge (sky/amber/violet), Nachweisart-Dropdown - Proxy: evidence_type Passthrough fuer controls + controls-count - Tests: 22 Tests fuer Klassifikations-Heuristik - Docs: Eigenes MkDocs-Kapitel mit Mermaid-Diagramm Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-25 21:53:40 +01:00
Benjamin Admin	6d3bdf8e74	feat: Control-Detail Provenance + Atomare Controls Seite ... Backend: provenance endpoint (obligations, doc refs, merged duplicates, regulations summary) + atomic-stats aggregation endpoint. Frontend: ControlDetail mit Provenance-Sektionen, klickbare Navigation, neue /sdk/atomic-controls Seite mit Stats-Bar und gefilterer Liste. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-24 10:38:34 +01:00
Benjamin Admin	ac6134ce6d	feat: control_parent_links population + traceability API + frontend ... - _write_atomic_control() now uses RETURNING id and inserts into control_parent_links (M:N) with source_regulation, source_article, and obligation_candidate_id parsed from parent's source_citation - New _parse_citation() helper for JSONB source_citation extraction - New GET /controls/{id}/traceability endpoint returning full chain: parent links with obligations, child controls, source_count - Backend: control_type filter (atomic/rich) for controls + count - Frontend: Rechtsgrundlagen section in ControlDetail showing all parent links per source regulation with obligation text + strength - Frontend: Atomic/Rich filter dropdown in Control Library list - Frontend: GenerationStrategyBadge recognizes 'pass0b' strategy - Tests: 3 new tests for parent_link creation + citation parsing, existing batch test mock updated for RETURNING clause Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-23 08:14:29 +01:00
Benjamin Admin	643b26618f	feat: Control Library UI, dedup migration, QA tooling, docs ... - Control Library: parent control display, ObligationTypeBadge, GenerationStrategyBadge variants, evidence string fallback - API: expose parent_control_uuid/id/title in canonical controls - Fix: DSFA SQLAlchemy 2.0 Row._mapping compatibility - Migration 074: control_parent_links + control_dedup_reviews tables - QA scripts: benchmark, gap analysis, OSCAL import, OWASP cleanup, phase5 normalize, phase74 gap fill, sync_db, run_job - Docs: dedup engine, RAG benchmark, lessons learned, pipeline docs Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-21 11:56:08 +01:00
Benjamin Admin	f2819b99af	feat(pipeline): v3 — scoped control applicability + source_type classification ... Phase 4: source_type (law/guideline/standard/restricted) on source_citation - NIST/OWASP/ENISA correctly shown as "Standard" instead of "Gesetzliche Grundlage" - Dynamic frontend labels based on source_type - Backfill endpoint POST /v1/canonical/generate/backfill-source-type Phase v3: Scoped Control Applicability - 3 new fields: applicable_industries, applicable_company_size, scope_conditions - LLM prompt extended with 39 industries, 5 company sizes, 10 scope signals - All 5 generation paths (Rule 1/2/3, batch structure, batch reform) updated - _build_control_from_json: parsing + validation (string→list, size validation) - _store_control: writes 3 new JSONB columns - API: response models, create/update requests, SELECT queries extended - Migration 063: 3 new JSONB columns with GIN indexes - 110 generator tests + 28 route tests = 138 total, all passing Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-18 16:28:05 +01:00
Benjamin Admin	3b2006ebce	feat(iace): add hazard-matching-engine with component library, tag system, and pattern engine ... Implements Phases 1-4 of the IACE Hazard-Matching-Engine: - 120 machine components (C001-C120) in 11 categories - 20 energy sources (EN01-EN20) - ~85 tag taxonomy across 5 domains - 44 hazard patterns with AND/NOT matching logic - Pattern engine with tag resolution and confidence scoring - 8 new API endpoints (component-library, energy-sources, tags, patterns, match/apply) - Completeness gate G09 for pattern matching - 320 tests passing (36 new) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-16 08:50:11 +01:00
Benjamin Admin	2ed1c08acf	feat: enhance legal basis display, add batch processing tests and docs ... - Backfill 81 controls with empty source_citation.source from generation_metadata - Add fallback to generation_metadata.source_regulation in ControlDetail blue box - Improve Rule 3 amber box text for reformulated controls - Add 30 new tests for batch processing (TestParseJsonArray, TestBatchSizeConfig, TestBatchProcessingLoop) — all 61 control generator tests passing - Fix stale test_config_defaults assertion (max_controls 50→0) - Update canonical-control-library.md with batch processing pipeline docs, processed chunks tracking, migration guide, and stats endpoint - Update testing.md with canonical control generator test section Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-14 23:51:52 +01:00
Benjamin Admin	49ce417428	feat: add compliance modules 2-5 (dashboard, security templates, process manager, evidence collector) ... Module 2: Extended Compliance Dashboard with roadmap, module-status, next-actions, snapshots, score-history Module 3: 7 German security document templates (IT-Sicherheitskonzept, Datenschutz, Backup, Logging, Incident-Response, Zugriff, Risikomanagement) Module 4: Compliance Process Manager with CRUD, complete/skip/seed, ~50 seed tasks, 3-tab UI Module 5: Evidence Collector Extended with automated checks, control-mapping, coverage report, 4-tab UI Also includes: canonical control library enhancements (verification method, categories, dedup), control generator improvements, RAG client extensions 52 tests pass, frontend builds clean. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-14 21:03:04 +01:00
Benjamin Admin	b6e6ffaaee	feat: add verification method, categories, and dedup UI to control library ... - Migration 047: verification_method + category columns, 17 category lookup table - Backend: new filters, GET /categories, GET /controls/{id}/similar (embedding-based) - Frontend: filter dropdowns, badges, dedup UI in ControlDetail with merge workflow - ControlForm: verification method + category selects - Provenance: verification methods, categories, master library strategy sections - Fix UUID cast syntax in generator routes (::uuid -> CAST) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-14 07:55:22 +01:00
Benjamin Admin	8a05fcc2f0	refactor: split control library into components, add generator UI ... - Extract ControlForm, ControlDetail, GeneratorModal, helpers into separate component files (max ~470 lines each, was 1210) - Add Collection selector in Generator modal - Add Job History view in Generator modal - Add Review Queue button with counter badge - Add review mode navigation (prev/next through review items) - Add vitest tests for helpers (getDomain, constants, options) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-13 18:52:42 +01:00