1113 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
Benjamin Admin	e536247c20	feat(quaidal): backend API + frontend tab for BSI QUAIDAL data-quality controls ... Wire the 195 Clean-Room QUAIDAL controls (from breakpilot-core migration 011) into the compliance SaaS UI. Backend: - GET /api/v1/quaidal/stats - counts by kind + source provenance - GET /api/v1/quaidal/controls - list, optional kind= filter - GET /api/v1/quaidal/controls/{id} - single derived control - GET /api/v1/quaidal/criteria - 10 QKB criteria - GET /api/v1/quaidal/criteria/{id} - QKB with QB/MA/QM tree Frontend: - /sdk/quality: new "Trainingsdaten-Qualität (BSI QUAIDAL)" tab with 10 QKB cards and a drill-down modal showing the full QB→MA→QM tree plus original BSI source link and license note. - /sdk/ai-act: Art. 10 tile on each high-risk/unacceptable result, linking to /sdk/quality?category=data_quality. Pattern matches existing IACE module DIN-reference handling: own wording, source section + URL preserved for due diligence. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-19 13:03:54 +02:00
Benjamin Admin	313982c6f1	feat(profile+report): P17 — 4 Polish-Items ... A) Cookie-Policy-Architecture-Block Fallback auf DSE-Text wenn cookie via P15 deduped wurde. Erkennt jetzt auch single-doc Sites (Safetykon-Pattern). B) Konkrete-Aufgaben-Liste: Per-Doc-Cap (3) entfernt + globaler Cap 10→20. Safetykon zeigt jetzt 7 statt 4 Aufgaben. C) business_type-Klassifizierer: B2B-Service-Cluster aus P14 als Boost. Bei 2+ Service-Indikatoren (CE-Zertifizierung/Compliance/Auditierung) wird b2b_score angehoben. Safetykon: "B2C consulting" → "B2B (consulting)". D) Vendor-Extract Fallback auf DSE-Text wenn cookie deduped + keine CMP- Payloads. LLM extrahiert dann Vendors aus dem DSE-Text. Safetykon: 0 → 1 Vendor (Google Analytics aus dem DSE-Text erkannt). Smoke-Test Safetykon: alle 4 Polish-Items wirken, kein Regression. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-19 12:22:05 +02:00
Benjamin Admin	f30a3ce471	Merge branch 'main' of ssh://gitea.meghsakha.com:22222/Benjamin_Boenisch/breakpilot-compliance	2026-05-19 11:47:45 +02:00
Benjamin Admin	479ce2225b	feat(profile): P14+P15+P16 — B2B-Heuristik + Doc-URL-Dedup + Homepage-Profile ... P14 — _detect_no_direct_sales erweitert um 3 Cluster: A) OEM-Konfigurator (BMW/Audi/Mercedes/VW/Porsche-Markennamen + Vertragshaendler-Pattern) B) B2B-Dienstleister (CE-Zertifizierung, Compliance-Beratung, Schulungen, Auditierung, TISAX, ISO-Normen, Arbeitssicherheit, ...) C) NGO/Verein/Public (Spendenkonto, Vereinsregister, gemeinnuetzig, ...) Schwelle: pos >= 2 pro Cluster UND pos > neg. Bisher: nur OEM. P15 — Doc-URL-Dedup im Worker: wenn mehrere Doc-Types DASSELBE Dokument referenzieren (Safetykon-Pattern: User gibt /datenschutz fuer dse, cookie UND widerruf), wird nur dem primaeren Doc-Type (Priority: dse > impressum > cookie > widerruf > agb > nutzungsbedingungen) der Text gegeben. Andere landen als "Nicht separat vorhanden — wird im Dokument 'X' mit-geprueft." Eliminiert die 8+8 systematischen widerruf/cookie False Positives. P16 — Profile-Detection auch Homepage-Text: Homepage-HTML wird mit kurzem Fetch (8s timeout) gezogen, getrippt und zum profile_input gemerged. Vor- her wirkte P14 nur wenn B2B-Indikatoren im DSE/Impressum-Pflichttext standen — bei Safetykon stehen sie nur im Homepage-Menue. Plus Bonus: TDM-Override-Submit-Button wird deaktiviert wenn Reason < 10 Zeichen — verhindert dass User wie heute in den Bug rein klickt. Smoke-Test Safetykon (B2B Compliance-Dienstleister): dse geprueft (kein err) impressum geprueft (kein err) cookie "Nicht separat vorhanden — wird in DSE mit-geprueft" agb "Nicht anwendbar — kein Direkt-Kaufvertrag" widerruf "Nicht anwendbar — kein Direkt-Kaufvertrag" nutzungsbedingungen "Nicht anwendbar — kein Direkt-Kaufvertrag" Vorher: 16 False Positives. Jetzt: 0. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-19 11:46:58 +02:00
Benjamin Admin	a1b380e211	fix(iace): getProject scan missed &p.CustomerName — single-project GET 500ed ... Migration 031 added customer_name to the SELECT statement in three places (GetProject, ListProjects, ListVariants), and the per-row Scan needed the matching destination. The replace_all caught ListProjects + ListVariants but missed GetProject because of an indentation difference (single tab vs row-scope indentation). Result: GET /projects/:id returned "get project: number of field descriptions must equal number of destinations, got 18 and 17" which the frontend interpreted as "project has no data" and surfaced an empty UI even though hazards/mitigations/components were intact (118/282/16 on Bremsscheibe). Single-line fix: add &p.CustomerName to the GetProject scan. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-19 11:46:34 +02:00
Sharang Parnerkar	077e0f1253	ci: force rebuild all services ... last-build/main tag deleted so detect-changes falls back to rebuild-all. Exercises the trigger-orca fix end-to-end. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-19 09:39:06 +02:00
Sharang Parnerkar	936c354547	fix(ci): trigger orca on per-job result, not needs.*.result spread ... Gitea act_runner evaluates contains(needs.*.result, 'success') to false when most upstream build jobs are skipped, so single-service changes never fired the orca redeploy. Gate trigger-orca on explicit needs.build-<service>.result == 'success' OR across all 8 build jobs. One green build now suffices to deploy. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-19 09:34:59 +02:00
Benjamin Admin	b87c27d104	fix(llm-verify): P13 — Default-Modell auf qwen3:30b-a3b (statt qwen3.5:35b-a3b) ... Bug: qwen3.5:35b-a3b liefert mit format='json' + Batch-Prompt leere Strings zurueck ('LLM batch: empty response from model'). Im echten Compliance-Check lief der LLM-Verifier deshalb wirkungslos — False-Positive-Findings wie 'Vorstand nicht erkannt' (BMW: Klammer- Liste) wurden nicht overturned. Fix: Default auf qwen3:30b-a3b umgestellt. Verifiziert mit BMW- Impressum-Text: representative_person wird mit Evidence 'Milan Nedeljkovic, Vorsitzender' overturned=True markiert. OLLAMA_VERIFY_MODEL Env-Var bleibt als Override-Moeglichkeit. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-19 09:11:01 +02:00
Benjamin Admin	78b27d4684	feat(compliance-check): P12 — TDM-Override mit dokumentierter Kunden-Erlaubnis ... Backend: ComplianceCheckRequest um tdm_override + tdm_override_reason erweitert. Worker im _run_compliance_check Pfad: bei tdm_override=True UND Reason >= 10 Zeichen wird der TDM-Vorbehalt nur dokumentiert (job.tdm_override.{reason, original_status}) und NICHT als Abbruch-Grund gewertet. Ohne Reason: Override ignoriert. Audit-Spur via logger.warning(reason). Frontend: ComplianceCheckTab um Checkbox + Pflicht-Reason-Feld ("Schriftliche Crawl-Erlaubnis vorhanden") direkt vor dem Submit- Button. Pflicht: Reason >= 10 Zeichen. Submit sendet die Flags ans Backend. Anwendungsfall: Safetykon-Pattern — robots.txt + ai.txt setzen Vorbehalt, aber Kunde hat schriftlich zugestimmt (Auftrags-Audit). [guardrail-change] ComplianceCheckTab.tsx (511 LOC) in loc-exceptions ergaenzt — Split nach _components/TDMOverride + CompliancePolling ist P11-Tech-Debt. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-19 08:56:50 +02:00
Benjamin Admin	a220f0d0a7	[guardrail-change] LOC-Exceptions: 4 grandfathered files fuer Coolify-Unblocker ... Diese 4 Pre-Existing-Files haben den Coolify-Build geblockt (LOC-CI-Step failed). Splits sind Phase-5+ Tech-Debt-Backlog, bis dahin als Exceptions getragen damit Production-Deploys nicht ausfallen. - cra_routes.py (1714) - vendor_redundancy.py (727) - cookie_knowledge_db.py (608) - cookie-banner-embed.ts (558) Jede Exception hat einen kurzen Rationale-Kommentar daruber. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-19 08:34:03 +02:00
Benjamin Admin	28a078ccb4	feat(compliance-check): P10 — Cookie-Policy-Architecture-Detection ... Neuer Service cookie_policy_architecture.detect_architecture(...) prueft vier Diagnose-Punkte der Cookie-Policy einer Website: 1. Layer-Trennung: single (BMW-Pattern: Banner + Info in EINER URL) | separate (Best Practice: getrennte Layer) 2. Versionierung: "Stand vom DD.MM.JJJJ" / "Version X.Y" / ... 3. Dynamic content: CMP-Capture auf Doc-URL oder Marker-Texte 4. Vendor-Count im Text: Indikator ob Liste statisch drinsteht Risiko-Ampel: - gruen: separate + versioned + statisch - gelb : single+unversioned (BMW) ODER separate+unversioned - rot : weder noch (Pflicht-Info fehlt) Wire-in im Compliance-Check-Worker: nach Exec-Summary-Block wird der Architecture-Block gerendert (build_architecture_html) mit konkreter Empfehlung. Bei BMW-Pattern: "Snapshot der dynamischen Vendor-Tabelle als versioniertes PDF im Archiv." Hintergrund: BMW hat eine HTML-Seite die GLEICHZEITIG Banner-Re-Trigger und Cookie-Richtlinie ist. Mindestanforderung nach §25 TDDDG + Art. 13 DSGVO erfuellt, aber bei einer Aufsichtsbehoerden-Pruefung kann nicht belegt werden welche Vendor-Liste an einem bestimmten Stichtag aktiv war. Das ist kein Verstoss aber best-practice-Luecke. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-19 01:01:48 +02:00
Benjamin Admin	0d37822b7c	fix(impressum): P9 — 7 False-Positive-Fixes in Pflichtangaben-Checks ... #1 Name des Anbieters: \b Word-Boundary verhindert "ag" in "samstag", plus "aktiengesellschaft" als Volltreffer. #2 Vertretungsberechtigte: Klammer-Liste-Pattern erkennt jetzt BMW- Format "Vorstand (Milan Nedeljkovic, Jochen Goller, ...)" plus "Vorsitzender des Aufsichtsrats: Name". #3 V.i.S.d.P.: war schon INFO, OK. #4 OS-Plattform/VSBG: bei no_direct_sales=True (OEM-Pattern) jetzt als "Nicht anwendbar" skipped statt 0/1 fail. Profile fliesst neu durch check_document_completeness -> runner. #5 Zustaendige Kammer: IHK + Handwerkskammer + Tieraerztekammer in Pattern aufgenommen + severity LOW -> INFO (konditional). #6 Stammkapital: war schon INFO, OK. #7 Link-Disclaimer: neue Check-Eigenschaft "invert"=True. Anti-Pattern ist passed wenn NICHT gefunden, fail wenn gefunden. Vorher feuerte das Finding immer, jetzt nur wenn ein illegaler Disclaimer im Text ist. Plus: L2-INFO-Checks (z.B. profession_chamber) zaehlen nicht mehr in correctness-pct und erzeugen keine DSI-DETAIL-Findings. Konsistent mit P8-Modell: INFO = "selbst pruefen", nicht "fail". Verifiziert mit BMW-Impressum-Text — alle 7 Faelle korrekt klassifiziert: name=passed, representative_person=passed, profession_chamber=INFO, illegal_disclaimer=passed (kein Disclaimer im Text), dispute_resolution=skipped (no_direct_sales), editorial_visdp=INFO, share_capital=INFO. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-19 00:52:03 +02:00
Benjamin Admin	575644c9c5	feat(audit): P8 — MC-Severity raus, Email nur harte Findings, MC-Audit als Checkliste ... Email-Hardening (mc_scorecard.top_fails): Neue _is_hard_finding-Heuristik filtert konditionale MCs ohne Negativ-Beleg aus den Top-Auffaelligkeiten. matched_text leer + Label enthaelt "falls/sofern/wenn/soweit/ggf." -> raus, landet nur noch im MC-Audit als "selbst pruefen". DATA-2066-A05 (kostenfreie Abschaltung Standortdaten) ist das prototypische Beispiel. MC-Audit-Frontend (audit/[checkId]/page.tsx): Severity-Spalte (CRITICAL/HIGH/MEDIUM/LOW) entfernt — der MC-Audit ist eine Checkliste, keine Severity-Drohung. Stattdessen: - Spalte "Prioritaet" mit 3-Tier aus regulation-Mapping: Gesetz (DSGVO/ePrivacy/TDDDG/...) / Behoerden-Leitlinie (EDPB/DSK/EuGH/...) / Best-Practice (ISO/NIST/BSI) - 3-Status: erfuellt (✓) / nicht erfuellt (✗) / selbst pruefen (?) / nicht anwendbar (—). rowReviewStatus() leitet "selbst pruefen" aus matched_text-leer + konditionalem Label ab. - Filter umgebaut auf 5 Stati statt 4 - Default-Filter "Nicht erfuellt" (vorher "Nur Fail") Bonus: f.payload.risk_label TS-Cast im FindingsTab clean gemacht (unknown -> string). Effekt: - Email an die GF zeigt nur noch echte Belege ("DSB fehlt", "Gebuehr fuer Widerruf") - MC-Audit ist eine sachliche Pruefliste fuer den Compliance-Officer Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-19 00:30:04 +02:00
Benjamin Admin	6c223c7c9b	feat(compliance-check): exec-summary + voll-audit + TDM-respect + cookie-KB-extended + saving-scan-funnel ... P1 — Exec-Summary oben im Email-Report (4 KPIs + 2 CTAs, dunkler Gradient) P3 — no_direct_sales-Flag fuer OEM-Konfigurator-Sites; AGB/Widerruf/AGB als "NICHT ANWENDBAR" (grau) statt "NICHT GEFUNDEN" (rot) P5 — Voll-Audit Unification: alle Findings (MC + Pflichtangaben + Vendor + Redundanz) in /data/compliance_audits.db.unified_findings; neuer /api/compliance/agent/findings/<id> Endpoint + FindingsTab im Audit-UI mit Filter + CSV-Export P7 — Crawl-Hardening: TDM-Reservation-Check (robots.txt / ai.txt / Header / Meta) vor jedem Run mit 24h-Cache; HeadlessChrome-UA (Firma noch nicht gegruendet — Switch via BREAKPILOT_BRANDED_UA env); per-Domain Rate-Limit 1 req/s + max 2 concurrent P2 — Cookie-Knowledge-DB additiv erweitert (35 -> 74 Cookies): Adobe, Meta, Microsoft, LinkedIn, TikTok, HubSpot, Marketo, Salesforce, Hotjar, FullStory, Mouseflow, Intercom, Drift, Zendesk, Cloudflare, Stripe, OneTrust/Cookiebot/Usercentrics, Matomo, Pinterest, Snapchat, X/Twitter, YouTube, Vimeo, Klaviyo, Mailchimp, Mixpanel, Segment, Amplitude, Optimizely, Datadog; Wire-in in cookie_function_classifier liefert compliance_risk-Label (kritisch/hoch/mittel/gering) pro Vendor A — k-Anonymitaets-Helper (benchmark_k_anonymity) fuer P6-Vorbereitung B — Cross-Tenant-Domain-Assertion im /findings-Endpoint (expected_domain Query-Param -> 403 bei Mismatch) C — Saving-Scan-Funnel: /api/compliance/agent/saving-scan/start mit Validierung + 24h-Rate-Limit pro Domain + Lead-Persistenz in saving_scan_leads + Auto-Discovery via _run_compliance_check; 6 Tests D — Risk-Badge im Email-Vendor-Row Rechtliche Leitplanken (Memory feedback_oem_data_legal.md): nur eigene Knapp-Bewertungen + Source-Pointer, keine 1:1-Kopien fremder CMP-Texte. TDM-Opt-Out-Respect nach § 44b UrhG. KEINE Schema-Aenderungen — alles in Sidecar-SQLite. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-18 23:48:34 +02:00
Benjamin Admin	a616b64273	feat(iace): Customer-Standard-Reuse across customer's prior projects ... [migration-approved] Task #22. The IACE module is used by a single Maschinenhersteller, but their plants land at many different end customers. When the safety expert commissions the second or third plant at the same customer, whole classes of mitigations (company-wide PPE rules, locked-out energy isolation, customer-standard signage) are already in place there — but rediscovered from scratch every project. Migration 031: iace_projects.customer_name TEXT + partial index. The customer is stored as a plain text field rather than a normalised iace_customers table (option A from the design discussion). A proper customer-management screen can promote this to a FK later without data loss. Backend store_customer_standards.go: - ListCustomerStandardSuggestions(projectID, includeVerified) collects mitigations from all non-archived prior projects sharing the same tenant_id AND case-insensitive customer_name. Aggregates by mitigation.name (since same-named measures from different prior projects collapse into one suggestion) and surfaces: • source_project_count + source_project_names • is_customer_standard / has_verified_instances flags includeVerified=false → strictly is_customer_standard=true includeVerified=true → also status='verified' - ImportCustomerStandardSuggestion(projectID, name): for every prior (mitigation.name → hazard.name) pairing, finds matching hazards in the current project (by name) and ensures a customer-standard mitigation exists. New rows via CreateMitigation (idempotent through the UNIQUE(hazard_id, name) from migration 030); existing rows are flipped to is_relevant=true + is_customer_standard=true + status='verified' via UPDATE. Routes: GET /api/v1/iace/projects/:id/customer-standards?include_verified= POST /api/v1/iace/projects/:id/customer-standards/import body {name} Frontend: - New page /sdk/iace/[projectId]/customer-standards with: • empty-state hint pointing to Auftrag → Kundenname • per-suggestion checkbox + per-row Übernehmen button • bulk "N übernehmen" button • toggle "Auch verifizierte einbeziehen" widening the pool • per-suggestion source_project_count + status badges - Sidebar item "Kundenstandards" (building icon) placed between Verifikation and Nachweise. - Order-page now mirrors Auftraggeber.Firmenname into the top-level customer_name column on save, so the Reuse feature is fed automatically without a separate input field. The same expert effect from migration 029's is_customer_standard flag — "I already know it's covered, no evidence needed" — now becomes a cross-project asset rather than a per-project annotation. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-18 22:31:30 +02:00
Benjamin Admin	27384aea09	feat(cra): Phase 5 — Technical Doc + DoC Generator (Annex V + VII) ... Migration 122: compliance_cra_documents with versioning + approval workflow - doc_type whitelist: doc_eu_conformity, doc_technical, doc_cvd_policy, doc_update_policy, doc_sbom_report - Status state machine: draft → reviewed → approved (+ superseded) - Snapshot generation_context for audit trail New module cra_doc_templates.py — pure-function generators (no DB access): - doc_eu_conformity: EU DoC structured per CRA Annex VII (all 7 mandatory fields) - doc_technical: Technische Dokumentation per CRA Annex V - doc_cvd_policy: ISO/IEC 29147-compliant CVD policy with SLA table - doc_update_policy: Patch/Update policy with Lifecycle + CSAF reference - doc_sbom_report: Latest SBOM summary with top-10 components Returns (title, markdown_content, requirements_coverage) — coverage tracks how many mandatory fields are filled vs placeholders. Backend endpoints: - POST /documents/generate — generates doc, supersedes previous version, increments version number atomically - GET /documents — lists all 5 doc types (also "not_generated" stubs) - GET /documents/{id} — full content_md - POST /documents/{id}/approve — set status + signed_by + signed_at Frontend: - /documents page: 5 doc-type cards with Generate/Re-Generate buttons, inline Markdown preview with .md download, 2-step approval flow (reviewed → approved with signature) - Optional params form: manufacturer, notified_body, security_contact - Dashboard: +1 button (Dokumente, 7 buttons total) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-18 22:10:23 +02:00
Benjamin Admin	cc80e59e5e	feat(cra): Phase 4 — Vulnerability Disclosure + Post-Market Monitoring ... Migration 121: compliance_cra_vulnerabilities table with full lifecycle tracking - Status state machine: reported → triaged → patched → disclosed (+ withdrawn) - CRA Art. 14(2) deadlines tracked: reported_to_enisa_at (24h), detailed_report_at (72h) - CVE-ID, severity, CVSS, affected_components (JSONB), embargo_until Backend endpoints in cra_routes.py: - POST /vulnerabilities — create with validation (severity, CVSS range) - GET /vulnerabilities — list with deadline-breach summary (24h/72h counters) - PATCH /vulnerabilities/{id} — update fields + auto-set lifecycle timestamps - DELETE /vulnerabilities/{id} — soft-delete (withdrawn) - GET /monitoring — combined view: CRA deadlines + vuln summary + post-market checklist Frontend: - /vuln page: intake form, vuln cards with 24h/72h-countdown buttons, status-transition flow with auto-timestamps - /monitoring page: CRA deadlines (11.06.26 / 11.09.26 / 11.12.27), breach banner if 24h/72h obligations missed, post-market checklist with deep-links - Dashboard: +2 buttons (Vulns, Monitoring) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-18 22:08:49 +02:00
Benjamin Admin	0a64da74bb	fix(iace/mitigations): idempotent CreateMitigation + UNIQUE(hazard_id, name) ... [migration-approved] The init-handler was non-idempotent. A second click on "Neu initialisieren in Grenzen" inserted every engine-suggested mitigation a second time — e.g. the Bremsscheibe project ended up with 5 (hazard_id, name) duplicate pairs (HMI-Usability-Pruefung, Eindeutiges visuelles Feedback, Betriebsarten-Anzeige, Sicher begrenzter Bewegungsbereich, …). 45 such duplicates accumulated across all projects. Migration 030_iace_mitigation_unique.sql: 1. Picks one winning row per (hazard_id, name) using a stable rank: is_relevant DESC (expert decision wins over engine default) status DESC (verified > implemented > planned) created_at DESC (newest beats older on otherwise-equal rows) and deletes the losers (Bremsscheibe: 5 rows; total: 45). 2. Adds UNIQUE constraint iace_mitigations_hazard_name_uniq (hazard_id, name). Store-Layer (CreateMitigation): INSERT … ON CONFLICT (hazard_id, name) DO NOTHING RETURNING id. pgx.ErrNoRows from RETURNING → look up the existing row and return that. Callers (engine init + manual add) always get a usable Mitigation; the second click is silently swallowed instead of failing. Frontend dedupe in groupByTitle stays — it covers any pre-existing duplicates that survived the migration in edge cases (multi-row write in flight, etc.). With the UNIQUE constraint live, the in-memory dedupe is a belt-and-suspenders safety net rather than the load-bearing mechanism. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-18 19:55:13 +02:00
Benjamin Admin	662327e8b4	feat(compliance-check): MC-Classification + Embedding + Vendor-Redundanz + Action-Recipes + Borlabs-Features ... Massiv-Update auf Basis BMW-Test-Iterationen (v1→v9): Core Compliance-Check - Sonnet check_type Klassifikation: text/process/review fuer alle 1874 MCs in compliance.doc_check_controls (script + Sidecar /data/mc_classification.db). rag_document_checker filtert auf check_type='text' fuer doc_check. Plus fits_doc_type-Audit (v2) + ui_only-Audit fuer DSA/E-Commerce-MCs in falscher doc_type-Schublade. - scope_requires-Filter: biometric/ai_decision/child_targeting MCs werden per business_profile gefiltert (FRT skipped fuer BMW etc.). - Embedding-Match (BGE-M3) als Phase-3 nach Regex-Match: Per-doc_type-Threshold-Override (impressum 0.50, dse/cookie 0.60), Short-Field-Rescue (15-Wort-Chunks) fuer Pflichtfelder im Impressum. Title+check_question als Embedding-Input fuer mehr Kontext. - Cookie-Text-Routing: consent-tester gibt cmp_cookie_text aus dem CMP-Reconstruct zurueck, Backend bevorzugt das gegen DOM-Extraction wenn richer (BMW 1824 vs 600 Worte). Vendor-Redundanz + EU-Alternativen + Cost-Saving - vendor_redundancy.analyze() — funktionale Kategorisierung der CMP-Vendors, Detektion von Mehrfach-Anbietern pro Kategorie, EU-Alternative-Lookup (Matomo, IONOS, HERE, Friendly Captcha, Smart AdServer, ...). - vendor_cost_estimator: Tier-Inferenz aus Cookie-Footprint (Cookie-Anzahl + Premium-Feature-Cookies + Third-Party-Quote → starter/professional/ enterprise/premier). - Self-Service-Werbung (Google/Meta/Pinterest/...) = 0 Lizenz-Kosten (nur Media-Spend, separat). DSP-Plattformen behalten enge Range. - Tier-aware Saving-Range: bei Enterprise/Premier nutzen wir den oberen 40-100%-Band der Listpreise, nicht starter→premier. - Multi-Function-Tools (Matomo Pro, SAP CX, IONOS Cloud, Userlike, Smart AdServer, HERE Maps, Vimeo Pro, LamaPoll) — ein Tool ersetzt mehrere Kategorien gleichzeitig. Cookie-Wissens-DB + Funktionale Klassifikation - cookie_knowledge_db: 50 kuratierte Top-Cookies (Google/Meta/Adobe/MS/...) mit vendor, exact_purpose, data_collected, IAB-TCF-IDs, reid_risk, schrems_ii_status, EuGH-Urteile, EU-Alternative. - cookie_function_classifier: pro Cookie funktionale Rolle (tracking_id, ad_pixel, session_id, ab_test, csrf, ...) + blocking_impact. Country-Inferenz aus Rechtsform - cookie_link_validator: Country-Field wird aus Vendor-Name abgeleitet (A/S=DK, GmbH=DE, Inc=US, B.V.=NL, ...) plus Vendor-Lookup-Table. Reduziert false-positive no_country-Flags bei eindeutig-EU-Vendors (Adform DK, Pinterest IE). Action-Recipes + Doc-Anchor-Locator - finding_action_recipes: pro Finding-Typ (no_cookies_listed, no_country, broken_opt_out, "Auftragsverarbeiter erwaehnen", "Art. 22 Profiling", ...) eine strukturierte Anweisung mit what/why/fix_text/where/example. Zum 1:1-Einfuegen in Kunden-Dokumente. - doc_anchor_locator: Embedding-basiert (BGE-M3 cosine) — sucht den passenden Absatz im existierenden Kundendokument fuer jeden Finding. Per-Run Thread-Local-Cache. Fallback: keyword-Match. - Email-Rendering integriert Recipe + Anchor pro Doc-Pruefungs-Fail + Vendor-Flag-Liste mit aufklappbarer Action-Liste. - Score-Erklaerung pro Vendor-Zeile (3/5-Untertitel + Tooltip). Migration-Pipeline (Compliance-Check -> Customer Banner/Documents) - migration_to_banner.py: Vendor-Liste -> CookieBannerConfig mit 4 Kategorien + Review-Flags. - migration_to_document.py: Vendor-Liste -> Cookie-Policy + VVT-Register + Privacy-Policy-Pre-Fills. - agent_migration_routes: 3 Preview-Endpoints (banner-preview, document-preview, summary). Persistierung der cmp_vendors in /data/compliance_audits.db check_payloads-Tabelle. Borlabs-Parity Cookie-Banner-Features - Consent-Historie im Banner: window.bpShowConsentHistory() + localStorage. - Content-Blocker: cookie-banner-content-blocker.ts — YouTube/Maps/Video Placeholder bis Einwilligung. - Google Consent Mode v2 erweitert: wait_for_update + region=EEA/CH/GB. - Consent-Log Export (CSV/JSON) per einwilligungen_export_routes. Bug-Fixes - canonical_control_routes: _jsonish-Helper fuer string-typed jsonb, similar-controls-Endpoint mit _has_embedding_col()-Cache (kein 500 mehr). - Control-Library Frontend: defensive .map-Coercer in 2 Detail-Views. - Embedding-Service-Batching (32er Batches statt 165 in einem Call). - KeyError 'control_id' in MC-Result-Aggregation (defensive .get). - Master-Controls-Klick-Through von /sdk/master-controls auf /sdk/control-library?control=<id> mit URL-Param-Auto-Open. - Dockerfile: /data pre-chowned auf appuser (Audit-DB-Schreibrecht). - Cookie-Text-Routing-Bug (cmp_reconstructed > DOM-extraction). - doc_type-aware MC-Filter (statt all-text-MCs). - Master-Contract-Dedup (60 BMW-Internal-Eintraege = 1 Adobe-Vertrag). - A3-v2-Audit hat 24 UI-Sprache-MCs als 'process' reklassifiziert. Tests - test_migration_mappers.py (9 Tests) - test_migration_endpoints.py (4 Tests) Skripte (one-shot) - classify_mc_check_type.py (v1) + _v2 (PK=control_id,doc_type) - audit_mc_doctype_fit.py (v1 fits) + _v2 (ui_only + scope_requires) BMW-Run-Bilanz v1 (broken) -> v9 (alle Fixes): DSE 7,5% -> 81-83% Impressum 4% -> 100% (6 echte MCs alle erfuellt) Cookie 0% -> 79-83% (CMP-Text-Routing + Embedding) Plus: 10 Konsolidierungs-Kategorien, geschaetzte Saving 200k-3M / Jahr Plus: Action-Recipes + Doc-Anchors fuer jeden Fail Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-18 18:30:08 +02:00
Benjamin Admin	52fb8b91e7	Merge branch 'main' of ssh://gitea.meghsakha.com:22222/Benjamin_Boenisch/breakpilot-compliance	2026-05-18 18:09:39 +02:00
Benjamin Admin	1cf5de1d45	feat(cra): CRA Compliance module Phase 1+2+3 (intake, scope, path, requirements, backlog, sbom, checks) ... Phase 1 — Intake + Scope + Path: - Migration 119: compliance_cra_projects table (intake + classification + path + status state machine) - Backend service cra_routes.py: CRUD + scope-check + path-select - Deterministic Annex III/IV classifier (verbatim mapping from migration 059 wiki) - Path validation per classification (CRITICAL → notified_body mandatory) - Frontend: project list, dashboard, 3-step wizard (intake/scope/path) - Sidebar entry under "CRA Compliance" (red) Phase 2 — Annex I Requirements + Priorisierungs-Backlog: - cra_annex_i_data.py: 40 Annex-I requirements (8 categories), 9 measures (M540-M548), 3 CRA deadlines - Endpoints: /requirements (40 items), /backlog (priority-sorted with deadline pressure) - Frontend: requirements table with filters + expandable details, backlog with deadline banner + score-ranked table - Dashboard KPI cards (Critical count, days to CE deadline, etc.) + top-10 backlog snippet Phase 3 — SBOM Upload + Automated Checks: - Migration 120: compliance_cra_sboms (versioned uploads, CycloneDX + SPDX) - SBOM endpoints: POST /sbom/upload (format detection, summary extraction), GET /sboms - Checks reuse compliance_evidence_checks: init creates 6 default CRA checks, run executes - Real implementations: cra_security_txt (HTTP + Contact: line) and cra_tls_cert_check (TLS handshake) - Frontend: SBOM file upload + version list, Checks page with per-check URL input + Run button Backend-Reuse: gap_projects (intake pre-population), compliance_evidence_checks/_check_results. Tenant scoping via existing X-Tenant-ID header pattern. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-18 17:56:52 +02:00
Benjamin Admin	3faa312b31	feat(iace/verification): derived view on relevant mitigations + 2 actions ... Task #21. The verification page used to manage a separate VerificationItem entity that the expert had to populate by hand — disjoint from the actual mitigations list. With the is_relevant flag from migration 029, the verification step has a natural definition: confirm completion for every mitigation the expert flagged as relevant for this project. Page is now a derived view on useMitigations(): filter is_relevant=true, group by title (same dedupe as Massnahmen page), expose two actions per hazard×mitigation row: 1. "Kundenstandard" — already implemented at the customer's site, no evidence file required. Sets is_customer_standard=true and status='verified'. 2. "Verifizieren…" — opens a modal asking for a textual evidence reference (Prüfprotokoll-Nr, audit reference, etc.). Calls the existing POST /mitigations/:mid/verify with verification_result. File upload is deferred to phase 2 once an object-storage backend is in place — the modal explains this. When a row is verified, a "Zurücksetzen" link reverts status to 'implemented' for accidental confirmations. Header counters: total relevant / open / verified / Kundenstandard. Maßnahmen-page polish (same commit): - "Lösch."-column header removed — the trash icon is self-explanatory - groupByTitle now additionally deduplicates by hazard_id within a group (engine occasionally emits duplicate (name, hazard_id) pairs when Reinit is clicked twice; a follow-up migration 030 will add a UNIQUE constraint to prevent these upstream) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-17 14:49:56 +02:00
Benjamin Admin	8f4f59f0e3	feat(iace/mitigations): is_relevant + is_customer_standard flags ... [migration-approved] Expert-driven workflow refinement on the Massnahmen page. The engine seeds ~80 mitigations per project, but for a concrete customer site most need a relevance decision before they're meaningful in verification: status: 'planned' | 'implemented' | 'verified' (existing — verification track) is_relevant bool (new) (does this apply to *this* site?) is_customer_standard bool (new) (already in place at customer — no evidence) Decision flow on the Mitigations tab: Engine-seeded → is_relevant=false (Default, waiting for expert) Expert checks "Relevant" → is_relevant=true → surfaces in verification Expert clicks trash → DELETE (banner warns: do not click Reinit afterwards or seeds come back) In verification, customer_standard=true bypasses evidence upload is_customer_standard implies is_relevant (DB CHECK constraint). Migration 029_iace_mitigation_relevance.sql: ALTER TABLE iace_mitigations ADD COLUMN is_relevant ..., is_customer_standard ... + CHECK constraint + partial index on is_relevant for the verification page's filter. Backend (Go): - Mitigation struct gains two bool fields - CreateMitigation: defaults to false/false (engine-seeded mitigations start unbewertet) - UpdateMitigation: new case clauses for both keys; setting is_customer_standard=true auto-flips is_relevant=true to satisfy the CHECK constraint - All three SELECT statements (ListMitigations, ListMitigationsByProject, getMitigation) extended with the two new columns Frontend: - Maßnahmen-page columns: [Relev. ☑] [Lösch. 🗑] Title | #Hazards | P·I·V - Group-header checkbox shows tri-state (indeterminate when partial), flips all instances in the group at once - Banner above the table: "Markiere jede Maßnahme als Relevant oder lösche sie. Nach Löschen kein Neu initialisieren mehr drücken." - Relevant rows tinted emerald, customer-standard label visible - Legacy bulk-select state + helpers removed (the Relevant checkbox now IS the primary mass action) - useMitigations gains handleSetRelevant, handleSetCustomerStandard, handleDeleteSilent (for non-confirm bulk deletes) Future use: is_customer_standard mitigations from a prior project at the same customer can later be auto-suggested when commissioning the next plant — turning expert knowledge into reusable customer-profile data. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-17 14:35:56 +02:00
Benjamin Admin	df7d83134b	feat(agent): migrate compliance-check results to banner + documents (M1-M5) ... After a compliance-check run finishes, the user can now apply the extracted vendor inventory directly to their own: - CookieBanner config (admin /sdk/einwilligungen) - Cookie-Policy / VVT-Register / Privacy-Policy templates (admin /sdk/document-generator) Backend: - migration_to_banner.py: vendor list -> CookieBannerConfig with ESSENTIAL/PERFORMANCE/PERSONALIZATION/EXTERNAL_MEDIA buckets + review flags (broken opt-out URLs, missing expiry, no cookies listed) - migration_to_document.py: vendor list -> pre-fills for 3 doc templates, recipient-type aware (INTERNAL/GROUP/PROCESSOR/CONTROLLER) - agent_migration_routes.py: GET /banner-preview, /document-preview, /summary keyed on check_id - compliance_audit_log: new check_payloads table persists cmp_vendors + extracted_profile so the preview survives an app restart - tests: 9 mapper units + 4 endpoint integration tests Frontend: - MigrationPanel.tsx: modal showing banner-config diff + document pre-fills, plus links into the existing editors - ComplianceCheckTab.tsx: replaces standalone audit link with the panel; net -3 lines, stays at the 500-cap Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-17 14:06:28 +02:00
Benjamin Admin	f4c9cea770	feat(iace/mitigations): group measure rows by title, collapse 21x→1 row ... The "Maßnahmen" page in the Bremsscheibe project showed a flat list with heavy redundancy — e.g. "Sicherheitszeichen nach ISO 7010" appeared on 21 separate rows, one per linked hazard. Same for "Gefahrenpiktogramme", "Flucht- und Rettungswege" etc. The signal got lost in the noise. This is a presentation-only regrouping. Each Hazard×Mitigation pair stays a separate DB row with its own status, notes and edit history (option B from the discussion: instances remain independently editable). The page now collapses rows that share the same `m.title` into one group row. Group row shows: - title + ISO 12100 sub-category (if encoded in description) - count of linked hazards on the right - compact status distribution "P · I · V" (Planned/Implemented/Verified) - shared checkbox that selects all instances in the group Click expands the group and reveals the individual hazard×measure rows, each with its own StatusBadge and detail-expand for MitigationHints. State additions: - expandedGroup: Set<string> with keys `${type}:${title}` so the same title across different reduction stages stays independently togglable - groupByTitle() helper trims the title, falls back to "(ohne Titel)" - statusCounts() helper for the P·I·V breakdown Pagination semantics swapped from 50 instances/page to 50 groups/page — makes the list far easier to scan at the ~80-instance scale this project exhibits. LOC: 267 → 346 (well under the 500 hard cap). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-17 13:50:45 +02:00
Benjamin Admin	6ed30dae5b	feat(agent): MC scorecard + audit drill-down + tenant trend (A1-A6) ... Now that all 1874 MCs run per check (Task #30 cap removal), the report was about to drown in noise. This commit adds the full aggregation / persistence / drill-down stack so each MC is actionable, not just counted. A1 mc_scorecard.py (new): build_scorecard(checks) -> per-regulation PASS/FAIL/SKIP + severity top_fails(checks, n) -> N most severe failed MCs full_audit_records(...) -> flat rows ready for sidecar SQLite A2 Email rendering: agent_doc_check_scorecard.py (new) builds an HTML scorecard table (regulation × passed/failed/HIGH/MEDIUM/score) shown at the top of the email. agent_doc_check_report._render_document now collapses the 500-MC L2 forest into 'X/Y bestanden (Z Fail)' summary plus a top-10 fails block per doc — old verbose render is gone. A3 compliance_audit_log.py (new) — sidecar SQLite at /data/compliance_audits.db (separate from compliance Postgres schema to comply with the no-new-migrations rule in CLAUDE.md): check_runs(check_id, ts, tenant_id, site_name, base_domain, doc_count, scorecard json, vvt_summary json) mc_results(check_id, doc_type, mc_id, label, passed, skipped, severity, regulation, matched_text, hint) Route persists every run after the email is sent. docker-compose.yml adds compliance-audit volume + env. A4 backfill_mc_regulation_llm.py (new) — Qwen-tagged backfill for the 1636 MCs the regex pass couldn't classify. Batches of 25, format=json, output constrained to the canonical regulation list. Run manually: docker exec bp-compliance-backend python3 \ /app/scripts/backfill_mc_regulation_llm.py [--dry-run] A5 Admin audit tab — GET /api/compliance/agent/audit/<check_id> proxied via /api/sdk/v1/agent/audit/<id>. New page /sdk/agent/audit/[checkId] renders scorecard + filterable MC table (status / doc_type / regulation, expandable rows with matched_text + hint). ComplianceCheckTab now shows 'Voll-Audit oeffnen' link. A6 Trend per tenant — GET /api/compliance/agent/audit/tenant/<id> returns recent runs. Email scorecard shows per-regulation delta badges ('(+12%)', '(-3%)') compared with the previous run for the same tenant + base_domain. Lookup is one SQLite query. Plumbing: rag_document_checker.py — SELECT now includes 'article'; MC results carry 'regulation' + 'article' through to CheckItem. agent_doc_check_routes.CheckItem schema gains regulation + article fields (defaults '') so old clients still parse. agent_compliance_check_routes — response gains 'check_id' so the frontend can build the audit link.	2026-05-17 13:45:58 +02:00
Benjamin Admin	6d29191e9b	fix(vvt): score INTERNAL/GROUP without opt-out/privacy penalty ... User feedback after BMW test: - 60 'BMW AG — XYZ' rows were rendered as ✗ for Opt-Out/Privacy and scored 38-52%. That's misleading: BMW processing for itself doesn't need a separate opt-out URL (cookie-banner is the consent mechanism) or a separate privacy policy (main DSI covers it). - Title 'Anbieter' was wrong for 60 of 90 rows (internal services). Three orthogonal fixes: 1. score_vendors becomes recipient_type aware: - INTERNAL/GROUP_COMPANY: opt_out_url, privacy_policy_url, country are NOT required (the user's main DSI + cookie-banner cover them). What IS required: name, purpose, cookies disclosed with name + expiry. Cookies-disclosure weight raised to 50 (was 15) so the VVT-relevant data is the score driver. - 'necessary' category: opt-out still skipped (§25 Abs. 2 TDDDG). - External (PROCESSOR/CONTROLLER): existing strict scoring stays. 2. _link_status_badge accepts na_label and renders a neutral em-dash with explanation tooltip instead of red ✗ when the column doesn't apply to that row. _render_vendor_row_full passes na_label based on recipient_type: - INTERNAL/GROUP -> 'Nicht erforderlich (eigene Verarbeitung)' - necessary -> 'Nicht erforderlich (§25 Abs. 2 TDDDG)' 3. Header + summary clarify the split: - h3 changed to 'Verarbeitungstaetigkeiten und Empfaenger aus der Cookie-Richtlinie' (was 'Drittanbieter aus Cookie-Richtlinie'). - Top line: '90 Verarbeitungen erfasst — 60 eigene + 30 externe Empfaenger'. - Disclaimer below: explains the INTERNAL/GROUP exemption so the reader understands why those rows don't show ✗ for missing URLs. - Section labels enriched with the relevant DSGVO article: 'Eigene Verarbeitungstaetigkeiten — fuer das VVT (Art. 30)', 'Auftragsverarbeiter — AVV erforderlich (Art. 28)', 'Joint Controller — Vereinbarung pruefen (Art. 26)'. Expected BMW result after fix: ~85% of the 60 BMW-AG rows jump from ~52% to 90-100% (the real issue, fehlende Cookies-Disclosure, stays flagged). The only true findings remaining are external links that return 4xx (e.g. Criteo 403, Teads 404).	2026-05-17 13:15:40 +02:00
Benjamin Admin	8a44e67293	feat(compliance-check): unlock all 1874 MCs + close gap-table items ... User: 'wir haben 1800 MCs erstellt um sie zu 10% zu nutzen — das ist Schwachsinn'. Fixed all 6 gaps from the audit. #1 max_controls=0 (was 20): - agent_compliance_check_routes _check_single: passes max_controls=0 to check_document_with_controls -> ALL MCs evaluated per doc_type. - 8 doc_types now use 1874 MCs instead of 160 (10x coverage). - Regex matching is cheap (<1s per doc); LLM-enrich cap of 10 stays. #2 LLM-verify fixed: - llm_verify.py was getting 0/N parsed. Causes: qwen3 thinking-mode wrapped output in <think>...</think>, /api/generate doesn't enforce JSON, prompt didn't handle code-fence wrappers. - Now uses /api/chat with format='json' (forces valid JSON). - _parse_batch_response strips <think> tags, accepts {results:[...]} AND bare [...], adds richer regex-fallback parse, logs raw head on total parse failure for diagnosis. #3 Loeschkonzept checklist (new): - doc_checks/loeschkonzept_checks.py — 9 L1 + 7 L2 checks per DIN 66398 + Art. 5(1)(e)/17/32 DSGVO: scope+responsibility, data categories, retention periods, legal basis refs (HGB/AO/BGB), deletion trigger, deletion process+technical+systems, deletion proof, exceptions + Art. 18 lock, review cycle, DSGVO references. - runner.py registered for loeschkonzept/loeschung/loeschfristen. #4 regulation backfill script: - backend-compliance/scripts/backfill_mc_regulation.py — regex-detects DSGVO/TDDDG/TMG/BGB/HGB/AO/MStV/UWG/VSBG/PAngV/GwG/BDSG/EU-VO references in MC title+question+pass_criteria, UPDATEs regulation + article fields. - Idempotent (only NULL rows), --dry-run flag, batched 200/UPDATE. - Run inside container: docker exec bp-compliance-backend python3 \ /app/scripts/backfill_mc_regulation.py #5 MC alias-fallback: - rag_document_checker._MC_ALIAS_FALLBACK maps doc_types without own MCs to a related set: nutzungsbedingungen->agb, social_media->dse, sub_processor/scc/tom_annex->avv, loeschfristen->loeschkonzept, eu_institution/dsb->dse. - _load_controls retries with the alias when the primary query returns 0 rows. - 14 additional doc_types now get MC coverage transparently. #6 cross-domain auto-discovery: - _autodiscover_missing builds a crawl plan: primary submitted base + up to 2 related domains sharing the owner SLD (e.g. BMW Group: bmw.de + bmwgroup.com + bmwgroup.jobs). - Detection: regex over submitted texts for https?://...<owner>... hostnames distinct from the primary base. - Each crawled base contributes documents + cmp_payloads to the discovery pool. Net effect for BMW: 1874 MCs evaluated (90 from cookie alone, was 20), Loeschkonzept Pflichtangaben benoten-bar, LLM overturns false regex FAILs, Joint-Controller policies on bmwgroup.jobs (Social Media) jetzt entdeckbar. Same wins will apply to CRA-Compliance check.	2026-05-17 13:07:50 +02:00
Benjamin Admin	fab1e35847	feat(vvt): recipient-type classification + 3-section VVT table ... Per user request: BMW (and others) put their own services AND external vendors in the same cookie-policy widget. The VVT-Tabelle now groups them by Art. 30(1)(d) DSGVO recipient category so the DSB can act on the right buckets: - INTERNAL — owner processing for itself ('BMW AG — XYZ') - GROUP_COMPANY — same brand family, different legal entity ('BMW Bank') - PROCESSOR — Auftragsverarbeiter, AVV-pflichtig (Adobe, Akamai) - CONTROLLER — independent / joint controller (Meta Pixel, Google Ads, LinkedIn — they run their own profiles) - AUTHORITY — government bodies (rare in cookies) - OTHER — fallback New module vendor_classifier.py: - owner_from_url(url) — derive site-owner token (bmw.de -> 'BMW', mercedes-benz.de -> 'Mercedes-Benz') - classify(name, category, owner) — strict 5-tier heuristic: * INTERNAL: vendor name first-token is '<Owner>' / '<Owner> AG' / '<Owner> SE' / '<Owner> GmbH' / '<Owner> AG & Co. KG' * GROUP_COMPANY: starts with '<Owner> ' but isn't '<Owner> AG' * CONTROLLER: matches a known joint-controller list (Meta, Google Ads, YouTube, LinkedIn Insight, TikTok, Pinterest, Taboola, Outbrain, Criteo, Twitter, Reddit, ...) * PROCESSOR: legal-form suffix in name (GmbH, AG, Inc., A/S, B.V., S.A., Ltd., LLC, ...) * OTHER: anything else vendor_extractor.extract_vendors_from_payloads now takes owner_name: - Passes it through to classify() for every extracted vendor record - The route derives owner_name via _company_name_from_url(doc_entries) - LLM-extracted vendors are classified the same way (so V3 fallback also produces tagged records) agent_doc_check_extras.build_vvt_table_html rewritten: - Buckets vendors by recipient_type - Renders one section per non-empty bucket, in canonical order (RECIPIENT_TYPE_SECTIONS), each with section header + count + bad count + nested table - Within each section: sorted by compliance_score ascending - Response JSON cmp_vendors includes recipient_type so the frontend can later import per-category into the VVT module Expected BMW result: ~60 INTERNAL rows (BMW AG own services), ~25 PROCESSOR rows (Adobe, Adform, Akamai, AWS, ...), ~5 CONTROLLER rows (Meta Pixel, Google, LinkedIn, Pinterest, Outbrain, Taboola).	2026-05-17 12:31:49 +02:00
Benjamin Admin	6c7d4c7552	fix(vvt): correct ePaaS schema mapping + category-aware scoring ... The first BMW VVT table rendered all 24 providers at 20% score because the ePaaS extractor was reading the wrong field names. Actual schema is nested: providers[].processings[].persistences[], NOT providers[] alone. Correct ePaaS schema (verified against bmw.com/epaas/.../de_DE.epaas.json): Provider: {id, name, description, processings[]} Processing: {id, name, description, categoryId, optOutLink, privacyPolicyLink, persistences[]} Persistence: {id, name, domain, type, expiry, description} Two structural changes: 1. One row per processing (not provider). BMW has 26 providers but ~91 processings spread across them (Adobe alone has ACMProcessing, AdobeAnalytics, AdobeCampaign, AdobeTargetAnalytics, AdobeTargetPers.). The cookie widget displays each processing separately — VVT now mirrors that. Display name format: 'Provider Name — Processing Name'. 2. Read optOutLink/privacyPolicyLink from PROCESSING (where they live), not provider. Persistences flatten to cookies[] with name + expiry + description. Plus category mapping: advertising -> marketing strictlyNecessary -> necessary statistics -> statistics functional -> functional Category-aware scoring (cookie_link_validator.score_vendors): - 'necessary' (technisch erforderliche, §25 Abs. 2 TDDDG): no opt-out required, no country required. Score weight shifts to purpose + cookie disclosure (essential cookies must list names + expiry). - All other categories: opt-out URL still mandatory; missing opt-out flags 'no_opt_out_url' and zeros that block of points. Expected BMW result after this fix: - ~91 rows (Adobe Analytics, Adform Retargeting, Akamai Infrastructure, AWS, ..., plus ~60 strictlyNecessary processings) - Marketing rows with present opt-out → ~75-90% - Necessary rows with cookie+expiry → ~85-95% - Rows missing fields → still flagged	2026-05-17 11:19:31 +02:00
Benjamin Admin	189918b043	fix(cmp): stricter heuristic + only replace DOM when CMP is strictly larger ... Two bugs observed in BMW BMW test run: 1. Generic JSON heuristic captured /de-de/login/bmw/api/flyout/data (4KB, user login fly-out data) and reconstruct_generic produced 56 words of noise. The CMP-prefer logic then 'replaced' the 185-word imprint DOM extraction with those 56 words because self_wc(185) < 300 — even though cmp_wc(56) < self_wc(185). 2. The strict prefilter list was too short. Login/auth/cart endpoints often have category-shaped JSON without being cookie policies. Fixes: - dsi_discovery: replace DOM with CMP only when cmp_wc > self_wc AND meets one of the existing conditions. Tiny captures can no longer silently destroy a bigger DOM extraction. - cmp_extractor: skip non-cookie URLs (/login, /auth, /user, /session, /cart, /checkout, /search, /flyout, /menu, /nav, /translation, /i18n, /locale, /feature-flag). - cmp_extractor: require ≥5KB payload size — real CMP policies are always larger (BMW ePaaS is ~393KB). Tiny matches drop out before reconstruction.	2026-05-17 10:50:19 +02:00
Benjamin Admin	873997c13b	feat(vvt): V3 — LLM vendor extraction fallback for unknown CMPs ... When the cookie text has no captured CMP payload (long-tail sites that don't use ePaaS/OneTrust/Cookiebot/etc.) we now fall back to a Qwen → OVH LLM cascade to extract a structured vendor list from the policy text. New module backend/compliance/services/vendor_llm_extractor.py: - extract_vendors_via_llm(cookie_text): runs Qwen first (local Ollama), then OVH if Qwen returns nothing usable. - System prompt instructs the model to return STRICT JSON only: {vendors: [{name, country, purpose, category, opt_out_url, privacy_policy_url, persistence, cookies: [...]}]} - Lenient JSON parser tolerates code-fences, prose wrappers, dict vs list. - _normalize() caps array sizes (80 vendors, 30 cookies each), validates URLs (must be http(s)), trims fields to reasonable lengths. Route integration (agent_compliance_check_routes.py): - After named-CMP extract: if cmp_vendors is empty AND the cookie text has ≥500 words (otherwise it's likely navigation chrome), invoke the LLM extractor. Progress message 'Vendor-Liste per LLM extrahieren...'. - Vendors then run through the same validate_vendor_urls + score_vendors pipeline → VVT table rendered identically regardless of source. docker-compose.yml: backend-compliance gains OLLAMA_URL, CMP_LLM_MODEL, OVH_LLM_URL/KEY/MODEL env vars (same names as consent-tester so the configuration is unified). This closes the 'every site eventually gets a VVT table' goal: - Known CMP → V1/V2 structured extraction (fast, exact) - Unknown CMP → V3 LLM extraction (slow, best-effort) - No text at all → no vendors, but other compliance checks still run.	2026-05-17 09:55:42 +02:00
Benjamin Admin	9c0cc0f59f	feat(vvt): V2 — vendor extractors for Cookiebot/Usercentrics/Didomi/TrustArc ... Backend vendor_extractor.py gets 4 new per-CMP dispatchers, mirroring the JSON schemas observed in each platform: - Cookiebot: 'Categories[*].Cookies[*]' with Vendor/Host, expiry, purpose - Usercentrics: 'services[*]' with cookieMaxAgeSeconds, processingCompanyCountry - Didomi: 'app.vendors[*]' with country + policyUrl - TrustArc: 'vendors[*]' + per-category 'Cookies' with provider All 6 named CMPs (ePaaS, OneTrust, Cookiebot, Usercentrics, Didomi, TrustArc) plus the generic-shape fallback are now mapped — every site hitting Phase B of the cascade gets a structured vendor list, scored opt-out links, and a VVT-Tabelle in the email.	2026-05-17 09:52:10 +02:00
Benjamin Admin	ea4dbb223f	feat(vvt): per-vendor extraction + opt-out check + VVT table in email (V1) ... When a known CMP (ePaaS, OneTrust) renders the cookie policy, we now extract structured vendor records, probe their opt-out + privacy URLs, score each vendor (0-100), and append a 'VVT-Vorschlag' table to the compliance email — one row per vendor, sortable by compliance score. consent-tester: - DSIDiscoveryResult.cmp_payloads: surfaces raw CMP JSON to callers - DSIDiscoveryResponse: new cmp_payloads field - discover_dsi_documents sets cmp_payloads from cmp_capture - cmp_library/{epaas,onetrust}.py: new extract_vendors(d) returning list[VendorRecord] backend: - _fetch_text() now returns (text, cmp_payloads) tuple - doc_entries store cmp_payloads per doc (mostly cookie) - _autodiscover_missing forwards homepage payloads to the cookie entry - New module vendor_extractor.py: dispatches ePaaS/OneTrust/generic schemas; dedupes vendors across multiple payloads - cookie_link_validator.py extended with validate_vendor_urls(vendors) and score_vendors(vendors) — 0-100 score per vendor based on name, purpose, country, opt-out reachable, privacy URL reachable, cookies with names + expiry - agent_doc_check_extras.build_vvt_table_html: renders the table - Route appends VVT HTML after the provider list, before the document-by-document report - Response JSON gains cmp_vendors for future frontend rendering Example for BMW: ~30 ePaaS providers → table with Name | Kategorie | Sitz | Cookies | Opt-Out (✓/✗) | Privacy (✓/✗) | Score. Sorted by score ascending so the worst-compliant vendors are at the top.	2026-05-17 09:50:11 +02:00
Benjamin Admin	c9c0fb5965	feat(cookie-check): enhanced patterns + active opt-out link validator ... cookie_checks.py: - cookie_names_listed: now also matches CMP placeholder notation (BMW: 'Adfpc###', 'CT###') and 'Diese Datenverarbeitung verwendet die folgenden Cookies oder ähnliche Technologien' as list-shape signal. Cryptic vendor names like 'audience', 'adformfrpid' are accepted via the surrounding markup, not by hard-coding each one. - cookie_providers_named: new pattern 'Gesetzt von: <Firma>' (BMW/ePaaS per-cookie vendor naming) + recognition of full legal-form names (Adform A/S, BMW AG, Adobe Systems Software Ireland Limited). - cookie_duration_values: now matches 'Ablauf: 1 Jahr' / 'Speicherdauer: 30 Tage' (BMW format) in addition to the legacy '<n> <unit>'. New L1 + L2 checks for controller in cookie-policy: - cookie_controller (L1): the cookie policy must name Verantwortlich(er) - cookie_controller_address (L2): PLZ + Ort or address keywords - cookie_controller_contact_or_link (L2): email/phone OR link back to Datenschutzerklärung (the practical equivalent — BMW does this) New L2 checks (parented under opt_out): - cookie_optout_links: detects per-provider opt-out URLs in the text - cookie_privacy_policy_links: per-provider privacy-policy URLs New service: cookie_link_validator.py - extract_links(text): pulls all https?://… URLs that follow 'Opt-Out Link:' / 'Link zur Privacy Policy:' (deduped) - validate_links(links): probes every URL concurrently (HEAD first, GET fallback for 405/403). 10 parallel, 8s per request, 60s batch cap. Returns reachable=True/False + status + final_url. - build_check_items(): renders 2 CheckItems (opt-out + privacy-policy), each pass if ALL links 2xx/3xx, fail with up-to-5 broken-link examples. Hook in _check_single: doc_type=='cookie' triggers the validator after regex+MC checks. Recomputes correctness with the new L2 items. This addresses two concrete BMW observations: 1. BMW's per-cookie structure (Name + Zweck + Ablauf, Gesetzt von: …, Opt-Out Link: …) now recognised → 'Konkrete Cookie-Namen aufgelistet' and 'Konkrete Speicherdauern' should pass. 2. Defective opt-out URLs surface as compliance findings rather than silently passing — Art. 7(3) DSGVO requires a working withdrawal path per provider.	2026-05-17 09:38:32 +02:00
Benjamin Admin	4a5924b8c4	feat(iace): CRA / DIN EN 40000-1-2 cyber-resilience spur ... [guardrail-change] Phase 18 adds an EU Cyber Resilience Act compliance track to IACE: the engine now fires patterns that surface the manufacturer-side CRA obligations whenever a project's components carry digital elements. Patterns (HP1910-HP1918, hazard_patterns_cra.go): HP1910 Missing SBOM HP1911 Unsigned firmware/software updates HP1912 Factory-default credentials still active HP1913 No coordinated vulnerability disclosure (CVD) policy HP1914 No documented security patch SLA HP1915 Missing user-facing hardening guide HP1916 No incident-notification process to ENISA / CSIRT HP1917 No security assessment prior to placing on market HP1918 AI component without cybersecurity risk assessment Each pattern carries ClarificationQuestionsDE so the operator gets auditor-grade questions to take back to the Anlagenbauer instead of the engine inventing prose. PatternMatch carries DefaultAvoidability (P=1 for all CRA patterns), feeding the PLr graph from Phase 17. Measures (M540-M548, measures_library_cra.go): M540 SBOM (SPDX or CycloneDX) with each machine release M541 Signed updates with rollback protection M542 Forced default-password change at first boot M543 Published CVD policy (security.txt / PSIRT) M544 Documented patch SLA with CVSS-tier response times M545 User-facing hardening guide in the machine docs M546 ENISA incident-notification process (24h/72h/14d) M547 Authenticated update channel + integrity check M548 Pre-market security assessment / pen-test The library is urheberrechtlich neutral: identifiers only (Verordnung (EU) 2024/2847, DIN EN 40000-1-2 Entwurf, IEC 62443, ETSI EN 303 645, ISO/IEC 5962, ISO/IEC 29147). No normative text is reproduced — DIN/Beuth proprietary content is referenced by section number only. Category-compatibility: cyber_resilience pattern category accepts measures with HazardCategory cyber_resilience, cyber_network, or software_control. Updated in both the runtime helper (iace_handler_init_helpers.go) and its test-mirror (pattern_coverage_test.go) — both must move in lockstep. Frontend (clarifications page): When at least one clarification references "2024/2847" or "40000-1-2" in its norm_references, a blue info-banner is rendered at the top of the page: "Cyber Resilience Act (CRA) — Hinweis zur Geltung Diese Klärungsliste enthält Fragen zur Verordnung (EU) 2024/2847 (CRA). Die CRA gilt für Produkte mit digitalen Elementen, die ab dem 11.12.2027 auf dem EU-Markt bereit- gestellt werden. ..." Reminds the user that the CRA pflichten are forward-looking while still allowing the manufacturer to bake them in now. LOC exceptions: Added three pre-existing files to .claude/rules/loc-exceptions.txt (manufacturer_safety_features.go, iace_handler_clarifications.go, routes.go). All three grew across Phases 16-17 and are tagged as Phase 5+ refactor backlog. [guardrail-change] marker required. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-17 02:15:51 +02:00
Benjamin Admin	2afa5a179b	feat(iace): Risikograph EN ISO 13849-1 PLr + Methoden-Kopf im Bericht ... Phase 17 of the risk-assessment polish. Two pieces: A) PLr per EN ISO 13849-1 Anhang A (Risikograph) - HazardPattern.DefaultAvoidability (1 = P1, 2 = P2). Optional; defaults to P1 if unset (conservative — operator can raise after review). - ComputePLr(s,f,p) implements the canonical 8-leaf binary tree (S1F1P1 -> a, ..., S2F2P2 -> e). Pinned by 8 table-driven tests. - SeverityToS / ExposureToF map the existing 1-5 fields to the binary S/F at the documented threshold (3). - At project initialise, every hazard's Description is appended with "Risikograph EN ISO 13849-1 (Anhang A): S2 · F1 · P1 -> PLr c" so the audit value is visible without leaving the hazard view. - PatternMatch carries DefaultAvoidability so the init handler can pick it up without a second pattern lookup. B) Methoden-Kopf am Bericht - GET /clarifications.html now opens with a standardised methodology block: ISO 12100 Anhang B (hazard ID) + ISO 13849-1 Anhang A (PLr graph) + ISO 12100 6.2/6.3/6.4 (reduction hierarchy). Same wording on every export, ready for the Anlagenbauer-Uebergabe. - Only norm identifiers — no norm text reproduced. C) ISO12100Section in Hazard Description - When a pattern is labeled with ISO12100Section, the hazard description gets a "Klassifikation: EN ISO 12100 Anhang B, Abschnitt 6.3.5.4" suffix. Provenance for the auditor. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-17 02:03:10 +02:00
Benjamin Admin	71d31c914b	feat(iace): ISO 12100 Anhang B mapping — split noise/vibration + section identifier ... Phase 16 of the Klaerungen / risk-assessment polish. Sources from EN ISO 12100 Anhang B Tabelle B.1 are now first-class: A) HazardPattern.ISO12100Section identifier (string), persisted only as the section number (e.g. "6.3.5.5") — not the norm text. Keeps the library urheberrechtlich neutral (DIN/Beuth license). 57 patterns labeled today; rest will follow on touch. B) Category split per ISO 12100 Nr. 4 vs Nr. 5: - 16 patterns reclassified noise_vibration -> noise_hazard - 7 patterns reclassified noise_vibration -> vibration_hazard - 1 pattern (HP228 UV-/Laermexposition) kept multi-cat acceptableMeasureCategories now accepts both new aliases plus the legacy noise_vibration. Coverage test recognises both as valid. C) 5 new ISO-12100-Annex-B gap patterns (HP1900-HP1904): - HP1900 Vakuum-Verletzung (6.3.5.5) - HP1901 Federenergie / elastische Elemente (6.2.10) - HP1902 Rutschen/Stolpern auf rauer Oberflaeche (6.3.5.6) - HP1903 Hochdruckinjektion (6.3.5.4) — includes clarifying "no hand-locating of leaks" question - HP1904 Ersticken durch Brustkorbquetschung (6.3.5.2) The library now mirrors the ISO 12100 Annex B structure for the gaps the Bremse benchmark surfaced. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-17 01:59:16 +02:00
Benjamin Admin	b090662524	fix(compliance-check): respect auto-discovery 'not found' verdict; DSB not canonical ... Two related bugs in the BMW test result: 1. AGB rendered as 'MANGELHAFT 0/13' even though BMW has no public AGB: - Auto-discovery correctly returned 'not found' for AGB (no link on bmw.de matches AGB keywords). - But auto_fill_from_dsi then found the substring 'AGB' in a section of the DSI and pseudo-filled the AGB entry with a 264-word DSI fragment. - cross_search_documents would have done the same. - Both now skip entries where discovery_attempted=True AND auto_discovered=False — the 'not found' verdict stands. 2. DSB-Kontakt rendered as a separate 100% OK document with 7566 words = the entire DSI text: - GDPR practice: the DSB is named *inside* the DSI as an email or contact block (Art. 13(1)(b)), not as a stand-alone page. - cross_search_documents had been assigning the full DSI to the DSB row because it matched 'datenschutzbeauftragte' keywords. - DSB removed from _ALL_DOC_TYPES — no longer canonical, no longer padded as missing, no longer auto-discovered. The frontend row remains so a tenant with a separate DSB page can still submit one. After this fix BMW should render: - DSE: OK - Impressum: LUECKENHAFT (unchanged — regex gaps to fix separately) - Cookie-Richtlinie: OK - Social Media: NICHT GEFUNDEN (bmw.de does not link to it) - AGB: NICHT GEFUNDEN (correct — BMW has no public AGB) - Nutzungsbedingungen: NICHT GEFUNDEN - Widerruf: NICHT GEFUNDEN	2026-05-17 01:53:09 +02:00
Benjamin Admin	c4be077c5d	feat(iace): Klaerungen Phase 3 — DB-Tabelle + Multi-User + PDF-Export ... [migration-approved] Three pieces complete the Klaerungen lifecycle: 1. Migration 028: iace_clarifications + iace_clarification_comments + iace_clarification_history. Deterministic clarification_key (UNIQUE per project) so engine re-inits don't lose answers. History table logs every status/answer transition. The previous JSONB-in-metadata storage is kept as read-only fallback for pre-migration projects until a one-shot upcopy script runs. 2. Multi-User-Workflow: - assigned_to field on every clarification (free-text user kuerzel for now; an FK to users can be added in a follow-up). - Comment thread per clarification (POST .../comment, GET .../detail returns the thread). - Status-history log written by UpsertClarification when the status or answer actually changes. - Frontend Modal: Zugewiesen-an + Bearbeiter fields, comment thread with inline post, collapsible history section. 3. PDF-Export via print-friendly HTML: - GET /clarifications.html returns a standalone A4-styled document with status badges, norm references, affected hazards and a signature row at the bottom. The Bediener opens the link and uses Strg-P / Cmd-P to save as PDF. No server-side PDF dependency added. - Frontend "PDF / Druck" button next to CSV export. Backend: - internal/iace/store_clarifications.go: UpsertClarification, ListClarificationsForProject, GetClarificationByKey, AddClarificationComment, ListClarificationComments, ListClarificationHistory. - internal/api/handlers/iace_handler_clarifications.go: - AnswerClarification now writes the SQL row, falls back to legacy JSONB read on list. - PostClarificationComment, ListClarificationDetail, ExportClarificationsHTML added. Migration must be applied manually on Mac Mini and prod via psql -f /migrations/028_iace_clarifications.sql — pattern as in scripts/apply_*_migration.sh. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-17 01:39:17 +02:00
Benjamin Admin	b2b4d77877	fix(auto-discovery): compute missing against canonical 8 types, not submitted ... Frontend filters out empty doc rows -> req.documents only contains the N submitted entries (3 in BMW case). The old auto-discovery loop computed 'missing' as 'entries in doc_entries with empty text', which was always empty for those N entries -> discovery never fired. Fix: - missing = _ALL_DOC_TYPES - {canonical doc_types in doc_entries} - For each missing type, APPEND a new entry to doc_entries with discovery_attempted=True. If a discovered doc matched, fill text/url and set auto_discovered=True. - Check loop: skip entries with no URL and no text (let padding label them). Entries with URL but no text keep the 'Kein Text' error so the user sees fetch failures explicitly.	2026-05-17 01:28:51 +02:00
Benjamin Admin	f19a75d83d	feat(iace): Klaerungen Phase 2 — Sidebar-Counter + CSV-Export + Hazard-Banner ... Three pieces complete the Klaerungen UX: 1. Sidebar-Counter: layout.tsx polls /clarifications and shows a colored open-count badge on the "Klaerungen" nav item. Refreshes whenever the user changes route. 2. CSV-Export: new backend endpoint GET /sdk/v1/iace/projects/:id/clarifications.csv produces a UTF-8- BOM-prefixed semicolon-separated CSV (Excel-friendly) with ID, Quelle, Kategorie, Frage, Status, Antwort, Begruendung, Bearbeiter, answered_at, anzahl Gefaehrdungen, Gefaehrdungs-Namen, Norm-Refs. Frontend Klaerungen-Seite bekommt einen "CSV-Export"-Button. 3. Hazard-Banner statt Fragentext im Benchmark-Detail: the previous bulleted clarification list was duplicated across 48 hazards for a single FANUC question. Phase 2 replaces it with a compact status badge — "N offene Klaerung(en) — Klaerungen-Seite oeffnen" (orange) or "Alle N Klaerungen beantwortet" (green) with a direct link. Backend cleanup: iace_handler_init.go no longer appends the "Mit Anlagenbauer zu klaeren" block to Hazard.Description. The description stays focused on the scenario; clarifications live in the dedicated endpoint and answers persist across re-inits via project.metadata. The aggregated "Referenzierte Normen" line on the hazard is kept. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-17 01:25:36 +02:00
Benjamin Admin	525038359a	feat(compliance-check): auto-discover missing doc types from homepage ... When the user leaves some doc-type rows empty, the tool now actively searches the website for them — only marks 'not found' as last resort. Flow: 1. User submits N URLs (e.g. just DSI) 2. For each canonical doc_type with no submitted URL/text, the route identifies the most-common base (scheme://netloc) from submitted URLs 3. Calls consent-tester /dsi-discovery on the homepage with max_documents=15 (180s timeout) 4. Classifies every discovered doc into a canonical doc_type via title/URL keyword rules (_DISCOVERY_RULES — covers cookie/widerruf/ social_media/agb/nutzungsbedingungen/dsb/impressum/dse) 5. Fills matching empty entries with the discovered text, marks auto_discovered=True and discovery_attempted=True Padding now differentiates: - 'Auf der Website nicht gefunden' — discovery was attempted, no doc matched. Amber badge, friendly hint to add URL manually. - 'Nicht eingereicht — Quelle nicht angegeben' — user gave NO URLs at all, nothing to crawl from. Grey badge. Email + frontend: - Status labels: NICHT GEFUNDEN (amber) vs NICHT EINGEREICHT (grey) - 'Gepruefte Quellen' table tags auto-discovered URLs with a small blue 'auto-entdeckt' badge so GF sees what tool found vs user submitted. Implementation only runs when ≥1 URL was submitted (no base to crawl from otherwise). Adds 30-90s for unsubmitted types but avoids the 'just say nicht gefunden' anti-pattern.	2026-05-17 01:14:05 +02:00
Benjamin Admin	79efa54898	feat(iace): Klaerungen MVP — Phase 1 ... New page "Klaerungen" between Massnahmen and Verifikation. Backend: - internal/iace/clarifications.go: Clarification struct + ClarificationAnswer + BuildProjectClarifications() — aggregates pattern-level + manufacturer- level questions from collectAllPatterns + GetManufacturerSafetyFeatures. Deterministic IDs ("pattern:HP1640:0", "manuf:fanuc:dual-check-safety-dcs:1") so persisted answers survive every re-init. - internal/api/handlers/iace_handler_clarifications.go: - GET /projects/:id/clarifications returns aggregated list with affected hazard names + persisted answer state, sorted (open first). - POST /projects/:id/clarifications/:cid/answer writes status/answer/ reasoning/answered_by/answered_at to project.metadata.clarification_- answers — no DB schema change. Frontend: - admin-compliance/app/sdk/iace/layout.tsx: new "Klaerungen" nav item. - app/sdk/iace/[projectId]/clarifications/page.tsx: table grouped by source (FANUC / Pattern HP1640 / …), Filter Offen/Beantwortet/Alle, search field, Antwort-Modal with status/answer/Begruendung/Bearbeiter. A clarification answered once applies to ALL referenced hazards — the operator no longer has to answer the same FANUC DCS question on 48 mechanical hazards individually. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-17 01:05:53 +02:00
Benjamin Admin	bc21480a2a	fix(compliance-check): always render 8 doc types + 4 BMW GT-gap fixes ... Always-show-8 (user-requested): - agent_compliance_check_routes.py: _pad_results_with_missing pads the results list to always include all 8 canonical doc_types in canonical order. Missing types get a placeholder DocCheckResult with error= 'Nicht eingereicht' + scenario='missing'. - agent_doc_check_report.py: NICHT EINGEREICHT status label (neutral), friendly grey body block instead of red error. - ChecklistView.tsx: 'Nicht eingereicht' chip (neutral grey, not red 'Fehler'); SCENARIO_LABELS adds missing entry + header chip counter. Impressum-Regression fix (#18): - _fetch_text(url, doc_type): cookie/dse/social_media -> max_documents=1 (CMP capture authoritative, sub-pages dilute). Other types -> =3 (Impressum needs Versicherungsvermittler, Aufsicht, Berufsrecht sub- pages). 15s networkidle bail keeps timing safe. ODR/Verbraucherstreitbeilegung filter (#19): - _apply_profile_filter: when profile.needs_odr=True (B2C), override the check's default B2B-oriented hint with action-oriented B2C guidance pointing at Art. 14 EU-VO 524/2013 + §36 VSBG. Previously the check contradicted itself: 'profile says B2C' + hint 'only relevant for B2C online vendors'. Registergericht regex (#20): - impressum_checks.py: accept colon/dot/dash between keyword and city (BMW writes 'registergericht: münchen hrb 42243'). Add 'sitz und registergericht: X' as separate pattern. Industry detection (#21): - business_profiler.py: 'automotive' keywords broadened (antriebs, motor, leasing, werkstatt, probefahrt, plus brand names BMW/Mercedes/ Audi/VW/Porsche/Opel). 'it_services' keywords narrowed — software/ cloud/hosting are mentioned in every privacy policy and were biasing the result toward IT for any tech-aware company.	2026-05-17 01:03:58 +02:00
Benjamin Admin	74f66c4c34	fix(admin/iace/benchmark): show Klaerungsfragen + Normen on Engine column ... The Go init handler appends two annotated blocks to Hazard.Description ("Mit Anlagenbauer zu klaeren: ..." and "Referenzierte Normen: ...") without changing the DB schema. The benchmark detail view only rendered hazard.scenario || hazard.description, so the appended blocks were silently hidden because scenario is always populated. Split the description into three structured pieces: 1. extractScenario() — pure scenario text, stripped of trailing blocks 2. extractClarifications() — bullet list of "Mit Anlagenbauer zu klaeren" 3. extractEngineNorms() — pipe-separated norm references Each piece is rendered as its own DetailRow. The FANUC DCS clarification that already lives in the DB (48/115 hazards on the Bremse project) is now visible in the Engine column. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-17 00:42:41 +02:00
Benjamin Admin	5f2da1de88	feat(consent-tester): Phase E — self-improving CMP library ... cmp_discovery_log.py: - sqlite log at /data/cmp_discoveries.db: every LLM-discovered CMP pattern recorded with domain, strategy, value, sample text - Auto-promote (user-chosen 'voll automatisch' mode): when LLM returns strategy=url AND extracted text >= 800 words, write a new module /data/auto_cmp/auto_<slug>.py with derived regex matcher + reconstruct - record_discovery() called from dsi_discovery._try_llm_cascade on success cmp_library/_registry.py: - Loads both hand-written modules from services/cmp_library/ AND auto-promoted modules from /data/auto_cmp/ (CMP_AUTO_DIR env) - Auto modules use importlib.util.spec_from_file_location, no package install needed; restart consent-tester to pick up new ones dsi_discovery.py: - _try_llm_cascade now calls record_discovery() on every successful LLM analysis (cached AND fresh) main.py: - GET /cmp-discoveries — admin endpoint listing all logged discoveries - DELETE /cmp-discoveries/{id} — rollback (unlinks auto_*.py) This closes the self-improving loop: first encounter with a new CMP fires the LLM (cost) → discovery is auto-promoted → all future runs against the same vendor pattern hit Phase B (Named CMP) at <50ms with no LLM call.	2026-05-16 23:09:23 +02:00
Benjamin Admin	2400aa6a9e	feat(consent-tester): Phase C+D — LLM cascade fallback (Qwen → OVH) ... New module consent-tester/services/cmp_llm_fallback.py: - LLMCookieExtractor: single-endpoint adapter (Ollama OR OpenAI-compat) - LLMCascade: tries Qwen (local Mac Mini Ollama) first; falls through to OVH (managed 120B) when Qwen returns no usable strategy - LLMCascade.from_env(): reads OLLAMA_URL/CMP_LLM_MODEL + OVH_LLM_URL/ OVH_LLM_KEY/OVH_LLM_MODEL from environment - LLM returns JSON {strategy: url|selector|text, value: ...} - Valkey-backed cache per netloc (cmp:hint:<netloc>, 7-day TTL) — next run against the same domain skips the LLM entirely dsi_discovery.py: - Wired network_log collector (URL/status/content-type/size of every JSON response on the page) — passed to LLM prompt as observation - After Named CMP (Phase B) + Heuristic (Phase A) both fail AND DOM < 300 words: invoke LLMCascade.analyze(...) - _apply_llm_hint executes the LLM's strategy: refetch URL via Playwright request context, query DOM selector, or use text directly - Cache HIT path: apply cached hint, only fall back to LLM if cache is stale docker-compose.yml: - consent-tester gets env vars + cmp-data volume (for Phase E) - All LLM endpoints configurable via env, sensible defaults consent-tester/requirements.txt: - redis>=5.0 (asyncio client, Valkey-compatible) - httpx>=0.27	2026-05-16 23:06:05 +02:00
Benjamin Admin	e9002175ac	feat(iace): manufacturer safety feature library (Stufe A — 50+ entries) ... Adds a curated database of safety-relevant features for the major manufacturers across mechanical/plant engineering, written entirely in own words with norm anchors. No verbatim manufacturer texts — therefore no copyright issue: - Markennennung (§ 23 MarkenG nominative use) is permitted. - Fakten ueber Produkt-Sicherheitsfunktionen are not protected by § 2 UrhG (only Werke, not facts). - NormReferences contain only the identifiers (e.g. "EN ISO 13849-1 PLd Kat.3"), never the norm text itself. Coverage (52 entries across 12 categories): Industrieroboter (10): FANUC DCS, KUKA SafeOperation, ABB SafeMove, Yaskawa FSU, Staeubli CS9, Kawasaki Cubic-S, Mitsubishi MELFA, Universal Robots PolyScope, Doosan PRS, Comau SafeNet CNC/WZM (8): DMG MORI, Mazak, TRUMPF, Okuma, Hermle, Heidenhain SPLC, GROB, Heller Pneumatik (4): Festo, SMC, AVENTICS, Parker Hydraulik (3): Bosch Rexroth, HAWE, HYDAC Safety-PLC / Sicherheitstechnik (8): PILZ, SICK, Schmersal, Euchner, Leuze, Phoenix Contact, Banner, Wieland Standard-PLC (5): Siemens, Beckhoff, Rockwell, Schneider, B&R Pressen (3): Schuler, Bruderer, AIDA Spritzguss (3): Arburg, KraussMaffei, ENGEL Verpackung (2): Krones, Bosch Packaging/Syntegon Laser/Schweissen (3): Bystronic, Amada, Fronius Foerdertechnik (2): Interroll, SEW EURODRIVE Engine integration: - LookupManufacturerFeaturesInText() scans the project narrative for any of the manufacturer aliases (case-insensitive, umlaut-tolerant). - Init-Handler appends matched feature clarifications to the relevant hazard's "Mit Anlagenbauer zu klaeren:" block — for the right HazardCategory only (e.g. FANUC DCS only on mechanical_hazard). - For a Bremse project narrative mentioning "Fanuc Robodrill", the engine now adds clarification questions like "Ist DCS am Roboter konfiguriert?" to relevant mechanical hazards automatically. Tests: 7 new pin tests — manufacturer count, norm prefixes, FANUC/KUKA detection in narrative, umlaut robustness (Staeubli vs Staubli). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-16 23:04:56 +02:00
Benjamin Admin	7e426c31f1	feat(consent-tester): Phase B — named CMP library + plugin architecture ... cmp_extractor.py refactored to thin coordinator (123 LOC, was 223). Discovers all CMP modules via cmp_library/_registry.py:load_all() at import time. Restart consent-tester to pick up new modules. New cmp_library/ folder: - _registry.py: auto-discovers all modules with MATCHER + reconstruct() - epaas.py: BMW Group ePaaS (extracted from cmp_extractor) - onetrust.py: cdn.cookielaw.org Groups/Cookies schema - cookiebot.py: consent.cookiebot.com Categories schema - usercentrics.py: api.usercentrics.eu services schema - didomi.py: sdk.privacy-center.org notice + vendors + purposes - trustarc.py: consent.trustarc.com categories + vendors Each module: - MATCHER: re.Pattern matching the CMP JSON endpoint URL - reconstruct(d: dict) -> str: builds German Markdown cookie-policy text Phase E (self-improving) will write auto_*.py files into the same folder; _registry already picks those up via pkgutil.iter_modules.	2026-05-16 22:59:48 +02:00