207 Commits

Author	SHA1	Message	Date
Benjamin Admin	c93c88577c	feat(audit): P88 PDF-Export via WeasyPrint ... GET /api/compliance/agent/snapshots/{id}/pdf liefert application/pdf mit dem vollen Audit-Mail-Inhalt im A4-Print-Layout (Header mit Site/Timestamp/Snapshot-ID, Seitenzahlen unten rechts). check_replay.py liefert jetzt zusaetzlich 'full_html' (nicht nur 500-char-preview), damit der PDF-Renderer das komplette HTML hat. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-21 17:06:48 +02:00
Benjamin Admin	3207acea3e	fix(audit): Replay-Pipeline um P35/P77/P78/P36 Signals-Block ergaenzen ... check_replay.py rendert jetzt auch die Textsignal-Findings (Save-Label- Ambiguitaet, Cookies-in-DSE-Akzeptanz, JC-Klausel positiv, Social-Embeds). Damit hat der Replay-Test parity mit der echten Mail-Pipeline. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-21 17:04:02 +02:00
Benjamin Admin	9f06911ff9	feat(audit): Cookie-Library-Fallback fuer VW-Pattern (kein bekanntes CMP) ... Wenn nach Standard-Extract + Phase-G + LLM-Cascade weiterhin < 3 cmp_vendors aber >= 5 Cookies im after_accept stehen (typisch: Custom-CMP wie VW 'cookiemgmt'), matcht der Fallback die Cookie-Namen gegen die compliance.cookie_library und rekonstruiert Vendor-Records aus den Library-Eintraegen. Hintergrund: VW Run de2a029e zeigt 4 Vendors trotz 28 after_accept-Cookies. cmp_payloads ist 0 (kein bekanntes IAB-Tool erkannt) und die hinterlegte Cookie-URL liefert 404. Die DSE ist mit 34k zwar substanziell, listet aber keine Vendor-Tabelle. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-21 17:00:49 +02:00
Benjamin Admin	4171cf0efd	feat(audit): P36 Social-Media-Einbindungs-Check ... check_social_embedding: erkennt direkte FB/Insta/Twitter/YouTube- Embeds (connect.facebook.net, platform.twitter.com etc) vs Heise-Shariff vs 2-Klick-Loesungen (Embetty). Direkte Embeds ohne Schutz = HIGH (EuGH C-40/17 Fashion-ID — der Site-Betreiber wird zum gemeinsam Verantwortlichen und braucht Einwilligung VOR dem Drittanbieter-Call). Shariff oder 2-Klick erkannt = INFO (positives Signal). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-21 16:45:12 +02:00
Benjamin Admin	30e43afba6	feat(audit): P86 Branchen-Benchmark + P35/P77/P78 Textsignale ... P86 — industry_benchmark.py: zieht alle Snapshots mit derselben scan_context.industry, berechnet Median + Percentile, rendert 'Sie 42% — Automotive-Median 58% (Stichprobe: 12)'. Min Sample 3. P35 — banner_text 'Speichern' ohne 'Ablehnen' = MEDIUM. Mehrdeutiges Label nach EDPB 03/2022 Deceptive-Design-Guidelines. P77 — DSE mit prominenter Cookie-Sektion (Vendor-Hints: Speicherdauer, Anbieter, Datenkategorie) ersetzt die Forderung nach separater Cookie-Richtlinie. Positives Signal statt False-Positive. P78 — Art. 26-Klausel im DSE-Text erkannt → positives Signal 'JC-Konstrukt dokumentiert'. Vermeidet False-Positive bei Konzern-Schwester-Kooperationen. Alle in Mail eingehaengt: Branchen-Block nach GF-1-Pager, Signale-Block nach Konsistenz-Check. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-21 16:43:15 +02:00
Benjamin Admin	df8832c521	feat(audit): P75 Banner-vs-CMP + P84 Diff-Mode + P74/P96/P97 Doc-Types ... P75 — check_banner_vs_cmp_partner_count: wenn Banner-Text 'N Partner' nennt und N < cmp_vendors * 0.6, HIGH-Finding (Art. 13(1)(e) DSGVO). Erkennt Verharmlosung der tatsaechlichen Vendor-Anzahl. P84 — run_diff.py: vergleicht aktuellen Lauf mit letztem Snapshot derselben Site (set-Diff auf normalisierten Finding-Labels). Block ueber dem GF-1-Pager: 'Seit letztem Lauf: X Findings weg, Y neue'. USP — keiner der grossen Anbieter hat das. P74/P96/P97 — Labels fuer legal_notice (Rechtliche Hinweise / IP / Forward-Looking), dsa (Art. 12+17 Digital Services Act), lizenzhinweise (OSS-Compliance) in _DOC_TYPE_LABELS registriert. Echte Pflichtangaben- Checks kommen separat. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-21 16:38:25 +02:00
Benjamin Admin	7842c95532	feat(audit): P92 CMP-Tool-Verfuegbarkeit + P94 Banner-vs-Cookie-Doc-Konsistenz ... P92 — Wenn der Nutzer 'Anpassen'/'Einstellungen' klickt und der CMP-Settings-Bereich kein Fehlerfreies Laden zeigt (Error, Timeout, <80 Zeichen ohne Kategorien, keine Toggles), ist das ein HIGH- Finding. Granulare Wahl formal vorhanden, faktisch nicht funktionsfaehig (Art. 7 (3) DSGVO + EDPB 03/2022). P94 — Cookie-Liste im Banner-Settings vs Cookie-Richtlinie. Heuristik extrahiert Cookie-Namen aus dem Cookie-Doc-Text (regex auf typische camelCase/_underscored Patterns + Vendor-Prefixes _ga/_gid/ot_/uc_). Wenn |only_in_doc| >= 5 ODER |only_in_banner| >= 3 → MEDIUM-Finding. |only_in_doc| >= 15 UND |only_in_banner| >= 5 → HIGH. Beide Findings landen im neuen Mail-Block 'Banner-Konsistenz-Pruefung' (amber-yellow) zwischen Mismatch-Block und VVT. Auch in check_replay.py eingehaengt. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-21 16:31:19 +02:00
Benjamin Admin	08671adfdf	feat(audit): P82 GF-1-Pager + P87 Konfidenz-Score pro Finding ... P82 — gf_one_pager.py: kompakte 5-Bullet-Kurzfassung ganz oben in der Mail. Score (gross + Farbe), Delta-zu-Vorlauf, Top-Findings nach HIGH/MEDIUM sortiert mit zustaendiger Rolle (DSB / Marketing / IT / Legal / Web-Team) und Klassifizierungsbits aus dem Wizard. Sachlicher Ton — keine 4%-Drohung, '4-8 Wochen' als realistischer Zeitrahmen. Eingehaengt vor Critical-Findings-Block in Mail-Composition und Replay-Pipeline. P87 — finding_confidence.py: 13 Regex-Regeln liefern (confidence_pct, reason) pro Finding-Label. Direkt im DOM beobachtbar = 95-98%, Library-Mismatch = 82%, Textmuster-Match auf Pflichtangaben = 75-88%. Im 1-Pager als kleines '(NN% Konfidenz)'-Tag mit Reason-Tooltip hinter jedem Finding gerendert. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-21 16:20:19 +02:00
Benjamin Admin	50fc0ecc59	feat(audit): P79 Pre-Scan-Wizard (8 Pflichtfelder) + P99 erweitert + P102 Replay-Fix ... P79: PreScanWizard.tsx mit 8 Pflichtfeldern (Branche, B2B/B2C, Direkt-Vertrieb, Rechtsform, Konzern-Struktur, MA-Zahl, Besondere Daten, Drittland). Scan-Button disabled bis alle 8 ausgefuellt. Werte landen in scan_context und ueber Backend in compliance_check_snapshots. P99: DOC_TYPES um dsa + legal_notice + lizenzhinweise + nutzungsbedingungen erweitert. URL-hinzufuegen-Button war schon da. P102 (Replay-Bug): check_replay.py liest jetzt e.get('text') statt nur full_text — Snapshot-Schema verwendet 'text'. Library-Mismatch- Block wird damit auch im Replay angezeigt. Backend: ComplianceCheckRequest.scan_context optional; save_snapshot persistiert ihn in compliance_check_snapshots.scan_context. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-21 15:59:01 +02:00
Benjamin Admin	94057b1536	feat(audit): VW-Cookie-Bug-Fix + P101/P102 Cookie-Library-Mismatch-Findings ... VW-Bug B1: extract_vendors_via_llm hatte max_text_chars=12000 -> bei VW-Cookie-Doc (60k chars, 100 Cookies in Tabelle) wurden 80% abgeschnitten, LLM extrahierte nur 1 Vendor. Fix: max_text_chars=50000, num_predict 6000->16000 fuer mehr Vendor-Output, Ollama-Timeout 120s->420s. P101 Aggregator-Script (backend-compliance/scripts/cookie_library_enrich.py) geht alle compliance_check_snapshots durch und extrahiert (cookie_name, declared_category, observed_sites). Erste Auswertung ueber 8 Snapshots: 101 unique Cookies, 47 in Library, 54 unbekannt, 18 Mismatches. P102 Cookie-Klassifikations-Pruefung als Mail-Block. Vergleicht Site-deklarierte Kategorie vs Library + Vendor-Doku. HIGH wenn Library sagt 'marketing' aber Site als 'essential'/'statistics' deklariert (faktische Drittland-/Werbe-Verarbeitung versteckt). MEDIUM sonst. In agent_compliance_check_routes Mail-Komposition + Replay-Pipeline eingebaut. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-21 15:47:11 +02:00
Benjamin Admin	50ed0f45af	fix(replay): P80 — DocCheckResult-Import entfernt (gibt es nicht in runner) ... Vorher hatte ich den Container hotfixed aber den Fix nicht committed. Beim naechsten Rebuild kam der Bug aus dem Image zurueck. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-21 12:25:04 +02:00
Benjamin Admin	e1df24cad7	fix(audit): P93+P95 — Reject-Wording erweitert + Vendor-zentrisches Cookie-Format akzeptiert ... P93: 'Cookies verbieten', 'Tracking ablehnen', 'verweigern' usw. zaehlen nun als expliziter Reject-Mechanismus. EDPB 5/2020 schreibt kein bestimmtes Wort vor — BMW False-Positive 'Kein Ablehnen-Mechanismus' weg. P95: cookie_table-Check akzeptiert nun zwei gleichwertige Formate: (a) klassische Tabelle, (b) Vendor-Detailseite mit Block pro Anbieter (Name+Anschrift, Zweck, Speicherdauer aggregiert, Cookie-Namen-Liste, Opt-Out-Link). BMW-Stil mit Adform-Block ist DSK-OH 2024 konform. False-Positive 'tabellarisches Cookie-Verzeichnis fehlt' wird seltener. Hinweis-Text in cookie_table umformuliert: nennt beide akzeptablen Formate, weniger normativ. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-21 12:21:29 +02:00
Benjamin Admin	4946571863	feat(audit-pipeline): P72-v2 Heuristik nachgeschaerft + P80 Mini-Replay-Endpoint ... P72-v2 MC-Scope-Classifier Heuristik v2 — v1 hatte 79% 'other'-Bucket (Patterns zu strict). v2 deckt deutlich breiter ab: - DSE: Art. 13/14 + Betroffenenrechte (Art. 15-22) + DSB + Aufsichtsbehoerde + Speicherdauer + besondere Kategorien - TOM: Art. 32 + Verschluesselung/Backup/Pseudonymisierung + Zugriffskontrolle + ISO 27001 + BSI-Grundschutz + Audit-Log - cookie_richtlinie: Tracking-Pixel + Webstorage + GA/Matomo/ Hotjar/Pixel/GTM - process: VVT (Art. 30) + DSFA (Art. 35) + Datenpannen (Art. 33/34) + HinSchG + Schulungen + Loeschkonzept Script `backfill_mc_scope_v2.py` re-classifiziert NUR den 'other'-Bucket (spezifische v1-Buckets bleiben unangetastet). P80 Mini-Replay-Endpoint (v1): POST /compliance-check/snapshots/{id}/replay ?recipient=foo@bar.com & dry_run=false Laedt Snapshot, rendert Mail mit AKTUELLEM Render-Code (P63-P67, P59b/P61/P62). Sendet [REPLAY]-prefixed Mail oder gibt nur HTML-Stats zurueck (dry_run). Effekt: 7min Re-Scan -> 2-5sec fuer Mail-Layout-Iterationen. v2 (spaeter): MC-Scorecard mit aktuellem scope_doc_type-Filter ueber Snapshot — erfordert _run_compliance_check Refactoring. Plus Bugfix: GET /snapshots/{id} raised jetzt HTTPException statt Tuple-Return (FastAPI hat Tuple als JSON-Array zurueckgegeben). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-21 10:21:56 +02:00
Benjamin Admin	cde670617e	feat(audit-pipeline): P72 MC-Scope-Classifier + P80 Snapshot/Replay-Foundation [migration-approved] ... P72 MC-Scope-Classifier — pro MC den ECHTEN Doc-Adressaten festlegen (cookie_richtlinie/dse/banner_implementation/cmp_audit/tom/avv/jc/ impressum/agb/widerruf/process/accounting/other). - Migration 145: scope_doc_type Spalte + Index auf canonical_controls - Backfill-Script mit Regex-Heuristik (12 Regeln, Prioritaet-sortiert) - Erste 11k-Sample-Distribution: 76% other (Heuristik v1 zu strict — v2 muss lockerere Patterns fuer DSE/TOM nachschaerfen) - Ziel: bevor MC-Scorecard filtert, weiss jeder MC welches Dokument er adressiert. Bisher landeten eHealth-/HGB-MCs im Cookie-Audit. P80 Snapshot + Replay-Foundation — Roh-Daten persistieren damit Audit-Pipeline ohne erneuten Crawl rebuildbar ist. - Migration 146: compliance_check_snapshots Tabelle (JSONB pro doc_entries/banner_result/profile/cmp_vendors/scan_context) - services.check_snapshot.save_snapshot/load_snapshot/list - Endpoints GET /snapshots, GET /snapshots/{id} - Hook in _run_compliance_check: nach Mail-Send automatischer Snapshot-Save via separater SessionLocal (background-task safe) - Replay-Endpoint folgt im naechsten PR (braucht Refactoring von _run_compliance_check in crawl_phase + interpret_phase) - Effekt: Test-Cycle 7min -> 5sec bei reinen Logik-Aenderungen (P73/P79/P81+ profitieren direkt). Snapshots dienen auch als Regression-Test-Corpus (P81 Golden-Truth-Library). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-21 08:53:31 +02:00
Benjamin Admin	603381a67f	feat(audit-mail): P58/P59c/P60b/P61/P62 — Mercedes-Cycle Phase 1 abgeschlossen ... P58 Anti-Audit-Detection robuster (script-domain + settings-spezifisch — war bereits im Code, jetzt sauber als completed dokumentiert). P59c DACH-Custom-Cookies in compliance.cookie_library: Borlabs, etracker, Matomo/Piwik, Userlike, Cookiebot/Cookieyes/Usercentrics, Akamai/Cloudflare/Datadome Bot-Manager + HubSpot. 21 neue Eintraege (3 von 24 schon via Open-Cookie-Database vorhanden). Script: backend-compliance/scripts/seed_dach_cookies.py. P60b Vendor-Pattern-Dedupe mit Fuzzy-Match (Jaccard >= 0.7) statt exakter Tuple-Equality. Vendors mit teilweise befuellten Feldern (z.B. Sitzland eingetragen) fallen nicht mehr aus der globalen Notice — Bug: Amazon/Psyma/Qualtrics hatten zuvor wiederholte per-row Actions. P61 "Untergeschobene Cookies"-Erkennung — wenn ein deklarierter Vendor (z.B. Google Tag Manager) automatisch weitere mitbringt (GA + GCL_AU + DoubleClick), werden diese als separater Mail-Block (gelb) mit COOKIE/VENDOR-Badges + Quellen-Doku ausgewiesen. Neuer Service: compliance.services.vendor_package_cookies (8 Primary-Vendors mit je 2-4 implicit Cookies/Vendors). P62 Marketing-Manager-Disclaimer "Was wir sehen / nicht sehen" als blauer Box-Block direkt unter dem Critical-Findings-Block. Erklaert Grenzen unseres Audits (Server-Side-Tracking, Vendor-interne Datenweitergabe, Cross-Page-Banner) und Risiko des Falschvertrauens in einen 100%-Score. Neuer Renderer: compliance.api.scope_disclaimer. Architektur: VVT-Tabellen-Renderer aus agent_doc_check_extras.py (552 LOC -> 242 LOC) in compliance.api.vvt_table_renderer ausgelagert, um den 500-LOC-Hardcap einzuhalten. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-21 08:01:27 +02:00
Benjamin Admin	57c0f940a2	feat(consent+report): P56-P67 Mercedes-Audit-Cycle (Anti-Audit, Phase G Vendors, Cookie-Behavior-Validator + 5 Mail-Polish-Items) [migration-approved] ... P56 Anti-Auditing-Detection als constructive Compliance-Finding (Audit-API- Empfehlung statt Anklage, weil Mercedes berechtigt Bots blockiert) P57 Phase G vendor_details Union mit cmp_vendors -> 42 Anbieter sichtbar P58 Anti-Audit-Detection robuster (Script-Domain-Check + Settings-spezifisch) P59 Cookie-Behavior-Validator (4 Layer, 3-Tier-Severity: MEDIUM=Kategorie- Mismatch / HIGH=Zweck-Mismatch / CRITICAL=beide=Vorsatz-Indiz) + Open Cookie Database (CC0) als Library-Seed (2264 Cookies) P59b Cookie-Behavior in Banner-Check verdrahtet + Mail-Block (BUGFIX: SessionLocal selbst oeffnen, db war im Background-Task nicht im Scope) Mail-Polish nach Mercedes-Review: P63 Banner-Footer-Links auch im wb7-link/role=link erkennen (Shadow-DOM- Walker label-based statt nur <a href>) P64 Re-Access-Severity: MEDIUM statt HIGH, wenn Footer "Einstellungen" oder Mercedes-typisch existiert; OEM-Footer-Detection (wb7-footer) P65 Text-Truncation: Word-Boundary statt Zeichen-Cut (kein "einfa"-Bruch mehr in Sofortmassnahmen) P66 GF-Aktionen: Service-Zweck vs Cookie-Zweck explizit erklaert (haeufige Verwechslung Marketing/GF: "Akamai-Beschreibung" != Cookie- Zweck pro DSK-OH 2024) P67 Stirring-Finding mit "Verlust-Framing"-Erklaerung + Alt-vs-Neutral- Beispiel, statt nur EDPB-Fachbegriff Compliance-Advisor FAQ (admin agent-core/soul): + CNIL/EDPB Top-Bussgelder (Google 100M, Meta 60M, Amazon 35M) + Deutsche Praezedenz (LG Muenchen Google Fonts, EuGH Planet49, BGH I ZR 7/16) + 4 Risiko-Pfade (Bussgeld/Abmahnung/Sammelklage/NOYB) + Berechnungs-Methodik Document-Generator Templates: AGB-DE (142), Impressum (140), Widerrufs- formular-Anlage (143), DSR-Process-Dedup (139), Cookie-Library (144). Architektur: doc_action_mappings.py + banner_dom_walkers.py + cookie_behavior_validator.py + vendor_detail_extractor.py rausgezogen, um die 500-LOC-Caps in agent_doc_check_report.py und banner_text_checker.py einzuhalten. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-21 06:28:25 +02:00
Benjamin Admin	badb356740	fix(founding-wizard): nested IF-Bloecke korrekt aufloesen (innermost-first)	2026-05-20 19:21:08 +02:00
Benjamin Admin	f08eb71480	fix(founding-wizard): default values fuer alle 8 Notar-Templates Platzhalter	2026-05-20 18:45:12 +02:00
Benjamin Admin	0477a2f2dc	fix(founding-wizard): RESSORT_N_NAME/_GF/_AUFGABEN aus GF-Liste ableiten	2026-05-20 18:42:36 +02:00
Benjamin Admin	93cedbecbd	fix(founding-wizard): missing context vars (P_INFO etc) + italic regex no longer eats snake_case underscores	2026-05-20 18:37:12 +02:00
Benjamin Admin	4478b7f479	fix(founding-wizard): mypy/ruff cleanup for CI ... - markdown_to_docx.py: type annotations + unused import - founding_wizard_routes.py: drop unused get_db import	2026-05-20 09:58:38 +02:00
Benjamin Admin	7a5f1e48dd	feat(founding-wizard): Gründungs-Wizard für 2-Mann GmbH + 14 Notar-Templates ... [migration-approved] Templates (Migrations 123-136): - 123 GO-GF (Geschäftsordnung Geschäftsführung) - 124 SHA (Shareholders' Agreement, 56 Platzhalter) - 125 Satzung (Articles of Association mit UG-Variante) - 126 GF-Dienstvertrag (Trennungsprinzip Organ/Anstellung) - 127 Arbeitsvertrag (AGG-neutral, NachwG, eAU) - 128 Gesellschafterliste (§ 40 GmbHG) - 129 GF-Bestellungsbeschluss (mit § 6 Abs. 2 Versicherung) - 130 HRB-Anmeldung (§§ 7, 8, 39 GmbHG, § 12 HGB) - 131 IP-Assignment Agreement (Gründer→GmbH) - 132 Term Sheet (Pre-Seed/Seed VC-Standard) - 133 Wandeldarlehensvertrag (Convertible Loan) - 134 Beteiligungsvertrag (Subscription Agreement) - 135 ESOP/VSOP-Plan (3 Varianten) - 136 Cap Table Kategorisierung (Migrations 137-138): - ALTER TABLE compliance_legal_templates ADD lifecycle_stage TEXT[], functional_category TEXT (mit CHECK Constraints + GIN-Index) - Backfill aller 105 Templates: lifecycle_stage (pre_founding|founding| startup|kmu|konzern) + functional_category (founding_legal|employment| investor_funding|...) Backend Founding-Wizard Service: - template_renderer.py: Handlebars-light ({{VAR}}, {{#IF FLAG}}...{{/IF}}) - wizard_to_context.py: Mapping Wizard-State → SCREAMING_SNAKE_CASE Vars - markdown_to_docx.py: Markdown → DOCX via python-docx - founding_wizard_routes.py: POST /v1/founding-wizard/generate → liefert base64-DOCX-Files für ausgewählte Templates Frontend Founding-Wizard (/sdk/founding-wizard): - 8-Step Wizard (Basics, Gesellschafter, GF, Kapital, Notar, SHA, GF-Verträge, Generate) - useFoundingWizardForm Hook mit localStorage-Persistenz - TypeScript Code-Registry (template-categories.ts) als Backup zur DB - Word-Download via data:URLs (base64) Tests: - 20 Unit-Tests grün (Renderer, Context-Mapping, DOCX-Conversion) - Playwright E2E-Test mit 2-Mann GmbH (Benjamin + Sharang) Test-Daten	2026-05-20 09:30:51 +02:00
Benjamin Admin	6f16507c5f	feat(banner): P19 + P20 — Per-Category-Click-Test + Frontend-Drilldown ... P19 (consent-tester): - dp-cookieconsent (TYPO3, Safetykon-Pattern) als CMP-Profil hinzu — Selektoren #dp--cookie-statistics/marketing + a.cc-allow Save-Button - Neues Signal provider_details_visible: nach Kategorie-Toggle prueft Playwright ob im Banner sichtbare Provider-/Cookie-Detail-Elemente erscheinen. Bei dp-cookieconsent (Banner ohne Listing) immer False -> HIGH-Violation "Kategorie zeigt keine Provider-/Cookie-Details — Nutzer kann nicht informiert einwilligen (Art. 7 Abs. 1 DSGVO)" - main.py serialisiert provider_details_visible + cookies_set pro Kategorie P20 (Frontend-Drilldown): - Backend: check_payloads-Tabelle um Spalte 'banner' (JSON) — voller banner_result persistiert (vorher nur in-memory). ALTER TABLE Migration idempotent. - Neuer Endpoint GET /api/compliance/agent/banner/<check_id> — liefert Quality-Score, Phases, Category-Tests, Banner-Checks, alle 46 structured_checks. - Frontend: BannerTab im /sdk/agent/audit/<id> mit Quality-Cards, 3-Phasen-Cookie-Tabelle, Per-Category-Listing (mit P19-Signal rot/gruen), Banner-Verstoesse + Rechtsgrundlagen, 46-Check-Drilldown filterbar nach Severity. - Tab-Switcher in page.tsx um "Cookie-Banner-Analyse" erweitert. - Bonus: 2 alte route.ts auf Next.js 15 Promise-params umgestellt (Build-Fix). Plus: Critical-Findings-Block nutzt provider_details_visible als primaeres Signal statt nur tracking_services-Anzahl. Smoke-Test Safetykon: 4 Critical Findings im Mail, banner-Endpoint liefert 46 checks + 3 phases + 2 categories mit provider_details_visible=False. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-19 14:31:13 +02:00
Benjamin Admin	313982c6f1	feat(profile+report): P17 — 4 Polish-Items ... A) Cookie-Policy-Architecture-Block Fallback auf DSE-Text wenn cookie via P15 deduped wurde. Erkennt jetzt auch single-doc Sites (Safetykon-Pattern). B) Konkrete-Aufgaben-Liste: Per-Doc-Cap (3) entfernt + globaler Cap 10→20. Safetykon zeigt jetzt 7 statt 4 Aufgaben. C) business_type-Klassifizierer: B2B-Service-Cluster aus P14 als Boost. Bei 2+ Service-Indikatoren (CE-Zertifizierung/Compliance/Auditierung) wird b2b_score angehoben. Safetykon: "B2C consulting" → "B2B (consulting)". D) Vendor-Extract Fallback auf DSE-Text wenn cookie deduped + keine CMP- Payloads. LLM extrahiert dann Vendors aus dem DSE-Text. Safetykon: 0 → 1 Vendor (Google Analytics aus dem DSE-Text erkannt). Smoke-Test Safetykon: alle 4 Polish-Items wirken, kein Regression. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-19 12:22:05 +02:00
Benjamin Admin	479ce2225b	feat(profile): P14+P15+P16 — B2B-Heuristik + Doc-URL-Dedup + Homepage-Profile ... P14 — _detect_no_direct_sales erweitert um 3 Cluster: A) OEM-Konfigurator (BMW/Audi/Mercedes/VW/Porsche-Markennamen + Vertragshaendler-Pattern) B) B2B-Dienstleister (CE-Zertifizierung, Compliance-Beratung, Schulungen, Auditierung, TISAX, ISO-Normen, Arbeitssicherheit, ...) C) NGO/Verein/Public (Spendenkonto, Vereinsregister, gemeinnuetzig, ...) Schwelle: pos >= 2 pro Cluster UND pos > neg. Bisher: nur OEM. P15 — Doc-URL-Dedup im Worker: wenn mehrere Doc-Types DASSELBE Dokument referenzieren (Safetykon-Pattern: User gibt /datenschutz fuer dse, cookie UND widerruf), wird nur dem primaeren Doc-Type (Priority: dse > impressum > cookie > widerruf > agb > nutzungsbedingungen) der Text gegeben. Andere landen als "Nicht separat vorhanden — wird im Dokument 'X' mit-geprueft." Eliminiert die 8+8 systematischen widerruf/cookie False Positives. P16 — Profile-Detection auch Homepage-Text: Homepage-HTML wird mit kurzem Fetch (8s timeout) gezogen, getrippt und zum profile_input gemerged. Vor- her wirkte P14 nur wenn B2B-Indikatoren im DSE/Impressum-Pflichttext standen — bei Safetykon stehen sie nur im Homepage-Menue. Plus Bonus: TDM-Override-Submit-Button wird deaktiviert wenn Reason < 10 Zeichen — verhindert dass User wie heute in den Bug rein klickt. Smoke-Test Safetykon (B2B Compliance-Dienstleister): dse geprueft (kein err) impressum geprueft (kein err) cookie "Nicht separat vorhanden — wird in DSE mit-geprueft" agb "Nicht anwendbar — kein Direkt-Kaufvertrag" widerruf "Nicht anwendbar — kein Direkt-Kaufvertrag" nutzungsbedingungen "Nicht anwendbar — kein Direkt-Kaufvertrag" Vorher: 16 False Positives. Jetzt: 0. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-19 11:46:58 +02:00
Benjamin Admin	b87c27d104	fix(llm-verify): P13 — Default-Modell auf qwen3:30b-a3b (statt qwen3.5:35b-a3b) ... Bug: qwen3.5:35b-a3b liefert mit format='json' + Batch-Prompt leere Strings zurueck ('LLM batch: empty response from model'). Im echten Compliance-Check lief der LLM-Verifier deshalb wirkungslos — False-Positive-Findings wie 'Vorstand nicht erkannt' (BMW: Klammer- Liste) wurden nicht overturned. Fix: Default auf qwen3:30b-a3b umgestellt. Verifiziert mit BMW- Impressum-Text: representative_person wird mit Evidence 'Milan Nedeljkovic, Vorsitzender' overturned=True markiert. OLLAMA_VERIFY_MODEL Env-Var bleibt als Override-Moeglichkeit. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-19 09:11:01 +02:00
Benjamin Admin	28a078ccb4	feat(compliance-check): P10 — Cookie-Policy-Architecture-Detection ... Neuer Service cookie_policy_architecture.detect_architecture(...) prueft vier Diagnose-Punkte der Cookie-Policy einer Website: 1. Layer-Trennung: single (BMW-Pattern: Banner + Info in EINER URL) | separate (Best Practice: getrennte Layer) 2. Versionierung: "Stand vom DD.MM.JJJJ" / "Version X.Y" / ... 3. Dynamic content: CMP-Capture auf Doc-URL oder Marker-Texte 4. Vendor-Count im Text: Indikator ob Liste statisch drinsteht Risiko-Ampel: - gruen: separate + versioned + statisch - gelb : single+unversioned (BMW) ODER separate+unversioned - rot : weder noch (Pflicht-Info fehlt) Wire-in im Compliance-Check-Worker: nach Exec-Summary-Block wird der Architecture-Block gerendert (build_architecture_html) mit konkreter Empfehlung. Bei BMW-Pattern: "Snapshot der dynamischen Vendor-Tabelle als versioniertes PDF im Archiv." Hintergrund: BMW hat eine HTML-Seite die GLEICHZEITIG Banner-Re-Trigger und Cookie-Richtlinie ist. Mindestanforderung nach §25 TDDDG + Art. 13 DSGVO erfuellt, aber bei einer Aufsichtsbehoerden-Pruefung kann nicht belegt werden welche Vendor-Liste an einem bestimmten Stichtag aktiv war. Das ist kein Verstoss aber best-practice-Luecke. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-19 01:01:48 +02:00
Benjamin Admin	0d37822b7c	fix(impressum): P9 — 7 False-Positive-Fixes in Pflichtangaben-Checks ... #1 Name des Anbieters: \b Word-Boundary verhindert "ag" in "samstag", plus "aktiengesellschaft" als Volltreffer. #2 Vertretungsberechtigte: Klammer-Liste-Pattern erkennt jetzt BMW- Format "Vorstand (Milan Nedeljkovic, Jochen Goller, ...)" plus "Vorsitzender des Aufsichtsrats: Name". #3 V.i.S.d.P.: war schon INFO, OK. #4 OS-Plattform/VSBG: bei no_direct_sales=True (OEM-Pattern) jetzt als "Nicht anwendbar" skipped statt 0/1 fail. Profile fliesst neu durch check_document_completeness -> runner. #5 Zustaendige Kammer: IHK + Handwerkskammer + Tieraerztekammer in Pattern aufgenommen + severity LOW -> INFO (konditional). #6 Stammkapital: war schon INFO, OK. #7 Link-Disclaimer: neue Check-Eigenschaft "invert"=True. Anti-Pattern ist passed wenn NICHT gefunden, fail wenn gefunden. Vorher feuerte das Finding immer, jetzt nur wenn ein illegaler Disclaimer im Text ist. Plus: L2-INFO-Checks (z.B. profession_chamber) zaehlen nicht mehr in correctness-pct und erzeugen keine DSI-DETAIL-Findings. Konsistent mit P8-Modell: INFO = "selbst pruefen", nicht "fail". Verifiziert mit BMW-Impressum-Text — alle 7 Faelle korrekt klassifiziert: name=passed, representative_person=passed, profession_chamber=INFO, illegal_disclaimer=passed (kein Disclaimer im Text), dispute_resolution=skipped (no_direct_sales), editorial_visdp=INFO, share_capital=INFO. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-19 00:52:03 +02:00
Benjamin Admin	575644c9c5	feat(audit): P8 — MC-Severity raus, Email nur harte Findings, MC-Audit als Checkliste ... Email-Hardening (mc_scorecard.top_fails): Neue _is_hard_finding-Heuristik filtert konditionale MCs ohne Negativ-Beleg aus den Top-Auffaelligkeiten. matched_text leer + Label enthaelt "falls/sofern/wenn/soweit/ggf." -> raus, landet nur noch im MC-Audit als "selbst pruefen". DATA-2066-A05 (kostenfreie Abschaltung Standortdaten) ist das prototypische Beispiel. MC-Audit-Frontend (audit/[checkId]/page.tsx): Severity-Spalte (CRITICAL/HIGH/MEDIUM/LOW) entfernt — der MC-Audit ist eine Checkliste, keine Severity-Drohung. Stattdessen: - Spalte "Prioritaet" mit 3-Tier aus regulation-Mapping: Gesetz (DSGVO/ePrivacy/TDDDG/...) / Behoerden-Leitlinie (EDPB/DSK/EuGH/...) / Best-Practice (ISO/NIST/BSI) - 3-Status: erfuellt (✓) / nicht erfuellt (✗) / selbst pruefen (?) / nicht anwendbar (—). rowReviewStatus() leitet "selbst pruefen" aus matched_text-leer + konditionalem Label ab. - Filter umgebaut auf 5 Stati statt 4 - Default-Filter "Nicht erfuellt" (vorher "Nur Fail") Bonus: f.payload.risk_label TS-Cast im FindingsTab clean gemacht (unknown -> string). Effekt: - Email an die GF zeigt nur noch echte Belege ("DSB fehlt", "Gebuehr fuer Widerruf") - MC-Audit ist eine sachliche Pruefliste fuer den Compliance-Officer Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-19 00:30:04 +02:00
Benjamin Admin	6c223c7c9b	feat(compliance-check): exec-summary + voll-audit + TDM-respect + cookie-KB-extended + saving-scan-funnel ... P1 — Exec-Summary oben im Email-Report (4 KPIs + 2 CTAs, dunkler Gradient) P3 — no_direct_sales-Flag fuer OEM-Konfigurator-Sites; AGB/Widerruf/AGB als "NICHT ANWENDBAR" (grau) statt "NICHT GEFUNDEN" (rot) P5 — Voll-Audit Unification: alle Findings (MC + Pflichtangaben + Vendor + Redundanz) in /data/compliance_audits.db.unified_findings; neuer /api/compliance/agent/findings/<id> Endpoint + FindingsTab im Audit-UI mit Filter + CSV-Export P7 — Crawl-Hardening: TDM-Reservation-Check (robots.txt / ai.txt / Header / Meta) vor jedem Run mit 24h-Cache; HeadlessChrome-UA (Firma noch nicht gegruendet — Switch via BREAKPILOT_BRANDED_UA env); per-Domain Rate-Limit 1 req/s + max 2 concurrent P2 — Cookie-Knowledge-DB additiv erweitert (35 -> 74 Cookies): Adobe, Meta, Microsoft, LinkedIn, TikTok, HubSpot, Marketo, Salesforce, Hotjar, FullStory, Mouseflow, Intercom, Drift, Zendesk, Cloudflare, Stripe, OneTrust/Cookiebot/Usercentrics, Matomo, Pinterest, Snapchat, X/Twitter, YouTube, Vimeo, Klaviyo, Mailchimp, Mixpanel, Segment, Amplitude, Optimizely, Datadog; Wire-in in cookie_function_classifier liefert compliance_risk-Label (kritisch/hoch/mittel/gering) pro Vendor A — k-Anonymitaets-Helper (benchmark_k_anonymity) fuer P6-Vorbereitung B — Cross-Tenant-Domain-Assertion im /findings-Endpoint (expected_domain Query-Param -> 403 bei Mismatch) C — Saving-Scan-Funnel: /api/compliance/agent/saving-scan/start mit Validierung + 24h-Rate-Limit pro Domain + Lead-Persistenz in saving_scan_leads + Auto-Discovery via _run_compliance_check; 6 Tests D — Risk-Badge im Email-Vendor-Row Rechtliche Leitplanken (Memory feedback_oem_data_legal.md): nur eigene Knapp-Bewertungen + Source-Pointer, keine 1:1-Kopien fremder CMP-Texte. TDM-Opt-Out-Respect nach § 44b UrhG. KEINE Schema-Aenderungen — alles in Sidecar-SQLite. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-18 23:48:34 +02:00
Benjamin Admin	662327e8b4	feat(compliance-check): MC-Classification + Embedding + Vendor-Redundanz + Action-Recipes + Borlabs-Features ... Massiv-Update auf Basis BMW-Test-Iterationen (v1→v9): Core Compliance-Check - Sonnet check_type Klassifikation: text/process/review fuer alle 1874 MCs in compliance.doc_check_controls (script + Sidecar /data/mc_classification.db). rag_document_checker filtert auf check_type='text' fuer doc_check. Plus fits_doc_type-Audit (v2) + ui_only-Audit fuer DSA/E-Commerce-MCs in falscher doc_type-Schublade. - scope_requires-Filter: biometric/ai_decision/child_targeting MCs werden per business_profile gefiltert (FRT skipped fuer BMW etc.). - Embedding-Match (BGE-M3) als Phase-3 nach Regex-Match: Per-doc_type-Threshold-Override (impressum 0.50, dse/cookie 0.60), Short-Field-Rescue (15-Wort-Chunks) fuer Pflichtfelder im Impressum. Title+check_question als Embedding-Input fuer mehr Kontext. - Cookie-Text-Routing: consent-tester gibt cmp_cookie_text aus dem CMP-Reconstruct zurueck, Backend bevorzugt das gegen DOM-Extraction wenn richer (BMW 1824 vs 600 Worte). Vendor-Redundanz + EU-Alternativen + Cost-Saving - vendor_redundancy.analyze() — funktionale Kategorisierung der CMP-Vendors, Detektion von Mehrfach-Anbietern pro Kategorie, EU-Alternative-Lookup (Matomo, IONOS, HERE, Friendly Captcha, Smart AdServer, ...). - vendor_cost_estimator: Tier-Inferenz aus Cookie-Footprint (Cookie-Anzahl + Premium-Feature-Cookies + Third-Party-Quote → starter/professional/ enterprise/premier). - Self-Service-Werbung (Google/Meta/Pinterest/...) = 0 Lizenz-Kosten (nur Media-Spend, separat). DSP-Plattformen behalten enge Range. - Tier-aware Saving-Range: bei Enterprise/Premier nutzen wir den oberen 40-100%-Band der Listpreise, nicht starter→premier. - Multi-Function-Tools (Matomo Pro, SAP CX, IONOS Cloud, Userlike, Smart AdServer, HERE Maps, Vimeo Pro, LamaPoll) — ein Tool ersetzt mehrere Kategorien gleichzeitig. Cookie-Wissens-DB + Funktionale Klassifikation - cookie_knowledge_db: 50 kuratierte Top-Cookies (Google/Meta/Adobe/MS/...) mit vendor, exact_purpose, data_collected, IAB-TCF-IDs, reid_risk, schrems_ii_status, EuGH-Urteile, EU-Alternative. - cookie_function_classifier: pro Cookie funktionale Rolle (tracking_id, ad_pixel, session_id, ab_test, csrf, ...) + blocking_impact. Country-Inferenz aus Rechtsform - cookie_link_validator: Country-Field wird aus Vendor-Name abgeleitet (A/S=DK, GmbH=DE, Inc=US, B.V.=NL, ...) plus Vendor-Lookup-Table. Reduziert false-positive no_country-Flags bei eindeutig-EU-Vendors (Adform DK, Pinterest IE). Action-Recipes + Doc-Anchor-Locator - finding_action_recipes: pro Finding-Typ (no_cookies_listed, no_country, broken_opt_out, "Auftragsverarbeiter erwaehnen", "Art. 22 Profiling", ...) eine strukturierte Anweisung mit what/why/fix_text/where/example. Zum 1:1-Einfuegen in Kunden-Dokumente. - doc_anchor_locator: Embedding-basiert (BGE-M3 cosine) — sucht den passenden Absatz im existierenden Kundendokument fuer jeden Finding. Per-Run Thread-Local-Cache. Fallback: keyword-Match. - Email-Rendering integriert Recipe + Anchor pro Doc-Pruefungs-Fail + Vendor-Flag-Liste mit aufklappbarer Action-Liste. - Score-Erklaerung pro Vendor-Zeile (3/5-Untertitel + Tooltip). Migration-Pipeline (Compliance-Check -> Customer Banner/Documents) - migration_to_banner.py: Vendor-Liste -> CookieBannerConfig mit 4 Kategorien + Review-Flags. - migration_to_document.py: Vendor-Liste -> Cookie-Policy + VVT-Register + Privacy-Policy-Pre-Fills. - agent_migration_routes: 3 Preview-Endpoints (banner-preview, document-preview, summary). Persistierung der cmp_vendors in /data/compliance_audits.db check_payloads-Tabelle. Borlabs-Parity Cookie-Banner-Features - Consent-Historie im Banner: window.bpShowConsentHistory() + localStorage. - Content-Blocker: cookie-banner-content-blocker.ts — YouTube/Maps/Video Placeholder bis Einwilligung. - Google Consent Mode v2 erweitert: wait_for_update + region=EEA/CH/GB. - Consent-Log Export (CSV/JSON) per einwilligungen_export_routes. Bug-Fixes - canonical_control_routes: _jsonish-Helper fuer string-typed jsonb, similar-controls-Endpoint mit _has_embedding_col()-Cache (kein 500 mehr). - Control-Library Frontend: defensive .map-Coercer in 2 Detail-Views. - Embedding-Service-Batching (32er Batches statt 165 in einem Call). - KeyError 'control_id' in MC-Result-Aggregation (defensive .get). - Master-Controls-Klick-Through von /sdk/master-controls auf /sdk/control-library?control=<id> mit URL-Param-Auto-Open. - Dockerfile: /data pre-chowned auf appuser (Audit-DB-Schreibrecht). - Cookie-Text-Routing-Bug (cmp_reconstructed > DOM-extraction). - doc_type-aware MC-Filter (statt all-text-MCs). - Master-Contract-Dedup (60 BMW-Internal-Eintraege = 1 Adobe-Vertrag). - A3-v2-Audit hat 24 UI-Sprache-MCs als 'process' reklassifiziert. Tests - test_migration_mappers.py (9 Tests) - test_migration_endpoints.py (4 Tests) Skripte (one-shot) - classify_mc_check_type.py (v1) + _v2 (PK=control_id,doc_type) - audit_mc_doctype_fit.py (v1 fits) + _v2 (ui_only + scope_requires) BMW-Run-Bilanz v1 (broken) -> v9 (alle Fixes): DSE 7,5% -> 81-83% Impressum 4% -> 100% (6 echte MCs alle erfuellt) Cookie 0% -> 79-83% (CMP-Text-Routing + Embedding) Plus: 10 Konsolidierungs-Kategorien, geschaetzte Saving 200k-3M / Jahr Plus: Action-Recipes + Doc-Anchors fuer jeden Fail Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-18 18:30:08 +02:00
Benjamin Admin	df7d83134b	feat(agent): migrate compliance-check results to banner + documents (M1-M5) ... After a compliance-check run finishes, the user can now apply the extracted vendor inventory directly to their own: - CookieBanner config (admin /sdk/einwilligungen) - Cookie-Policy / VVT-Register / Privacy-Policy templates (admin /sdk/document-generator) Backend: - migration_to_banner.py: vendor list -> CookieBannerConfig with ESSENTIAL/PERFORMANCE/PERSONALIZATION/EXTERNAL_MEDIA buckets + review flags (broken opt-out URLs, missing expiry, no cookies listed) - migration_to_document.py: vendor list -> pre-fills for 3 doc templates, recipient-type aware (INTERNAL/GROUP/PROCESSOR/CONTROLLER) - agent_migration_routes.py: GET /banner-preview, /document-preview, /summary keyed on check_id - compliance_audit_log: new check_payloads table persists cmp_vendors + extracted_profile so the preview survives an app restart - tests: 9 mapper units + 4 endpoint integration tests Frontend: - MigrationPanel.tsx: modal showing banner-config diff + document pre-fills, plus links into the existing editors - ComplianceCheckTab.tsx: replaces standalone audit link with the panel; net -3 lines, stays at the 500-cap Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-17 14:06:28 +02:00
Benjamin Admin	6ed30dae5b	feat(agent): MC scorecard + audit drill-down + tenant trend (A1-A6) ... Now that all 1874 MCs run per check (Task #30 cap removal), the report was about to drown in noise. This commit adds the full aggregation / persistence / drill-down stack so each MC is actionable, not just counted. A1 mc_scorecard.py (new): build_scorecard(checks) -> per-regulation PASS/FAIL/SKIP + severity top_fails(checks, n) -> N most severe failed MCs full_audit_records(...) -> flat rows ready for sidecar SQLite A2 Email rendering: agent_doc_check_scorecard.py (new) builds an HTML scorecard table (regulation × passed/failed/HIGH/MEDIUM/score) shown at the top of the email. agent_doc_check_report._render_document now collapses the 500-MC L2 forest into 'X/Y bestanden (Z Fail)' summary plus a top-10 fails block per doc — old verbose render is gone. A3 compliance_audit_log.py (new) — sidecar SQLite at /data/compliance_audits.db (separate from compliance Postgres schema to comply with the no-new-migrations rule in CLAUDE.md): check_runs(check_id, ts, tenant_id, site_name, base_domain, doc_count, scorecard json, vvt_summary json) mc_results(check_id, doc_type, mc_id, label, passed, skipped, severity, regulation, matched_text, hint) Route persists every run after the email is sent. docker-compose.yml adds compliance-audit volume + env. A4 backfill_mc_regulation_llm.py (new) — Qwen-tagged backfill for the 1636 MCs the regex pass couldn't classify. Batches of 25, format=json, output constrained to the canonical regulation list. Run manually: docker exec bp-compliance-backend python3 \ /app/scripts/backfill_mc_regulation_llm.py [--dry-run] A5 Admin audit tab — GET /api/compliance/agent/audit/<check_id> proxied via /api/sdk/v1/agent/audit/<id>. New page /sdk/agent/audit/[checkId] renders scorecard + filterable MC table (status / doc_type / regulation, expandable rows with matched_text + hint). ComplianceCheckTab now shows 'Voll-Audit oeffnen' link. A6 Trend per tenant — GET /api/compliance/agent/audit/tenant/<id> returns recent runs. Email scorecard shows per-regulation delta badges ('(+12%)', '(-3%)') compared with the previous run for the same tenant + base_domain. Lookup is one SQLite query. Plumbing: rag_document_checker.py — SELECT now includes 'article'; MC results carry 'regulation' + 'article' through to CheckItem. agent_doc_check_routes.CheckItem schema gains regulation + article fields (defaults '') so old clients still parse. agent_compliance_check_routes — response gains 'check_id' so the frontend can build the audit link.	2026-05-17 13:45:58 +02:00
Benjamin Admin	6d29191e9b	fix(vvt): score INTERNAL/GROUP without opt-out/privacy penalty ... User feedback after BMW test: - 60 'BMW AG — XYZ' rows were rendered as ✗ for Opt-Out/Privacy and scored 38-52%. That's misleading: BMW processing for itself doesn't need a separate opt-out URL (cookie-banner is the consent mechanism) or a separate privacy policy (main DSI covers it). - Title 'Anbieter' was wrong for 60 of 90 rows (internal services). Three orthogonal fixes: 1. score_vendors becomes recipient_type aware: - INTERNAL/GROUP_COMPANY: opt_out_url, privacy_policy_url, country are NOT required (the user's main DSI + cookie-banner cover them). What IS required: name, purpose, cookies disclosed with name + expiry. Cookies-disclosure weight raised to 50 (was 15) so the VVT-relevant data is the score driver. - 'necessary' category: opt-out still skipped (§25 Abs. 2 TDDDG). - External (PROCESSOR/CONTROLLER): existing strict scoring stays. 2. _link_status_badge accepts na_label and renders a neutral em-dash with explanation tooltip instead of red ✗ when the column doesn't apply to that row. _render_vendor_row_full passes na_label based on recipient_type: - INTERNAL/GROUP -> 'Nicht erforderlich (eigene Verarbeitung)' - necessary -> 'Nicht erforderlich (§25 Abs. 2 TDDDG)' 3. Header + summary clarify the split: - h3 changed to 'Verarbeitungstaetigkeiten und Empfaenger aus der Cookie-Richtlinie' (was 'Drittanbieter aus Cookie-Richtlinie'). - Top line: '90 Verarbeitungen erfasst — 60 eigene + 30 externe Empfaenger'. - Disclaimer below: explains the INTERNAL/GROUP exemption so the reader understands why those rows don't show ✗ for missing URLs. - Section labels enriched with the relevant DSGVO article: 'Eigene Verarbeitungstaetigkeiten — fuer das VVT (Art. 30)', 'Auftragsverarbeiter — AVV erforderlich (Art. 28)', 'Joint Controller — Vereinbarung pruefen (Art. 26)'. Expected BMW result after fix: ~85% of the 60 BMW-AG rows jump from ~52% to 90-100% (the real issue, fehlende Cookies-Disclosure, stays flagged). The only true findings remaining are external links that return 4xx (e.g. Criteo 403, Teads 404).	2026-05-17 13:15:40 +02:00
Benjamin Admin	8a44e67293	feat(compliance-check): unlock all 1874 MCs + close gap-table items ... User: 'wir haben 1800 MCs erstellt um sie zu 10% zu nutzen — das ist Schwachsinn'. Fixed all 6 gaps from the audit. #1 max_controls=0 (was 20): - agent_compliance_check_routes _check_single: passes max_controls=0 to check_document_with_controls -> ALL MCs evaluated per doc_type. - 8 doc_types now use 1874 MCs instead of 160 (10x coverage). - Regex matching is cheap (<1s per doc); LLM-enrich cap of 10 stays. #2 LLM-verify fixed: - llm_verify.py was getting 0/N parsed. Causes: qwen3 thinking-mode wrapped output in <think>...</think>, /api/generate doesn't enforce JSON, prompt didn't handle code-fence wrappers. - Now uses /api/chat with format='json' (forces valid JSON). - _parse_batch_response strips <think> tags, accepts {results:[...]} AND bare [...], adds richer regex-fallback parse, logs raw head on total parse failure for diagnosis. #3 Loeschkonzept checklist (new): - doc_checks/loeschkonzept_checks.py — 9 L1 + 7 L2 checks per DIN 66398 + Art. 5(1)(e)/17/32 DSGVO: scope+responsibility, data categories, retention periods, legal basis refs (HGB/AO/BGB), deletion trigger, deletion process+technical+systems, deletion proof, exceptions + Art. 18 lock, review cycle, DSGVO references. - runner.py registered for loeschkonzept/loeschung/loeschfristen. #4 regulation backfill script: - backend-compliance/scripts/backfill_mc_regulation.py — regex-detects DSGVO/TDDDG/TMG/BGB/HGB/AO/MStV/UWG/VSBG/PAngV/GwG/BDSG/EU-VO references in MC title+question+pass_criteria, UPDATEs regulation + article fields. - Idempotent (only NULL rows), --dry-run flag, batched 200/UPDATE. - Run inside container: docker exec bp-compliance-backend python3 \ /app/scripts/backfill_mc_regulation.py #5 MC alias-fallback: - rag_document_checker._MC_ALIAS_FALLBACK maps doc_types without own MCs to a related set: nutzungsbedingungen->agb, social_media->dse, sub_processor/scc/tom_annex->avv, loeschfristen->loeschkonzept, eu_institution/dsb->dse. - _load_controls retries with the alias when the primary query returns 0 rows. - 14 additional doc_types now get MC coverage transparently. #6 cross-domain auto-discovery: - _autodiscover_missing builds a crawl plan: primary submitted base + up to 2 related domains sharing the owner SLD (e.g. BMW Group: bmw.de + bmwgroup.com + bmwgroup.jobs). - Detection: regex over submitted texts for https?://...<owner>... hostnames distinct from the primary base. - Each crawled base contributes documents + cmp_payloads to the discovery pool. Net effect for BMW: 1874 MCs evaluated (90 from cookie alone, was 20), Loeschkonzept Pflichtangaben benoten-bar, LLM overturns false regex FAILs, Joint-Controller policies on bmwgroup.jobs (Social Media) jetzt entdeckbar. Same wins will apply to CRA-Compliance check.	2026-05-17 13:07:50 +02:00
Benjamin Admin	fab1e35847	feat(vvt): recipient-type classification + 3-section VVT table ... Per user request: BMW (and others) put their own services AND external vendors in the same cookie-policy widget. The VVT-Tabelle now groups them by Art. 30(1)(d) DSGVO recipient category so the DSB can act on the right buckets: - INTERNAL — owner processing for itself ('BMW AG — XYZ') - GROUP_COMPANY — same brand family, different legal entity ('BMW Bank') - PROCESSOR — Auftragsverarbeiter, AVV-pflichtig (Adobe, Akamai) - CONTROLLER — independent / joint controller (Meta Pixel, Google Ads, LinkedIn — they run their own profiles) - AUTHORITY — government bodies (rare in cookies) - OTHER — fallback New module vendor_classifier.py: - owner_from_url(url) — derive site-owner token (bmw.de -> 'BMW', mercedes-benz.de -> 'Mercedes-Benz') - classify(name, category, owner) — strict 5-tier heuristic: * INTERNAL: vendor name first-token is '<Owner>' / '<Owner> AG' / '<Owner> SE' / '<Owner> GmbH' / '<Owner> AG & Co. KG' * GROUP_COMPANY: starts with '<Owner> ' but isn't '<Owner> AG' * CONTROLLER: matches a known joint-controller list (Meta, Google Ads, YouTube, LinkedIn Insight, TikTok, Pinterest, Taboola, Outbrain, Criteo, Twitter, Reddit, ...) * PROCESSOR: legal-form suffix in name (GmbH, AG, Inc., A/S, B.V., S.A., Ltd., LLC, ...) * OTHER: anything else vendor_extractor.extract_vendors_from_payloads now takes owner_name: - Passes it through to classify() for every extracted vendor record - The route derives owner_name via _company_name_from_url(doc_entries) - LLM-extracted vendors are classified the same way (so V3 fallback also produces tagged records) agent_doc_check_extras.build_vvt_table_html rewritten: - Buckets vendors by recipient_type - Renders one section per non-empty bucket, in canonical order (RECIPIENT_TYPE_SECTIONS), each with section header + count + bad count + nested table - Within each section: sorted by compliance_score ascending - Response JSON cmp_vendors includes recipient_type so the frontend can later import per-category into the VVT module Expected BMW result: ~60 INTERNAL rows (BMW AG own services), ~25 PROCESSOR rows (Adobe, Adform, Akamai, AWS, ...), ~5 CONTROLLER rows (Meta Pixel, Google, LinkedIn, Pinterest, Outbrain, Taboola).	2026-05-17 12:31:49 +02:00
Benjamin Admin	6c7d4c7552	fix(vvt): correct ePaaS schema mapping + category-aware scoring ... The first BMW VVT table rendered all 24 providers at 20% score because the ePaaS extractor was reading the wrong field names. Actual schema is nested: providers[].processings[].persistences[], NOT providers[] alone. Correct ePaaS schema (verified against bmw.com/epaas/.../de_DE.epaas.json): Provider: {id, name, description, processings[]} Processing: {id, name, description, categoryId, optOutLink, privacyPolicyLink, persistences[]} Persistence: {id, name, domain, type, expiry, description} Two structural changes: 1. One row per processing (not provider). BMW has 26 providers but ~91 processings spread across them (Adobe alone has ACMProcessing, AdobeAnalytics, AdobeCampaign, AdobeTargetAnalytics, AdobeTargetPers.). The cookie widget displays each processing separately — VVT now mirrors that. Display name format: 'Provider Name — Processing Name'. 2. Read optOutLink/privacyPolicyLink from PROCESSING (where they live), not provider. Persistences flatten to cookies[] with name + expiry + description. Plus category mapping: advertising -> marketing strictlyNecessary -> necessary statistics -> statistics functional -> functional Category-aware scoring (cookie_link_validator.score_vendors): - 'necessary' (technisch erforderliche, §25 Abs. 2 TDDDG): no opt-out required, no country required. Score weight shifts to purpose + cookie disclosure (essential cookies must list names + expiry). - All other categories: opt-out URL still mandatory; missing opt-out flags 'no_opt_out_url' and zeros that block of points. Expected BMW result after this fix: - ~91 rows (Adobe Analytics, Adform Retargeting, Akamai Infrastructure, AWS, ..., plus ~60 strictlyNecessary processings) - Marketing rows with present opt-out → ~75-90% - Necessary rows with cookie+expiry → ~85-95% - Rows missing fields → still flagged	2026-05-17 11:19:31 +02:00
Benjamin Admin	873997c13b	feat(vvt): V3 — LLM vendor extraction fallback for unknown CMPs ... When the cookie text has no captured CMP payload (long-tail sites that don't use ePaaS/OneTrust/Cookiebot/etc.) we now fall back to a Qwen → OVH LLM cascade to extract a structured vendor list from the policy text. New module backend/compliance/services/vendor_llm_extractor.py: - extract_vendors_via_llm(cookie_text): runs Qwen first (local Ollama), then OVH if Qwen returns nothing usable. - System prompt instructs the model to return STRICT JSON only: {vendors: [{name, country, purpose, category, opt_out_url, privacy_policy_url, persistence, cookies: [...]}]} - Lenient JSON parser tolerates code-fences, prose wrappers, dict vs list. - _normalize() caps array sizes (80 vendors, 30 cookies each), validates URLs (must be http(s)), trims fields to reasonable lengths. Route integration (agent_compliance_check_routes.py): - After named-CMP extract: if cmp_vendors is empty AND the cookie text has ≥500 words (otherwise it's likely navigation chrome), invoke the LLM extractor. Progress message 'Vendor-Liste per LLM extrahieren...'. - Vendors then run through the same validate_vendor_urls + score_vendors pipeline → VVT table rendered identically regardless of source. docker-compose.yml: backend-compliance gains OLLAMA_URL, CMP_LLM_MODEL, OVH_LLM_URL/KEY/MODEL env vars (same names as consent-tester so the configuration is unified). This closes the 'every site eventually gets a VVT table' goal: - Known CMP → V1/V2 structured extraction (fast, exact) - Unknown CMP → V3 LLM extraction (slow, best-effort) - No text at all → no vendors, but other compliance checks still run.	2026-05-17 09:55:42 +02:00
Benjamin Admin	9c0cc0f59f	feat(vvt): V2 — vendor extractors for Cookiebot/Usercentrics/Didomi/TrustArc ... Backend vendor_extractor.py gets 4 new per-CMP dispatchers, mirroring the JSON schemas observed in each platform: - Cookiebot: 'Categories[*].Cookies[*]' with Vendor/Host, expiry, purpose - Usercentrics: 'services[*]' with cookieMaxAgeSeconds, processingCompanyCountry - Didomi: 'app.vendors[*]' with country + policyUrl - TrustArc: 'vendors[*]' + per-category 'Cookies' with provider All 6 named CMPs (ePaaS, OneTrust, Cookiebot, Usercentrics, Didomi, TrustArc) plus the generic-shape fallback are now mapped — every site hitting Phase B of the cascade gets a structured vendor list, scored opt-out links, and a VVT-Tabelle in the email.	2026-05-17 09:52:10 +02:00
Benjamin Admin	ea4dbb223f	feat(vvt): per-vendor extraction + opt-out check + VVT table in email (V1) ... When a known CMP (ePaaS, OneTrust) renders the cookie policy, we now extract structured vendor records, probe their opt-out + privacy URLs, score each vendor (0-100), and append a 'VVT-Vorschlag' table to the compliance email — one row per vendor, sortable by compliance score. consent-tester: - DSIDiscoveryResult.cmp_payloads: surfaces raw CMP JSON to callers - DSIDiscoveryResponse: new cmp_payloads field - discover_dsi_documents sets cmp_payloads from cmp_capture - cmp_library/{epaas,onetrust}.py: new extract_vendors(d) returning list[VendorRecord] backend: - _fetch_text() now returns (text, cmp_payloads) tuple - doc_entries store cmp_payloads per doc (mostly cookie) - _autodiscover_missing forwards homepage payloads to the cookie entry - New module vendor_extractor.py: dispatches ePaaS/OneTrust/generic schemas; dedupes vendors across multiple payloads - cookie_link_validator.py extended with validate_vendor_urls(vendors) and score_vendors(vendors) — 0-100 score per vendor based on name, purpose, country, opt-out reachable, privacy URL reachable, cookies with names + expiry - agent_doc_check_extras.build_vvt_table_html: renders the table - Route appends VVT HTML after the provider list, before the document-by-document report - Response JSON gains cmp_vendors for future frontend rendering Example for BMW: ~30 ePaaS providers → table with Name | Kategorie | Sitz | Cookies | Opt-Out (✓/✗) | Privacy (✓/✗) | Score. Sorted by score ascending so the worst-compliant vendors are at the top.	2026-05-17 09:50:11 +02:00
Benjamin Admin	c9c0fb5965	feat(cookie-check): enhanced patterns + active opt-out link validator ... cookie_checks.py: - cookie_names_listed: now also matches CMP placeholder notation (BMW: 'Adfpc###', 'CT###') and 'Diese Datenverarbeitung verwendet die folgenden Cookies oder ähnliche Technologien' as list-shape signal. Cryptic vendor names like 'audience', 'adformfrpid' are accepted via the surrounding markup, not by hard-coding each one. - cookie_providers_named: new pattern 'Gesetzt von: <Firma>' (BMW/ePaaS per-cookie vendor naming) + recognition of full legal-form names (Adform A/S, BMW AG, Adobe Systems Software Ireland Limited). - cookie_duration_values: now matches 'Ablauf: 1 Jahr' / 'Speicherdauer: 30 Tage' (BMW format) in addition to the legacy '<n> <unit>'. New L1 + L2 checks for controller in cookie-policy: - cookie_controller (L1): the cookie policy must name Verantwortlich(er) - cookie_controller_address (L2): PLZ + Ort or address keywords - cookie_controller_contact_or_link (L2): email/phone OR link back to Datenschutzerklärung (the practical equivalent — BMW does this) New L2 checks (parented under opt_out): - cookie_optout_links: detects per-provider opt-out URLs in the text - cookie_privacy_policy_links: per-provider privacy-policy URLs New service: cookie_link_validator.py - extract_links(text): pulls all https?://… URLs that follow 'Opt-Out Link:' / 'Link zur Privacy Policy:' (deduped) - validate_links(links): probes every URL concurrently (HEAD first, GET fallback for 405/403). 10 parallel, 8s per request, 60s batch cap. Returns reachable=True/False + status + final_url. - build_check_items(): renders 2 CheckItems (opt-out + privacy-policy), each pass if ALL links 2xx/3xx, fail with up-to-5 broken-link examples. Hook in _check_single: doc_type=='cookie' triggers the validator after regex+MC checks. Recomputes correctness with the new L2 items. This addresses two concrete BMW observations: 1. BMW's per-cookie structure (Name + Zweck + Ablauf, Gesetzt von: …, Opt-Out Link: …) now recognised → 'Konkrete Cookie-Namen aufgelistet' and 'Konkrete Speicherdauern' should pass. 2. Defective opt-out URLs surface as compliance findings rather than silently passing — Art. 7(3) DSGVO requires a working withdrawal path per provider.	2026-05-17 09:38:32 +02:00
Benjamin Admin	b090662524	fix(compliance-check): respect auto-discovery 'not found' verdict; DSB not canonical ... Two related bugs in the BMW test result: 1. AGB rendered as 'MANGELHAFT 0/13' even though BMW has no public AGB: - Auto-discovery correctly returned 'not found' for AGB (no link on bmw.de matches AGB keywords). - But auto_fill_from_dsi then found the substring 'AGB' in a section of the DSI and pseudo-filled the AGB entry with a 264-word DSI fragment. - cross_search_documents would have done the same. - Both now skip entries where discovery_attempted=True AND auto_discovered=False — the 'not found' verdict stands. 2. DSB-Kontakt rendered as a separate 100% OK document with 7566 words = the entire DSI text: - GDPR practice: the DSB is named *inside* the DSI as an email or contact block (Art. 13(1)(b)), not as a stand-alone page. - cross_search_documents had been assigning the full DSI to the DSB row because it matched 'datenschutzbeauftragte' keywords. - DSB removed from _ALL_DOC_TYPES — no longer canonical, no longer padded as missing, no longer auto-discovered. The frontend row remains so a tenant with a separate DSB page can still submit one. After this fix BMW should render: - DSE: OK - Impressum: LUECKENHAFT (unchanged — regex gaps to fix separately) - Cookie-Richtlinie: OK - Social Media: NICHT GEFUNDEN (bmw.de does not link to it) - AGB: NICHT GEFUNDEN (correct — BMW has no public AGB) - Nutzungsbedingungen: NICHT GEFUNDEN - Widerruf: NICHT GEFUNDEN	2026-05-17 01:53:09 +02:00
Benjamin Admin	bc21480a2a	fix(compliance-check): always render 8 doc types + 4 BMW GT-gap fixes ... Always-show-8 (user-requested): - agent_compliance_check_routes.py: _pad_results_with_missing pads the results list to always include all 8 canonical doc_types in canonical order. Missing types get a placeholder DocCheckResult with error= 'Nicht eingereicht' + scenario='missing'. - agent_doc_check_report.py: NICHT EINGEREICHT status label (neutral), friendly grey body block instead of red error. - ChecklistView.tsx: 'Nicht eingereicht' chip (neutral grey, not red 'Fehler'); SCENARIO_LABELS adds missing entry + header chip counter. Impressum-Regression fix (#18): - _fetch_text(url, doc_type): cookie/dse/social_media -> max_documents=1 (CMP capture authoritative, sub-pages dilute). Other types -> =3 (Impressum needs Versicherungsvermittler, Aufsicht, Berufsrecht sub- pages). 15s networkidle bail keeps timing safe. ODR/Verbraucherstreitbeilegung filter (#19): - _apply_profile_filter: when profile.needs_odr=True (B2C), override the check's default B2B-oriented hint with action-oriented B2C guidance pointing at Art. 14 EU-VO 524/2013 + §36 VSBG. Previously the check contradicted itself: 'profile says B2C' + hint 'only relevant for B2C online vendors'. Registergericht regex (#20): - impressum_checks.py: accept colon/dot/dash between keyword and city (BMW writes 'registergericht: münchen hrb 42243'). Add 'sitz und registergericht: X' as separate pattern. Industry detection (#21): - business_profiler.py: 'automotive' keywords broadened (antriebs, motor, leasing, werkstatt, probefahrt, plus brand names BMW/Mercedes/ Audi/VW/Porsche/Opel). 'it_services' keywords narrowed — software/ cloud/hosting are mentioned in every privacy policy and were biasing the result toward IT for any tech-aware company.	2026-05-17 01:03:58 +02:00
Benjamin Admin	e61e9d9e2a	feat(agent): progress_pct + 6 BMW-Run Verbesserungen ... Backend (agent_compliance_check_routes.py): - progress_pct (0-100%) im Job-State, ueber alle Phasen verteilt (Laden 0-30, Profil 35-40, Pruefen 40-80, Banner 80-92, Report 95-100) - Status-Texte vereinheitlicht ("Texte laden X/N", "Pruefen X/N") - Firmenname fuer Email-Subject jetzt aus URL abgeleitet (bmw.de -> "BMW", mercedes-benz.de -> "Mercedes-Benz") statt unzuverlaessigem extracted_profile.companyName (matchte oft juris.de) - E-Mail-Report enthaelt jetzt Banner+TCF-Vendor-Liste (build_provider_list_html) Backend (agent_doc_check_extras.py — neu): - build_scanned_urls_html: gepruefte URLs als Tabelle oben im Report (transparent fuer GF, welche Quellen wirklich gezogen wurden) - Cross-Domain-Hinweis bei >1 netloc (BMW: bmw.de / bmwgroup.com / bmwgroup.jobs — Auffindbarkeit nach Art. 12 DSGVO) - build_provider_list_html: Banner-Box + TCF-Vendor-Tabelle mit Spalten Name | Kategorie | Zweck | Drittland | Rechtsgrundlage Backend (business_profiler.py): - §34d-GewO Versicherungsvermittler-Hinweise zaehlen nicht mehr als "finance"-Industrie (BMW wurde dadurch falsch als B2B/finance erkannt) - Neue Industry "automotive" (Fahrzeug/KFZ/Konfigurator/Modellpalette) - B2B-Keywords: generische Begriffe wie "unternehmen", "beratung", "consulting" entfernt (matchten in jedem Konzerntext) - B2C-Fallback: bei Verbraucher-Signalen ("widerruf", "kunde", redaktioneller Inhalt) tendiert auf b2c statt b2b Frontend (ComplianceCheckTab.tsx): - Progress-Balken mit Width-% und XX%-Anzeige rechts - liest data.progress_pct aus Polling-Response Consent-Tester (dsi_discovery.py): - Cookie-Policy-Extraktion kritisch fixt: wait_for_function bis body.innerText > 500 chars (BMW SPA-Rendering brauchte mehr Zeit) - _extract_text_robust: 3-Strategien-Extraktion (Selektoren -> Body- Cleanup -> P/LI/TD-Tags) - _extract_text_from_iframes: liest OneTrust/Sourcepoint/Usercentrics Iframe-Inhalte (manche Cookie-Policies leben dort) Adressiert alle Findings aus dem BMW-Ground-Truth-Vergleich.	2026-05-16 17:53:14 +02:00
Benjamin Admin	bd2d6976d6	fix(cross-doc): also check entries with wrong text, not just empty ones ... Cross-search now validates if existing text matches the expected doc_type using keyword scoring. If text is present but doesn't match (e.g. Nutzungsbedingungen in Widerruf row), searches other texts and creates a finding explaining the mismatch. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-05-15 00:19:40 +02:00
Benjamin Admin	4e9043f26d	feat(cross-doc): search all texts for all doc_types + misplacement finding ... Cross-Document Intelligence: When a doc_type row is empty, searches ALL other loaded documents for that content. If found (e.g. Widerruf in AGB), extracts the section, runs the check, AND creates a finding: "Widerrufsbelehrung in falschem Dokument gefunden — schwer auffindbar" Keywords for: widerruf, cookie, social_media, impressum, agb, dsb. Integrated as Step 1c in compliance check pipeline. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-05-14 23:19:39 +02:00
Benjamin Admin	33bf2b7c5a	feat(service-detector): detect 118 services in legal texts (was 20) ... New service_detector.py uses service_registry (88 entries) plus 30+ extra text patterns to detect services mentioned in DSI/legal texts. Results on Spiegel: 31/32 services detected (97%, was 5/32 = 16%). Includes metadata: name, category, country, EU adequacy status. - Profiler now uses detect_services_in_text() instead of 20-entry list - Profile extractor adds detected_services with full metadata - Auto-generates scope hint for non-EU services (Drittlandtransfer) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-05-13 16:00:15 +02:00
Benjamin Admin	5e317d2f0f	fix: text extraction 50k char limit was root cause of all Spiegel FNs ... ROOT CAUSE: main.py line 338 truncated full_text at 50,000 chars. Spiegel DSI has 107,720 chars (13,705 words) — only 47% was extracted. DSB, Art. 77, Betroffenenrechte were all in the truncated portion. Fixes: 1. Raise text limit from 50k to 200k chars in API response + discovery 2. click_button(): add iframe fallback for Sourcepoint/Quantcast 3. dsi_helpers: iterate ALL page.frames for consent buttons 4. Profiler: only check impressum (not full text) for regulated professions, and "rechtsanwalt" must be in first 500 chars (company description) 5. GT: save full Spiegel DSI text (13,705 words) as reference Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-05-13 15:22:38 +02:00
Benjamin Admin	c702260ec1	fix: 5 regex bugs + text extraction scroll + GT update ... Root cause: Spiegel DSI text was truncated (lazy-loading) — the rights/DSB/complaints sections at the bottom were never extracted. Fixes: 1. Text extraction: scroll to bottom before innerText (dsi_discovery.py) 2. V.i.S.d.P.: add "verantwortlicher i.s.v." + "§18 Abs. N MStV" pattern 3. USt-IdNr: add "umsatzsteuer-id" + "DE 212 442 423" (with spaces) 4. Profiler: remove generic "anwalt"/"praxis" (false positive on Spiegel "Redaktionsanwalt"), keep only "rechtsanwalt", "kanzlei" etc. 5. Section splitter: auto_fill_from_dsi() fills empty Cookie/Social-Media rows from sections found in the DSI text Ground Truth 06-spiegel.md fully rewritten with verified data from live website — 3 L1 False Negatives identified (DSB, Beschwerderecht, Betroffenenrechte all present on website but not in extracted text). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-05-13 01:20:55 +02:00
Benjamin Admin	0b9150f16f	feat(vendor-assessment): Pruefprotokoll + Frontend + Sidebar ... Phase 4-5: Professional Pruefprotokoll report builder with styled HTML output (Kopfdaten, Kategorie-Scores, L1/L2 Check-Hierarchie, Findings, Freigabe-Block). Frontend at /sdk/vendor-assessment with 3-step flow: DocumentUploader → AssessmentProgress → PruefprotokollView. Sidebar: "Use-Case Audits" → "Vertragspruefung" renamed. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-05-12 23:24:12 +02:00