36 Commits

Author	SHA1	Message	Date
Benjamin Admin	313982c6f1	feat(profile+report): P17 — 4 Polish-Items ... A) Cookie-Policy-Architecture-Block Fallback auf DSE-Text wenn cookie via P15 deduped wurde. Erkennt jetzt auch single-doc Sites (Safetykon-Pattern). B) Konkrete-Aufgaben-Liste: Per-Doc-Cap (3) entfernt + globaler Cap 10→20. Safetykon zeigt jetzt 7 statt 4 Aufgaben. C) business_type-Klassifizierer: B2B-Service-Cluster aus P14 als Boost. Bei 2+ Service-Indikatoren (CE-Zertifizierung/Compliance/Auditierung) wird b2b_score angehoben. Safetykon: "B2C consulting" → "B2B (consulting)". D) Vendor-Extract Fallback auf DSE-Text wenn cookie deduped + keine CMP- Payloads. LLM extrahiert dann Vendors aus dem DSE-Text. Safetykon: 0 → 1 Vendor (Google Analytics aus dem DSE-Text erkannt). Smoke-Test Safetykon: alle 4 Polish-Items wirken, kein Regression. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-19 12:22:05 +02:00
Benjamin Admin	479ce2225b	feat(profile): P14+P15+P16 — B2B-Heuristik + Doc-URL-Dedup + Homepage-Profile ... P14 — _detect_no_direct_sales erweitert um 3 Cluster: A) OEM-Konfigurator (BMW/Audi/Mercedes/VW/Porsche-Markennamen + Vertragshaendler-Pattern) B) B2B-Dienstleister (CE-Zertifizierung, Compliance-Beratung, Schulungen, Auditierung, TISAX, ISO-Normen, Arbeitssicherheit, ...) C) NGO/Verein/Public (Spendenkonto, Vereinsregister, gemeinnuetzig, ...) Schwelle: pos >= 2 pro Cluster UND pos > neg. Bisher: nur OEM. P15 — Doc-URL-Dedup im Worker: wenn mehrere Doc-Types DASSELBE Dokument referenzieren (Safetykon-Pattern: User gibt /datenschutz fuer dse, cookie UND widerruf), wird nur dem primaeren Doc-Type (Priority: dse > impressum > cookie > widerruf > agb > nutzungsbedingungen) der Text gegeben. Andere landen als "Nicht separat vorhanden — wird im Dokument 'X' mit-geprueft." Eliminiert die 8+8 systematischen widerruf/cookie False Positives. P16 — Profile-Detection auch Homepage-Text: Homepage-HTML wird mit kurzem Fetch (8s timeout) gezogen, getrippt und zum profile_input gemerged. Vor- her wirkte P14 nur wenn B2B-Indikatoren im DSE/Impressum-Pflichttext standen — bei Safetykon stehen sie nur im Homepage-Menue. Plus Bonus: TDM-Override-Submit-Button wird deaktiviert wenn Reason < 10 Zeichen — verhindert dass User wie heute in den Bug rein klickt. Smoke-Test Safetykon (B2B Compliance-Dienstleister): dse geprueft (kein err) impressum geprueft (kein err) cookie "Nicht separat vorhanden — wird in DSE mit-geprueft" agb "Nicht anwendbar — kein Direkt-Kaufvertrag" widerruf "Nicht anwendbar — kein Direkt-Kaufvertrag" nutzungsbedingungen "Nicht anwendbar — kein Direkt-Kaufvertrag" Vorher: 16 False Positives. Jetzt: 0. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-19 11:46:58 +02:00
Benjamin Admin	78b27d4684	feat(compliance-check): P12 — TDM-Override mit dokumentierter Kunden-Erlaubnis ... Backend: ComplianceCheckRequest um tdm_override + tdm_override_reason erweitert. Worker im _run_compliance_check Pfad: bei tdm_override=True UND Reason >= 10 Zeichen wird der TDM-Vorbehalt nur dokumentiert (job.tdm_override.{reason, original_status}) und NICHT als Abbruch-Grund gewertet. Ohne Reason: Override ignoriert. Audit-Spur via logger.warning(reason). Frontend: ComplianceCheckTab um Checkbox + Pflicht-Reason-Feld ("Schriftliche Crawl-Erlaubnis vorhanden") direkt vor dem Submit- Button. Pflicht: Reason >= 10 Zeichen. Submit sendet die Flags ans Backend. Anwendungsfall: Safetykon-Pattern — robots.txt + ai.txt setzen Vorbehalt, aber Kunde hat schriftlich zugestimmt (Auftrags-Audit). [guardrail-change] ComplianceCheckTab.tsx (511 LOC) in loc-exceptions ergaenzt — Split nach _components/TDMOverride + CompliancePolling ist P11-Tech-Debt. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-19 08:56:50 +02:00
Benjamin Admin	28a078ccb4	feat(compliance-check): P10 — Cookie-Policy-Architecture-Detection ... Neuer Service cookie_policy_architecture.detect_architecture(...) prueft vier Diagnose-Punkte der Cookie-Policy einer Website: 1. Layer-Trennung: single (BMW-Pattern: Banner + Info in EINER URL) | separate (Best Practice: getrennte Layer) 2. Versionierung: "Stand vom DD.MM.JJJJ" / "Version X.Y" / ... 3. Dynamic content: CMP-Capture auf Doc-URL oder Marker-Texte 4. Vendor-Count im Text: Indikator ob Liste statisch drinsteht Risiko-Ampel: - gruen: separate + versioned + statisch - gelb : single+unversioned (BMW) ODER separate+unversioned - rot : weder noch (Pflicht-Info fehlt) Wire-in im Compliance-Check-Worker: nach Exec-Summary-Block wird der Architecture-Block gerendert (build_architecture_html) mit konkreter Empfehlung. Bei BMW-Pattern: "Snapshot der dynamischen Vendor-Tabelle als versioniertes PDF im Archiv." Hintergrund: BMW hat eine HTML-Seite die GLEICHZEITIG Banner-Re-Trigger und Cookie-Richtlinie ist. Mindestanforderung nach §25 TDDDG + Art. 13 DSGVO erfuellt, aber bei einer Aufsichtsbehoerden-Pruefung kann nicht belegt werden welche Vendor-Liste an einem bestimmten Stichtag aktiv war. Das ist kein Verstoss aber best-practice-Luecke. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-19 01:01:48 +02:00
Benjamin Admin	0d37822b7c	fix(impressum): P9 — 7 False-Positive-Fixes in Pflichtangaben-Checks ... #1 Name des Anbieters: \b Word-Boundary verhindert "ag" in "samstag", plus "aktiengesellschaft" als Volltreffer. #2 Vertretungsberechtigte: Klammer-Liste-Pattern erkennt jetzt BMW- Format "Vorstand (Milan Nedeljkovic, Jochen Goller, ...)" plus "Vorsitzender des Aufsichtsrats: Name". #3 V.i.S.d.P.: war schon INFO, OK. #4 OS-Plattform/VSBG: bei no_direct_sales=True (OEM-Pattern) jetzt als "Nicht anwendbar" skipped statt 0/1 fail. Profile fliesst neu durch check_document_completeness -> runner. #5 Zustaendige Kammer: IHK + Handwerkskammer + Tieraerztekammer in Pattern aufgenommen + severity LOW -> INFO (konditional). #6 Stammkapital: war schon INFO, OK. #7 Link-Disclaimer: neue Check-Eigenschaft "invert"=True. Anti-Pattern ist passed wenn NICHT gefunden, fail wenn gefunden. Vorher feuerte das Finding immer, jetzt nur wenn ein illegaler Disclaimer im Text ist. Plus: L2-INFO-Checks (z.B. profession_chamber) zaehlen nicht mehr in correctness-pct und erzeugen keine DSI-DETAIL-Findings. Konsistent mit P8-Modell: INFO = "selbst pruefen", nicht "fail". Verifiziert mit BMW-Impressum-Text — alle 7 Faelle korrekt klassifiziert: name=passed, representative_person=passed, profession_chamber=INFO, illegal_disclaimer=passed (kein Disclaimer im Text), dispute_resolution=skipped (no_direct_sales), editorial_visdp=INFO, share_capital=INFO. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-19 00:52:03 +02:00
Benjamin Admin	6c223c7c9b	feat(compliance-check): exec-summary + voll-audit + TDM-respect + cookie-KB-extended + saving-scan-funnel ... P1 — Exec-Summary oben im Email-Report (4 KPIs + 2 CTAs, dunkler Gradient) P3 — no_direct_sales-Flag fuer OEM-Konfigurator-Sites; AGB/Widerruf/AGB als "NICHT ANWENDBAR" (grau) statt "NICHT GEFUNDEN" (rot) P5 — Voll-Audit Unification: alle Findings (MC + Pflichtangaben + Vendor + Redundanz) in /data/compliance_audits.db.unified_findings; neuer /api/compliance/agent/findings/<id> Endpoint + FindingsTab im Audit-UI mit Filter + CSV-Export P7 — Crawl-Hardening: TDM-Reservation-Check (robots.txt / ai.txt / Header / Meta) vor jedem Run mit 24h-Cache; HeadlessChrome-UA (Firma noch nicht gegruendet — Switch via BREAKPILOT_BRANDED_UA env); per-Domain Rate-Limit 1 req/s + max 2 concurrent P2 — Cookie-Knowledge-DB additiv erweitert (35 -> 74 Cookies): Adobe, Meta, Microsoft, LinkedIn, TikTok, HubSpot, Marketo, Salesforce, Hotjar, FullStory, Mouseflow, Intercom, Drift, Zendesk, Cloudflare, Stripe, OneTrust/Cookiebot/Usercentrics, Matomo, Pinterest, Snapchat, X/Twitter, YouTube, Vimeo, Klaviyo, Mailchimp, Mixpanel, Segment, Amplitude, Optimizely, Datadog; Wire-in in cookie_function_classifier liefert compliance_risk-Label (kritisch/hoch/mittel/gering) pro Vendor A — k-Anonymitaets-Helper (benchmark_k_anonymity) fuer P6-Vorbereitung B — Cross-Tenant-Domain-Assertion im /findings-Endpoint (expected_domain Query-Param -> 403 bei Mismatch) C — Saving-Scan-Funnel: /api/compliance/agent/saving-scan/start mit Validierung + 24h-Rate-Limit pro Domain + Lead-Persistenz in saving_scan_leads + Auto-Discovery via _run_compliance_check; 6 Tests D — Risk-Badge im Email-Vendor-Row Rechtliche Leitplanken (Memory feedback_oem_data_legal.md): nur eigene Knapp-Bewertungen + Source-Pointer, keine 1:1-Kopien fremder CMP-Texte. TDM-Opt-Out-Respect nach § 44b UrhG. KEINE Schema-Aenderungen — alles in Sidecar-SQLite. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-18 23:48:34 +02:00
Benjamin Admin	662327e8b4	feat(compliance-check): MC-Classification + Embedding + Vendor-Redundanz + Action-Recipes + Borlabs-Features ... Massiv-Update auf Basis BMW-Test-Iterationen (v1→v9): Core Compliance-Check - Sonnet check_type Klassifikation: text/process/review fuer alle 1874 MCs in compliance.doc_check_controls (script + Sidecar /data/mc_classification.db). rag_document_checker filtert auf check_type='text' fuer doc_check. Plus fits_doc_type-Audit (v2) + ui_only-Audit fuer DSA/E-Commerce-MCs in falscher doc_type-Schublade. - scope_requires-Filter: biometric/ai_decision/child_targeting MCs werden per business_profile gefiltert (FRT skipped fuer BMW etc.). - Embedding-Match (BGE-M3) als Phase-3 nach Regex-Match: Per-doc_type-Threshold-Override (impressum 0.50, dse/cookie 0.60), Short-Field-Rescue (15-Wort-Chunks) fuer Pflichtfelder im Impressum. Title+check_question als Embedding-Input fuer mehr Kontext. - Cookie-Text-Routing: consent-tester gibt cmp_cookie_text aus dem CMP-Reconstruct zurueck, Backend bevorzugt das gegen DOM-Extraction wenn richer (BMW 1824 vs 600 Worte). Vendor-Redundanz + EU-Alternativen + Cost-Saving - vendor_redundancy.analyze() — funktionale Kategorisierung der CMP-Vendors, Detektion von Mehrfach-Anbietern pro Kategorie, EU-Alternative-Lookup (Matomo, IONOS, HERE, Friendly Captcha, Smart AdServer, ...). - vendor_cost_estimator: Tier-Inferenz aus Cookie-Footprint (Cookie-Anzahl + Premium-Feature-Cookies + Third-Party-Quote → starter/professional/ enterprise/premier). - Self-Service-Werbung (Google/Meta/Pinterest/...) = 0 Lizenz-Kosten (nur Media-Spend, separat). DSP-Plattformen behalten enge Range. - Tier-aware Saving-Range: bei Enterprise/Premier nutzen wir den oberen 40-100%-Band der Listpreise, nicht starter→premier. - Multi-Function-Tools (Matomo Pro, SAP CX, IONOS Cloud, Userlike, Smart AdServer, HERE Maps, Vimeo Pro, LamaPoll) — ein Tool ersetzt mehrere Kategorien gleichzeitig. Cookie-Wissens-DB + Funktionale Klassifikation - cookie_knowledge_db: 50 kuratierte Top-Cookies (Google/Meta/Adobe/MS/...) mit vendor, exact_purpose, data_collected, IAB-TCF-IDs, reid_risk, schrems_ii_status, EuGH-Urteile, EU-Alternative. - cookie_function_classifier: pro Cookie funktionale Rolle (tracking_id, ad_pixel, session_id, ab_test, csrf, ...) + blocking_impact. Country-Inferenz aus Rechtsform - cookie_link_validator: Country-Field wird aus Vendor-Name abgeleitet (A/S=DK, GmbH=DE, Inc=US, B.V.=NL, ...) plus Vendor-Lookup-Table. Reduziert false-positive no_country-Flags bei eindeutig-EU-Vendors (Adform DK, Pinterest IE). Action-Recipes + Doc-Anchor-Locator - finding_action_recipes: pro Finding-Typ (no_cookies_listed, no_country, broken_opt_out, "Auftragsverarbeiter erwaehnen", "Art. 22 Profiling", ...) eine strukturierte Anweisung mit what/why/fix_text/where/example. Zum 1:1-Einfuegen in Kunden-Dokumente. - doc_anchor_locator: Embedding-basiert (BGE-M3 cosine) — sucht den passenden Absatz im existierenden Kundendokument fuer jeden Finding. Per-Run Thread-Local-Cache. Fallback: keyword-Match. - Email-Rendering integriert Recipe + Anchor pro Doc-Pruefungs-Fail + Vendor-Flag-Liste mit aufklappbarer Action-Liste. - Score-Erklaerung pro Vendor-Zeile (3/5-Untertitel + Tooltip). Migration-Pipeline (Compliance-Check -> Customer Banner/Documents) - migration_to_banner.py: Vendor-Liste -> CookieBannerConfig mit 4 Kategorien + Review-Flags. - migration_to_document.py: Vendor-Liste -> Cookie-Policy + VVT-Register + Privacy-Policy-Pre-Fills. - agent_migration_routes: 3 Preview-Endpoints (banner-preview, document-preview, summary). Persistierung der cmp_vendors in /data/compliance_audits.db check_payloads-Tabelle. Borlabs-Parity Cookie-Banner-Features - Consent-Historie im Banner: window.bpShowConsentHistory() + localStorage. - Content-Blocker: cookie-banner-content-blocker.ts — YouTube/Maps/Video Placeholder bis Einwilligung. - Google Consent Mode v2 erweitert: wait_for_update + region=EEA/CH/GB. - Consent-Log Export (CSV/JSON) per einwilligungen_export_routes. Bug-Fixes - canonical_control_routes: _jsonish-Helper fuer string-typed jsonb, similar-controls-Endpoint mit _has_embedding_col()-Cache (kein 500 mehr). - Control-Library Frontend: defensive .map-Coercer in 2 Detail-Views. - Embedding-Service-Batching (32er Batches statt 165 in einem Call). - KeyError 'control_id' in MC-Result-Aggregation (defensive .get). - Master-Controls-Klick-Through von /sdk/master-controls auf /sdk/control-library?control=<id> mit URL-Param-Auto-Open. - Dockerfile: /data pre-chowned auf appuser (Audit-DB-Schreibrecht). - Cookie-Text-Routing-Bug (cmp_reconstructed > DOM-extraction). - doc_type-aware MC-Filter (statt all-text-MCs). - Master-Contract-Dedup (60 BMW-Internal-Eintraege = 1 Adobe-Vertrag). - A3-v2-Audit hat 24 UI-Sprache-MCs als 'process' reklassifiziert. Tests - test_migration_mappers.py (9 Tests) - test_migration_endpoints.py (4 Tests) Skripte (one-shot) - classify_mc_check_type.py (v1) + _v2 (PK=control_id,doc_type) - audit_mc_doctype_fit.py (v1 fits) + _v2 (ui_only + scope_requires) BMW-Run-Bilanz v1 (broken) -> v9 (alle Fixes): DSE 7,5% -> 81-83% Impressum 4% -> 100% (6 echte MCs alle erfuellt) Cookie 0% -> 79-83% (CMP-Text-Routing + Embedding) Plus: 10 Konsolidierungs-Kategorien, geschaetzte Saving 200k-3M / Jahr Plus: Action-Recipes + Doc-Anchors fuer jeden Fail Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-18 18:30:08 +02:00
Benjamin Admin	df7d83134b	feat(agent): migrate compliance-check results to banner + documents (M1-M5) ... After a compliance-check run finishes, the user can now apply the extracted vendor inventory directly to their own: - CookieBanner config (admin /sdk/einwilligungen) - Cookie-Policy / VVT-Register / Privacy-Policy templates (admin /sdk/document-generator) Backend: - migration_to_banner.py: vendor list -> CookieBannerConfig with ESSENTIAL/PERFORMANCE/PERSONALIZATION/EXTERNAL_MEDIA buckets + review flags (broken opt-out URLs, missing expiry, no cookies listed) - migration_to_document.py: vendor list -> pre-fills for 3 doc templates, recipient-type aware (INTERNAL/GROUP/PROCESSOR/CONTROLLER) - agent_migration_routes.py: GET /banner-preview, /document-preview, /summary keyed on check_id - compliance_audit_log: new check_payloads table persists cmp_vendors + extracted_profile so the preview survives an app restart - tests: 9 mapper units + 4 endpoint integration tests Frontend: - MigrationPanel.tsx: modal showing banner-config diff + document pre-fills, plus links into the existing editors - ComplianceCheckTab.tsx: replaces standalone audit link with the panel; net -3 lines, stays at the 500-cap Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-17 14:06:28 +02:00
Benjamin Admin	6ed30dae5b	feat(agent): MC scorecard + audit drill-down + tenant trend (A1-A6) ... Now that all 1874 MCs run per check (Task #30 cap removal), the report was about to drown in noise. This commit adds the full aggregation / persistence / drill-down stack so each MC is actionable, not just counted. A1 mc_scorecard.py (new): build_scorecard(checks) -> per-regulation PASS/FAIL/SKIP + severity top_fails(checks, n) -> N most severe failed MCs full_audit_records(...) -> flat rows ready for sidecar SQLite A2 Email rendering: agent_doc_check_scorecard.py (new) builds an HTML scorecard table (regulation × passed/failed/HIGH/MEDIUM/score) shown at the top of the email. agent_doc_check_report._render_document now collapses the 500-MC L2 forest into 'X/Y bestanden (Z Fail)' summary plus a top-10 fails block per doc — old verbose render is gone. A3 compliance_audit_log.py (new) — sidecar SQLite at /data/compliance_audits.db (separate from compliance Postgres schema to comply with the no-new-migrations rule in CLAUDE.md): check_runs(check_id, ts, tenant_id, site_name, base_domain, doc_count, scorecard json, vvt_summary json) mc_results(check_id, doc_type, mc_id, label, passed, skipped, severity, regulation, matched_text, hint) Route persists every run after the email is sent. docker-compose.yml adds compliance-audit volume + env. A4 backfill_mc_regulation_llm.py (new) — Qwen-tagged backfill for the 1636 MCs the regex pass couldn't classify. Batches of 25, format=json, output constrained to the canonical regulation list. Run manually: docker exec bp-compliance-backend python3 \ /app/scripts/backfill_mc_regulation_llm.py [--dry-run] A5 Admin audit tab — GET /api/compliance/agent/audit/<check_id> proxied via /api/sdk/v1/agent/audit/<id>. New page /sdk/agent/audit/[checkId] renders scorecard + filterable MC table (status / doc_type / regulation, expandable rows with matched_text + hint). ComplianceCheckTab now shows 'Voll-Audit oeffnen' link. A6 Trend per tenant — GET /api/compliance/agent/audit/tenant/<id> returns recent runs. Email scorecard shows per-regulation delta badges ('(+12%)', '(-3%)') compared with the previous run for the same tenant + base_domain. Lookup is one SQLite query. Plumbing: rag_document_checker.py — SELECT now includes 'article'; MC results carry 'regulation' + 'article' through to CheckItem. agent_doc_check_routes.CheckItem schema gains regulation + article fields (defaults '') so old clients still parse. agent_compliance_check_routes — response gains 'check_id' so the frontend can build the audit link.	2026-05-17 13:45:58 +02:00
Benjamin Admin	8a44e67293	feat(compliance-check): unlock all 1874 MCs + close gap-table items ... User: 'wir haben 1800 MCs erstellt um sie zu 10% zu nutzen — das ist Schwachsinn'. Fixed all 6 gaps from the audit. #1 max_controls=0 (was 20): - agent_compliance_check_routes _check_single: passes max_controls=0 to check_document_with_controls -> ALL MCs evaluated per doc_type. - 8 doc_types now use 1874 MCs instead of 160 (10x coverage). - Regex matching is cheap (<1s per doc); LLM-enrich cap of 10 stays. #2 LLM-verify fixed: - llm_verify.py was getting 0/N parsed. Causes: qwen3 thinking-mode wrapped output in <think>...</think>, /api/generate doesn't enforce JSON, prompt didn't handle code-fence wrappers. - Now uses /api/chat with format='json' (forces valid JSON). - _parse_batch_response strips <think> tags, accepts {results:[...]} AND bare [...], adds richer regex-fallback parse, logs raw head on total parse failure for diagnosis. #3 Loeschkonzept checklist (new): - doc_checks/loeschkonzept_checks.py — 9 L1 + 7 L2 checks per DIN 66398 + Art. 5(1)(e)/17/32 DSGVO: scope+responsibility, data categories, retention periods, legal basis refs (HGB/AO/BGB), deletion trigger, deletion process+technical+systems, deletion proof, exceptions + Art. 18 lock, review cycle, DSGVO references. - runner.py registered for loeschkonzept/loeschung/loeschfristen. #4 regulation backfill script: - backend-compliance/scripts/backfill_mc_regulation.py — regex-detects DSGVO/TDDDG/TMG/BGB/HGB/AO/MStV/UWG/VSBG/PAngV/GwG/BDSG/EU-VO references in MC title+question+pass_criteria, UPDATEs regulation + article fields. - Idempotent (only NULL rows), --dry-run flag, batched 200/UPDATE. - Run inside container: docker exec bp-compliance-backend python3 \ /app/scripts/backfill_mc_regulation.py #5 MC alias-fallback: - rag_document_checker._MC_ALIAS_FALLBACK maps doc_types without own MCs to a related set: nutzungsbedingungen->agb, social_media->dse, sub_processor/scc/tom_annex->avv, loeschfristen->loeschkonzept, eu_institution/dsb->dse. - _load_controls retries with the alias when the primary query returns 0 rows. - 14 additional doc_types now get MC coverage transparently. #6 cross-domain auto-discovery: - _autodiscover_missing builds a crawl plan: primary submitted base + up to 2 related domains sharing the owner SLD (e.g. BMW Group: bmw.de + bmwgroup.com + bmwgroup.jobs). - Detection: regex over submitted texts for https?://...<owner>... hostnames distinct from the primary base. - Each crawled base contributes documents + cmp_payloads to the discovery pool. Net effect for BMW: 1874 MCs evaluated (90 from cookie alone, was 20), Loeschkonzept Pflichtangaben benoten-bar, LLM overturns false regex FAILs, Joint-Controller policies on bmwgroup.jobs (Social Media) jetzt entdeckbar. Same wins will apply to CRA-Compliance check.	2026-05-17 13:07:50 +02:00
Benjamin Admin	fab1e35847	feat(vvt): recipient-type classification + 3-section VVT table ... Per user request: BMW (and others) put their own services AND external vendors in the same cookie-policy widget. The VVT-Tabelle now groups them by Art. 30(1)(d) DSGVO recipient category so the DSB can act on the right buckets: - INTERNAL — owner processing for itself ('BMW AG — XYZ') - GROUP_COMPANY — same brand family, different legal entity ('BMW Bank') - PROCESSOR — Auftragsverarbeiter, AVV-pflichtig (Adobe, Akamai) - CONTROLLER — independent / joint controller (Meta Pixel, Google Ads, LinkedIn — they run their own profiles) - AUTHORITY — government bodies (rare in cookies) - OTHER — fallback New module vendor_classifier.py: - owner_from_url(url) — derive site-owner token (bmw.de -> 'BMW', mercedes-benz.de -> 'Mercedes-Benz') - classify(name, category, owner) — strict 5-tier heuristic: * INTERNAL: vendor name first-token is '<Owner>' / '<Owner> AG' / '<Owner> SE' / '<Owner> GmbH' / '<Owner> AG & Co. KG' * GROUP_COMPANY: starts with '<Owner> ' but isn't '<Owner> AG' * CONTROLLER: matches a known joint-controller list (Meta, Google Ads, YouTube, LinkedIn Insight, TikTok, Pinterest, Taboola, Outbrain, Criteo, Twitter, Reddit, ...) * PROCESSOR: legal-form suffix in name (GmbH, AG, Inc., A/S, B.V., S.A., Ltd., LLC, ...) * OTHER: anything else vendor_extractor.extract_vendors_from_payloads now takes owner_name: - Passes it through to classify() for every extracted vendor record - The route derives owner_name via _company_name_from_url(doc_entries) - LLM-extracted vendors are classified the same way (so V3 fallback also produces tagged records) agent_doc_check_extras.build_vvt_table_html rewritten: - Buckets vendors by recipient_type - Renders one section per non-empty bucket, in canonical order (RECIPIENT_TYPE_SECTIONS), each with section header + count + bad count + nested table - Within each section: sorted by compliance_score ascending - Response JSON cmp_vendors includes recipient_type so the frontend can later import per-category into the VVT module Expected BMW result: ~60 INTERNAL rows (BMW AG own services), ~25 PROCESSOR rows (Adobe, Adform, Akamai, AWS, ...), ~5 CONTROLLER rows (Meta Pixel, Google, LinkedIn, Pinterest, Outbrain, Taboola).	2026-05-17 12:31:49 +02:00
Benjamin Admin	873997c13b	feat(vvt): V3 — LLM vendor extraction fallback for unknown CMPs ... When the cookie text has no captured CMP payload (long-tail sites that don't use ePaaS/OneTrust/Cookiebot/etc.) we now fall back to a Qwen → OVH LLM cascade to extract a structured vendor list from the policy text. New module backend/compliance/services/vendor_llm_extractor.py: - extract_vendors_via_llm(cookie_text): runs Qwen first (local Ollama), then OVH if Qwen returns nothing usable. - System prompt instructs the model to return STRICT JSON only: {vendors: [{name, country, purpose, category, opt_out_url, privacy_policy_url, persistence, cookies: [...]}]} - Lenient JSON parser tolerates code-fences, prose wrappers, dict vs list. - _normalize() caps array sizes (80 vendors, 30 cookies each), validates URLs (must be http(s)), trims fields to reasonable lengths. Route integration (agent_compliance_check_routes.py): - After named-CMP extract: if cmp_vendors is empty AND the cookie text has ≥500 words (otherwise it's likely navigation chrome), invoke the LLM extractor. Progress message 'Vendor-Liste per LLM extrahieren...'. - Vendors then run through the same validate_vendor_urls + score_vendors pipeline → VVT table rendered identically regardless of source. docker-compose.yml: backend-compliance gains OLLAMA_URL, CMP_LLM_MODEL, OVH_LLM_URL/KEY/MODEL env vars (same names as consent-tester so the configuration is unified). This closes the 'every site eventually gets a VVT table' goal: - Known CMP → V1/V2 structured extraction (fast, exact) - Unknown CMP → V3 LLM extraction (slow, best-effort) - No text at all → no vendors, but other compliance checks still run.	2026-05-17 09:55:42 +02:00
Benjamin Admin	ea4dbb223f	feat(vvt): per-vendor extraction + opt-out check + VVT table in email (V1) ... When a known CMP (ePaaS, OneTrust) renders the cookie policy, we now extract structured vendor records, probe their opt-out + privacy URLs, score each vendor (0-100), and append a 'VVT-Vorschlag' table to the compliance email — one row per vendor, sortable by compliance score. consent-tester: - DSIDiscoveryResult.cmp_payloads: surfaces raw CMP JSON to callers - DSIDiscoveryResponse: new cmp_payloads field - discover_dsi_documents sets cmp_payloads from cmp_capture - cmp_library/{epaas,onetrust}.py: new extract_vendors(d) returning list[VendorRecord] backend: - _fetch_text() now returns (text, cmp_payloads) tuple - doc_entries store cmp_payloads per doc (mostly cookie) - _autodiscover_missing forwards homepage payloads to the cookie entry - New module vendor_extractor.py: dispatches ePaaS/OneTrust/generic schemas; dedupes vendors across multiple payloads - cookie_link_validator.py extended with validate_vendor_urls(vendors) and score_vendors(vendors) — 0-100 score per vendor based on name, purpose, country, opt-out reachable, privacy URL reachable, cookies with names + expiry - agent_doc_check_extras.build_vvt_table_html: renders the table - Route appends VVT HTML after the provider list, before the document-by-document report - Response JSON gains cmp_vendors for future frontend rendering Example for BMW: ~30 ePaaS providers → table with Name | Kategorie | Sitz | Cookies | Opt-Out (✓/✗) | Privacy (✓/✗) | Score. Sorted by score ascending so the worst-compliant vendors are at the top.	2026-05-17 09:50:11 +02:00
Benjamin Admin	c9c0fb5965	feat(cookie-check): enhanced patterns + active opt-out link validator ... cookie_checks.py: - cookie_names_listed: now also matches CMP placeholder notation (BMW: 'Adfpc###', 'CT###') and 'Diese Datenverarbeitung verwendet die folgenden Cookies oder ähnliche Technologien' as list-shape signal. Cryptic vendor names like 'audience', 'adformfrpid' are accepted via the surrounding markup, not by hard-coding each one. - cookie_providers_named: new pattern 'Gesetzt von: <Firma>' (BMW/ePaaS per-cookie vendor naming) + recognition of full legal-form names (Adform A/S, BMW AG, Adobe Systems Software Ireland Limited). - cookie_duration_values: now matches 'Ablauf: 1 Jahr' / 'Speicherdauer: 30 Tage' (BMW format) in addition to the legacy '<n> <unit>'. New L1 + L2 checks for controller in cookie-policy: - cookie_controller (L1): the cookie policy must name Verantwortlich(er) - cookie_controller_address (L2): PLZ + Ort or address keywords - cookie_controller_contact_or_link (L2): email/phone OR link back to Datenschutzerklärung (the practical equivalent — BMW does this) New L2 checks (parented under opt_out): - cookie_optout_links: detects per-provider opt-out URLs in the text - cookie_privacy_policy_links: per-provider privacy-policy URLs New service: cookie_link_validator.py - extract_links(text): pulls all https?://… URLs that follow 'Opt-Out Link:' / 'Link zur Privacy Policy:' (deduped) - validate_links(links): probes every URL concurrently (HEAD first, GET fallback for 405/403). 10 parallel, 8s per request, 60s batch cap. Returns reachable=True/False + status + final_url. - build_check_items(): renders 2 CheckItems (opt-out + privacy-policy), each pass if ALL links 2xx/3xx, fail with up-to-5 broken-link examples. Hook in _check_single: doc_type=='cookie' triggers the validator after regex+MC checks. Recomputes correctness with the new L2 items. This addresses two concrete BMW observations: 1. BMW's per-cookie structure (Name + Zweck + Ablauf, Gesetzt von: …, Opt-Out Link: …) now recognised → 'Konkrete Cookie-Namen aufgelistet' and 'Konkrete Speicherdauern' should pass. 2. Defective opt-out URLs surface as compliance findings rather than silently passing — Art. 7(3) DSGVO requires a working withdrawal path per provider.	2026-05-17 09:38:32 +02:00
Benjamin Admin	b090662524	fix(compliance-check): respect auto-discovery 'not found' verdict; DSB not canonical ... Two related bugs in the BMW test result: 1. AGB rendered as 'MANGELHAFT 0/13' even though BMW has no public AGB: - Auto-discovery correctly returned 'not found' for AGB (no link on bmw.de matches AGB keywords). - But auto_fill_from_dsi then found the substring 'AGB' in a section of the DSI and pseudo-filled the AGB entry with a 264-word DSI fragment. - cross_search_documents would have done the same. - Both now skip entries where discovery_attempted=True AND auto_discovered=False — the 'not found' verdict stands. 2. DSB-Kontakt rendered as a separate 100% OK document with 7566 words = the entire DSI text: - GDPR practice: the DSB is named *inside* the DSI as an email or contact block (Art. 13(1)(b)), not as a stand-alone page. - cross_search_documents had been assigning the full DSI to the DSB row because it matched 'datenschutzbeauftragte' keywords. - DSB removed from _ALL_DOC_TYPES — no longer canonical, no longer padded as missing, no longer auto-discovered. The frontend row remains so a tenant with a separate DSB page can still submit one. After this fix BMW should render: - DSE: OK - Impressum: LUECKENHAFT (unchanged — regex gaps to fix separately) - Cookie-Richtlinie: OK - Social Media: NICHT GEFUNDEN (bmw.de does not link to it) - AGB: NICHT GEFUNDEN (correct — BMW has no public AGB) - Nutzungsbedingungen: NICHT GEFUNDEN - Widerruf: NICHT GEFUNDEN	2026-05-17 01:53:09 +02:00
Benjamin Admin	b2b4d77877	fix(auto-discovery): compute missing against canonical 8 types, not submitted ... Frontend filters out empty doc rows -> req.documents only contains the N submitted entries (3 in BMW case). The old auto-discovery loop computed 'missing' as 'entries in doc_entries with empty text', which was always empty for those N entries -> discovery never fired. Fix: - missing = _ALL_DOC_TYPES - {canonical doc_types in doc_entries} - For each missing type, APPEND a new entry to doc_entries with discovery_attempted=True. If a discovered doc matched, fill text/url and set auto_discovered=True. - Check loop: skip entries with no URL and no text (let padding label them). Entries with URL but no text keep the 'Kein Text' error so the user sees fetch failures explicitly.	2026-05-17 01:28:51 +02:00
Benjamin Admin	525038359a	feat(compliance-check): auto-discover missing doc types from homepage ... When the user leaves some doc-type rows empty, the tool now actively searches the website for them — only marks 'not found' as last resort. Flow: 1. User submits N URLs (e.g. just DSI) 2. For each canonical doc_type with no submitted URL/text, the route identifies the most-common base (scheme://netloc) from submitted URLs 3. Calls consent-tester /dsi-discovery on the homepage with max_documents=15 (180s timeout) 4. Classifies every discovered doc into a canonical doc_type via title/URL keyword rules (_DISCOVERY_RULES — covers cookie/widerruf/ social_media/agb/nutzungsbedingungen/dsb/impressum/dse) 5. Fills matching empty entries with the discovered text, marks auto_discovered=True and discovery_attempted=True Padding now differentiates: - 'Auf der Website nicht gefunden' — discovery was attempted, no doc matched. Amber badge, friendly hint to add URL manually. - 'Nicht eingereicht — Quelle nicht angegeben' — user gave NO URLs at all, nothing to crawl from. Grey badge. Email + frontend: - Status labels: NICHT GEFUNDEN (amber) vs NICHT EINGEREICHT (grey) - 'Gepruefte Quellen' table tags auto-discovered URLs with a small blue 'auto-entdeckt' badge so GF sees what tool found vs user submitted. Implementation only runs when ≥1 URL was submitted (no base to crawl from otherwise). Adds 30-90s for unsubmitted types but avoids the 'just say nicht gefunden' anti-pattern.	2026-05-17 01:14:05 +02:00
Benjamin Admin	bc21480a2a	fix(compliance-check): always render 8 doc types + 4 BMW GT-gap fixes ... Always-show-8 (user-requested): - agent_compliance_check_routes.py: _pad_results_with_missing pads the results list to always include all 8 canonical doc_types in canonical order. Missing types get a placeholder DocCheckResult with error= 'Nicht eingereicht' + scenario='missing'. - agent_doc_check_report.py: NICHT EINGEREICHT status label (neutral), friendly grey body block instead of red error. - ChecklistView.tsx: 'Nicht eingereicht' chip (neutral grey, not red 'Fehler'); SCENARIO_LABELS adds missing entry + header chip counter. Impressum-Regression fix (#18): - _fetch_text(url, doc_type): cookie/dse/social_media -> max_documents=1 (CMP capture authoritative, sub-pages dilute). Other types -> =3 (Impressum needs Versicherungsvermittler, Aufsicht, Berufsrecht sub- pages). 15s networkidle bail keeps timing safe. ODR/Verbraucherstreitbeilegung filter (#19): - _apply_profile_filter: when profile.needs_odr=True (B2C), override the check's default B2B-oriented hint with action-oriented B2C guidance pointing at Art. 14 EU-VO 524/2013 + §36 VSBG. Previously the check contradicted itself: 'profile says B2C' + hint 'only relevant for B2C online vendors'. Registergericht regex (#20): - impressum_checks.py: accept colon/dot/dash between keyword and city (BMW writes 'registergericht: münchen hrb 42243'). Add 'sitz und registergericht: X' as separate pattern. Industry detection (#21): - business_profiler.py: 'automotive' keywords broadened (antriebs, motor, leasing, werkstatt, probefahrt, plus brand names BMW/Mercedes/ Audi/VW/Porsche/Opel). 'it_services' keywords narrowed — software/ cloud/hosting are mentioned in every privacy policy and were biasing the result toward IT for any tech-aware company.	2026-05-17 01:03:58 +02:00
Benjamin Admin	9814b56f2f	fix(cookie-extract): max_documents=1 + faster networkidle bail (Phase 0 fix) ... Root cause of the recurring 603-word BMW result: - DSI discovery for cookie-policy URL was hitting 4x networkidle timeouts (60s each = ~240s total). - Backend httpx timeout (180s after the previous fix) gave up before the consent-tester finished, falling through to the raw HTTP fetch which returned BMWs SSR navigation chrome (603 words) as the 'cookie policy'. Two orthogonal fixes: 1. _fetch_text now passes max_documents=1 for user-specified URLs. We only want self-extraction of THAT page; link-following is unnecessary noise. 2. networkidle wait_until window dropped 60s -> 15s. SPAs like BMW/Daimler never reach networkidle anyway; the 60s wait was pure latency. Falls through to domcontentloaded+5s render-wait, same as before.	2026-05-16 22:53:23 +02:00
Benjamin Admin	6689b37f95	fix(agent): bump _fetch_text timeout 60s->180s ... The dsi-discovery in consent-tester does self-extraction + follows up to 3 sub-links + waits for CMP JSON payloads. On big SPAs (BMW, Daimler) this routinely exceeds 60s. When it timed out, the HTTP fallback returned the SSR shell as text — for the BMW cookie page that's 603 words of site navigation, which then registered as 'Cookie-Richtlinie nicht im eingereichten Text' (33%). With 180s the consent-tester finishes cleanly and we get the CMP-captured 1824 words of real policy.	2026-05-16 22:00:42 +02:00
Benjamin Admin	e61e9d9e2a	feat(agent): progress_pct + 6 BMW-Run Verbesserungen ... Backend (agent_compliance_check_routes.py): - progress_pct (0-100%) im Job-State, ueber alle Phasen verteilt (Laden 0-30, Profil 35-40, Pruefen 40-80, Banner 80-92, Report 95-100) - Status-Texte vereinheitlicht ("Texte laden X/N", "Pruefen X/N") - Firmenname fuer Email-Subject jetzt aus URL abgeleitet (bmw.de -> "BMW", mercedes-benz.de -> "Mercedes-Benz") statt unzuverlaessigem extracted_profile.companyName (matchte oft juris.de) - E-Mail-Report enthaelt jetzt Banner+TCF-Vendor-Liste (build_provider_list_html) Backend (agent_doc_check_extras.py — neu): - build_scanned_urls_html: gepruefte URLs als Tabelle oben im Report (transparent fuer GF, welche Quellen wirklich gezogen wurden) - Cross-Domain-Hinweis bei >1 netloc (BMW: bmw.de / bmwgroup.com / bmwgroup.jobs — Auffindbarkeit nach Art. 12 DSGVO) - build_provider_list_html: Banner-Box + TCF-Vendor-Tabelle mit Spalten Name | Kategorie | Zweck | Drittland | Rechtsgrundlage Backend (business_profiler.py): - §34d-GewO Versicherungsvermittler-Hinweise zaehlen nicht mehr als "finance"-Industrie (BMW wurde dadurch falsch als B2B/finance erkannt) - Neue Industry "automotive" (Fahrzeug/KFZ/Konfigurator/Modellpalette) - B2B-Keywords: generische Begriffe wie "unternehmen", "beratung", "consulting" entfernt (matchten in jedem Konzerntext) - B2C-Fallback: bei Verbraucher-Signalen ("widerruf", "kunde", redaktioneller Inhalt) tendiert auf b2c statt b2b Frontend (ComplianceCheckTab.tsx): - Progress-Balken mit Width-% und XX%-Anzeige rechts - liest data.progress_pct aus Polling-Response Consent-Tester (dsi_discovery.py): - Cookie-Policy-Extraktion kritisch fixt: wait_for_function bis body.innerText > 500 chars (BMW SPA-Rendering brauchte mehr Zeit) - _extract_text_robust: 3-Strategien-Extraktion (Selektoren -> Body- Cleanup -> P/LI/TD-Tags) - _extract_text_from_iframes: liest OneTrust/Sourcepoint/Usercentrics Iframe-Inhalte (manche Cookie-Policies leben dort) Adressiert alle Findings aus dem BMW-Ground-Truth-Vergleich.	2026-05-16 17:53:14 +02:00
Benjamin Admin	d45e08e25f	fix: reduce Playwright timeout 180s→60s, increase poll limit 15→25min	2026-05-16 00:47:28 +02:00
Benjamin Admin	3dbf3aa34a	feat: HTTP fallback for text extraction when Playwright times out ... BMW Impressum/Cookie pages timeout in Playwright (>180s) because the SPA has many sub-links to follow. But the HTML source already contains the text (SSR). New fallback: direct HTTP GET + HTML tag stripping. Order: 1. Consent-tester (Playwright, 180s) → 2. HTTP GET (30s) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-05-15 23:16:10 +02:00
Benjamin Admin	f34305c0a1	fix: increase dsi-discovery timeout 90s→300s, reduce max_documents 10→5	2026-05-15 14:21:13 +02:00
Benjamin Admin	fca67c1f43	fix: accordion close bug + merge multi-page DSIs (BMW fix) ... 1. _expand_all_interactive(): Only click aria-expanded="false" buttons. Before: clicked ALL accordion buttons including open ones → BMW's pre-expanded accordions got CLOSED, reducing text from 1151 to 361w. 2. _fetch_text() + /extract-text: merge ALL documents found on a page (max_documents=10 instead of 1). BMW splits DSI across 5 sub-pages that the discovery finds as separate documents — now merged. 3. Tab panels: unhide hidden tabpanels instead of clicking tabs (clicking tabs can hide the currently visible panel). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-05-15 13:32:04 +02:00
Benjamin Admin	9f87bc5a2c	fix: include website/company name in compliance-check email subject	2026-05-15 10:15:34 +02:00
Benjamin Admin	d72aa10691	feat: management summary for GF + batch GT test script ... 1. Management Summary (agent_doc_check_report.py): - Plain-language action items for Geschaeftsfuehrer - Maps technical checks to business actions ("Ihren DSB erwaehnen", "Beschwerderecht ergaenzen", "Loeschfristen dokumentieren") - Shows at top of compliance check email before detail report - Max 10 actions, max 3 per document 2. Batch GT Test (zeroclaw/scripts/batch_gt_test.py): - Runs all 10 GT websites through compliance-check API - Prints comparison table with L1 scores, word counts, services - Saves raw JSON results for analysis - Usage: python3 batch_gt_test.py --sites 1,6 --backend-url URL Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-05-15 09:39:19 +02:00
Benjamin Admin	826ce2a1b8	fix(cross-doc): suppress false positives when regex checks already pass ... Cross-search "not in text" findings are only shown when regex L1 completeness < 50%. This prevents false positives where the text IS the right doc_type but doesn't contain the specific cross-search keywords (e.g. Impressum passes 9/13 checks but lacks "§5 TMG"). Also: cross-search now checks entries with wrong text, not just empty. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-05-15 00:54:33 +02:00
Benjamin Admin	4e9043f26d	feat(cross-doc): search all texts for all doc_types + misplacement finding ... Cross-Document Intelligence: When a doc_type row is empty, searches ALL other loaded documents for that content. If found (e.g. Widerruf in AGB), extracts the section, runs the check, AND creates a finding: "Widerrufsbelehrung in falschem Dokument gefunden — schwer auffindbar" Keywords for: widerruf, cookie, social_media, impressum, agb, dsb. Integrated as Step 1c in compliance check pipeline. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-05-14 23:19:39 +02:00
Benjamin Admin	c702260ec1	fix: 5 regex bugs + text extraction scroll + GT update ... Root cause: Spiegel DSI text was truncated (lazy-loading) — the rights/DSB/complaints sections at the bottom were never extracted. Fixes: 1. Text extraction: scroll to bottom before innerText (dsi_discovery.py) 2. V.i.S.d.P.: add "verantwortlicher i.s.v." + "§18 Abs. N MStV" pattern 3. USt-IdNr: add "umsatzsteuer-id" + "DE 212 442 423" (with spaces) 4. Profiler: remove generic "anwalt"/"praxis" (false positive on Spiegel "Redaktionsanwalt"), keep only "rechtsanwalt", "kanzlei" etc. 5. Section splitter: auto_fill_from_dsi() fills empty Cookie/Social-Media rows from sections found in the DSI text Ground Truth 06-spiegel.md fully rewritten with verified data from live website — 3 L1 False Negatives identified (DSB, Beschwerderecht, Betroffenenrechte all present on website but not in extracted text). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-05-13 01:20:55 +02:00
Benjamin Admin	c867478791	feat(tcf-vendors): GVL cache + vendor extraction + VVT mapping ... Phase 1-2 of the closed quality loop: - GVL cache (consent-tester/services/gvl_cache.py): downloads and caches IAB Global Vendor List with 24h TTL, resolves vendor IDs to names, purposes, policy URLs, retention, country - Vendor extraction (consent_interceptor.py): extract_tcf_vendors() reads __tcfapi after accept phase, resolves via GVL - Scan response: tcf_vendors field added to /scan endpoint - VVT mapper (vendor_vvt_mapper.py): maps TCF vendors to VVT format with purpose labels, Rechtsgrundlage, Drittland detection - Vendor cross-check (banner_cookie_cross_check.py): checks all TCF vendors against DSI text — missing vendors, undocumented transfers - Compliance check integrates Step 3d: TCF vendors vs DSI Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-05-12 18:18:50 +02:00
Benjamin Admin	7be34552bb	feat(compliance-check): profile extraction + scenario classification ... - New profile_extractor.py: extracts Company Profile fields (name, legal form, address, DPO, USt-IdNr) and Compliance Scope hints (Art. 9 data, third country, profiling) from document texts - Scenario per document: regenerate (<30%), fix (30-95%), import (>95%) - Widerruf for B2B: no longer skipped, instead all checks flagged as INFO with "not needed for B2B" hint - Move _build_profile_html to report builder module - DocCheckResult gets scenario field Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-05-12 17:34:33 +02:00
Benjamin Admin	be9cfdc2d4	feat(compliance-check): skip Widerruf for B2B, limit MCs, fix industry ... - Skip Widerrufsbelehrung check entirely for B2B/B2G businesses - Limit MC checks to top 20 per doc_type (by severity) to reduce noise (e.g. 75 impressum MCs → 20, avoiding 55 irrelevant FAILs) - Add consulting/manufacturing industry keywords (arbeitssicherheit, brandschutz, werkzeugbau, etc.) - Lower industry detection threshold from 2 to 1 keyword hit Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-05-12 17:03:57 +02:00
Benjamin Admin	74f00bbb0f	feat(compliance-check): split shared URLs into sections per doc_type ... When the same URL is used for multiple document types (e.g. /datenschutz for DSI + Cookie + DSB), the section splitter now: - Detects duplicate URLs and fetches text only once - Splits text at classified headings (Cookie, Google Analytics, etc.) - Assigns matching sections to each doc_type - DSI always keeps the full text Extracted to section_splitter.py (170 LOC) to keep routes under 500. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-05-12 12:49:57 +02:00
Benjamin Admin	b6ad958b69	feat(compliance-check): integrate banner cross-check + extract to module ... Add automatic banner check (Step 3b) and banner-vs-cookie cross-check (Step 3c) to unified compliance check. Extract cross-check logic to banner_cookie_cross_check.py to keep routes under 500 LOC. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-05-12 00:08:47 +02:00
Benjamin Admin	0d0e705117	feat: Unified Compliance-Check — 8 document types in one form ... New 3-tab structure: Website-Scan, Compliance-Check, Banner-Check. Compliance-Check Tab (replaces Dokumenten-Pruefung + Impressum-Check): - 8 document rows: DSI, Impressum, Social Media, Cookie, AGB, Nutzungsbedingungen, Widerruf, DSB-Kontakt - Each row: URL input + "Text laden" + file upload + manual text - "Text laden" extracts via consent-tester, shows in editable textarea - User verifies/corrects text before checking - Empty fields = "not present" → own finding Business Profiler (business_profiler.py): - Detects B2B/B2C/B2G from all documents together - Recognizes regulated professions, online shops, editorial content - Context-aware: INFO checks become PASS/FAIL based on profile Backend: /compliance-check + /extract-text endpoints Frontend: ComplianceCheckTab.tsx + DocumentRow.tsx API proxies: compliance-check/route.ts + extract-text/route.ts Also: Impressum regex fixes (Telefon, AG, Geschaeftsfuehrung) and INFO severity for context-dependent checks. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-05-11 20:56:10 +02:00