16 Commits

Author	SHA1	Message	Date
Benjamin Admin	0b29d1fada	fix(cookie-inventory): fuzzy prefix-match + BMW-GT-File ... BMW-Mail zeigte 738 deklariert / 31 Browser / **0 OK** — alle Browser-Cookies landeten als UNDOC, alle deklarierten als ORPH. Ursache: exact-string-match scheitert bei Suffix-Cookies. _norm_for_match() + _matches() Helper: - Strippt Wildcards (`*`, `.*`, `<id>`, `{var}`) + Lower-Case - Erhält führende Underscores (`__cf_bm`, `_ga` sind meaningful) - Prefix-Match in BEIDE Richtungen, min 3 Chars (kein "_"-Garbage) build_cookie_inventory(): - Für jeden Browser-Cookie: längster Prefix-Match in declared wählen - browser-to-decl Index + decl-match-Index für O(N×M) → O(N+M) - matched browser-keys werden aus all_keys entfernt → kein Double-Count (vorher: ORPH + UNDOC parallel) Realistischer BMW-Match-Test: declared=[_ga, _gid, __cf_bm, AMP_TOKEN, _fbp, intercom-session, _pk_id.*, OptanonConsent] browser= [_ga_K8YL3M9T, _gid_xyz, __cf_bm_actual_hash, AMP_TOKEN_runtime, _fbp_123, intercom-session-2026, _pk_id.5.7d8, OptanonConsent] → 8 OK (vorher 0) BMW-GT-File (zeroclaw/docs/ground-truth/bmw_de_2026-06-07.json): - OneTrust CMP + 14 erwartete Vendoren - Cookie-Count-Ranges (browser 80-250, deklariert 300-800) - 7 expected findings inkl. neuem COOKIE-INVENTORY-MATCH-001 als Benchmark gegen den Fuzzy-Match-Bug Tests: 14/14 grün (4 _norm_for_match + 5 _matches + 5 build_cookie_inventory inkl. realistic_bmw_pattern). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-06-07 21:29:21 +02:00
Benjamin Admin	a2cae94526	fix(b9)+test: real-world false-positives + multi-site GT-bench ... Real-World-Smoke gegen Westfield Hamburg (englische DSE) deckte B9-Bug auf: Pattern matched "If mfi Immobilien Marketing GmbH", "Discover our Se", "Centre Se" usw. als angebliche Entitäten — englische Connector-Worte + abgeschnittene "Services"-Strings. B9 Fix: - _name_is_blocked() strenger: min 2 Worte, mind. einer ≥4 Chars UND capitalized (vor Legal-Form-Suffix). Filtert "Se", "ag", "If ...", "Centre Se" zuverlässig. - _clean_entity_name() strippt jetzt führende Lowercase- Connector-Worte (kontextuelle Verben wie "by", "If", "according to"). - _dedup_substring() collapses "mfi Immobilien Marketing GmbH" + "Marketing GmbH" zum längeren. - Anwendung sowohl im HRB-Pfad als auch im Fallback-Pfad. Multi-Site-Bench (2 neue GTs, 2 Engine-Runs): - zeroclaw/docs/ground-truth/westfield_hamburg_2026-06-07.json: iAdvize-Chatbot bekannt, Unibail-Management-Verantwortlicher. - zeroclaw/docs/ground-truth/allianz_reise_chatbot_2026-06-07.json: Twilio-Infrastruktur (US-Transfer), lit. f + 2-Mo-Retention. - zeroclaw/docs/audits/2026-06-07-multi-site-walk-results.md: Sprint-Briefing mit Detektor × Site Matrix, Audit-Walk-DSMS- CIDs, identifizierte Real-World-Bugs + Backlog. Audit-Walk-Endstand (B17 Stufen 1-3): - Westfield: 400 KB Video, CID Qm…WJYfYDt…BXgwt - Allianz: 1 MB Video, CID Qm…XFuiC4z…9mSMM Beide DSMS-persistiert, Reviewer kann jederzeit verifizieren. Tests: 21/21 grün (test_impressum/test_elli_gt_coverage). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-06-07 17:51:17 +02:00
Benjamin Admin	8e3d05f172	test(elli-gt): GT-Coverage-Integration-Test + Sprint-Briefing ... - tests/test_elli_gt_coverage.py: 7 Charakterisierungstests die einen synthetischen Elli-State konstruieren und sicherstellen, dass die 5 neuen Detektoren (B13-B16 + B9-Cleanup) genau die erwarteten GT-IDs fangen. Regressionsschutz. - zeroclaw/docs/audits/2026-06-06-elli-gt-coverage-sprint.md: Sprint-Zusammenfassung mit GT-Bilanz (12/13 voll, 1/13 wartet auf #7), Commit-Liste und Morgen-Agenda-Kandidaten. Combined Sprint-Test-Run: 72/72 grün. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-06-07 00:28:29 +02:00
Benjamin Admin	d0e3621192	feat(audit): V2 mail render + 5 new findings (B4/B5/B6/B7/B8) + LLM-Plausibility-Phase ... Mail Render V2 (compliance/services/mail_render_v2/) — 11-Modul-Subpackage das einen einheitlichen Audit-Mail-Output erzeugt mit: - Header + KPI-Kacheln (Score / Findings / Docs / Vendors) - TOC + Sprung-Links - 3-Bucket-Trennung: Kritische Befunde / Manuelle Prüfung / Interne Reminder - Cookie-Inventar (Name·Vendor·Kategorie·Speicherdauer·Löschfrist·Sitzland·Quelle·Status) - Sofortmaßnahmen-Aggregator ("Sitzland ergänzen für 11 Cookies") - 24 Legacy-Wrappers — alle alten build_*_html in V2-Sections - Scope-Filter: FIN/GOV/MED/INS/EDU/LEG aus Berichten wenn nicht relevant - Hint/Action-Dedup: keine doppelten Sätze pro Card mehr Aktiviert via env MAIL_RENDER_V2=true (Default: legacy renderer). 5 neue deterministische Findings als Phase D-2b/B4/B5/B6/B7/B8: B4 vendor_consistency_check — Cross-Doc-Provider-Widerspruch (Elli: DSE nennt Vertex AI für Chatbot, /de/cookies nennt Iadvize → HIGH). 6 Service-Types: chatbot/analytics/tag_manager/pixel/cdn/cmp. B5 ai_act_transparency_check — AI Act Art. 50 Transparenzpflicht (Elli: Vertex AI vorhanden ohne Pre-Chat-Disclosure → HIGH). Plus B5-Erweiterung: Rechtsgrundlage Art-6-Abs-1-lit-f bei AI → MED (Einwilligung empfehlen). B6 cross_doc_dpo_check — DPO in DSE genannt, nicht im Impressum (LOW). B7 doc_staleness_check — Datum-Extraktion aus DSE/AGB/Nutzungsbedingungen. Cap: AGB/NB 3y, DSE 2y. Älter → MEDIUM (Elli NB Stand 2018 → HIGH). B8 cmp_fingerprint_check — Banner detected, aber CMP-Provider generic (kein Usercentrics/OneTrust/Cookiebot/etc → MED). B3-Erweiterung detect_intra_doc_contradictions — Widersprüchliche Speicherdauer im SELBEN Doc (Elli: Logfile 7d vs 30d → HIGH). LLM-Plausibility-Phase (Phase D-2b, finding_plausibility_check.py): - Läuft AFTER MC pipeline, BEFORE D3 render - Prompt mit Beispiel-IDs + 3-Phase-Mapping: exact-ID / position-fallback / fuzzy-tail-match - Stempelt llm_title / llm_severity / llm_recommendation / llm_drop auf jeden FAIL CheckItem - V2-Render zeigt "🤖 LLM-Plausibility:" Box pro Finding wenn gestempelt - KNOWN ISSUE: qwen3:30b-a3b liefert oft empty content auf format='json' + 8000-char-excerpt prompts. Pipeline läuft mit stamped=0 weiter. Task #16. Coverage gegen Elli Ground Truth (zeroclaw/docs/ground-truth/elli_eco_2026-06-06.json, 13 expected findings via WebFetch-Agent-Crawl): - 4/4 HIGH-Findings ✓ (COOKIE-CONSENT-UX-001 + WIDERRUFSBELEHRUNG-001 + VENDOR-CONSISTENCY-001 + AI-ACT-TRANSPARENCY-001) - 4/6 MEDIUM ✓ - 2/3 LOW ✓ - Total: 10/13 = 77% (Sprung von 4/13 = 31%) Restliche 3 Gaps als Task #17: IMPRESSUM-001 (multi-entity USt-IdNr), TRANSFER-001 (Vendor-Mechanismus DPF/SCC), TH-RETENTION-002 (AI-Retention pro Datenkategorie). V2-Mail-Preview in Mailpit: 'v2all@local.test' Subject '[V2 ALL] ELLI'. Backend healthy, B1+B3+B4+B5+B6+B7+B8 alle live im Orchestrator. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-06-06 21:19:49 +02:00
Benjamin Admin	70af018da5	docs(gt): BMW cross-domain finding — 3 domains, no AGB, Social Media on jobs portal	2026-05-15 13:21:27 +02:00
Benjamin Admin	0182c91ef9	docs(gt): BMW fully verified — URLs, DSB, Impressum, Social Media data	2026-05-15 12:01:20 +02:00
Benjamin Admin	a67cfa7c4a	fix(gt): update BMW URLs (all old URLs are 404 since 2026)	2026-05-15 10:38:07 +02:00
Benjamin Admin	b175212516	docs(gt): update Spiegel GT with verified 2026-05-14 results ... DSI: 9/9 L1 (was 6/9), 13698 words (was 6461), all FNs resolved. Social Media: 10/10 L1 (was 9/10). Services: 31 detected (was 5). Impressum: 9/13 (USt-IdNr + V.i.S.d.P. fixed). Widerruf: NOT correctly tested (wrong text assigned, needs Cross-Doc Intelligence). Full service list (31 providers) documented with country + EU status. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-05-14 23:07:42 +02:00
Benjamin Admin	5e317d2f0f	fix: text extraction 50k char limit was root cause of all Spiegel FNs ... ROOT CAUSE: main.py line 338 truncated full_text at 50,000 chars. Spiegel DSI has 107,720 chars (13,705 words) — only 47% was extracted. DSB, Art. 77, Betroffenenrechte were all in the truncated portion. Fixes: 1. Raise text limit from 50k to 200k chars in API response + discovery 2. click_button(): add iframe fallback for Sourcepoint/Quantcast 3. dsi_helpers: iterate ALL page.frames for consent buttons 4. Profiler: only check impressum (not full text) for regulated professions, and "rechtsanwalt" must be in first 500 chars (company description) 5. GT: save full Spiegel DSI text (13,705 words) as reference Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-05-13 15:22:38 +02:00
Benjamin Admin	c702260ec1	fix: 5 regex bugs + text extraction scroll + GT update ... Root cause: Spiegel DSI text was truncated (lazy-loading) — the rights/DSB/complaints sections at the bottom were never extracted. Fixes: 1. Text extraction: scroll to bottom before innerText (dsi_discovery.py) 2. V.i.S.d.P.: add "verantwortlicher i.s.v." + "§18 Abs. N MStV" pattern 3. USt-IdNr: add "umsatzsteuer-id" + "DE 212 442 423" (with spaces) 4. Profiler: remove generic "anwalt"/"praxis" (false positive on Spiegel "Redaktionsanwalt"), keep only "rechtsanwalt", "kanzlei" etc. 5. Section splitter: auto_fill_from_dsi() fills empty Cookie/Social-Media rows from sections found in the DSI text Ground Truth 06-spiegel.md fully rewritten with verified data from live website — 3 L1 False Negatives identified (DSB, Beschwerderecht, Betroffenenrechte all present on website but not in extracted text). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-05-13 01:20:55 +02:00
Benjamin Admin	407a9503e4	fix(profiler): fix B2G false positive + add consulting/manufacturing ... - Remove generic B2G keywords (behörde, amt, öffentlich) that match in every DSI due to "Aufsichtsbehörde", "Amtsgericht", "veröffentlichen" - Remove "server" from it_services (too generic, appears in every DSI) - Add consulting, manufacturing, media industries - Add B2B fallback for GmbH/AG without B2C signals - Add 10 ground truth files for unified compliance check Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-05-12 12:20:44 +02:00
Benjamin Admin	36c6101b91	Merge feat/zeroclaw-compliance-agent into main ... Brings all compliance doc-check features: - 162 regex checks + 1874 Master Controls - LLM-agnostic agent with tool calling - Banner check (46 checks, 30 CMPs, stealth, Shadow DOM) - Impressum check (24 checks) - Deep consent verification (DataLayer, GCM, TCF) - CMP E2E tests (39 tests) - HTML email reports, FAQ, persistent history Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-05-11 11:44:20 +02:00
Benjamin Admin	1b5c6bd340	docs: Batch test results for 9 websites + EUIPO analysis ... Tested BMW, Stadt Koeln, BfDI, Sparkasse, Caritas, TUEV Sued, Spiegel, ETO Gruppe, EUIPO. Key findings: - Stadt Koeln + ETO Gruppe best (95% correctness) - BMW, Sparkasse, Spiegel genuinely deficient (verified) - EUIPO uses EU Regulation 2018/1725, not GDPR — needs separate checklist - ~0-2 false positives per website after LLM verification 7 regex fixes emerged from batch testing (soft hyphens, word insertions, numbered headings, German section names, etc.) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-05-08 00:41:28 +02:00
Benjamin Admin	fa4fd87102	fix: 7 regex bugs from IHK Konstanz ground truth analysis ... Fixes based on manual verification of all 30 failed checks: 1. Cookie table: recognize "folgende cookies" + column headers as text 2. Cookie names: add JSESSIONID, cookieinfo, et_id, BT_* patterns 3. Essential justified: match "sitzung zuordnen", "betrieb der website" 4. Social bookmarks: recognize as 2-click alternative 5. DSFA plural: "kanaelen" now matches alongside "kanal" 6. Section splitter: skip-headings no longer lose subsequent text (Risikoabwaegung section was cut from DSFA, losing risk scores) 7. Cookie legal basis: accept Art. 6(1)(f) in cookie context Reduces false positives from 7 to ~1-2 for IHK Konstanz test case. Ground truth table: zeroclaw/docs/ground-truth-ihk-konstanz.md Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-05-07 14:51:09 +02:00
Benjamin Admin	4f92e5056c	docs: Complete agent architecture reference for reuse in other agents ... Full documentation of the ZeroClaw compliance agent architecture: - System overview diagram (Frontend → Backend → LLM → Playwright) - Detailed request flow for Website-Scan mode (7 steps) - All 5 components: Frontend, Backend, Consent-Tester, Ollama, Soul Files - 20 banner checks across 3 files - LLM call patterns (/api/generate + /api/chat + think-mode stripping) - Blueprint for creating new agents (5 steps: Soul, Route, Page, Proxy, Docker) - Timeouts, environment variables, file reference with LOC counts Designed as reusable blueprint for Sales, HR, Finance, or other agents. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-05-04 22:26:56 +02:00
Benjamin Admin	e318215cc5	refactor: split agent_analyze_routes (420→309 LOC) + agent docs + migration ... - Extracted website compliance checks + helpers to website_compliance_checks.py - Created agent documentation (zeroclaw/docs/compliance-agent.md) - DB migration 086 executed (compliance_agent_scans table) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-05-02 08:22:52 +02:00