21 Commits

Author	SHA1	Message	Date
Benjamin Admin	21c01d6405	fix: Heading detection allows digit-start (e.g. "5. Soziale Medien") ... Headings starting with numbers (numbered sections like "5. Soziale Medien", "6. Analyse-Tools") were not detected because the check required stripped[0].isupper(). Now also accepts stripped[0].isdigit(). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-05-08 00:16:36 +02:00
Benjamin Admin	a3a83e5677	fix: Section classifier strips leading numbers + recognizes German headings ... - "5. Soziale Medien" now stripped to "soziale medien" before classification - Added "soziale medien/netzwerke" as social_media heading pattern - Fixes etogruppe.com where Social Media section wasn't detected Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-05-08 00:03:37 +02:00
Benjamin Admin	7c17321089	feat: Cookie Banner Check as standalone tab in Compliance Agent ... New "Banner-Check" tab with: - URL input → Playwright 3-phase test (before/reject/accept) - Shield icon + provider detection - Progress bar with pass/fail percentage - 3-phase summary (cookies + scripts per phase) - Violations (red) and passes (green) in structured list Backend: new POST /api/compliance/agent/banner-check endpoint that proxies to consent-tester:8094/scan. Next step: Upgrade banner checks to L1/L2 format with expert hints (same quality as document checks). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-05-07 17:39:44 +02:00
Benjamin Admin	a4b75dc6b1	fix: Section splitter only splits at classified headings + LLM gets full text ... Two critical fixes: 1. Section splitter: Only lines that classify as a known doc_type (cookie, social_media, dsfa, etc.) trigger section splits. Random short lines ("Typen", "Funktionale Cookies") no longer split sections — they all had blank lines before them in the extracted HTML text. 2. LLM verification: Sub-section checks now pass the full document text to the LLM, not just the section fragment. This lets the LLM find content that the section splitter missed. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-05-07 16:28:17 +02:00
Benjamin Admin	f51671737a	fix: Correct Ollama model name + strict blank-line heading detection ... 1. LLM model: qwen3:32b → qwen3.5:35b-a3b (actual model on Mac Mini) 2. Section splitter: headings MUST be preceded by a blank line. This prevents cookie table entries ("Funktionale Cookies", "Session Cookies") from splitting the cookie section. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-05-07 15:53:53 +02:00
Benjamin Admin	4f29e5ff3c	feat: LLM verification for regex FAILs + section-split hardening ... Path to 100% correctness: Regex finds 80%, LLM catches the rest. 1. LLM verification (llm_verify.py): - Every regex FAIL is re-checked by Qwen (qwen3:32b) - Binary YES/NO question with evidence extraction - Overturned checks marked with [LLM] prefix in matched_text - Graceful fallback if LLM unavailable 2. Section splitter hardening: - Short lines (<16 chars) only treated as headings if preceded by blank line — prevents table column headers ("Funktion", "Speicherdauer") from splitting cookie sections - Fixes IHK cookie section: 288 words → full section 3. DSFA documentation patterns expanded: - Recognizes "4.) Ergebnis:" numbered result sections - Matches risk assessment conclusions Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-05-07 15:34:07 +02:00
Benjamin Admin	a3287cd5e6	feat: HTML email report with hints + fix duplicate Social Media sections ... 1. Email report now renders as styled HTML (matching frontend design): - Progress bars (green=completeness, blue=correctness) - Hierarchical L1→L2 check display - Red hint boxes under failed checks explaining what to fix - Matched text evidence for passed checks 2. Section splitter deduplicates: two "Social Media" headings on the same page are merged into one section instead of creating duplicates. 3. Extracted report builder to agent_doc_check_report.py (175 LOC) to keep routes file under 500 LOC (386 LOC). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-05-07 15:13:00 +02:00
Benjamin Admin	fa4fd87102	fix: 7 regex bugs from IHK Konstanz ground truth analysis ... Fixes based on manual verification of all 30 failed checks: 1. Cookie table: recognize "folgende cookies" + column headers as text 2. Cookie names: add JSESSIONID, cookieinfo, et_id, BT_* patterns 3. Essential justified: match "sitzung zuordnen", "betrieb der website" 4. Social bookmarks: recognize as 2-click alternative 5. DSFA plural: "kanaelen" now matches alongside "kanal" 6. Section splitter: skip-headings no longer lose subsequent text (Risikoabwaegung section was cut from DSFA, losing risk scores) 7. Cookie legal basis: accept Art. 6(1)(f) in cookie context Reduces false positives from 7 to ~1-2 for IHK Konstanz test case. Ground truth table: zeroclaw/docs/ground-truth-ihk-konstanz.md Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-05-07 14:51:09 +02:00
Benjamin Admin	293c58d0dd	feat: Add actionable hints to all 138 compliance checks ... Each check now has a "hint" field explaining what is missing and what the customer should do to fix it. Hints are shown in the frontend below failed checks in red text. Examples: - "Bei Verarbeitung auf Basis von Art. 6(1)(f) muss dokumentiert werden, warum Ihr berechtigtes Interesse die Rechte der Betroffenen ueberwiegt." - "Die ladungsfaehige Anschrift fehlt. Erforderlich: Strasse, Hausnummer, PLZ und Ort." Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-05-07 14:05:01 +02:00
Benjamin Admin	b363c28539	feat: Add 76 Level-2 regex checks for document correctness verification ... Split dsi_document_checker.py (466 LOC) into doc_checks/ package (9 files). Two-pass L1→L2 logic: L1 checks "Is it mentioned?", L2 checks "Is it correct?" (e.g. controller has full address, specific Art. 6 lit., concrete time periods). 138 total checks (62 L1 + 76 L2) across 7 doc types: - DSE Art. 13: 31, Impressum §5 TMG: 16, Cookie §25 TDDDG: 15 - Widerruf §355: 15, AGB §305ff: 21, Social Media Art. 26: 20, DSFA Art. 35: 18 Frontend: hierarchical L1→L2 display with dual progress bars (green=completeness, blue=correctness). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-05-07 12:37:03 +02:00
Benjamin Admin	3c12e06faf	feat: Fix DSFA dedup + expand all checklists to 56 total checks ... Fixes: - 'Risikoabwaegung' is sub-section of DSFA → added to SKIP_HEADINGS - 'Social Media' standalone heading → recognized as social_media DSE - Removed 'risikobew' from DSFA pattern (was too broad) Expanded checklists: - Widerruf: 4→7 checks (+Empfaenger, kein Grund, §312k Button) - AGB: 4→9 checks (+Zahlung, Lieferung, Gewaehrleistung, Kuendigung, Datenschutz) - Social Media: +1 (Social Bookmarks) - DSFA: +1 (LFDI Richtlinie) Total: 47→56 Regex-Checks across 7 document types: DSI=9, Cookie=5, Social Media=10, DSFA=8, Impressum=6, Widerruf=7, AGB=9 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-05-07 11:55:29 +02:00
Benjamin Admin	58234ac18b	fix: DSFA must be matched before social_media in SECTION_TYPE_MAP ... 'Datenschutzfolgeabschätzung...Social Media' was matching as social_media (Art. 26) instead of dsfa (Art. 35) because the social_media pattern 'datenschutz.*social media' matched first. Fixed: DSFA patterns checked before social_media patterns. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-05-07 11:35:10 +02:00
Benjamin Admin	3853a0838a	feat: Art. 26 Joint Controller + DSFA checklists for Social Media sections ... New checklists: - JOINT_CONTROLLER_CHECKLIST (Art. 26 DSGVO, 7 checks): Joint parties, arrangement, contact point, processing split, data categories, third-country transfer (USA), rights - DSFA_CHECKLIST (Art. 35 DSGVO, 5 checks): Description, necessity, risk assessment, measures, DSB involvement Section detection: 'Datenschutzerklaerung fuer Social Media' → social_media, 'Datenschutzfolgeabschaetzung/Risikoanalyse' → dsfa classify_document_type: DSFA and social_media detected before generic DSE Frontend: DOC_TYPES dropdown + ChecklistView labels updated Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-05-07 10:49:32 +02:00
Benjamin Admin	5188411828	disable: Control Library checks until doc-check Master Controls are ready ... 8 false positives from generic canonical_controls. Regex checks (9+5) are accurate. Re-enable when ~80 specific doc-check controls exist. See INSTRUCTION-master-controls-for-doc-check.md Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-05-07 10:28:25 +02:00
Benjamin Admin	fa45b5793c	feat: Control Library check via SQL (canonical_controls) instead of Qdrant ... Complete rewrite of rag_document_checker.py: - Queries canonical_controls table (294K controls, 10K data_protection) - Filters by category + title keywords per document type - Uses test_procedure field as actual check instructions - Regex pre-check extracts key terms from procedure → fast match - LLM fallback only for regex misses (saves tokens) - /no_think prefix for direct JSON output SQL approach advantages: - Structured data with test_procedure, pass_criteria, fail_criteria - Category filtering (data_protection, compliance, governance) - No Qdrant API key issues - Controls are actual check criteria, not general legal texts Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-05-06 20:26:56 +02:00
Benjamin Admin	7e7f31c344	disable: RAG checks until Master Controls (G1 Decision Trace) are ready ... Current 144K controls are general legal texts, not specific check criteria. RAG integration code stays (rag_document_checker.py), just disabled in the doc-check endpoint. Re-enable when G1-G4 block is complete and 25K Master Controls with Decision Trace are available. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-05-06 17:11:58 +02:00
Benjamin Admin	1ff34227bf	debug: Add logging to RAG check integration ... Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-05-06 14:57:30 +02:00
Benjamin Admin	090da0f71b	feat: RAG-based document verification against 144K Control Library ... New module: rag_document_checker.py - Searches RAG (Qdrant) for controls relevant to document type - Filters by regulation (DSGVO Art.13, TDDDG §25, BGB §355 etc.) - LLM (Qwen 3.5:35b) verifies each control against document text - Returns fulfilled/missing with evidence text + severity - Supports: DSI, Cookie, Impressum, Widerruf, AGB, DSFA, AVV, Loeschkonzept Integration in doc-check endpoint: - Regex checklist runs first (fast, deterministic) - RAG checks run after (semantic, catches what regex misses) - Both results combined in single response LLM prompt returns JSON: {fulfilled, evidence, issue, severity} Think-tags stripped, JSON extracted from response. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-05-06 13:19:15 +02:00
Benjamin Admin	13c5880f51	fix: Restrict sub-section detection to genuinely separate document types ... Only Cookie and Widerruf sections are checked as separate documents. Social Media, DSFA, Betroffenenrechte, Dienste von Drittanbietern are part of the parent DSI and no longer generate false findings. Added PLAN-rag-document-check.md for Phase 2: - RAG-based checks with document-type-specific Controls - DSFA checklist (Art. 35 + Landes-Listen) - AVV checklist (Art. 28) - Reference detection (sub-doc → parent doc) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-05-06 11:02:36 +02:00
Benjamin Admin	539bc824fd	feat: Auto-detect sub-sections within a page and check each separately ... When a single URL contains multiple document sections (e.g. IHK DSI page with Cookies, Social Media, Dienste von Drittanbietern), the system now: 1. Extracts full page text (main document check as before) 2. Splits text at heading boundaries (short uppercase lines) 3. Classifies each section: Cookie→cookie checklist, Social Media→DSI etc. 4. Runs type-specific checklist per section 5. Returns all results: main doc + sub-sections Section type detection via SECTION_TYPE_MAP patterns: - 'Cookie*' → §25 TDDDG checklist - 'Dienste von Drittanbietern' → DSI checklist - 'Social Media' → DSI checklist (Art. 26 joint controllership) - 'Widerrufsrecht' → §355 BGB checklist - 'Impressum' → §5 TMG checklist Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-05-06 10:44:42 +02:00
Benjamin Admin	4c68caac4e	feat: Multi-URL Document Check with full checklist visibility ... New "Dokumenten-Pruefung" tab in Compliance Agent: - User adds multiple URLs with document type (DSI, AGB, Impressum, Cookie, Widerruf) - Each document loaded via Playwright, accordions expanded, text extracted - Checked against type-specific legal checklist - Optional: Cookie banner check via checkbox Checklisten-UX (solves "100% looks like nothing was checked"): - All checks shown per document: green checkmark + matched text excerpt - Red X for missing fields with legal reference - Builds user trust: "9 Punkte geprueft, alle bestanden" - Expandable per document with completeness bar New checklists: - Impressum: §5 TMG (6 fields: name, address, contact, register, VAT, representative) - Cookie-Richtlinie: §25 TDDDG (5 fields: types, purposes, retention, third-party, opt-out) Backend: - POST /agent/doc-check — async with polling (same pattern as /scan) - DocCheckResult includes checks[] with passed/failed + matched_text - dsi_document_checker returns all_checks in SCORE finding - Email report shows per-document checklist Files: agent_doc_check_routes.py (280 LOC), DocCheckTab.tsx (248 LOC), ChecklistView.tsx (130 LOC), dsi_document_checker.py (+70 LOC) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-05-06 10:08:40 +02:00