Green pill button: 'KI-Agent aus' / 'KI-Agent aktiv (1.874 MCs)'
Toggles use_agent flag which is passed through the full chain:
Frontend → DocCheckRequest → _run_doc_check → _check_single_document
→ check_document_with_controls(use_agent=True)
→ ComplianceAgent with tool calling
Default: OFF (deterministic regex). User can enable per scan.
Also works via env var COMPLIANCE_USE_AGENT=true for always-on.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Extend banner consent records with consent_method, banner_version,
banner_config_hash, geo, page_url, referrer, device info, session_id
and consent_scope for full Art. 7 DSGVO proof with any tracking vendor.
Migration 107, backward-compatible (all fields nullable).
Admin detail modal shows tracking context, device info and technical data.
Fix pre-existing str|None → Optional[str] for Python 3.9 compat.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
New agent architecture for intelligent MC evaluation:
agent_tools.py (367 LOC):
- 5 tools in OpenAI function-calling format
- query_controls: async DB query for MCs by doc_type
- evaluate_controls_batch: deterministic keyword matching
- search_document: text search with context
- get_document_stats: word count, sections, language
- submit_results: finalize check results
compliance_agent.py (398 LOC):
- ComplianceAgent class with agent loop
- 3 LLM providers: Ollama, OpenAI-compatible (OVH), Anthropic
- Tool call dispatch + result collection
- System prompt for systematic compliance analysis
- run_compliance_check() convenience function
Hybrid mode:
- COMPLIANCE_USE_AGENT=false (default): deterministic regex
- COMPLIANCE_USE_AGENT=true: LLM agent with tool calling
- Agent fallback to regex if LLM unavailable
Works with Qwen 35B (Ollama), Qwen 120B (OVH vLLM), Claude.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Merges two separate consent views into one unified page at /sdk/einwilligungen:
- Tab "Website-Besucher": device-based banner consents with site selector
- Tab "Login-Nutzer": user-based DSGVO consents (existing, unchanged)
Backend:
- New endpoint GET /admin/consents for paginated banner consent records
- Fix: categories JSON string parsing (was iterating chars instead of array)
CMP Dashboard:
- Dynamic site selector replacing hardcoded "preview-test-site"
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Deterministic pass/fail stays unchanged. After keyword checking,
ONE batched LLM call enriches the top 10 severity FAILs with
context-specific recommendations based on the actual document.
Example: If document uses Google Analytics but lacks transfer
mechanism → LLM generates: "Sie nutzen Google Analytics (USA).
Ergaenzen Sie einen Verweis auf das EU-US Data Privacy Framework
und pruefen Sie die DPF-Zertifizierung unter dataprivacyframework.gov."
- Pass/fail: deterministic (keyword matching, reproducible)
- Hint enrichment: LLM (contextual, one call for all fails)
- Temperature 0.3 for consistency
- Graceful fallback if Ollama unavailable
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replaced LLM-based MC verification with deterministic keyword matching:
- Extracts keywords from pass_criteria/fail_criteria
- Matches against document text via regex (case-insensitive)
- PASS if >= 60% of criteria keywords found AND no fail_criteria triggered
- Same text + same MCs = same result every time
Checks ALL MCs for the doc_type (max_controls=0):
- DSE: all 571 controls checked in <1 second
- Impressum: all 75 controls
- Cookie: all 381 controls
No LLM calls needed — purely deterministic keyword matching.
Bigram extraction for compound terms (e.g. "standardvertragsklauseln").
Stop word filtering for German legal text.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Rewritten rag_document_checker.py to use doc_check_controls table
instead of generic canonical_controls. Each MC has:
- check_question: binary YES/NO for LLM
- pass_criteria: JSONB list of concrete requirements
- fail_criteria: JSONB list of common mistakes
Flow: Regex checks (fast) → LLM verify FAILs → MC deep check (15 per doc)
MC results appear as additional L2 checks in the report.
Coverage: 571 DSE, 381 Cookie, 309 Loeschkonzept, 153 Widerruf,
147 DSFA, 125 AVV, 113 AGB, 75 Impressum = 1.874 total.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When checking impressum/agb/widerruf, the DSI discovery would follow
links away from the page and return the wrong document (e.g.
/impressum → finds link to /datenschutz → returns datenschutz text).
Now: for non-DSE doc_types, prefer the html_full_page document
(self-extracted from the actual URL the user provided) over linked
pages found by the crawler.
Fixes safetykon.de/impressum returning datenschutz text.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1. localStorage Persistenz: URL, letztes Ergebnis, Historie (30 Eintraege)
2. Historie: Zeigt URL, Datum, Provider, Violations, Prozent
3. Letztes Ergebnis bleibt nach Tab-Wechsel/Reload sichtbar
4. E-Mail-Report: HTML-formatiert mit Violations + Hints an mailpit
5. Email-Status Anzeige im Frontend
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Headings starting with numbers (numbered sections like "5. Soziale
Medien", "6. Analyse-Tools") were not detected because the check
required stripped[0].isupper(). Now also accepts stripped[0].isdigit().
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- "5. Soziale Medien" now stripped to "soziale medien" before classification
- Added "soziale medien/netzwerke" as social_media heading pattern
- Fixes etogruppe.com where Social Media section wasn't detected
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1. Soft hyphens (/\xad) stripped before regex matching —
fixes "Datenübertragbarkeit" not matching
2. Art. 15/17/20: allow adjectives between "Recht auf" and keyword
("Recht auf unentgeltliche Auskunft" now matches)
3. DSB contact: regex spans up to 300 chars across newlines
(DSB section with company address between heading and email)
4. Löschkonzept: added "Fortfall", "Entfall", "Beendigung" as
deletion trigger words alongside "Ablauf"/"Wegfall"
Reduces etogruppe FPs from 5 to ~1.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
New "Banner-Check" tab with:
- URL input → Playwright 3-phase test (before/reject/accept)
- Shield icon + provider detection
- Progress bar with pass/fail percentage
- 3-phase summary (cookies + scripts per phase)
- Violations (red) and passes (green) in structured list
Backend: new POST /api/compliance/agent/banner-check endpoint
that proxies to consent-tester:8094/scan.
Next step: Upgrade banner checks to L1/L2 format with expert
hints (same quality as document checks).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Every hint now reads like a mini-consultation from a data protection
lawyer — with specific legal references, court rulings, and common
mistakes. Examples:
- EuGH C-210/16 (Fanpage), C-298/17 (Kontaktpflicht), C-311/18 (Schrems II)
- BGH I ZR 228/03 (ladungsfaehige Anschrift), XI ZR 388/10 (AGB)
- EDSA Guidelines 2/2019 (lit. b misuse), WP 248 Rev.01 (DSFA)
- DSK-Orientierungshilfe, CNIL-Leitlinien, SDM, BSI-IT-Grundschutz
- §25 TDDDG, §38 BDSG, §309 BGB, §312k BGB, Art. 246a EGBGB
This is the core value proposition: no lawyer can deliver this level
of specific, actionable compliance feedback in 60 seconds.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Hint now explicitly warns that EU-US Privacy Shield is invalid since
Schrems II (July 2020) and recommends DPF or SCC as replacements.
This is the kind of specific, actionable feedback that makes the tool
valuable — catching outdated legal references no human would spot
in under a minute.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Two critical fixes:
1. Section splitter: Only lines that classify as a known doc_type
(cookie, social_media, dsfa, etc.) trigger section splits.
Random short lines ("Typen", "Funktionale Cookies") no longer
split sections — they all had blank lines before them in the
extracted HTML text.
2. LLM verification: Sub-section checks now pass the full document
text to the LLM, not just the section fragment. This lets the
LLM find content that the section splitter missed.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1. LLM model: qwen3:32b → qwen3.5:35b-a3b (actual model on Mac Mini)
2. Section splitter: headings MUST be preceded by a blank line.
This prevents cookie table entries ("Funktionale Cookies",
"Session Cookies") from splitting the cookie section.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1. Email report now renders as styled HTML (matching frontend design):
- Progress bars (green=completeness, blue=correctness)
- Hierarchical L1→L2 check display
- Red hint boxes under failed checks explaining what to fix
- Matched text evidence for passed checks
2. Section splitter deduplicates: two "Social Media" headings on the
same page are merged into one section instead of creating duplicates.
3. Extracted report builder to agent_doc_check_report.py (175 LOC)
to keep routes file under 500 LOC (386 LOC).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Fixes based on manual verification of all 30 failed checks:
1. Cookie table: recognize "folgende cookies" + column headers as text
2. Cookie names: add JSESSIONID, cookieinfo, et_id, BT_* patterns
3. Essential justified: match "sitzung zuordnen", "betrieb der website"
4. Social bookmarks: recognize as 2-click alternative
5. DSFA plural: "kanaelen" now matches alongside "kanal"
6. Section splitter: skip-headings no longer lose subsequent text
(Risikoabwaegung section was cut from DSFA, losing risk scores)
7. Cookie legal basis: accept Art. 6(1)(f) in cookie context
Reduces false positives from 7 to ~1-2 for IHK Konstanz test case.
Ground truth table: zeroclaw/docs/ground-truth-ihk-konstanz.md
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Each check now has a "hint" field explaining what is missing and
what the customer should do to fix it. Hints are shown in the
frontend below failed checks in red text.
Examples:
- "Bei Verarbeitung auf Basis von Art. 6(1)(f) muss dokumentiert
werden, warum Ihr berechtigtes Interesse die Rechte der
Betroffenen ueberwiegt."
- "Die ladungsfaehige Anschrift fehlt. Erforderlich: Strasse,
Hausnummer, PLZ und Ort."
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Patterns ran on text.lower() but searched [A-Z] — changed to [a-z].
Also accept D-12345 prefix (common German format).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
'Datenschutzfolgeabschätzung...Social Media' was matching as social_media
(Art. 26) instead of dsfa (Art. 35) because the social_media pattern
'datenschutz.*social media' matched first.
Fixed: DSFA patterns checked before social_media patterns.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
8 false positives from generic canonical_controls. Regex checks (9+5)
are accurate. Re-enable when ~80 specific doc-check controls exist.
See INSTRUCTION-master-controls-for-doc-check.md
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Only use controls whose test_procedure mentions document-type-specific terms:
- DSI: test_procedure must contain 'datenschutzerkl' or 'art. 13/14'
- Cookie: must contain 'cookie', 'einwilligung', 'consent'
- Impressum: must contain 'impressum'
This filters out internal governance controls (Datenmodelle, Infrastruktur)
that are irrelevant for public document checks.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Complete rewrite of rag_document_checker.py:
- Queries canonical_controls table (294K controls, 10K data_protection)
- Filters by category + title keywords per document type
- Uses test_procedure field as actual check instructions
- Regex pre-check extracts key terms from procedure → fast match
- LLM fallback only for regex misses (saves tokens)
- /no_think prefix for direct JSON output
SQL approach advantages:
- Structured data with test_procedure, pass_criteria, fail_criteria
- Category filtering (data_protection, compliance, governance)
- No Qdrant API key issues
- Controls are actual check criteria, not general legal texts
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Current 144K controls are general legal texts, not specific check criteria.
RAG integration code stays (rag_document_checker.py), just disabled in
the doc-check endpoint. Re-enable when G1-G4 block is complete and
25K Master Controls with Decision Trace are available.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
LLM returns {fulfilled: true} instead of {"fulfilled": true}.
Now fixes unquoted keys, True→true, and falls back to text-based
boolean extraction when JSON parsing fails entirely.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Qwen 3.5 uses all tokens for thinking, leaving response empty.
Using /no_think prefix to get direct JSON output.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Qwen 3.5 with latest Ollama returns structured thinking in separate
'thinking' field, leaving 'response' empty. Now checks both fields.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replaces scroll+filter approach with proper semantic search:
1. Embed query via bp-core-embedding-service (bge-m3, 1024 dim)
2. Vector search in Qdrant (bp_compliance_datenschutz + bp_compliance_gesetze)
3. Sort by cosine similarity score
4. No API key needed — local Qdrant on Mac Mini
Falls back gracefully: SDK first, then semantic Qdrant, then empty.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Go SDK points to external Qdrant (qdrant-dev.breakpilot.ai) with expired API key.
Fallback: search directly in local Qdrant (bp-core-qdrant:6333) which has
all collections: bp_compliance_datenschutz, bp_compliance_gesetze, atomic_controls_dedup.
Search strategy:
1. Try Go SDK RAG endpoint (preferred, has embedding-based search)
2. Fallback: Qdrant scroll with text-based regulation filter
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
New module: rag_document_checker.py
- Searches RAG (Qdrant) for controls relevant to document type
- Filters by regulation (DSGVO Art.13, TDDDG §25, BGB §355 etc.)
- LLM (Qwen 3.5:35b) verifies each control against document text
- Returns fulfilled/missing with evidence text + severity
- Supports: DSI, Cookie, Impressum, Widerruf, AGB, DSFA, AVV, Loeschkonzept
Integration in doc-check endpoint:
- Regex checklist runs first (fast, deterministic)
- RAG checks run after (semantic, catches what regex misses)
- Both results combined in single response
LLM prompt returns JSON: {fulfilled, evidence, issue, severity}
Think-tags stripped, JSON extracted from response.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Only Cookie and Widerruf sections are checked as separate documents.
Social Media, DSFA, Betroffenenrechte, Dienste von Drittanbietern are
part of the parent DSI and no longer generate false findings.
Added PLAN-rag-document-check.md for Phase 2:
- RAG-based checks with document-type-specific Controls
- DSFA checklist (Art. 35 + Landes-Listen)
- AVV checklist (Art. 28)
- Reference detection (sub-doc → parent doc)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When a single URL contains multiple document sections (e.g. IHK DSI page
with Cookies, Social Media, Dienste von Drittanbietern), the system now:
1. Extracts full page text (main document check as before)
2. Splits text at heading boundaries (short uppercase lines)
3. Classifies each section: Cookie→cookie checklist, Social Media→DSI etc.
4. Runs type-specific checklist per section
5. Returns all results: main doc + sub-sections
Section type detection via SECTION_TYPE_MAP patterns:
- 'Cookie*' → §25 TDDDG checklist
- 'Dienste von Drittanbietern' → DSI checklist
- 'Social Media' → DSI checklist (Art. 26 joint controllership)
- 'Widerrufsrecht' → §355 BGB checklist
- 'Impressum' → §5 TMG checklist
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
New "Dokumenten-Pruefung" tab in Compliance Agent:
- User adds multiple URLs with document type (DSI, AGB, Impressum, Cookie, Widerruf)
- Each document loaded via Playwright, accordions expanded, text extracted
- Checked against type-specific legal checklist
- Optional: Cookie banner check via checkbox
Checklisten-UX (solves "100% looks like nothing was checked"):
- All checks shown per document: green checkmark + matched text excerpt
- Red X for missing fields with legal reference
- Builds user trust: "9 Punkte geprueft, alle bestanden"
- Expandable per document with completeness bar
New checklists:
- Impressum: §5 TMG (6 fields: name, address, contact, register, VAT, representative)
- Cookie-Richtlinie: §25 TDDDG (5 fields: types, purposes, retention, third-party, opt-out)
Backend:
- POST /agent/doc-check — async with polling (same pattern as /scan)
- DocCheckResult includes checks[] with passed/failed + matched_text
- dsi_document_checker returns all_checks in SCORE finding
- Email report shows per-document checklist
Files: agent_doc_check_routes.py (280 LOC), DocCheckTab.tsx (248 LOC),
ChecklistView.tsx (130 LOC), dsi_document_checker.py (+70 LOC)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Service detection:
- Only search script tags + src/href attributes for service patterns
- Prevents false positives from DSE text mentioning services
(e.g. IHK DSE describes etracker, 'google analytics' in text)
- Technical patterns (with regex chars) still checked in full HTML
Short documents:
- Documents with < 200 words flagged as 'Kurzhinweis' instead of
'MANGELHAFT' — too short for Art. 13 completeness check
- Prevents 96-word navigation pages from showing 8 missing fields
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Benjamin Admin