BMW Impressum/Cookie pages timeout in Playwright (>180s) because the
SPA has many sub-links to follow. But the HTML source already contains
the text (SSR). New fallback: direct HTTP GET + HTML tag stripping.
Order: 1. Consent-tester (Playwright, 180s) → 2. HTTP GET (30s)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1. _expand_all_interactive(): Only click aria-expanded="false" buttons.
Before: clicked ALL accordion buttons including open ones → BMW's
pre-expanded accordions got CLOSED, reducing text from 1151 to 361w.
2. _fetch_text() + /extract-text: merge ALL documents found on a page
(max_documents=10 instead of 1). BMW splits DSI across 5 sub-pages
that the discovery finds as separate documents — now merged.
3. Tab panels: unhide hidden tabpanels instead of clicking tabs
(clicking tabs can hide the currently visible panel).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1. Management Summary (agent_doc_check_report.py):
- Plain-language action items for Geschaeftsfuehrer
- Maps technical checks to business actions ("Ihren DSB erwaehnen",
"Beschwerderecht ergaenzen", "Loeschfristen dokumentieren")
- Shows at top of compliance check email before detail report
- Max 10 actions, max 3 per document
2. Batch GT Test (zeroclaw/scripts/batch_gt_test.py):
- Runs all 10 GT websites through compliance-check API
- Prints comparison table with L1 scores, word counts, services
- Saves raw JSON results for analysis
- Usage: python3 batch_gt_test.py --sites 1,6 --backend-url URL
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Cross-search "not in text" findings are only shown when regex L1
completeness < 50%. This prevents false positives where the text IS
the right doc_type but doesn't contain the specific cross-search
keywords (e.g. Impressum passes 9/13 checks but lacks "§5 TMG").
Also: cross-search now checks entries with wrong text, not just empty.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Cross-Document Intelligence: When a doc_type row is empty, searches
ALL other loaded documents for that content. If found (e.g. Widerruf
in AGB), extracts the section, runs the check, AND creates a finding:
"Widerrufsbelehrung in falschem Dokument gefunden — schwer auffindbar"
Keywords for: widerruf, cookie, social_media, impressum, agb, dsb.
Integrated as Step 1c in compliance check pipeline.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Root cause: Spiegel DSI text was truncated (lazy-loading) — the
rights/DSB/complaints sections at the bottom were never extracted.
Fixes:
1. Text extraction: scroll to bottom before innerText (dsi_discovery.py)
2. V.i.S.d.P.: add "verantwortlicher i.s.v." + "§18 Abs. N MStV" pattern
3. USt-IdNr: add "umsatzsteuer-id" + "DE 212 442 423" (with spaces)
4. Profiler: remove generic "anwalt"/"praxis" (false positive on Spiegel
"Redaktionsanwalt"), keep only "rechtsanwalt", "kanzlei" etc.
5. Section splitter: auto_fill_from_dsi() fills empty Cookie/Social-Media
rows from sections found in the DSI text
Ground Truth 06-spiegel.md fully rewritten with verified data from
live website — 3 L1 False Negatives identified (DSB, Beschwerderecht,
Betroffenenrechte all present on website but not in extracted text).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Phase 1-2 of the closed quality loop:
- GVL cache (consent-tester/services/gvl_cache.py): downloads and caches
IAB Global Vendor List with 24h TTL, resolves vendor IDs to names,
purposes, policy URLs, retention, country
- Vendor extraction (consent_interceptor.py): extract_tcf_vendors()
reads __tcfapi after accept phase, resolves via GVL
- Scan response: tcf_vendors field added to /scan endpoint
- VVT mapper (vendor_vvt_mapper.py): maps TCF vendors to VVT format
with purpose labels, Rechtsgrundlage, Drittland detection
- Vendor cross-check (banner_cookie_cross_check.py): checks all TCF
vendors against DSI text — missing vendors, undocumented transfers
- Compliance check integrates Step 3d: TCF vendors vs DSI
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- New profile_extractor.py: extracts Company Profile fields (name,
legal form, address, DPO, USt-IdNr) and Compliance Scope hints
(Art. 9 data, third country, profiling) from document texts
- Scenario per document: regenerate (<30%), fix (30-95%), import (>95%)
- Widerruf for B2B: no longer skipped, instead all checks flagged as
INFO with "not needed for B2B" hint
- Move _build_profile_html to report builder module
- DocCheckResult gets scenario field
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Skip Widerrufsbelehrung check entirely for B2B/B2G businesses
- Limit MC checks to top 20 per doc_type (by severity) to reduce noise
(e.g. 75 impressum MCs → 20, avoiding 55 irrelevant FAILs)
- Add consulting/manufacturing industry keywords (arbeitssicherheit,
brandschutz, werkzeugbau, etc.)
- Lower industry detection threshold from 2 to 1 keyword hit
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When the same URL is used for multiple document types (e.g. /datenschutz
for DSI + Cookie + DSB), the section splitter now:
- Detects duplicate URLs and fetches text only once
- Splits text at classified headings (Cookie, Google Analytics, etc.)
- Assigns matching sections to each doc_type
- DSI always keeps the full text
Extracted to section_splitter.py (170 LOC) to keep routes under 500.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add automatic banner check (Step 3b) and banner-vs-cookie cross-check
(Step 3c) to unified compliance check. Extract cross-check logic to
banner_cookie_cross_check.py to keep routes under 500 LOC.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Benjamin Admin