feat(audit): A Audit-Transparenz + B Tabellen-Parse + D HTML-Tables aus DOM
CI / detect-changes (push) Successful in 10s
CI / branch-name (push) Has been skipped
CI / guardrail-integrity (push) Has been skipped
CI / nodejs-lint (push) Has been skipped
CI / nodejs-build (push) Has been skipped
CI / test-python-backend (push) Successful in 45s
CI / secret-scan (push) Has been skipped
CI / dep-audit (push) Has been skipped
CI / sbom-scan (push) Has been skipped
CI / validate-canonical-controls (push) Successful in 20s
CI / loc-budget (push) Failing after 17s
CI / go-lint (push) Has been skipped
CI / python-lint (push) Has been skipped
CI / test-go (push) Has been skipped
CI / iace-gt-coverage (push) Has been skipped
CI / test-python-document-crawler (push) Has been skipped
CI / test-python-dsms-gateway (push) Has been skipped

Drei zusammenhaengende Fixes fuer den VW-Befund (6 Vendors statt 100+):

A — audit_quality_checks.py: drei systemische Vorbehalte die IMMER prominent
gezeigt werden:
* banner_detected=False trotz Cookie-Doc → HIGH 'CMP-Tool ungeladen'
* cookie_doc >= 30k chars aber cmp_vendors < 15 → HIGH/MEDIUM
  'Vendor-Liste auffaellig kurz fuer Doc-Groesse'
* submitted URL aber 0/Mini-Text → MEDIUM 'URL nicht ladbar'
Rote Audit-Vorbehalt-Box ueber dem GF-1-Pager. GF-Summary sagt
'Audit unvollstaendig' statt faelschlich 'Keine kritischen Themen'.
gf_one_pager nimmt audit_quality_findings in top_findings auf
(BEVOR andere Findings).

B — cookies_table_parser laeuft jetzt auch auf gecrawltem Cookie-Doc-
Text (nicht nur bei User-Paste). Wenn der dsi-discovery-Response Tab/
Pipe-getrennte Tabellen-Reihen liefert, parsen wir sie deterministisch.

D — consent-tester/dsi-discovery extrahiert jetzt zusaetzlich zum
Text die <table>-Elemente aus dem DOM als list[str] (Tab-getrennt pro
Zeile, mind. 2 Zellen, mind. 3 Zeilen, max 10 Tabellen pro Doc). Backend
schleust diese als 'html_table'-cmp_payload ein und jagt sie zuerst durch
cookies_table_parser → 100% deterministische Vendor-Extraktion ohne LLM.

VW-Erwartung: aus der 65k-Cookie-Tabelle werden jetzt 30-50 Vendors
deterministisch geparst statt 6 vom LLM-Cascade.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Benjamin Admin
2026-05-21 20:21:28 +02:00
parent e411c4f0d3
commit cb5dad1a2f
5 changed files with 382 additions and 16 deletions
+5
View File
@@ -292,6 +292,10 @@ class DSIDocumentInfo(BaseModel):
word_count: int = 0
text_preview: str = ""
full_text: str = ""
# D — Tab-getrennte HTML-Tabellen aus dem DOM (z.B. Cookie-Tabellen).
# Pro Tabelle ein Array von Zeilen, jede Zeile Tab-getrennt.
# Backend nutzt das fuer deterministischen Cookie-Tabellen-Parse.
tables: list[list[str]] = []
class DSIDiscoveryResponse(BaseModel):
@@ -347,6 +351,7 @@ async def dsi_discovery(req: DSIDiscoveryRequest):
word_count=d.word_count,
text_preview=d.text[:500] if d.text else "",
full_text=d.text[:200000] if d.text else "",
tables=getattr(d, "tables", []) or [],
)
for d in result.documents
],
+26
View File
@@ -159,6 +159,10 @@ class DiscoveredDSI:
text: str = "" # Extracted full text
sections: list[dict] = field(default_factory=list) # Parsed sections
word_count: int = 0
# D — Tab-getrennte HTML-Tabellen aus dem DOM. Pro Tabelle eine
# Liste von Zeilen, jede Zeile ein Tab-getrennter String. Erlaubt
# dem Backend deterministischen Cookie-Tabellen-Parse ohne LLM.
tables: list[list[str]] = field(default_factory=list)
@dataclass
class DSIDiscoveryResult:
@@ -523,12 +527,34 @@ async def discover_dsi_documents(
return (body.innerText || body.textContent || '').trim();
}
""")
# D — HTML-Tabellen separat extrahieren. Pro Tabelle ein
# Array von Zeilen, jede Zeile ein Tab-getrennter String.
# Das erlaubt dem Backend deterministischen Spalten-Parse
# (cookies_table_parser) ohne LLM-Halluzinationen.
tables = await page.evaluate("""
() => {
const out = [];
document.querySelectorAll('table').forEach(t => {
const rows = [];
t.querySelectorAll('tr').forEach(tr => {
const cells = [];
tr.querySelectorAll('th, td').forEach(c => {
cells.push((c.innerText || c.textContent || '').trim().replace(/\\s+/g, ' '));
});
if (cells.length >= 2) rows.push(cells.join('\\t'));
});
if (rows.length >= 3) out.push(rows);
});
return out;
}
""")
if text and len(text) > 50:
result.documents.append(DiscoveredDSI(
title=title, url=href, source_url=url,
language=lang,
doc_type="cross_domain" if not _is_allowed_domain(href, base_domain) else "html_page",
text=text[:200000], word_count=len(text.split()),
tables=(tables or [])[:10],
))
# Recursive: search THIS page for more DSI links