feat(audit): A Audit-Transparenz + B Tabellen-Parse + D HTML-Tables aus DOM
CI / detect-changes (push) Successful in 10s
CI / branch-name (push) Has been skipped
CI / guardrail-integrity (push) Has been skipped
CI / nodejs-lint (push) Has been skipped
CI / nodejs-build (push) Has been skipped
CI / test-python-backend (push) Successful in 45s
CI / secret-scan (push) Has been skipped
CI / dep-audit (push) Has been skipped
CI / sbom-scan (push) Has been skipped
CI / validate-canonical-controls (push) Successful in 20s
CI / loc-budget (push) Failing after 17s
CI / go-lint (push) Has been skipped
CI / python-lint (push) Has been skipped
CI / test-go (push) Has been skipped
CI / iace-gt-coverage (push) Has been skipped
CI / test-python-document-crawler (push) Has been skipped
CI / test-python-dsms-gateway (push) Has been skipped
CI / detect-changes (push) Successful in 10s
CI / branch-name (push) Has been skipped
CI / guardrail-integrity (push) Has been skipped
CI / nodejs-lint (push) Has been skipped
CI / nodejs-build (push) Has been skipped
CI / test-python-backend (push) Successful in 45s
CI / secret-scan (push) Has been skipped
CI / dep-audit (push) Has been skipped
CI / sbom-scan (push) Has been skipped
CI / validate-canonical-controls (push) Successful in 20s
CI / loc-budget (push) Failing after 17s
CI / go-lint (push) Has been skipped
CI / python-lint (push) Has been skipped
CI / test-go (push) Has been skipped
CI / iace-gt-coverage (push) Has been skipped
CI / test-python-document-crawler (push) Has been skipped
CI / test-python-dsms-gateway (push) Has been skipped
Drei zusammenhaengende Fixes fuer den VW-Befund (6 Vendors statt 100+): A — audit_quality_checks.py: drei systemische Vorbehalte die IMMER prominent gezeigt werden: * banner_detected=False trotz Cookie-Doc → HIGH 'CMP-Tool ungeladen' * cookie_doc >= 30k chars aber cmp_vendors < 15 → HIGH/MEDIUM 'Vendor-Liste auffaellig kurz fuer Doc-Groesse' * submitted URL aber 0/Mini-Text → MEDIUM 'URL nicht ladbar' Rote Audit-Vorbehalt-Box ueber dem GF-1-Pager. GF-Summary sagt 'Audit unvollstaendig' statt faelschlich 'Keine kritischen Themen'. gf_one_pager nimmt audit_quality_findings in top_findings auf (BEVOR andere Findings). B — cookies_table_parser laeuft jetzt auch auf gecrawltem Cookie-Doc- Text (nicht nur bei User-Paste). Wenn der dsi-discovery-Response Tab/ Pipe-getrennte Tabellen-Reihen liefert, parsen wir sie deterministisch. D — consent-tester/dsi-discovery extrahiert jetzt zusaetzlich zum Text die <table>-Elemente aus dem DOM als list[str] (Tab-getrennt pro Zeile, mind. 2 Zellen, mind. 3 Zeilen, max 10 Tabellen pro Doc). Backend schleust diese als 'html_table'-cmp_payload ein und jagt sie zuerst durch cookies_table_parser → 100% deterministische Vendor-Extraktion ohne LLM. VW-Erwartung: aus der 65k-Cookie-Tabelle werden jetzt 30-50 Vendors deterministisch geparst statt 6 vom LLM-Cascade. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -292,6 +292,10 @@ class DSIDocumentInfo(BaseModel):
|
||||
word_count: int = 0
|
||||
text_preview: str = ""
|
||||
full_text: str = ""
|
||||
# D — Tab-getrennte HTML-Tabellen aus dem DOM (z.B. Cookie-Tabellen).
|
||||
# Pro Tabelle ein Array von Zeilen, jede Zeile Tab-getrennt.
|
||||
# Backend nutzt das fuer deterministischen Cookie-Tabellen-Parse.
|
||||
tables: list[list[str]] = []
|
||||
|
||||
|
||||
class DSIDiscoveryResponse(BaseModel):
|
||||
@@ -347,6 +351,7 @@ async def dsi_discovery(req: DSIDiscoveryRequest):
|
||||
word_count=d.word_count,
|
||||
text_preview=d.text[:500] if d.text else "",
|
||||
full_text=d.text[:200000] if d.text else "",
|
||||
tables=getattr(d, "tables", []) or [],
|
||||
)
|
||||
for d in result.documents
|
||||
],
|
||||
|
||||
@@ -159,6 +159,10 @@ class DiscoveredDSI:
|
||||
text: str = "" # Extracted full text
|
||||
sections: list[dict] = field(default_factory=list) # Parsed sections
|
||||
word_count: int = 0
|
||||
# D — Tab-getrennte HTML-Tabellen aus dem DOM. Pro Tabelle eine
|
||||
# Liste von Zeilen, jede Zeile ein Tab-getrennter String. Erlaubt
|
||||
# dem Backend deterministischen Cookie-Tabellen-Parse ohne LLM.
|
||||
tables: list[list[str]] = field(default_factory=list)
|
||||
|
||||
@dataclass
|
||||
class DSIDiscoveryResult:
|
||||
@@ -523,12 +527,34 @@ async def discover_dsi_documents(
|
||||
return (body.innerText || body.textContent || '').trim();
|
||||
}
|
||||
""")
|
||||
# D — HTML-Tabellen separat extrahieren. Pro Tabelle ein
|
||||
# Array von Zeilen, jede Zeile ein Tab-getrennter String.
|
||||
# Das erlaubt dem Backend deterministischen Spalten-Parse
|
||||
# (cookies_table_parser) ohne LLM-Halluzinationen.
|
||||
tables = await page.evaluate("""
|
||||
() => {
|
||||
const out = [];
|
||||
document.querySelectorAll('table').forEach(t => {
|
||||
const rows = [];
|
||||
t.querySelectorAll('tr').forEach(tr => {
|
||||
const cells = [];
|
||||
tr.querySelectorAll('th, td').forEach(c => {
|
||||
cells.push((c.innerText || c.textContent || '').trim().replace(/\\s+/g, ' '));
|
||||
});
|
||||
if (cells.length >= 2) rows.push(cells.join('\\t'));
|
||||
});
|
||||
if (rows.length >= 3) out.push(rows);
|
||||
});
|
||||
return out;
|
||||
}
|
||||
""")
|
||||
if text and len(text) > 50:
|
||||
result.documents.append(DiscoveredDSI(
|
||||
title=title, url=href, source_url=url,
|
||||
language=lang,
|
||||
doc_type="cross_domain" if not _is_allowed_domain(href, base_domain) else "html_page",
|
||||
text=text[:200000], word_count=len(text.split()),
|
||||
tables=(tables or [])[:10],
|
||||
))
|
||||
|
||||
# Recursive: search THIS page for more DSI links
|
||||
|
||||
Reference in New Issue
Block a user