Now that all 1874 MCs run per check (Task #30 cap removal), the report
was about to drown in noise. This commit adds the full aggregation /
persistence / drill-down stack so each MC is actionable, not just
counted.
A1 mc_scorecard.py (new):
build_scorecard(checks) -> per-regulation PASS/FAIL/SKIP + severity
top_fails(checks, n) -> N most severe failed MCs
full_audit_records(...) -> flat rows ready for sidecar SQLite
A2 Email rendering:
agent_doc_check_scorecard.py (new) builds an HTML scorecard table
(regulation × passed/failed/HIGH/MEDIUM/score) shown at the top of
the email. agent_doc_check_report._render_document now collapses
the 500-MC L2 forest into 'X/Y bestanden (Z Fail)' summary plus
a top-10 fails block per doc — old verbose render is gone.
A3 compliance_audit_log.py (new) — sidecar SQLite at
/data/compliance_audits.db (separate from compliance Postgres
schema to comply with the no-new-migrations rule in CLAUDE.md):
check_runs(check_id, ts, tenant_id, site_name, base_domain,
doc_count, scorecard json, vvt_summary json)
mc_results(check_id, doc_type, mc_id, label, passed, skipped,
severity, regulation, matched_text, hint)
Route persists every run after the email is sent.
docker-compose.yml adds compliance-audit volume + env.
A4 backfill_mc_regulation_llm.py (new) — Qwen-tagged backfill for
the 1636 MCs the regex pass couldn't classify. Batches of 25,
format=json, output constrained to the canonical regulation list.
Run manually: docker exec bp-compliance-backend python3 \
/app/scripts/backfill_mc_regulation_llm.py [--dry-run]
A5 Admin audit tab — GET /api/compliance/agent/audit/<check_id>
proxied via /api/sdk/v1/agent/audit/<id>. New page
/sdk/agent/audit/[checkId] renders scorecard + filterable MC table
(status / doc_type / regulation, expandable rows with matched_text
+ hint). ComplianceCheckTab now shows 'Voll-Audit oeffnen' link.
A6 Trend per tenant — GET /api/compliance/agent/audit/tenant/<id>
returns recent runs. Email scorecard shows per-regulation delta
badges ('(+12%)', '(-3%)') compared with the previous run for the
same tenant + base_domain. Lookup is one SQLite query.
Plumbing:
rag_document_checker.py — SELECT now includes 'article'; MC results
carry 'regulation' + 'article' through to CheckItem.
agent_doc_check_routes.CheckItem schema gains regulation + article
fields (defaults '') so old clients still parse.
agent_compliance_check_routes — response gains 'check_id' so the
frontend can build the audit link.
User: 'wir haben 1800 MCs erstellt um sie zu 10% zu nutzen — das ist
Schwachsinn'. Fixed all 6 gaps from the audit.
#1 max_controls=0 (was 20):
- agent_compliance_check_routes _check_single: passes max_controls=0 to
check_document_with_controls -> ALL MCs evaluated per doc_type.
- 8 doc_types now use 1874 MCs instead of 160 (10x coverage).
- Regex matching is cheap (<1s per doc); LLM-enrich cap of 10 stays.
#2 LLM-verify fixed:
- llm_verify.py was getting 0/N parsed. Causes: qwen3 thinking-mode
wrapped output in <think>...</think>, /api/generate doesn't enforce
JSON, prompt didn't handle code-fence wrappers.
- Now uses /api/chat with format='json' (forces valid JSON).
- _parse_batch_response strips <think> tags, accepts {results:[...]}
AND bare [...], adds richer regex-fallback parse, logs raw head on
total parse failure for diagnosis.
#3 Loeschkonzept checklist (new):
- doc_checks/loeschkonzept_checks.py — 9 L1 + 7 L2 checks per DIN 66398
+ Art. 5(1)(e)/17/32 DSGVO: scope+responsibility, data categories,
retention periods, legal basis refs (HGB/AO/BGB), deletion trigger,
deletion process+technical+systems, deletion proof, exceptions +
Art. 18 lock, review cycle, DSGVO references.
- runner.py registered for loeschkonzept/loeschung/loeschfristen.
#4 regulation backfill script:
- backend-compliance/scripts/backfill_mc_regulation.py — regex-detects
DSGVO/TDDDG/TMG/BGB/HGB/AO/MStV/UWG/VSBG/PAngV/GwG/BDSG/EU-VO
references in MC title+question+pass_criteria, UPDATEs regulation +
article fields.
- Idempotent (only NULL rows), --dry-run flag, batched 200/UPDATE.
- Run inside container: docker exec bp-compliance-backend python3 \
/app/scripts/backfill_mc_regulation.py
#5 MC alias-fallback:
- rag_document_checker._MC_ALIAS_FALLBACK maps doc_types without own
MCs to a related set: nutzungsbedingungen->agb, social_media->dse,
sub_processor/scc/tom_annex->avv, loeschfristen->loeschkonzept,
eu_institution/dsb->dse.
- _load_controls retries with the alias when the primary query
returns 0 rows.
- 14 additional doc_types now get MC coverage transparently.
#6 cross-domain auto-discovery:
- _autodiscover_missing builds a crawl plan: primary submitted base
+ up to 2 related domains sharing the owner SLD (e.g. BMW Group:
bmw.de + bmwgroup.com + bmwgroup.jobs).
- Detection: regex over submitted texts for https?://...<owner>...
hostnames distinct from the primary base.
- Each crawled base contributes documents + cmp_payloads to the
discovery pool.
Net effect for BMW: 1874 MCs evaluated (90 from cookie alone, was
20), Loeschkonzept Pflichtangaben benoten-bar, LLM overturns false
regex FAILs, Joint-Controller policies on bmwgroup.jobs (Social
Media) jetzt entdeckbar. Same wins will apply to CRA-Compliance check.
Add 29 new document types (IT security, data, personnel, vendor, BCM
policies) to VALID_DOCUMENT_TYPES and 5 category pills to the document
generator UI. Include seed script for production DB population.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Benjamin Admin