Benjamin_Boenisch/breakpilot-compliance
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs+tests: Phase 2 RAG audit — missing tests, dev docs, SDK flow page
All checks were successful
CI / go-lint (push) Has been skipped
Details
CI / python-lint (push) Has been skipped
Details
CI / nodejs-lint (push) Has been skipped
Details
CI / test-go-ai-compliance (push) Successful in 33s
Details
CI / test-python-backend-compliance (push) Successful in 27s
Details
CI / test-python-document-crawler (push) Successful in 20s
Details
CI / test-python-dsms-gateway (push) Successful in 16s
Details

Browse Source 
- Add rag-query.test.ts (7 Jest tests for shared queryRAG utility)
- Add test_routes_legal_context.py (3 tests for ?include_legal_context param)
- Update ARCHITECTURE.md with multi-collection RAG section (3.3)
- Update DEVELOPER.md with RAG usage examples, collection table, error tolerance
- Add SDK flow page with updated requirements + DSFA RAG descriptions

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Benjamin Admin
2026-03-02 09:26:57 +01:00
parent 14a99322eb
commit d9f819e5be
6 changed files with 2090 additions and 41 deletions
Download Patch File Download Diff File Expand all files Collapse all files

77
docs-src/services/ai-compliance-sdk/ARCHITECTURE.md
View File

@@ -144,53 +144,88 @@ Das Eskalationssystem routet kritische Assessments zur menschlichen Prüfung.
└─────────────────────────────────────────────────────────────────┘
```
### 3.3 Legal RAG (`internal/llm/legal_rag.go`)
### 3.3 Legal RAG (`internal/ucca/legal_rag.go`)
Semantische Suche in 19 EU-Regulierungen für kontextbasierte Erklärungen.
Semantische Suche in 19 EU-Regulierungen fuer kontextbasierte Erklaerungen.
Unterstuetzt Multi-Collection-Suche fuer unterschiedliche Rechtsgebiete.
```
┌─────────────────────────────────────────────────────────────────┐
│ Legal RAG System │
├─────────────────────────────────────────────────────────────────┤
│ │
│ Explain Request ──────────────────────────────────────────────> │
│ Search Request (Query + Collection) ────────────────────────── │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ Qdrant Vector DB │ │
│ │ Collection: bp_legal_corpus │ │
│ │ 2,274 Chunks, 1024-dim BGE-M3 │ │
│ │ 6 Collections, 1024-dim BGE-M3 │ │
│ ├─────────────────────────────────────────────────────────────┤ │
│ │ │ │
│ │ EU-Verordnungen: │ │
│ │ ├── DSGVO (128) ├── AI Act (96) ├── NIS2 (128) │ │
│ │ ├── CRA (256) ├── Data Act (256) ├── DSA (256) │ │
│ │ ├── DGA (32) ├── EUCSA (32) ├── DPF (714) │ │
│ │ └── ... │ │
│ │ bp_compliance_ce (Default) │ │
│ │ ├── DSGVO, AI Act, NIS2, CRA, Data Act, DSA, DPF, ... │ │
│ │ └── EU-Verordnungen & CE-Regulierungen │ │
│ │ │ │
│ │ Deutsche Gesetze: │ │
│ │ ├── TDDDG (1) ├── SCC (32) ├── ... │ │
│ │ bp_compliance_recht │ │
│ │ ├── BDSG, TDDDG, DDG, UrhG, TMG, TKG │ │
│ │ └── Deutsche Gesetze │ │
│ │ │ │
│ │ BSI-Standards: │ │
│ │ ├── TR-03161-1 (6) ├── TR-03161-2 (6) ├── TR-03161-3 │ │
│ │ bp_compliance_gesetze — Regulierungstexte (Modul-Matching) │ │
│ │ bp_compliance_datenschutz — DSGVO Datenschutzmassnahmen │ │
│ │ bp_dsfa_corpus — DSFA Templates & Bewertungskriterien │ │
│ │ bp_legal_templates — Rechtsdokument-Vorlagen │ │
│ │ │ │
│ └─────────────────────────────────────────────────────────────┘ │
│ │ │
│ │ Hybrid Search (Dense + Sparse) │
│ │ Re-Ranking (Cross-Encoder) │
│ │ Dense Search (BGE-M3 Embedding) │
│ ▼ │
│ Top-K Relevant Passages ─────────────────────────────────────> │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ LLM Explanation │ │
│ │ Provider: Ollama (local) / Anthropic (fallback) │ │
│ │ Prompt: Assessment + Legal Context → Erklärung │ │
│ │ Consumer Modules │ │
│ │ ├── UCCA Explain (LLM-Erklaerung fuer Assessments) │ │
│ │ ├── Requirements AI (interpret + suggest_controls) │ │
│ │ ├── DSFA Drafting v1/v2 (Rechtskontext im Draft) │ │
│ │ └── Requirements API (?include_legal_context=true) │ │
│ └─────────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────┘
```
#### 3.3.1 Multi-Collection Search API
```
POST /sdk/v1/rag/search
{
"query": "DSGVO Art. 35 Datenschutz-Folgenabschaetzung",
"collection": "bp_compliance_recht", // optional, default: bp_compliance_ce
"regulations": ["eu_2016_679"], // optional Filter
"top_k": 5 // optional, default: 5, max: 20
}
Erlaubte Collections (Whitelist):
bp_compliance_ce, bp_compliance_recht, bp_compliance_gesetze,
bp_compliance_datenschutz, bp_dsfa_corpus, bp_legal_templates
```
#### 3.3.2 Collection-Routing (Requirements AI)
Das Python-Backend waehlt automatisch die passende Collection:
| Regulation | Collection |
|------------|-----------|
| DSGVO, GDPR, AI_ACT, NIS2, CRA | `bp_compliance_ce` |
| BDSG, TDDDG, DDG, UrhG, TMG, TKG | `bp_compliance_recht` |
| Unbekannt / Fallback | `bp_compliance_ce` |
#### 3.3.3 Fehlertoleranz
RAG-Ausfaelle brechen **nie** die Hauptfunktion:
- Go SDK: `SearchCollection()` gibt leere Liste bei Fehler
- Python: `ComplianceRAGClient.search()` gibt `[]` zurueck, wirft nie
- TypeScript: `queryRAG()` gibt `''` zurueck, wirft nie
---
## 4. Datenfluss
@@ -943,5 +978,5 @@ policies/obligations/
---
*Dokumentation erstellt: 2026-01-29*
*Version: 2.1.0*
*Dokumentation erstellt: 2026-01-29, aktualisiert: 2026-03-02*
*Version: 2.2.0 (Phase 2: Multi-Collection RAG)*

148
docs-src/services/ai-compliance-sdk/DEVELOPER.md
View File

@@ -273,36 +273,112 @@ if facts.Publisher == "DIN_MEDIA" && facts.AIUsePermitted != "YES" {
## 5. Legal RAG Integration
### Übersicht
### Uebersicht
Das Legal RAG System (`internal/ucca/legal_rag.go`) generiert Erklärungen mit rechtlichem Kontext.
Das Legal RAG System (`internal/ucca/legal_rag.go`) stellt semantische Suche in Qdrant-Collections bereit. Es wird von mehreren Modulen konsumiert:
### Verwendung
- **UCCA Explain**: LLM-Erklaerungen fuer Assessments
- **Requirements AI** (Python): `interpret_requirement()` und `suggest_controls()`
- **DSFA Drafting** (TypeScript): v1 + v2 Draft-Pipelines
- **Requirements API** (Python): `?include_legal_context=true` Endpunkt
### Go SDK — SearchCollection
```go
import "ai-compliance-sdk/internal/ucca"
rag := ucca.NewLegalRAGService(qdrantClient, llmClient, "bp_legal_corpus")
client := ucca.NewLegalRAGClient()
// Erklärung generieren
explanation, err := rag.Explain(ctx, result, intake)
if err != nil {
log.Error(err)
}
// Standard-Suche (bp_compliance_ce)
results, err := client.Search(ctx, "DSGVO Art. 35", nil, 5)
fmt.Println("Erklärung:", explanation.Text)
fmt.Println("Rechtsquellen:", explanation.Sources)
// Suche in spezifischer Collection
results, err := client.SearchCollection(ctx, "bp_compliance_recht", "BDSG §26", nil, 3)
// Leere Collection → Fallback auf Default (bp_compliance_ce)
results, err := client.SearchCollection(ctx, "", "DSGVO Art. 35", nil, 5)
```
### Rechtsquellen im RAG
### Python RAG Client (Proxy)
| Quelle | Chunks | Beschreibung |
|--------|--------|--------------|
| DSGVO | 128 | EU Datenschutz-Grundverordnung |
| AI Act | 96 | EU AI-Verordnung |
| NIS2 | 128 | Netzwerk-Informationssicherheit |
| SCC | 32 | Standardvertragsklauseln |
| DPF | 714 | Data Privacy Framework |
```python
from compliance.services.rag_client import get_rag_client
rag = get_rag_client()
# Async-Suche via Go SDK
results = await rag.search(
"DSGVO Art. 35 Risikobewertung",
collection="bp_compliance_recht",
top_k=3
)
# Fuer LLM-Prompt formatieren
context_str = rag.format_for_prompt(results)
# → "## Relevanter Rechtskontext\n1. **DSGVO** (eu_2016_679) — Art. 35\n..."
```
### TypeScript Shared Utility (Drafting Engine)
```typescript
import { queryRAG } from '@/lib/sdk/drafting-engine/rag-query'
// Sucht via klausur-service DSFA-RAG
const ragContext = await queryRAG('DSFA Art. 35 DSGVO', 3)
// → "[Quelle 1: DSGVO]\nArt. 35 regelt die DSFA..."
```
### RAG Search API
```bash
# Suche in spezifischer Collection
curl -X POST http://localhost:8090/sdk/v1/rag/search \
-H "Content-Type: application/json" \
-d '{
"query": "Datenschutz-Folgenabschaetzung Art. 35",
"collection": "bp_compliance_recht",
"top_k": 3
}'
# Antwort
{
"query": "Datenschutz-Folgenabschaetzung Art. 35",
"results": [
{
"text": "...",
"regulation_code": "eu_2016_679",
"regulation_name": "DSGVO",
"regulation_short": "DSGVO",
"score": 0.92
}
],
"count": 1
}
```
### Erlaubte Collections (Whitelist)
| Collection | Inhalt |
|------------|--------|
| `bp_compliance_ce` | EU-Verordnungen (DSGVO, AI Act, NIS2, CRA, ...) |
| `bp_compliance_recht` | Deutsche Gesetze (BDSG, TDDDG, DDG, ...) |
| `bp_compliance_gesetze` | Regulierungstexte fuer Modul-Matching |
| `bp_compliance_datenschutz` | DSGVO Datenschutzmassnahmen |
| `bp_dsfa_corpus` | DSFA Templates & Bewertungskriterien |
| `bp_legal_templates` | Rechtsdokument-Vorlagen (DSE, AGB, AVV) |
Unbekannte Collections → `400 Bad Request`.
### Fehlertoleranz
RAG-Ausfaelle brechen **nie** die Hauptfunktion:
| Schicht | Verhalten bei Fehler |
|---------|---------------------|
| Go SDK (`SearchCollection`) | Gibt `error` zurueck (Caller entscheidet) |
| Python (`ComplianceRAGClient`) | Gibt `[]` zurueck, loggt WARNING |
| TypeScript (`queryRAG`) | Gibt `''` zurueck, kein throw |
| Requirements API | `legal_context: []` statt HTTP 500 |
---
@@ -391,6 +467,15 @@ go monitor.Start(ctx)
| GET | `/sdk/v1/ucca/wizard/schema` | Wizard-Schema abrufen |
| POST | `/sdk/v1/ucca/wizard/ask` | Legal Assistant fragen |
### RAG Endpoints
| Method | Endpoint | Beschreibung |
|--------|----------|--------------|
| POST | `/sdk/v1/rag/search` | Multi-Collection RAG-Suche |
| GET | `/sdk/v1/rag/regulations` | Verfuegbare Regulierungen |
| GET | `/sdk/v1/rag/corpus-status` | Corpus-Versions-Status |
| GET | `/sdk/v1/rag/corpus-versions/:collection` | Versionshistorie |
### License Endpoints
| Method | Endpoint | Beschreibung |
@@ -732,6 +817,10 @@ func TestAIActModule_HighRiskEmploymentAI(t *testing.T) {
|-------|--------------|
| `internal/ucca/policy_engine.go` | Haupt-Policy-Engine |
| `internal/ucca/license_policy.go` | License Policy Engine |
| `internal/ucca/legal_rag.go` | Legal RAG Client (Multi-Collection Search) |
| `internal/ucca/legal_rag_test.go` | Tests fuer SearchCollection, Fallback |
| `internal/api/handlers/rag_handlers.go` | RAG Search API (Collection-Whitelist) |
| `internal/api/handlers/rag_handlers_test.go` | Tests fuer RAG Handler |
| `internal/ucca/obligations_framework.go` | Obligations Interfaces & Typen |
| `internal/ucca/obligations_registry.go` | Modul-Registry |
| `internal/ucca/nis2_module.go` | NIS2 Decision Tree |
@@ -741,6 +830,25 @@ func TestAIActModule_HighRiskEmploymentAI(t *testing.T) {
| `internal/api/handlers/obligations_handlers.go` | Obligations API |
| `policies/obligations/*.yaml` | Pflichten-Kataloge |
### Python Backend (RAG-Integration)
| Datei | Beschreibung |
|-------|--------------|
| `backend-compliance/compliance/services/rag_client.py` | ComplianceRAGClient (Proxy zum Go SDK) |
| `backend-compliance/compliance/services/ai_compliance_assistant.py` | AI Assistant mit RAG-Anreicherung |
| `backend-compliance/compliance/api/routes.py` | Requirements API mit `?include_legal_context` |
| `backend-compliance/tests/test_rag_client.py` | Tests fuer RAG Client + Collection Mapping |
| `backend-compliance/tests/test_routes_legal_context.py` | Tests fuer Legal Context API |
### TypeScript Frontend (Drafting Engine RAG)
| Datei | Beschreibung |
|-------|--------------|
| `admin-compliance/lib/sdk/drafting-engine/rag-query.ts` | Shared `queryRAG()` Utility |
| `admin-compliance/lib/sdk/drafting-engine/__tests__/rag-query.test.ts` | Tests fuer queryRAG |
| `admin-compliance/app/api/sdk/drafting-engine/draft/route.ts` | Draft v1/v2 mit RAG-Kontext |
| `admin-compliance/app/api/sdk/drafting-engine/chat/route.ts` | Chat mit shared queryRAG Import |
---
*Dokumentationsstand: 2026-01-29*
*Dokumentationsstand: 2026-03-02*