fix: Restore all files lost during destructive rebase
A previous `git pull --rebase origin main` dropped 177 local commits,
losing 3400+ files across admin-v2, backend, studio-v2, website,
klausur-service, and many other services. The partial restore attempt
(660295e2) only recovered some files.
This commit restores all missing files from pre-rebase ref 98933f5e
while preserving post-rebase additions (night-scheduler, night-mode UI,
NightModeWidget dashboard integration).
Restored features include:
- AI Module Sidebar (FAB), OCR Labeling, OCR Compare
- GPU Dashboard, RAG Pipeline, Magic Help
- Klausur-Korrektur (8 files), Abitur-Archiv (5+ files)
- Companion, Zeugnisse-Crawler, Screen Flow
- Full backend, studio-v2, website, klausur-service
- All compliance SDKs, agent-core, voice-service
- CI/CD configs, documentation, scripts
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Benjamin Admin
307
backend/compliance/README.md
Normal file
307
backend/compliance/README.md
Normal file
@@ -0,0 +1,307 @@
|
||||
# Breakpilot Compliance & Audit Framework
|
||||
|
||||
## Uebersicht
|
||||
|
||||
Enterprise-ready GRC (Governance, Risk, Compliance) Framework fuer die Breakpilot EdTech-Plattform.
|
||||
|
||||
### Kernfunktionen
|
||||
|
||||
| Feature | Status | Beschreibung |
|
||||
|---------|--------|--------------|
|
||||
| **19 EU-Regulations** | Aktiv | DSGVO, AI Act, CRA, NIS2, Data Act, etc. |
|
||||
| **558 Requirements** | Aktiv | Automatisch extrahiert aus EUR-Lex + BSI-TR PDFs |
|
||||
| **44 Controls** | Aktiv | Technische und organisatorische Massnahmen |
|
||||
| **474 Control-Mappings** | Aktiv | Keyword-basiertes Auto-Mapping |
|
||||
| **KI-Interpretation** | Aktiv | Claude API fuer Anforderungsanalyse |
|
||||
| **Executive Dashboard** | Aktiv | Ampel-Status, Trends, Top-Risiken |
|
||||
|
||||
## Architektur
|
||||
|
||||
```
|
||||
backend/compliance/
|
||||
├── api/
|
||||
│ ├── routes.py # 52 FastAPI Endpoints
|
||||
│ └── schemas.py # Pydantic Response Models
|
||||
├── db/
|
||||
│ ├── models.py # SQLAlchemy Models
|
||||
│ └── repository.py # CRUD Operations
|
||||
├── data/
|
||||
│ ├── regulations.py # 19 Regulations Seed
|
||||
│ ├── controls.py # 44 Controls Seed
|
||||
│ ├── requirements.py # Requirements Seed
|
||||
│ └── service_modules.py # 30 Service-Module
|
||||
├── services/
|
||||
│ ├── ai_compliance_assistant.py # Claude Integration
|
||||
│ ├── llm_provider.py # LLM Abstraction Layer
|
||||
│ ├── pdf_extractor.py # BSI-TR PDF Parser
|
||||
│ └── regulation_scraper.py # EUR-Lex Scraper
|
||||
└── tests/ # Pytest Tests (in /backend/tests/)
|
||||
```
|
||||
|
||||
## Schnellstart
|
||||
|
||||
### 1. Backend starten
|
||||
|
||||
```bash
|
||||
cd backend
|
||||
docker-compose up -d
|
||||
# ODER
|
||||
uvicorn main:app --reload --port 8000
|
||||
```
|
||||
|
||||
### 2. Datenbank initialisieren
|
||||
|
||||
```bash
|
||||
# Regulations, Controls, Requirements seeden
|
||||
curl -X POST http://localhost:8000/api/v1/compliance/seed \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"force": false}'
|
||||
|
||||
# Service-Module seeden
|
||||
curl -X POST http://localhost:8000/api/v1/compliance/modules/seed \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"force": false}'
|
||||
```
|
||||
|
||||
### 3. KI-Interpretation aktivieren
|
||||
|
||||
```bash
|
||||
# Vault-gesteuerte API-Keys
|
||||
export VAULT_ADDR=http://localhost:8200
|
||||
export VAULT_TOKEN=breakpilot-dev-token
|
||||
|
||||
# Status pruefen
|
||||
curl http://localhost:8000/api/v1/compliance/ai/status
|
||||
|
||||
# Einzelne Anforderung interpretieren
|
||||
curl -X POST http://localhost:8000/api/v1/compliance/ai/interpret \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"requirement_id": "REQ-ID", "save_to_db": true}'
|
||||
```
|
||||
|
||||
## API-Endpoints
|
||||
|
||||
### Dashboard & Executive View
|
||||
|
||||
| Method | Endpoint | Beschreibung |
|
||||
|--------|----------|--------------|
|
||||
| GET | `/api/v1/compliance/dashboard` | Dashboard-Daten mit Scores |
|
||||
| GET | `/api/v1/compliance/dashboard/executive` | Executive Dashboard (Ampel, Trends) |
|
||||
| GET | `/api/v1/compliance/dashboard/trend` | Score-Trend (12 Monate) |
|
||||
|
||||
### Regulations & Requirements
|
||||
|
||||
| Method | Endpoint | Beschreibung |
|
||||
|--------|----------|--------------|
|
||||
| GET | `/api/v1/compliance/regulations` | Alle 19 Regulations |
|
||||
| GET | `/api/v1/compliance/regulations/{code}` | Eine Regulation |
|
||||
| GET | `/api/v1/compliance/requirements` | 558 Requirements (paginiert) |
|
||||
| GET | `/api/v1/compliance/requirements/{id}` | Einzelnes Requirement |
|
||||
|
||||
### Controls & Mappings
|
||||
|
||||
| Method | Endpoint | Beschreibung |
|
||||
|--------|----------|--------------|
|
||||
| GET | `/api/v1/compliance/controls` | Alle 44 Controls |
|
||||
| GET | `/api/v1/compliance/controls/{id}` | Ein Control |
|
||||
| GET | `/api/v1/compliance/controls/by-domain/{domain}` | Controls nach Domain |
|
||||
| GET | `/api/v1/compliance/mappings` | 474 Control-Mappings |
|
||||
|
||||
### KI-Features
|
||||
|
||||
| Method | Endpoint | Beschreibung |
|
||||
|--------|----------|--------------|
|
||||
| GET | `/api/v1/compliance/ai/status` | LLM Provider Status |
|
||||
| POST | `/api/v1/compliance/ai/interpret` | Requirement interpretieren |
|
||||
| POST | `/api/v1/compliance/ai/batch` | Batch-Interpretation |
|
||||
| POST | `/api/v1/compliance/ai/suggest-controls` | Control-Vorschlaege |
|
||||
|
||||
### Scraper & Import
|
||||
|
||||
| Method | Endpoint | Beschreibung |
|
||||
|--------|----------|--------------|
|
||||
| POST | `/api/v1/compliance/scraper/fetch` | EUR-Lex Live-Fetch |
|
||||
| POST | `/api/v1/compliance/scraper/extract-pdf` | BSI-TR PDF Extraktion |
|
||||
| GET | `/api/v1/compliance/scraper/status` | Scraper-Status |
|
||||
|
||||
### Evidence & Risks
|
||||
|
||||
| Method | Endpoint | Beschreibung |
|
||||
|--------|----------|--------------|
|
||||
| GET | `/api/v1/compliance/evidence` | Alle Nachweise |
|
||||
| POST | `/api/v1/compliance/evidence/collect` | CI/CD Evidence Upload |
|
||||
| GET | `/api/v1/compliance/risks` | Risk Register |
|
||||
| GET | `/api/v1/compliance/risks/matrix` | Risk Matrix View |
|
||||
|
||||
## Datenmodell
|
||||
|
||||
### RegulationDB
|
||||
|
||||
```python
|
||||
class RegulationDB(Base):
|
||||
id: str # UUID
|
||||
code: str # "GDPR", "AIACT", etc.
|
||||
name: str # Kurzname
|
||||
full_name: str # Vollstaendiger Name
|
||||
regulation_type: enum # eu_regulation, bsi_standard, etc.
|
||||
source_url: str # EUR-Lex URL
|
||||
effective_date: date # Inkrafttreten
|
||||
```
|
||||
|
||||
### RequirementDB
|
||||
|
||||
```python
|
||||
class RequirementDB(Base):
|
||||
id: str # UUID
|
||||
regulation_id: str # FK zu Regulation
|
||||
article: str # "Art. 32"
|
||||
paragraph: str # "(1)(a)"
|
||||
title: str # Kurztitel
|
||||
requirement_text: str # Original-Text
|
||||
breakpilot_interpretation: str # KI-Interpretation
|
||||
priority: int # 1-5
|
||||
```
|
||||
|
||||
### ControlDB
|
||||
|
||||
```python
|
||||
class ControlDB(Base):
|
||||
id: str # UUID
|
||||
control_id: str # "PRIV-001"
|
||||
domain: enum # gov, priv, iam, crypto, sdlc, ops, ai
|
||||
control_type: enum # preventive, detective, corrective
|
||||
title: str # Kontroll-Titel
|
||||
pass_criteria: str # Messbare Kriterien
|
||||
code_reference: str # z.B. "middleware/pii_redactor.py:45"
|
||||
status: enum # pass, partial, fail, planned
|
||||
```
|
||||
|
||||
## Frontend-Integration
|
||||
|
||||
### Compliance Dashboard
|
||||
|
||||
```
|
||||
/admin/compliance # Haupt-Dashboard
|
||||
/admin/compliance/controls # Control Catalogue
|
||||
/admin/compliance/evidence # Evidence Management
|
||||
/admin/compliance/risks # Risk Matrix
|
||||
/admin/compliance/scraper # Regulation Scraper
|
||||
/admin/compliance/audit-workspace # Audit Workspace
|
||||
```
|
||||
|
||||
### Neue Komponenten (Sprint 1+2)
|
||||
|
||||
- `ComplianceTrendChart.tsx` - Recharts-basierter Trend-Chart
|
||||
- `TrafficLightIndicator.tsx` - Ampel-Status Anzeige
|
||||
- `LanguageSwitch.tsx` - DE/EN Terminologie-Umschaltung
|
||||
- `GlossaryTooltip.tsx` - Erklaerungen fuer Fachbegriffe
|
||||
|
||||
### i18n-System
|
||||
|
||||
```typescript
|
||||
import { getTerm, Language } from '@/lib/compliance-i18n'
|
||||
|
||||
// Nutzung
|
||||
const label = getTerm('de', 'control') // "Massnahme"
|
||||
const label = getTerm('en', 'control') // "Control"
|
||||
```
|
||||
|
||||
## Tests
|
||||
|
||||
```bash
|
||||
# Alle Compliance-Tests ausfuehren
|
||||
cd backend
|
||||
pytest tests/test_compliance_*.py -v
|
||||
|
||||
# Einzelne Test-Dateien
|
||||
pytest tests/test_compliance_api.py -v # API Endpoints
|
||||
pytest tests/test_compliance_ai.py -v # KI-Integration
|
||||
pytest tests/test_compliance_repository.py -v # Repository
|
||||
pytest tests/test_compliance_pdf_extractor.py -v # PDF Parser
|
||||
```
|
||||
|
||||
## Umgebungsvariablen
|
||||
|
||||
```bash
|
||||
# LLM Provider
|
||||
COMPLIANCE_LLM_PROVIDER=anthropic # oder "mock" fuer Tests
|
||||
ANTHROPIC_API_KEY=sk-ant-... # Falls nicht ueber Vault
|
||||
|
||||
# Vault Integration
|
||||
VAULT_ADDR=http://localhost:8200
|
||||
VAULT_TOKEN=breakpilot-dev-token
|
||||
|
||||
# Datenbank
|
||||
DATABASE_URL=postgresql://user:pass@localhost:5432/breakpilot
|
||||
```
|
||||
|
||||
## Regulations-Uebersicht
|
||||
|
||||
| Code | Name | Typ | Requirements |
|
||||
|------|------|-----|--------------|
|
||||
| GDPR | DSGVO | EU-Verordnung | ~50 |
|
||||
| AIACT | AI Act | EU-Verordnung | ~80 |
|
||||
| CRA | Cyber Resilience Act | EU-Verordnung | ~60 |
|
||||
| NIS2 | NIS2-Richtlinie | EU-Richtlinie | ~40 |
|
||||
| DATAACT | Data Act | EU-Verordnung | ~35 |
|
||||
| DGA | Data Governance Act | EU-Verordnung | ~30 |
|
||||
| DSA | Digital Services Act | EU-Verordnung | ~25 |
|
||||
| EUCSA | EU Cybersecurity Act | EU-Verordnung | ~20 |
|
||||
| EAA | European Accessibility Act | EU-Richtlinie | ~15 |
|
||||
| BSI-TR-03161-1 | Mobile Anwendungen Teil 1 | BSI-Standard | ~30 |
|
||||
| BSI-TR-03161-2 | Mobile Anwendungen Teil 2 | BSI-Standard | ~100 |
|
||||
| BSI-TR-03161-3 | Mobile Anwendungen Teil 3 | BSI-Standard | ~50 |
|
||||
| ... | 7 weitere | ... | ~50 |
|
||||
|
||||
## Control-Domains
|
||||
|
||||
| Domain | Beschreibung | Anzahl Controls |
|
||||
|--------|--------------|-----------------|
|
||||
| `gov` | Governance & Organisation | 5 |
|
||||
| `priv` | Datenschutz & Privacy | 7 |
|
||||
| `iam` | Identity & Access Management | 5 |
|
||||
| `crypto` | Kryptografie | 4 |
|
||||
| `sdlc` | Secure Development | 6 |
|
||||
| `ops` | Betrieb & Monitoring | 5 |
|
||||
| `ai` | KI-spezifisch | 5 |
|
||||
| `cra` | CRA & Supply Chain | 4 |
|
||||
| `aud` | Audit & Nachvollziehbarkeit | 3 |
|
||||
|
||||
## Erweiterungen
|
||||
|
||||
### Neue Regulation hinzufuegen
|
||||
|
||||
1. Eintrag in `data/regulations.py`
|
||||
2. Requirements ueber Scraper importieren
|
||||
3. Control-Mappings generieren
|
||||
|
||||
```bash
|
||||
# EUR-Lex Regulation importieren
|
||||
curl -X POST http://localhost:8000/api/v1/compliance/scraper/fetch \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"regulation_code": "NEW_REG", "url": "https://eur-lex.europa.eu/..."}'
|
||||
```
|
||||
|
||||
### Neues Control hinzufuegen
|
||||
|
||||
1. Eintrag in `data/controls.py`
|
||||
2. Re-Seed ausfuehren
|
||||
3. Mappings werden automatisch generiert
|
||||
|
||||
## Changelog
|
||||
|
||||
### v2.0 (2026-01-17)
|
||||
- Executive Dashboard mit Ampel-Status
|
||||
- Trend-Charts (Recharts)
|
||||
- DE/EN Terminologie-Umschaltung
|
||||
- 52 API-Endpoints
|
||||
- 558 Requirements aus 19 Regulations
|
||||
- 474 Auto-Mappings
|
||||
- KI-Interpretation (Claude API)
|
||||
|
||||
### v1.0 (2026-01-16)
|
||||
- Basis-Dashboard
|
||||
- EUR-Lex Scraper
|
||||
- BSI-TR PDF Parser
|
||||
- Control Catalogue
|
||||
- Evidence Management
|
||||
275
backend/compliance/README_AI.md
Normal file
275
backend/compliance/README_AI.md
Normal file
@@ -0,0 +1,275 @@
|
||||
# Compliance AI Integration - Quick Start
|
||||
|
||||
## Schnellstart (5 Minuten)
|
||||
|
||||
### 1. Environment Variables setzen
|
||||
|
||||
```bash
|
||||
# In backend/.env
|
||||
COMPLIANCE_LLM_PROVIDER=mock # Für Testing ohne API-Key
|
||||
# ODER
|
||||
COMPLIANCE_LLM_PROVIDER=anthropic
|
||||
ANTHROPIC_API_KEY=sk-ant-...
|
||||
```
|
||||
|
||||
### 2. Backend starten
|
||||
|
||||
```bash
|
||||
cd backend
|
||||
docker-compose up -d
|
||||
# ODER
|
||||
uvicorn main:app --reload
|
||||
```
|
||||
|
||||
### 3. Datenbank seeden
|
||||
|
||||
```bash
|
||||
# Requirements und Module laden
|
||||
curl -X POST http://localhost:8000/api/v1/compliance/seed \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"force": false}'
|
||||
|
||||
curl -X POST http://localhost:8000/api/v1/compliance/modules/seed \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"force": false}'
|
||||
```
|
||||
|
||||
### 4. AI-Features testen
|
||||
|
||||
```bash
|
||||
# Test-Script ausfuhren
|
||||
python backend/scripts/test_compliance_ai_endpoints.py
|
||||
```
|
||||
|
||||
## API Endpoints
|
||||
|
||||
Alle Endpoints unter: `http://localhost:8000/api/v1/compliance/ai/`
|
||||
|
||||
### 1. Status prufen
|
||||
|
||||
```bash
|
||||
curl http://localhost:8000/api/v1/compliance/ai/status
|
||||
```
|
||||
|
||||
### 2. Requirement interpretieren
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:8000/api/v1/compliance/ai/interpret \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"requirement_id": "YOUR_REQUIREMENT_ID"
|
||||
}'
|
||||
```
|
||||
|
||||
### 3. Controls vorschlagen
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:8000/api/v1/compliance/ai/suggest-controls \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"requirement_id": "YOUR_REQUIREMENT_ID"
|
||||
}'
|
||||
```
|
||||
|
||||
### 4. Modul-Risiko bewerten
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:8000/api/v1/compliance/ai/assess-risk \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"module_id": "consent-service"
|
||||
}'
|
||||
```
|
||||
|
||||
### 5. Gap-Analyse
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:8000/api/v1/compliance/ai/gap-analysis \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"requirement_id": "YOUR_REQUIREMENT_ID"
|
||||
}'
|
||||
```
|
||||
|
||||
### 6. Batch-Interpretation
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:8000/api/v1/compliance/ai/batch-interpret \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"requirement_ids": ["id1", "id2"],
|
||||
"rate_limit": 1.0
|
||||
}'
|
||||
```
|
||||
|
||||
## Provider-Konfiguration
|
||||
|
||||
### Option 1: Mock (Testing)
|
||||
|
||||
```bash
|
||||
export COMPLIANCE_LLM_PROVIDER=mock
|
||||
```
|
||||
|
||||
Vorteile:
|
||||
- Keine API-Keys erforderlich
|
||||
- Schnell
|
||||
- Deterministisch
|
||||
|
||||
Nachteile:
|
||||
- Keine echten AI-Antworten
|
||||
|
||||
### Option 2: Anthropic Claude (Empfohlen)
|
||||
|
||||
```bash
|
||||
export COMPLIANCE_LLM_PROVIDER=anthropic
|
||||
export ANTHROPIC_API_KEY=sk-ant-...
|
||||
export ANTHROPIC_MODEL=claude-sonnet-4-20250514
|
||||
```
|
||||
|
||||
Vorteile:
|
||||
- Beste Qualitat
|
||||
- Zuverlassig
|
||||
- Breakpilot-optimiert
|
||||
|
||||
Nachteile:
|
||||
- API-Kosten (~$3 per 1M input tokens)
|
||||
|
||||
### Option 3: Self-Hosted (Ollama/vLLM)
|
||||
|
||||
```bash
|
||||
export COMPLIANCE_LLM_PROVIDER=self_hosted
|
||||
export SELF_HOSTED_LLM_URL=http://localhost:11434
|
||||
export SELF_HOSTED_LLM_MODEL=llama3.1:8b
|
||||
```
|
||||
|
||||
Vorteile:
|
||||
- Kostenlos
|
||||
- Privacy (on-premise)
|
||||
- Keine Rate-Limits
|
||||
|
||||
Nachteile:
|
||||
- Geringere Qualitat als Claude
|
||||
- Benotigt GPU/CPU-Ressourcen
|
||||
|
||||
## Beispiel-Response
|
||||
|
||||
### Interpretation
|
||||
|
||||
```json
|
||||
{
|
||||
"requirement_id": "req-123",
|
||||
"summary": "Art. 32 DSGVO verlangt angemessene technische Maßnahmen zur Datensicherheit.",
|
||||
"applicability": "Gilt für alle Breakpilot-Module die personenbezogene Daten verarbeiten.",
|
||||
"technical_measures": [
|
||||
"Verschlüsselung personenbezogener Daten (AES-256)",
|
||||
"TLS 1.3 für Datenübertragung",
|
||||
"Regelmäßige Sicherheitsaudits",
|
||||
"Zugriffskontrolle mit IAM"
|
||||
],
|
||||
"affected_modules": [
|
||||
"consent-service",
|
||||
"klausur-service",
|
||||
"backend"
|
||||
],
|
||||
"risk_level": "high",
|
||||
"implementation_hints": [
|
||||
"SOPS mit Age-Keys für Secret-Management",
|
||||
"PostgreSQL transparent encryption",
|
||||
"Nginx TLS-Konfiguration prüfen"
|
||||
],
|
||||
"confidence_score": 0.85,
|
||||
"error": null
|
||||
}
|
||||
```
|
||||
|
||||
### Control-Suggestion
|
||||
|
||||
```json
|
||||
{
|
||||
"requirement_id": "req-123",
|
||||
"suggestions": [
|
||||
{
|
||||
"control_id": "PRIV-042",
|
||||
"domain": "priv",
|
||||
"title": "Verschlüsselung personenbezogener Daten",
|
||||
"description": "Alle personenbezogenen Daten müssen verschlüsselt gespeichert werden",
|
||||
"pass_criteria": "100% der PII in PostgreSQL sind AES-256 verschlüsselt",
|
||||
"implementation_guidance": "Verwende SOPS mit Age-Keys für Secrets. Aktiviere PostgreSQL transparent data encryption.",
|
||||
"is_automated": true,
|
||||
"automation_tool": "SOPS",
|
||||
"priority": "high",
|
||||
"confidence_score": 0.9
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Problem: "AI provider is not available"
|
||||
|
||||
Lösung:
|
||||
```bash
|
||||
# Prüfe Status
|
||||
curl http://localhost:8000/api/v1/compliance/ai/status
|
||||
|
||||
# Prüfe Environment Variables
|
||||
echo $COMPLIANCE_LLM_PROVIDER
|
||||
echo $ANTHROPIC_API_KEY
|
||||
|
||||
# Fallback auf Mock
|
||||
export COMPLIANCE_LLM_PROVIDER=mock
|
||||
```
|
||||
|
||||
### Problem: "Requirement not found"
|
||||
|
||||
Lösung:
|
||||
```bash
|
||||
# Datenbank seeden
|
||||
curl -X POST http://localhost:8000/api/v1/compliance/seed \
|
||||
-d '{"force": false}'
|
||||
|
||||
# Requirements auflisten
|
||||
curl http://localhost:8000/api/v1/compliance/requirements
|
||||
```
|
||||
|
||||
### Problem: Timeout bei Anthropic
|
||||
|
||||
Lösung:
|
||||
```bash
|
||||
# Timeout erhöhen
|
||||
export COMPLIANCE_LLM_TIMEOUT=120.0
|
||||
|
||||
# Oder Mock-Provider verwenden
|
||||
export COMPLIANCE_LLM_PROVIDER=mock
|
||||
```
|
||||
|
||||
## Unit Tests
|
||||
|
||||
```bash
|
||||
cd backend
|
||||
|
||||
# Alle Tests
|
||||
pytest tests/test_compliance_ai.py -v
|
||||
|
||||
# Nur Mock-Tests
|
||||
pytest tests/test_compliance_ai.py::TestMockProvider -v
|
||||
|
||||
# Integration Tests (benötigt API-Key)
|
||||
pytest tests/test_compliance_ai.py -v --integration
|
||||
```
|
||||
|
||||
## Weitere Dokumentation
|
||||
|
||||
- **Vollständige Dokumentation**: `backend/docs/compliance_ai_integration.md`
|
||||
- **API Schemas**: `backend/compliance/api/schemas.py`
|
||||
- **LLM Provider**: `backend/compliance/services/llm_provider.py`
|
||||
- **AI Assistant**: `backend/compliance/services/ai_compliance_assistant.py`
|
||||
|
||||
## Support
|
||||
|
||||
Bei Problemen:
|
||||
1. Prüfe `/api/v1/compliance/ai/status`
|
||||
2. Prüfe Logs: `docker logs breakpilot-backend`
|
||||
3. Teste mit Mock: `COMPLIANCE_LLM_PROVIDER=mock`
|
||||
4. Siehe: `backend/docs/compliance_ai_integration.md`
|
||||
297
backend/compliance/SERVICE_COVERAGE.md
Normal file
297
backend/compliance/SERVICE_COVERAGE.md
Normal file
@@ -0,0 +1,297 @@
|
||||
# Breakpilot Service Coverage - Sprint 3
|
||||
|
||||
## Übersicht
|
||||
|
||||
Vollständige Dokumentation aller 36 Breakpilot Services in der Compliance-Registry.
|
||||
|
||||
## Service-Kategorien
|
||||
|
||||
### Backend Services (11)
|
||||
|
||||
| Service | Port | PII | AI | Criticality | GDPR | AI Act | BSI-TR |
|
||||
|---------|------|-----|----|----|------|--------|--------|
|
||||
| python-backend | 8000 | ✓ | - | critical | ✓✓✓ | ✓✓ | ✓✓ |
|
||||
| consent-service | 8081 | ✓ | - | critical | ✓✓✓ | - | ✓✓ |
|
||||
| billing-service | 8083 | ✓ | - | critical | ✓✓✓ | - | - |
|
||||
| school-service | 8084 | ✓ | - | high | ✓✓✓ | - | ✓✓ |
|
||||
| calendar-service | 8085 | ✓ | - | medium | ✓✓ | - | - |
|
||||
| h5p-service | 8082 | ✓ | - | medium | ✓✓ | - | - |
|
||||
| website | 3000 | ✓ | - | high | ✓✓ | - | ✓✓ |
|
||||
| dsms-gateway | 8082 | ✓ | - | medium | ✓✓ | - | - |
|
||||
| erpnext | 8080 | ✓ | - | high | ✓✓✓ | - | - |
|
||||
| camunda | 8089 | ✓ | - | medium | ✓✓ | - | - |
|
||||
| compliance-module | - | - | ✓ | high | ✓✓ | ✓ | - |
|
||||
|
||||
### AI Services (4)
|
||||
|
||||
| Service | Port | PII | AI | Criticality | GDPR | AI Act | Notes |
|
||||
|---------|------|-----|----|-------------|------|--------|-------|
|
||||
| klausur-service | 8086 | ✓ | ✓ | high | ✓✓✓ | ✓✓✓ | High-Risk KI (Bildung) |
|
||||
| embedding-service | 8087 | - | ✓ | medium | ✓ | ✓✓ | RAG/Embeddings |
|
||||
| transcription-worker | - | ✓ | ✓ | medium | ✓✓ | ✓✓ | Whisper ASR |
|
||||
| llm-gateway | 8088 | ✓ | ✓ | high | ✓✓ | ✓✓✓ | LLM Orchestration |
|
||||
| breakpilot-drive | 3001 | ✓ | ✓ | medium | ✓✓ | ✓✓ | Unity + LLM |
|
||||
|
||||
### Databases (5)
|
||||
|
||||
| Service | Port | Type | PII | Criticality | GDPR | BSI-TR |
|
||||
|---------|------|------|-----|-------------|------|--------|
|
||||
| postgresql | 5432 | Relational | ✓ | critical | ✓✓✓ | ✓✓✓ |
|
||||
| qdrant | 6333 | Vector | - | medium | ✓ | ✓✓ |
|
||||
| valkey | 6379 | Cache | ✓ | high | ✓✓ | ✓✓ |
|
||||
| content-db | 5433 | Relational | - | medium | - | ✓✓ |
|
||||
| erpnext-db | 3306 | MariaDB | ✓ | high | ✓✓ | ✓✓ |
|
||||
|
||||
### Communication Services (6)
|
||||
|
||||
| Service | Port | PII | Criticality | GDPR | DSA | Notes |
|
||||
|---------|------|-----|-------------|------|-----|-------|
|
||||
| matrix-synapse | 8008 | ✓ | high | ✓✓✓ | ✓✓ | E2EE Chat |
|
||||
| synapse-db | 5432 | ✓ | high | ✓✓✓ | - | Matrix DB |
|
||||
| jitsi-meet | 8443 | ✓ | high | ✓✓✓ | - | Video Frontend |
|
||||
| jitsi-prosody | 5222 | ✓ | high | ✓✓ | - | XMPP Server |
|
||||
| jitsi-jicofo | - | - | medium | ✓ | - | Conference Focus |
|
||||
| jitsi-jvb | 10000 | ✓ | high | ✓✓ | - | Video Bridge |
|
||||
| jibri | - | ✓ | high | ✓✓✓ | - | Recording |
|
||||
|
||||
### Storage Services (2)
|
||||
|
||||
| Service | Port | Type | PII | Criticality | GDPR | BSI-TR |
|
||||
|---------|------|------|-----|-------------|------|--------|
|
||||
| minio | 9000 | S3 | ✓ | critical | ✓✓✓ | ✓✓ |
|
||||
| dsms-node | 5001 | IPFS | ✓ | medium | ✓✓ | ✓✓ |
|
||||
|
||||
### Infrastructure Services (5)
|
||||
|
||||
| Service | Port | PII | Criticality | GDPR | NIS2 | Notes |
|
||||
|---------|------|-----|-------------|------|------|-------|
|
||||
| vault | 8200 | - | critical | ✓✓ | - | Secrets Management |
|
||||
| traefik | 443 | ✓ | critical | - | ✓✓ | Reverse Proxy |
|
||||
| mailpit | 8025 | ✓ | low | ✓ | - | Dev Mail Server |
|
||||
| backup | - | ✓ | critical | ✓✓✓ | - | DB Backups |
|
||||
|
||||
### Monitoring Services (3)
|
||||
|
||||
| Service | Port | PII | Criticality | GDPR | BSI-TR | Notes |
|
||||
|---------|------|-----|-------------|------|--------|-------|
|
||||
| loki | 3100 | ✓ | high | ✓✓ | ✓✓ | Log Aggregation |
|
||||
| grafana | 3000 | - | medium | - | ✓✓ | Dashboards |
|
||||
| prometheus | 9090 | - | medium | - | ✓✓ | Metrics |
|
||||
|
||||
### Security Services (1)
|
||||
|
||||
| Service | Port | PII | Criticality | GDPR | BSI-TR | Notes |
|
||||
|---------|------|-----|-------------|------|--------|-------|
|
||||
| vault | 8200 | - | critical | ✓✓ | ✓✓✓ | Encryption as a Service |
|
||||
|
||||
## Statistiken
|
||||
|
||||
### Gesamt
|
||||
- **36 Services** dokumentiert
|
||||
- **26 Services** (72%) verarbeiten PII
|
||||
- **5 Services** (14%) enthalten KI-Komponenten
|
||||
- **9 Services** (25%) sind als "critical" eingestuft
|
||||
|
||||
### Nach Service-Typ
|
||||
```
|
||||
Backend: 11 (31%)
|
||||
Communication: 6 (17%)
|
||||
Database: 5 (14%)
|
||||
AI: 5 (14%)
|
||||
Infrastructure: 5 (14%)
|
||||
Monitoring: 3 (8%)
|
||||
Storage: 2 (6%)
|
||||
Security: 1 (3%)
|
||||
```
|
||||
|
||||
### Technologie-Stack (Top 10)
|
||||
```
|
||||
Python: 15 Services
|
||||
PostgreSQL: 8 Services
|
||||
FastAPI: 7 Services
|
||||
Go: 4 Services
|
||||
Java: 3 Services
|
||||
JavaScript: 2 Services
|
||||
WebRTC: 2 Services
|
||||
Redis/Valkey: 2 Services
|
||||
Nginx: 2 Services
|
||||
Docker: 36 Services (alle)
|
||||
```
|
||||
|
||||
### Compliance-Abdeckung
|
||||
|
||||
#### GDPR
|
||||
- **Critical**: 15 Services (consent, billing, school, postgresql, minio, backup, etc.)
|
||||
- **High**: 10 Services (python-backend, klausur-service, matrix-synapse, etc.)
|
||||
- **Medium**: 8 Services (calendar, embedding, dsms, etc.)
|
||||
- **Low**: 3 Services (mailpit, etc.)
|
||||
|
||||
#### AI Act
|
||||
- **Critical**: 3 Services (klausur-service, llm-gateway)
|
||||
- **High**: 2 Services (python-backend)
|
||||
- **Medium**: 5 Services (embedding-service, transcription-worker, compliance-module, etc.)
|
||||
|
||||
#### BSI-TR-03161
|
||||
- **Critical**: 4 Services (postgresql, vault, backup)
|
||||
- **High**: 8 Services (consent-service, school-service, matrix-synapse, etc.)
|
||||
- **Medium**: 12 Services (qdrant, valkey, minio, etc.)
|
||||
|
||||
## Port-Übersicht
|
||||
|
||||
### Häufig genutzte Ports
|
||||
```
|
||||
8000 - python-backend
|
||||
8008 - matrix-synapse
|
||||
8025 - mailpit (Web UI)
|
||||
8081 - consent-service
|
||||
8082 - h5p-service / dsms-gateway (Konflikt möglich)
|
||||
8083 - billing-service
|
||||
8084 - school-service
|
||||
8085 - calendar-service
|
||||
8086 - klausur-service
|
||||
8087 - embedding-service
|
||||
8088 - llm-gateway
|
||||
8089 - camunda
|
||||
8090 - erpnext-frontend
|
||||
8200 - vault
|
||||
8443 - jitsi-meet
|
||||
|
||||
3000 - website / grafana (Konflikt möglich)
|
||||
3001 - breakpilot-drive
|
||||
3100 - loki
|
||||
3306 - erpnext-db (MariaDB)
|
||||
|
||||
5001 - dsms-node (IPFS API)
|
||||
5222 - jitsi-prosody (XMPP)
|
||||
5432 - postgresql / synapse-db
|
||||
5433 - content-db
|
||||
|
||||
6333 - qdrant
|
||||
6379 - valkey (Redis)
|
||||
|
||||
9000 - minio (S3 API)
|
||||
9001 - minio (Console)
|
||||
9090 - prometheus
|
||||
|
||||
10000 - jitsi-jvb (UDP)
|
||||
```
|
||||
|
||||
### Erkannte Port-Konflikte
|
||||
- **Port 8082**: h5p-service, dsms-gateway (beide in service_modules.py)
|
||||
- **Port 3000**: website, grafana (beide in service_modules.py)
|
||||
- **Port 5432**: postgresql, synapse-db (separater Service)
|
||||
|
||||
**Hinweis**: Konflikte in docker-compose.yml durch unterschiedliche Profile oder Host-Ports gelöst.
|
||||
|
||||
## PII-Verarbeitung
|
||||
|
||||
### Services die PII verarbeiten (26)
|
||||
|
||||
**Critical PII Processing:**
|
||||
- consent-service (Einwilligungen)
|
||||
- billing-service (Zahlungsdaten)
|
||||
- school-service (Schülerdaten)
|
||||
- postgresql (alle persistenten Daten)
|
||||
- minio (Dateispeicher)
|
||||
- backup (Datensicherung)
|
||||
|
||||
**High PII Processing:**
|
||||
- python-backend (User-Daten, Dokumente)
|
||||
- klausur-service (Klausuren, Korrekturen)
|
||||
- matrix-synapse (Chat-Inhalte)
|
||||
- jitsi-meet/jvb (Video/Audio)
|
||||
- jibri (Aufzeichnungen)
|
||||
- transcription-worker (Sprachaufnahmen)
|
||||
|
||||
## KI-Komponenten
|
||||
|
||||
### Services mit KI (5)
|
||||
|
||||
1. **klausur-service** (High-Risk AI)
|
||||
- Claude API für Klausurkorrektur
|
||||
- AI Act Art. 6 (Bildungsbereich)
|
||||
- GDPR Art. 22 (automatisierte Entscheidungen)
|
||||
|
||||
2. **embedding-service**
|
||||
- SentenceTransformers (lokal)
|
||||
- General-Purpose AI System
|
||||
|
||||
3. **transcription-worker**
|
||||
- Whisper ASR (OpenAI)
|
||||
- Biometrische Daten (GDPR)
|
||||
|
||||
4. **llm-gateway**
|
||||
- LLM Orchestrierung
|
||||
- Externe API-Calls
|
||||
|
||||
5. **breakpilot-drive**
|
||||
- Unity + LLM Integration
|
||||
- Lernspiel mit KI
|
||||
|
||||
## Kritikalität
|
||||
|
||||
### Critical Services (9)
|
||||
Ausfall führt zu System-Shutdown oder schwerwiegendem Datenverlust:
|
||||
- python-backend
|
||||
- consent-service
|
||||
- billing-service
|
||||
- postgresql
|
||||
- minio
|
||||
- vault
|
||||
- traefik
|
||||
- backup
|
||||
|
||||
### High Services (10)
|
||||
Wichtige Funktionalität, aber System kann degradiert weiterlaufen:
|
||||
- klausur-service
|
||||
- school-service
|
||||
- website
|
||||
- matrix-synapse
|
||||
- jitsi-meet/jvb
|
||||
- valkey
|
||||
- loki
|
||||
- erpnext
|
||||
- erpnext-db
|
||||
|
||||
### Medium Services (14)
|
||||
Standard-Funktionalität:
|
||||
- calendar-service
|
||||
- embedding-service
|
||||
- transcription-worker
|
||||
- h5p-service
|
||||
- qdrant
|
||||
- dsms-node/gateway
|
||||
- jitsi-jicofo
|
||||
- grafana
|
||||
- prometheus
|
||||
- compliance-module
|
||||
- camunda
|
||||
- breakpilot-drive
|
||||
|
||||
### Low Services (3)
|
||||
Nur für Entwicklung/Testing:
|
||||
- mailpit
|
||||
- content-db
|
||||
|
||||
## Nächste Schritte
|
||||
|
||||
### Sprint 4 Planung
|
||||
- [ ] Port-Konflikte auflösen (8082, 3000)
|
||||
- [ ] Compliance-Score Berechnung
|
||||
- [ ] Automatische Dependency-Graph-Erstellung
|
||||
- [ ] Service-Health-Checks integrieren
|
||||
- [ ] Gap-Analyse pro Service
|
||||
- [ ] Dashboard für Service-Overview
|
||||
|
||||
### Fehlende Services
|
||||
Services in docker-compose.yml aber nicht kritisch für Compliance:
|
||||
- erpnext-redis-queue
|
||||
- erpnext-redis-cache
|
||||
- erpnext-create-site (Init-Service)
|
||||
- erpnext-backend
|
||||
- erpnext-websocket
|
||||
- erpnext-scheduler
|
||||
- erpnext-worker-long
|
||||
- erpnext-worker-short
|
||||
|
||||
**Grund**: Interne ERPNext Worker, keine separate Compliance-Relevanz.
|
||||
393
backend/compliance/SPRINT3_INTEGRATION.md
Normal file
393
backend/compliance/SPRINT3_INTEGRATION.md
Normal file
@@ -0,0 +1,393 @@
|
||||
# Sprint 3 Integration Guide - Service Module Registry
|
||||
|
||||
## Übersicht
|
||||
|
||||
Sprint 3 erweitert das Compliance-Modul um eine vollständige Service-Registry mit:
|
||||
- 30+ dokumentierte Breakpilot Services
|
||||
- Service ↔ Regulation Mappings
|
||||
- Automatisches Seeding
|
||||
- Validierung und Statistiken
|
||||
|
||||
## Dateien
|
||||
|
||||
### Neue/Aktualisierte Dateien
|
||||
|
||||
```
|
||||
backend/compliance/
|
||||
├── data/
|
||||
│ ├── service_modules.py (AKTUALISIERT - +4 neue Services)
|
||||
│ └── README.md (NEU - Dokumentation)
|
||||
├── services/
|
||||
│ └── seeder.py (AKTUALISIERT - Service-Seeding)
|
||||
└── scripts/
|
||||
├── __init__.py (NEU)
|
||||
├── seed_service_modules.py (NEU - Seeding-Script)
|
||||
└── validate_service_modules.py (NEU - Validierung)
|
||||
```
|
||||
|
||||
## Neue Services
|
||||
|
||||
Folgende Services wurden zu `service_modules.py` hinzugefügt:
|
||||
|
||||
1. **dsms-node** (Port 5001) - IPFS Dezentralspeicher
|
||||
2. **dsms-gateway** (Port 8082) - DSMS REST API
|
||||
3. **mailpit** (Port 8025) - Dev Mail Server
|
||||
4. **backup** - PostgreSQL Backup Service
|
||||
5. **breakpilot-drive** (Port 3001) - Unity WebGL Game
|
||||
6. **camunda** (Port 8089) - BPMN Workflow Engine
|
||||
|
||||
## Installation & Setup
|
||||
|
||||
### 1. Datenbank-Migration
|
||||
|
||||
```bash
|
||||
cd backend
|
||||
alembic revision --autogenerate -m "Add service module registry"
|
||||
alembic upgrade head
|
||||
```
|
||||
|
||||
### 2. Validierung der Service-Daten
|
||||
|
||||
```bash
|
||||
python -m compliance.scripts.validate_service_modules
|
||||
```
|
||||
|
||||
Erwartete Ausgabe:
|
||||
```
|
||||
=== Validating Required Fields ===
|
||||
✓ All services have required fields
|
||||
|
||||
=== Checking Port Conflicts ===
|
||||
✓ No port conflicts detected
|
||||
|
||||
=== Validating Regulation Mappings ===
|
||||
✓ All regulation mappings are valid
|
||||
|
||||
=== Service Module Statistics ===
|
||||
Total Services: 30+
|
||||
...
|
||||
```
|
||||
|
||||
### 3. Seeding
|
||||
|
||||
Nur Service-Module:
|
||||
```bash
|
||||
python -m compliance.scripts.seed_service_modules --mode modules
|
||||
```
|
||||
|
||||
Vollständige Compliance-DB:
|
||||
```bash
|
||||
python -m compliance.scripts.seed_service_modules --mode all
|
||||
```
|
||||
|
||||
## API-Integration
|
||||
|
||||
### Bestehende Endpoints erweitern
|
||||
|
||||
Die Service-Module sind bereits im Datenmodell integriert:
|
||||
|
||||
```python
|
||||
# compliance/db/models.py
|
||||
class ServiceModuleDB(Base):
|
||||
__tablename__ = 'compliance_service_modules'
|
||||
# ... bereits definiert
|
||||
|
||||
class ModuleRegulationMappingDB(Base):
|
||||
__tablename__ = 'compliance_module_regulations'
|
||||
# ... bereits definiert
|
||||
```
|
||||
|
||||
### Neue API-Endpoints (Optional)
|
||||
|
||||
Füge zu `compliance/api/routes.py` hinzu:
|
||||
|
||||
```python
|
||||
@router.get("/modules", response_model=List[ServiceModuleSchema])
|
||||
async def list_service_modules(
|
||||
db: Session = Depends(get_db),
|
||||
service_type: Optional[str] = None,
|
||||
processes_pii: Optional[bool] = None,
|
||||
):
|
||||
"""List all service modules with optional filtering."""
|
||||
query = db.query(ServiceModuleDB)
|
||||
|
||||
if service_type:
|
||||
query = query.filter(ServiceModuleDB.service_type == service_type)
|
||||
if processes_pii is not None:
|
||||
query = query.filter(ServiceModuleDB.processes_pii == processes_pii)
|
||||
|
||||
return query.all()
|
||||
|
||||
|
||||
@router.get("/modules/{module_id}", response_model=ServiceModuleDetailSchema)
|
||||
async def get_service_module(
|
||||
module_id: str,
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""Get detailed information about a service module."""
|
||||
module = db.query(ServiceModuleDB).filter(
|
||||
ServiceModuleDB.id == module_id
|
||||
).first()
|
||||
|
||||
if not module:
|
||||
raise HTTPException(status_code=404, detail="Service module not found")
|
||||
|
||||
return module
|
||||
|
||||
|
||||
@router.get("/modules/{module_id}/regulations")
|
||||
async def get_module_regulations(
|
||||
module_id: str,
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""Get all applicable regulations for a service."""
|
||||
mappings = db.query(ModuleRegulationMappingDB).filter(
|
||||
ModuleRegulationMappingDB.module_id == module_id
|
||||
).all()
|
||||
|
||||
return mappings
|
||||
```
|
||||
|
||||
### Schemas hinzufügen
|
||||
|
||||
Füge zu `compliance/api/schemas.py` hinzu:
|
||||
|
||||
```python
|
||||
from pydantic import BaseModel
|
||||
from typing import List, Optional
|
||||
|
||||
class ServiceModuleSchema(BaseModel):
|
||||
id: str
|
||||
name: str
|
||||
display_name: str
|
||||
service_type: str
|
||||
port: Optional[int]
|
||||
processes_pii: bool
|
||||
ai_components: bool
|
||||
criticality: str
|
||||
|
||||
class Config:
|
||||
from_attributes = True
|
||||
|
||||
|
||||
class ServiceModuleDetailSchema(ServiceModuleSchema):
|
||||
description: Optional[str]
|
||||
technology_stack: List[str]
|
||||
data_categories: List[str]
|
||||
owner_team: Optional[str]
|
||||
compliance_score: Optional[float]
|
||||
|
||||
class Config:
|
||||
from_attributes = True
|
||||
```
|
||||
|
||||
## Verwendung im Code
|
||||
|
||||
### Python
|
||||
|
||||
```python
|
||||
from compliance.data.service_modules import (
|
||||
BREAKPILOT_SERVICES,
|
||||
get_service_count,
|
||||
get_services_by_type,
|
||||
get_services_processing_pii,
|
||||
)
|
||||
|
||||
# Alle KI-Services finden
|
||||
ai_services = get_services_with_ai()
|
||||
for service in ai_services:
|
||||
print(f"{service['name']}: {service['regulations']}")
|
||||
|
||||
# Services mit kritischer GDPR-Relevanz
|
||||
from compliance.db.models import ServiceModuleDB, ModuleRegulationMappingDB
|
||||
from classroom_engine.database import SessionLocal
|
||||
|
||||
db = SessionLocal()
|
||||
critical_gdpr = db.query(ServiceModuleDB).join(
|
||||
ModuleRegulationMappingDB
|
||||
).filter(
|
||||
ModuleRegulationMappingDB.relevance_level == "critical"
|
||||
).all()
|
||||
```
|
||||
|
||||
### Frontend (React/Next.js)
|
||||
|
||||
```typescript
|
||||
// Fetch all service modules
|
||||
const modules = await fetch('/api/compliance/modules').then(r => r.json());
|
||||
|
||||
// Filter AI services
|
||||
const aiServices = modules.filter(m => m.ai_components);
|
||||
|
||||
// Get regulations for a service
|
||||
const regulations = await fetch(
|
||||
`/api/compliance/modules/${moduleId}/regulations`
|
||||
).then(r => r.json());
|
||||
```
|
||||
|
||||
## Testing
|
||||
|
||||
### Unit Tests
|
||||
|
||||
Beispiel für `backend/tests/test_service_modules.py`:
|
||||
|
||||
```python
|
||||
import pytest
|
||||
from compliance.data.service_modules import (
|
||||
get_service_count,
|
||||
get_services_by_type,
|
||||
get_critical_services,
|
||||
)
|
||||
|
||||
def test_service_count():
|
||||
"""Test that we have all expected services."""
|
||||
count = get_service_count()
|
||||
assert count >= 30, "Should have at least 30 services"
|
||||
|
||||
|
||||
def test_critical_services():
|
||||
"""Test that critical services are properly marked."""
|
||||
critical = get_critical_services()
|
||||
critical_names = [s['name'] for s in critical]
|
||||
|
||||
assert 'consent-service' in critical_names
|
||||
assert 'postgresql' in critical_names
|
||||
assert 'vault' in critical_names
|
||||
|
||||
|
||||
def test_ai_services_have_regulations():
|
||||
"""Test that all AI services have AI Act regulations."""
|
||||
from compliance.data.service_modules import get_services_with_ai
|
||||
|
||||
ai_services = get_services_with_ai()
|
||||
for service in ai_services:
|
||||
regulation_codes = [r['code'] for r in service['regulations']]
|
||||
assert 'AIACT' in regulation_codes or 'GDPR' in regulation_codes
|
||||
```
|
||||
|
||||
### Integration Test
|
||||
|
||||
```python
|
||||
def test_seeder_creates_modules(db_session):
|
||||
"""Test that seeder creates service modules."""
|
||||
from compliance.services.seeder import ComplianceSeeder
|
||||
|
||||
seeder = ComplianceSeeder(db_session)
|
||||
result = seeder.seed_service_modules_only()
|
||||
|
||||
assert result > 0
|
||||
|
||||
# Verify consent-service was created
|
||||
from compliance.db.models import ServiceModuleDB
|
||||
module = db_session.query(ServiceModuleDB).filter(
|
||||
ServiceModuleDB.name == "consent-service"
|
||||
).first()
|
||||
|
||||
assert module is not None
|
||||
assert module.port == 8081
|
||||
assert module.processes_pii == True
|
||||
```
|
||||
|
||||
## Maintenance
|
||||
|
||||
### Neue Services hinzufügen
|
||||
|
||||
1. Service zu `BREAKPILOT_SERVICES` in `service_modules.py` hinzufügen
|
||||
2. Validierung ausführen: `python -m compliance.scripts.validate_service_modules`
|
||||
3. Re-seeding: `python -m compliance.scripts.seed_service_modules`
|
||||
|
||||
### Service aktualisieren
|
||||
|
||||
```python
|
||||
# Via API (wenn implementiert)
|
||||
PATCH /api/compliance/modules/{module_id}
|
||||
{
|
||||
"port": 8082,
|
||||
"technology_stack": ["Python", "FastAPI", "IPFS"]
|
||||
}
|
||||
|
||||
# Oder via Code
|
||||
from compliance.db.models import ServiceModuleDB
|
||||
from classroom_engine.database import SessionLocal
|
||||
|
||||
db = SessionLocal()
|
||||
module = db.query(ServiceModuleDB).filter(
|
||||
ServiceModuleDB.name == "dsms-gateway"
|
||||
).first()
|
||||
|
||||
module.port = 8082
|
||||
db.commit()
|
||||
```
|
||||
|
||||
## Monitoring & Reporting
|
||||
|
||||
### Compliance-Score pro Service
|
||||
|
||||
```python
|
||||
# Zukünftige Implementierung
|
||||
def calculate_service_compliance_score(module_id: str) -> float:
|
||||
"""
|
||||
Berechnet Compliance-Score basierend auf:
|
||||
- Anzahl erfüllter Requirements
|
||||
- Implementierungsstatus der Controls
|
||||
- Kritikalität der offenen Gaps
|
||||
"""
|
||||
# TODO: Implementierung
|
||||
pass
|
||||
```
|
||||
|
||||
### Gap-Analyse
|
||||
|
||||
```sql
|
||||
-- Services mit offenen kritischen GDPR-Requirements
|
||||
SELECT
|
||||
sm.name,
|
||||
sm.display_name,
|
||||
COUNT(r.id) as open_requirements
|
||||
FROM compliance_service_modules sm
|
||||
JOIN compliance_module_regulations mr ON sm.id = mr.module_id
|
||||
JOIN compliance_regulations reg ON mr.regulation_id = reg.id
|
||||
JOIN compliance_requirements r ON r.regulation_id = reg.id
|
||||
WHERE reg.code = 'GDPR'
|
||||
AND r.implementation_status != 'implemented'
|
||||
AND r.priority = 1
|
||||
GROUP BY sm.id, sm.name, sm.display_name
|
||||
ORDER BY open_requirements DESC;
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Import-Fehler
|
||||
|
||||
```bash
|
||||
# Stelle sicher, dass der Python-Path korrekt ist
|
||||
export PYTHONPATH="${PYTHONPATH}:/Users/benjaminadmin/Projekte/breakpilot-pwa/backend"
|
||||
```
|
||||
|
||||
### Seeding-Fehler
|
||||
|
||||
```bash
|
||||
# Prüfe Datenbankverbindung
|
||||
python -c "from classroom_engine.database import SessionLocal; db = SessionLocal(); print('DB OK')"
|
||||
|
||||
# Prüfe, ob Regulations bereits geseedet sind
|
||||
python -m compliance.scripts.seed_service_modules --mode all
|
||||
```
|
||||
|
||||
### Port-Konflikte
|
||||
|
||||
```bash
|
||||
# Validierung zeigt Port-Konflikte
|
||||
python -m compliance.scripts.validate_service_modules
|
||||
|
||||
# Konflikte manuell in service_modules.py beheben
|
||||
```
|
||||
|
||||
## Nächste Schritte (Sprint 4+)
|
||||
|
||||
- [ ] Compliance-Score Berechnung implementieren
|
||||
- [ ] Automatische Gap-Analyse
|
||||
- [ ] Dashboard für Service-Overview
|
||||
- [ ] CI/CD Integration (Auto-Validierung)
|
||||
- [ ] Service-Health-Checks integrieren
|
||||
- [ ] Abhängigkeits-Graph visualisieren
|
||||
285
backend/compliance/SPRINT_4_SUMMARY.md
Normal file
285
backend/compliance/SPRINT_4_SUMMARY.md
Normal file
@@ -0,0 +1,285 @@
|
||||
# Sprint 4: KI-Integration - Implementierungs-Zusammenfassung
|
||||
|
||||
## Status: ✅ VOLLSTÄNDIG IMPLEMENTIERT
|
||||
|
||||
Alle geforderten Komponenten für Sprint 4 (KI-Integration für automatische Requirement-Interpretation) sind bereits vollständig implementiert und einsatzbereit.
|
||||
|
||||
## Implementierte Komponenten
|
||||
|
||||
### 1. LLM Provider Abstraction ✅
|
||||
**Datei**: `/backend/compliance/services/llm_provider.py` (453 Zeilen)
|
||||
|
||||
**Implementiert**:
|
||||
- ✅ Abstrakte `LLMProvider` Basisklasse
|
||||
- ✅ `AnthropicProvider` für Claude API
|
||||
- ✅ `SelfHostedProvider` für Ollama/vLLM (mit Auto-Detection)
|
||||
- ✅ `MockProvider` für Testing
|
||||
- ✅ `get_llm_provider()` Factory Funktion
|
||||
- ✅ `LLMConfig` Dataclass für Konfiguration
|
||||
- ✅ `LLMResponse` Dataclass für Responses
|
||||
- ✅ Batch-Processing mit Rate-Limiting
|
||||
|
||||
**Features**:
|
||||
- Unterstützt Anthropic Claude API (https://api.anthropic.com)
|
||||
- Unterstützt Self-Hosted LLMs (Ollama, vLLM, LocalAI)
|
||||
- Auto-Detection von API-Formaten (Ollama vs OpenAI-kompatibel)
|
||||
- Konfigurierbare Timeouts, Temperatures, Max-Tokens
|
||||
- Error Handling und Fallback auf Mock bei fehlenden Credentials
|
||||
- Singleton Pattern für Provider-Reuse
|
||||
|
||||
### 2. AI Compliance Assistant ✅
|
||||
**Datei**: `/backend/compliance/services/ai_compliance_assistant.py` (500 Zeilen)
|
||||
|
||||
**Implementierte Methoden**:
|
||||
- ✅ `interpret_requirement()` - Interpretiert regulatorische Anforderungen
|
||||
- ✅ `suggest_controls()` - Schlägt passende Controls vor
|
||||
- ✅ `assess_module_risk()` - Bewertet Modul-Risiken
|
||||
- ✅ `analyze_gap()` - Gap-Analyse zwischen Requirements und Controls
|
||||
- ✅ `batch_interpret_requirements()` - Batch-Verarbeitung mit Rate-Limiting
|
||||
|
||||
**Features**:
|
||||
- Deutsche Prompts, Breakpilot-spezifisch (EdTech SaaS mit KI)
|
||||
- Strukturierte JSON-Responses
|
||||
- Robustes JSON-Parsing (Markdown-safe)
|
||||
- Confidence Scores
|
||||
- Error Handling mit Fallback-Responses
|
||||
- Singleton Pattern für Assistant-Reuse
|
||||
|
||||
**Dataclasses**:
|
||||
- `RequirementInterpretation`
|
||||
- `ControlSuggestion`
|
||||
- `RiskAssessment`
|
||||
- `GapAnalysis`
|
||||
|
||||
### 3. API Endpoints ✅
|
||||
**Datei**: `/backend/compliance/api/routes.py` (2683 Zeilen)
|
||||
|
||||
**Implementierte Endpoints** (alle unter `/api/v1/compliance/ai/`):
|
||||
|
||||
| Endpoint | Method | Status | Beschreibung |
|
||||
|----------|--------|--------|--------------|
|
||||
| `/ai/status` | GET | ✅ | Prüft Status des AI Providers |
|
||||
| `/ai/interpret` | POST | ✅ | Interpretiert eine Anforderung |
|
||||
| `/ai/suggest-controls` | POST | ✅ | Schlägt Controls vor |
|
||||
| `/ai/assess-risk` | POST | ✅ | Bewertet Modul-Risiko |
|
||||
| `/ai/gap-analysis` | POST | ✅ | Analysiert Coverage-Lücken |
|
||||
| `/ai/batch-interpret` | POST | ✅ | Batch-Interpretation mehrerer Requirements |
|
||||
|
||||
**Features**:
|
||||
- DB-Integration (lädt Requirements, Modules, Regulations)
|
||||
- Strukturierte Request/Response Schemas
|
||||
- Error Handling mit HTTP Status Codes
|
||||
- Background Task Support für große Batches
|
||||
- Rate-Limiting
|
||||
|
||||
### 4. Pydantic Schemas ✅
|
||||
**Datei**: `/backend/compliance/api/schemas.py` (766 Zeilen)
|
||||
|
||||
**Implementierte Schemas**:
|
||||
- ✅ `AIStatusResponse`
|
||||
- ✅ `AIInterpretationRequest` / `AIInterpretationResponse`
|
||||
- ✅ `AIBatchInterpretationRequest` / `AIBatchInterpretationResponse`
|
||||
- ✅ `AIControlSuggestionRequest` / `AIControlSuggestionResponse`
|
||||
- ✅ `AIControlSuggestionItem`
|
||||
- ✅ `AIRiskAssessmentRequest` / `AIRiskAssessmentResponse`
|
||||
- ✅ `AIRiskFactor`
|
||||
- ✅ `AIGapAnalysisRequest` / `AIGapAnalysisResponse`
|
||||
|
||||
### 5. Environment Variables ✅
|
||||
**Datei**: `/backend/.env.example` (erweitert)
|
||||
|
||||
**Hinzugefügte Variablen**:
|
||||
```bash
|
||||
# Provider Selection
|
||||
COMPLIANCE_LLM_PROVIDER=anthropic # oder: self_hosted, mock
|
||||
|
||||
# Anthropic Claude (empfohlen)
|
||||
ANTHROPIC_API_KEY=sk-ant-...
|
||||
ANTHROPIC_MODEL=claude-sonnet-4-20250514
|
||||
|
||||
# Self-Hosted Alternative
|
||||
SELF_HOSTED_LLM_URL=http://localhost:11434
|
||||
SELF_HOSTED_LLM_MODEL=llama3.1:8b
|
||||
SELF_HOSTED_LLM_KEY=optional-api-key
|
||||
|
||||
# Advanced Settings
|
||||
COMPLIANCE_LLM_MAX_TOKENS=4096
|
||||
COMPLIANCE_LLM_TEMPERATURE=0.3
|
||||
COMPLIANCE_LLM_TIMEOUT=60.0
|
||||
```
|
||||
|
||||
## Zusätzlich Erstellt
|
||||
|
||||
### 6. Dokumentation ✅
|
||||
- ✅ **Vollständige Dokumentation**: `/backend/docs/compliance_ai_integration.md`
|
||||
- Architektur-Übersicht
|
||||
- Komponenten-Beschreibung
|
||||
- API-Endpoint Dokumentation
|
||||
- Konfiguration (ENV Variables)
|
||||
- Verwendungs-Beispiele
|
||||
- Best Practices
|
||||
- Troubleshooting
|
||||
- Roadmap
|
||||
|
||||
- ✅ **Quick-Start Guide**: `/backend/compliance/README_AI.md`
|
||||
- 5-Minuten Setup
|
||||
- API-Beispiele (curl)
|
||||
- Provider-Konfiguration
|
||||
- Troubleshooting
|
||||
- Testing-Anleitung
|
||||
|
||||
### 7. Tests ✅
|
||||
- ✅ **Unit Tests**: `/backend/tests/test_compliance_ai.py` (380 Zeilen)
|
||||
- MockProvider Tests
|
||||
- Factory Tests
|
||||
- AIComplianceAssistant Tests
|
||||
- JSON-Parsing Tests
|
||||
- Integration Test Markers (für echte APIs)
|
||||
|
||||
- ✅ **Integration Test Script**: `/backend/scripts/test_compliance_ai_endpoints.py` (300 Zeilen)
|
||||
- Automatisiertes Testen aller Endpoints
|
||||
- Sample Data Fetching
|
||||
- Strukturierte Test-Reports
|
||||
- Error Handling
|
||||
|
||||
## Prompts
|
||||
|
||||
Alle Prompts sind:
|
||||
- ✅ Auf **Deutsch**
|
||||
- ✅ **Breakpilot-spezifisch**:
|
||||
- Erwähnt EdTech-Kontext (Schulverwaltung, Noten, Zeugnisse)
|
||||
- Kennt KI-Funktionen (Klausurkorrektur, Feedback)
|
||||
- Versteht Breakpilot-Module (consent-service, klausur-service, etc.)
|
||||
- Berücksichtigt DSGVO-Anforderungen
|
||||
- Self-Hosted in Deutschland
|
||||
|
||||
## Testing
|
||||
|
||||
### Bereits getestet:
|
||||
- ✅ Mock-Provider funktioniert
|
||||
- ✅ JSON-Parsing robust (Markdown-safe)
|
||||
- ✅ Error Handling korrekt
|
||||
- ✅ Batch-Processing mit Rate-Limiting
|
||||
|
||||
### Manuelles Testing:
|
||||
```bash
|
||||
# 1. Status prüfen
|
||||
curl http://localhost:8000/api/v1/compliance/ai/status
|
||||
|
||||
# 2. Test-Script ausführen
|
||||
export COMPLIANCE_LLM_PROVIDER=mock
|
||||
python backend/scripts/test_compliance_ai_endpoints.py
|
||||
|
||||
# 3. Unit Tests
|
||||
pytest backend/tests/test_compliance_ai.py -v
|
||||
```
|
||||
|
||||
## Code-Statistik
|
||||
|
||||
| Komponente | Dateien | Zeilen | Status |
|
||||
|------------|---------|--------|--------|
|
||||
| LLM Provider | 1 | 453 | ✅ Implementiert |
|
||||
| AI Assistant | 1 | 500 | ✅ Implementiert |
|
||||
| API Routes | 1 | 2683 | ✅ Implementiert |
|
||||
| Schemas | 1 | 766 | ✅ Implementiert |
|
||||
| Tests | 2 | 680 | ✅ Implementiert |
|
||||
| Dokumentation | 2 | - | ✅ Erstellt |
|
||||
| **Total** | **8** | **5082** | **✅ Komplett** |
|
||||
|
||||
## Verwendung
|
||||
|
||||
### Schnellstart (5 Minuten)
|
||||
|
||||
```bash
|
||||
# 1. Environment setzen
|
||||
export COMPLIANCE_LLM_PROVIDER=mock # Für Testing ohne API-Key
|
||||
|
||||
# 2. Backend starten
|
||||
cd backend && uvicorn main:app --reload
|
||||
|
||||
# 3. Datenbank seeden
|
||||
curl -X POST http://localhost:8000/api/v1/compliance/seed
|
||||
|
||||
# 4. AI testen
|
||||
python backend/scripts/test_compliance_ai_endpoints.py
|
||||
```
|
||||
|
||||
### Produktion (mit Claude API)
|
||||
|
||||
```bash
|
||||
# 1. API-Key setzen
|
||||
export COMPLIANCE_LLM_PROVIDER=anthropic
|
||||
export ANTHROPIC_API_KEY=sk-ant-...
|
||||
export ANTHROPIC_MODEL=claude-sonnet-4-20250514
|
||||
|
||||
# 2. Backend starten
|
||||
docker-compose up -d
|
||||
|
||||
# 3. Requirement interpretieren
|
||||
curl -X POST http://localhost:8000/api/v1/compliance/ai/interpret \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"requirement_id": "YOUR_ID"}'
|
||||
```
|
||||
|
||||
## Nächste Schritte (Optional)
|
||||
|
||||
Obwohl Sprint 4 vollständig ist, könnten folgende Erweiterungen sinnvoll sein:
|
||||
|
||||
### Sprint 5 (Caching & Optimization)
|
||||
- [ ] Response-Caching in DB (spart API-Kosten)
|
||||
- [ ] Background Job Queue für große Batches
|
||||
- [ ] Webhook-Support für Async Processing
|
||||
|
||||
### Sprint 6 (Fine-Tuning)
|
||||
- [ ] Fine-Tuning auf Breakpilot-Daten
|
||||
- [ ] Multi-Model Ensemble
|
||||
- [ ] Automatisches Re-Training basierend auf Auditor-Feedback
|
||||
|
||||
## Erfolgs-Kriterien ✅
|
||||
|
||||
Sprint 4 ist erfolgreich abgeschlossen, alle geforderten Komponenten sind implementiert:
|
||||
|
||||
1. ✅ `/backend/compliance/services/llm_provider.py` existiert
|
||||
- ✅ Abstrakte LLMProvider Basisklasse
|
||||
- ✅ AnthropicProvider für Claude API
|
||||
- ✅ SelfHostedProvider für Ollama/vLLM
|
||||
- ✅ get_llm_provider() Factory Funktion
|
||||
|
||||
2. ✅ `/backend/compliance/services/ai_compliance_assistant.py` existiert
|
||||
- ✅ AIComplianceAssistant Klasse
|
||||
- ✅ interpret_requirement()
|
||||
- ✅ suggest_controls()
|
||||
- ✅ assess_module_risk()
|
||||
- ✅ Batch-Verarbeitung mit Rate-Limiting
|
||||
|
||||
3. ✅ API-Endpoints in routes.py hinzugefügt
|
||||
- ✅ POST /api/v1/compliance/ai/interpret
|
||||
- ✅ POST /api/v1/compliance/ai/suggest-controls
|
||||
- ✅ POST /api/v1/compliance/ai/assess-risk
|
||||
- ✅ POST /api/v1/compliance/ai/batch-interpret
|
||||
|
||||
4. ✅ Environment Variables konfiguriert
|
||||
- ✅ COMPLIANCE_LLM_PROVIDER
|
||||
- ✅ ANTHROPIC_API_KEY
|
||||
- ✅ ANTHROPIC_MODEL
|
||||
- ✅ SELF_HOSTED_LLM_URL/MODEL
|
||||
|
||||
5. ✅ Pydantic Schemas hinzugefügt
|
||||
- ✅ AIInterpretationRequest/Response
|
||||
- ✅ AIControlSuggestionRequest/Response
|
||||
- ✅ AIRiskAssessmentRequest/Response
|
||||
- ✅ Und weitere...
|
||||
|
||||
6. ✅ Prompts sind auf Deutsch und Breakpilot-spezifisch
|
||||
|
||||
## Fazit
|
||||
|
||||
🎉 **Sprint 4 ist vollständig implementiert und produktionsbereit!**
|
||||
|
||||
Die KI-Integration für automatische Requirement-Interpretation ist:
|
||||
- ✅ Vollständig implementiert
|
||||
- ✅ Gut dokumentiert
|
||||
- ✅ Getestet
|
||||
- ✅ Produktionsbereit
|
||||
|
||||
Alle Komponenten sind bereits vorhanden und funktionieren. Die Integration kann sofort mit dem Mock-Provider getestet werden, oder mit echten APIs (Anthropic Claude oder Self-Hosted LLM) in Produktion eingesetzt werden.
|
||||
18
backend/compliance/__init__.py
Normal file
18
backend/compliance/__init__.py
Normal file
@@ -0,0 +1,18 @@
|
||||
"""
|
||||
Breakpilot Compliance & Audit Framework
|
||||
|
||||
Provides regulatory compliance management for:
|
||||
- 16 EU Regulations (GDPR, AI Act, CRA, NIS2, etc.)
|
||||
- BSI TR-03161 (Mobile Applications Security)
|
||||
- Control Catalogue with ~45 controls in 9 domains
|
||||
- Evidence Management with file upload
|
||||
- Risk Matrix (Likelihood x Impact)
|
||||
- Audit Export (ZIP packages for external auditors)
|
||||
|
||||
Integration points:
|
||||
- Uses RAG infrastructure for document parsing
|
||||
- Uses GPU infrastructure for LLM-based analysis
|
||||
"""
|
||||
|
||||
__version__ = "1.0.0"
|
||||
__author__ = "Breakpilot Team"
|
||||
33
backend/compliance/api/__init__.py
Normal file
33
backend/compliance/api/__init__.py
Normal file
@@ -0,0 +1,33 @@
|
||||
"""API routes for Compliance module."""
|
||||
|
||||
from .routes import router
|
||||
from .audit_routes import router as audit_router
|
||||
from .ai_routes import router as ai_router
|
||||
from .evidence_routes import router as evidence_router
|
||||
from .risk_routes import router as risk_router
|
||||
from .dashboard_routes import router as dashboard_router
|
||||
from .scraper_routes import router as scraper_router
|
||||
from .module_routes import router as module_router
|
||||
from .isms_routes import router as isms_router
|
||||
|
||||
# Include sub-routers
|
||||
router.include_router(audit_router)
|
||||
router.include_router(ai_router)
|
||||
router.include_router(evidence_router)
|
||||
router.include_router(risk_router)
|
||||
router.include_router(dashboard_router)
|
||||
router.include_router(scraper_router)
|
||||
router.include_router(module_router)
|
||||
router.include_router(isms_router)
|
||||
|
||||
__all__ = [
|
||||
"router",
|
||||
"audit_router",
|
||||
"ai_router",
|
||||
"evidence_router",
|
||||
"risk_router",
|
||||
"dashboard_router",
|
||||
"scraper_router",
|
||||
"module_router",
|
||||
"isms_router",
|
||||
]
|
||||
909
backend/compliance/api/ai_routes.py
Normal file
909
backend/compliance/api/ai_routes.py
Normal file
@@ -0,0 +1,909 @@
|
||||
"""
|
||||
FastAPI routes for AI Compliance Assistant.
|
||||
|
||||
Endpoints:
|
||||
- /ai/status: Get AI provider status
|
||||
- /ai/interpret: Interpret a requirement
|
||||
- /ai/suggest-controls: Get AI-suggested controls
|
||||
- /ai/assess-risk: Assess module risk
|
||||
- /ai/gap-analysis: Analyze coverage gaps
|
||||
- /ai/batch-interpret: Batch interpret requirements
|
||||
- /ai/auto-map-controls: Auto-map controls to requirements
|
||||
- /ai/batch-map-controls: Batch map controls
|
||||
- /ai/switch-provider: Switch LLM provider
|
||||
- /ai/providers: List available providers
|
||||
- /pdf/*: PDF extraction endpoints
|
||||
"""
|
||||
|
||||
import logging
|
||||
import os
|
||||
from typing import Optional, List
|
||||
|
||||
from pydantic import BaseModel
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query, BackgroundTasks
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from classroom_engine.database import get_db
|
||||
|
||||
from ..db import (
|
||||
RegulationRepository,
|
||||
RequirementRepository,
|
||||
ControlRepository,
|
||||
)
|
||||
from ..db.models import RegulationDB, RequirementDB
|
||||
from .schemas import (
|
||||
# AI Assistant schemas
|
||||
AIInterpretationRequest, AIInterpretationResponse,
|
||||
AIBatchInterpretationRequest, AIBatchInterpretationResponse,
|
||||
AIControlSuggestionRequest, AIControlSuggestionResponse, AIControlSuggestionItem,
|
||||
AIRiskAssessmentRequest, AIRiskAssessmentResponse, AIRiskFactor,
|
||||
AIGapAnalysisRequest, AIGapAnalysisResponse,
|
||||
AIStatusResponse,
|
||||
# PDF extraction schemas
|
||||
BSIAspectResponse, PDFExtractionResponse,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter(tags=["compliance-ai"])
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# AI Assistant Endpoints (Sprint 4)
|
||||
# ============================================================================
|
||||
|
||||
@router.get("/ai/status", response_model=AIStatusResponse)
|
||||
async def get_ai_status():
|
||||
"""Get the status of the AI provider."""
|
||||
from ..services.llm_provider import get_shared_provider, LLMProviderType
|
||||
|
||||
try:
|
||||
provider = get_shared_provider()
|
||||
return AIStatusResponse(
|
||||
provider=provider.provider_name,
|
||||
model=provider.config.model,
|
||||
is_available=True,
|
||||
is_mock=provider.provider_name == "mock",
|
||||
error=None,
|
||||
)
|
||||
except Exception as e:
|
||||
return AIStatusResponse(
|
||||
provider="unknown",
|
||||
model="unknown",
|
||||
is_available=False,
|
||||
is_mock=True,
|
||||
error=str(e),
|
||||
)
|
||||
|
||||
|
||||
@router.post("/ai/interpret", response_model=AIInterpretationResponse)
|
||||
async def interpret_requirement(
|
||||
request: AIInterpretationRequest,
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""Generate AI interpretation for a requirement."""
|
||||
from ..services.ai_compliance_assistant import get_ai_assistant
|
||||
|
||||
# Get requirement from DB
|
||||
req_repo = RequirementRepository(db)
|
||||
requirement = req_repo.get_by_id(request.requirement_id)
|
||||
|
||||
if not requirement:
|
||||
raise HTTPException(status_code=404, detail=f"Requirement {request.requirement_id} not found")
|
||||
|
||||
# Get regulation info
|
||||
reg_repo = RegulationRepository(db)
|
||||
regulation = reg_repo.get_by_id(requirement.regulation_id)
|
||||
|
||||
try:
|
||||
assistant = get_ai_assistant()
|
||||
result = await assistant.interpret_requirement(
|
||||
requirement_id=requirement.id,
|
||||
article=requirement.article,
|
||||
title=requirement.title,
|
||||
requirement_text=requirement.requirement_text or requirement.description or "",
|
||||
regulation_code=regulation.code if regulation else "UNKNOWN",
|
||||
regulation_name=regulation.name if regulation else "Unknown Regulation",
|
||||
)
|
||||
|
||||
return AIInterpretationResponse(
|
||||
requirement_id=result.requirement_id,
|
||||
summary=result.summary,
|
||||
applicability=result.applicability,
|
||||
technical_measures=result.technical_measures,
|
||||
affected_modules=result.affected_modules,
|
||||
risk_level=result.risk_level,
|
||||
implementation_hints=result.implementation_hints,
|
||||
confidence_score=result.confidence_score,
|
||||
error=result.error,
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error(f"AI interpretation failed: {e}")
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
|
||||
|
||||
@router.post("/ai/suggest-controls", response_model=AIControlSuggestionResponse)
|
||||
async def suggest_controls(
|
||||
request: AIControlSuggestionRequest,
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""Get AI-suggested controls for a requirement."""
|
||||
from ..services.ai_compliance_assistant import get_ai_assistant
|
||||
|
||||
# Get requirement from DB
|
||||
req_repo = RequirementRepository(db)
|
||||
requirement = req_repo.get_by_id(request.requirement_id)
|
||||
|
||||
if not requirement:
|
||||
raise HTTPException(status_code=404, detail=f"Requirement {request.requirement_id} not found")
|
||||
|
||||
# Get regulation info
|
||||
reg_repo = RegulationRepository(db)
|
||||
regulation = reg_repo.get_by_id(requirement.regulation_id)
|
||||
|
||||
try:
|
||||
assistant = get_ai_assistant()
|
||||
suggestions = await assistant.suggest_controls(
|
||||
requirement_title=requirement.title,
|
||||
requirement_text=requirement.requirement_text or requirement.description or "",
|
||||
regulation_name=regulation.name if regulation else "Unknown",
|
||||
affected_modules=[], # Could be populated from previous interpretation
|
||||
)
|
||||
|
||||
return AIControlSuggestionResponse(
|
||||
requirement_id=request.requirement_id,
|
||||
suggestions=[
|
||||
AIControlSuggestionItem(
|
||||
control_id=s.control_id,
|
||||
domain=s.domain,
|
||||
title=s.title,
|
||||
description=s.description,
|
||||
pass_criteria=s.pass_criteria,
|
||||
implementation_guidance=s.implementation_guidance,
|
||||
is_automated=s.is_automated,
|
||||
automation_tool=s.automation_tool,
|
||||
priority=s.priority,
|
||||
confidence_score=s.confidence_score,
|
||||
)
|
||||
for s in suggestions
|
||||
],
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error(f"AI control suggestion failed: {e}")
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
|
||||
|
||||
@router.post("/ai/assess-risk", response_model=AIRiskAssessmentResponse)
|
||||
async def assess_module_risk(
|
||||
request: AIRiskAssessmentRequest,
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""Get AI risk assessment for a service module."""
|
||||
from ..services.ai_compliance_assistant import get_ai_assistant
|
||||
from ..db.repository import ServiceModuleRepository
|
||||
|
||||
# Get module from DB
|
||||
module_repo = ServiceModuleRepository(db)
|
||||
module = module_repo.get_by_id(request.module_id)
|
||||
|
||||
if not module:
|
||||
module = module_repo.get_by_name(request.module_id)
|
||||
|
||||
if not module:
|
||||
raise HTTPException(status_code=404, detail=f"Module {request.module_id} not found")
|
||||
|
||||
# Get regulations for this module
|
||||
module_detail = module_repo.get_with_regulations(module.id)
|
||||
regulations = []
|
||||
if module_detail and module_detail.get("regulation_mappings"):
|
||||
for mapping in module_detail["regulation_mappings"]:
|
||||
regulations.append({
|
||||
"code": mapping.get("regulation_code", ""),
|
||||
"relevance": mapping.get("relevance_level", "medium"),
|
||||
})
|
||||
|
||||
try:
|
||||
assistant = get_ai_assistant()
|
||||
result = await assistant.assess_module_risk(
|
||||
module_name=module.name,
|
||||
service_type=module.service_type.value if module.service_type else "unknown",
|
||||
description=module.description or "",
|
||||
processes_pii=module.processes_pii,
|
||||
ai_components=module.ai_components,
|
||||
criticality=module.criticality or "medium",
|
||||
data_categories=module.data_categories or [],
|
||||
regulations=regulations,
|
||||
)
|
||||
|
||||
return AIRiskAssessmentResponse(
|
||||
module_name=result.module_name,
|
||||
overall_risk=result.overall_risk,
|
||||
risk_factors=[
|
||||
AIRiskFactor(
|
||||
factor=f.get("factor", ""),
|
||||
severity=f.get("severity", "medium"),
|
||||
likelihood=f.get("likelihood", "medium"),
|
||||
)
|
||||
for f in result.risk_factors
|
||||
],
|
||||
recommendations=result.recommendations,
|
||||
compliance_gaps=result.compliance_gaps,
|
||||
confidence_score=result.confidence_score,
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error(f"AI risk assessment failed: {e}")
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
|
||||
|
||||
@router.post("/ai/gap-analysis", response_model=AIGapAnalysisResponse)
|
||||
async def analyze_gap(
|
||||
request: AIGapAnalysisRequest,
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""Analyze coverage gaps between a requirement and existing controls."""
|
||||
from ..services.ai_compliance_assistant import get_ai_assistant
|
||||
|
||||
# Get requirement from DB
|
||||
req_repo = RequirementRepository(db)
|
||||
requirement = req_repo.get_by_id(request.requirement_id)
|
||||
|
||||
if not requirement:
|
||||
raise HTTPException(status_code=404, detail=f"Requirement {request.requirement_id} not found")
|
||||
|
||||
# Get regulation info
|
||||
reg_repo = RegulationRepository(db)
|
||||
regulation = reg_repo.get_by_id(requirement.regulation_id)
|
||||
|
||||
# Get existing control mappings from eager-loaded relationship
|
||||
ctrl_repo = ControlRepository(db)
|
||||
existing_controls = []
|
||||
|
||||
if requirement.control_mappings:
|
||||
for mapping in requirement.control_mappings:
|
||||
if mapping.control:
|
||||
existing_controls.append({
|
||||
"control_id": mapping.control.control_id,
|
||||
"title": mapping.control.title,
|
||||
"status": mapping.control.status.value if mapping.control.status else "unknown",
|
||||
})
|
||||
|
||||
try:
|
||||
assistant = get_ai_assistant()
|
||||
result = await assistant.analyze_gap(
|
||||
requirement_id=requirement.id,
|
||||
requirement_title=requirement.title,
|
||||
requirement_text=requirement.requirement_text or requirement.description or "",
|
||||
regulation_code=regulation.code if regulation else "UNKNOWN",
|
||||
existing_controls=existing_controls,
|
||||
)
|
||||
|
||||
return AIGapAnalysisResponse(
|
||||
requirement_id=result.requirement_id,
|
||||
requirement_title=result.requirement_title,
|
||||
coverage_level=result.coverage_level,
|
||||
existing_controls=result.existing_controls,
|
||||
missing_coverage=result.missing_coverage,
|
||||
suggested_actions=result.suggested_actions,
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error(f"AI gap analysis failed: {e}")
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
|
||||
|
||||
@router.post("/ai/batch-interpret", response_model=AIBatchInterpretationResponse)
|
||||
async def batch_interpret_requirements(
|
||||
request: AIBatchInterpretationRequest,
|
||||
background_tasks: BackgroundTasks,
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""
|
||||
Batch interpret multiple requirements.
|
||||
|
||||
For large batches, this runs in the background and returns immediately.
|
||||
"""
|
||||
from ..services.ai_compliance_assistant import get_ai_assistant
|
||||
|
||||
req_repo = RequirementRepository(db)
|
||||
reg_repo = RegulationRepository(db)
|
||||
|
||||
# Build list of requirements to process
|
||||
requirements_to_process = []
|
||||
|
||||
if request.requirement_ids:
|
||||
for req_id in request.requirement_ids:
|
||||
req = req_repo.get_by_id(req_id)
|
||||
if req:
|
||||
reg = reg_repo.get_by_id(req.regulation_id)
|
||||
requirements_to_process.append({
|
||||
"id": req.id,
|
||||
"article": req.article,
|
||||
"title": req.title,
|
||||
"requirement_text": req.requirement_text or req.description or "",
|
||||
"regulation_code": reg.code if reg else "UNKNOWN",
|
||||
"regulation_name": reg.name if reg else "Unknown",
|
||||
})
|
||||
|
||||
elif request.regulation_code:
|
||||
# Get all requirements for a regulation
|
||||
reg = reg_repo.get_by_code(request.regulation_code)
|
||||
if reg:
|
||||
reqs = req_repo.get_by_regulation(reg.id)
|
||||
for req in reqs[:50]: # Limit to 50 for batch processing
|
||||
requirements_to_process.append({
|
||||
"id": req.id,
|
||||
"article": req.article,
|
||||
"title": req.title,
|
||||
"requirement_text": req.requirement_text or req.description or "",
|
||||
"regulation_code": reg.code,
|
||||
"regulation_name": reg.name,
|
||||
})
|
||||
|
||||
if not requirements_to_process:
|
||||
raise HTTPException(status_code=400, detail="No requirements found to process")
|
||||
|
||||
# For small batches, process synchronously
|
||||
if len(requirements_to_process) <= 5:
|
||||
assistant = get_ai_assistant()
|
||||
results = await assistant.batch_interpret_requirements(
|
||||
requirements_to_process,
|
||||
rate_limit=request.rate_limit,
|
||||
)
|
||||
|
||||
return AIBatchInterpretationResponse(
|
||||
total=len(requirements_to_process),
|
||||
processed=len(results),
|
||||
interpretations=[
|
||||
AIInterpretationResponse(
|
||||
requirement_id=r.requirement_id,
|
||||
summary=r.summary,
|
||||
applicability=r.applicability,
|
||||
technical_measures=r.technical_measures,
|
||||
affected_modules=r.affected_modules,
|
||||
risk_level=r.risk_level,
|
||||
implementation_hints=r.implementation_hints,
|
||||
confidence_score=r.confidence_score,
|
||||
error=r.error,
|
||||
)
|
||||
for r in results
|
||||
],
|
||||
)
|
||||
|
||||
# For large batches, return immediately with info
|
||||
# (Background processing would be added in a production version)
|
||||
return AIBatchInterpretationResponse(
|
||||
total=len(requirements_to_process),
|
||||
processed=0,
|
||||
interpretations=[],
|
||||
)
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# PDF Extraction (Sprint 2)
|
||||
# ============================================================================
|
||||
|
||||
@router.get("/pdf/available")
|
||||
async def list_available_pdfs():
|
||||
"""List available PDF documents for extraction."""
|
||||
from pathlib import Path
|
||||
|
||||
docs_path = Path("/app/docs") if Path("/app/docs").exists() else Path("docs")
|
||||
|
||||
available = []
|
||||
bsi_files = list(docs_path.glob("BSI-TR-*.pdf"))
|
||||
|
||||
for pdf_file in bsi_files:
|
||||
available.append({
|
||||
"filename": pdf_file.name,
|
||||
"path": str(pdf_file),
|
||||
"size_bytes": pdf_file.stat().st_size,
|
||||
"type": "bsi_standard",
|
||||
})
|
||||
|
||||
return {
|
||||
"available_pdfs": available,
|
||||
"total": len(available),
|
||||
}
|
||||
|
||||
|
||||
@router.post("/pdf/extract/{doc_code}", response_model=PDFExtractionResponse)
|
||||
async def extract_pdf_requirements(
|
||||
doc_code: str,
|
||||
save_to_db: bool = Query(True, description="Save extracted requirements to database"),
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""
|
||||
Extract requirements/aspects from a BSI-TR PDF document.
|
||||
|
||||
doc_code examples:
|
||||
- BSI-TR-03161-1: General security requirements
|
||||
- BSI-TR-03161-2: Web application security
|
||||
- BSI-TR-03161-3: Backend/server security
|
||||
"""
|
||||
from pathlib import Path
|
||||
from ..services.pdf_extractor import BSIPDFExtractor
|
||||
from ..db.models import RegulationTypeEnum
|
||||
|
||||
# Find the PDF file
|
||||
docs_path = Path("/app/docs") if Path("/app/docs").exists() else Path("docs")
|
||||
pdf_path = docs_path / f"{doc_code}.pdf"
|
||||
|
||||
if not pdf_path.exists():
|
||||
raise HTTPException(status_code=404, detail=f"PDF not found: {doc_code}.pdf")
|
||||
|
||||
# Extract aspects
|
||||
extractor = BSIPDFExtractor()
|
||||
try:
|
||||
aspects = extractor.extract_from_file(str(pdf_path), source_name=doc_code)
|
||||
except Exception as e:
|
||||
logger.error(f"PDF extraction failed: {e}")
|
||||
raise HTTPException(status_code=500, detail=f"PDF extraction failed: {str(e)}")
|
||||
|
||||
# Find or create the regulation
|
||||
reg_repo = RegulationRepository(db)
|
||||
regulation = reg_repo.get_by_code(doc_code)
|
||||
|
||||
if not regulation:
|
||||
regulation = reg_repo.create(
|
||||
code=doc_code,
|
||||
name=f"BSI Technical Guideline {doc_code.split('-')[-1]}",
|
||||
full_name=f"BSI Technische Richtlinie {doc_code}",
|
||||
regulation_type=RegulationTypeEnum.BSI_STANDARD,
|
||||
local_pdf_path=str(pdf_path),
|
||||
)
|
||||
|
||||
# Save to database if requested
|
||||
saved_count = 0
|
||||
if save_to_db and aspects:
|
||||
req_repo = RequirementRepository(db)
|
||||
for aspect in aspects:
|
||||
# Check if requirement already exists
|
||||
existing = db.query(RequirementDB).filter(
|
||||
RequirementDB.regulation_id == regulation.id,
|
||||
RequirementDB.article == aspect.aspect_id,
|
||||
).first()
|
||||
|
||||
if not existing:
|
||||
try:
|
||||
req_repo.create(
|
||||
regulation_id=regulation.id,
|
||||
article=aspect.aspect_id,
|
||||
title=aspect.title[:300] if aspect.title else "",
|
||||
description=f"Category: {aspect.category.value}",
|
||||
requirement_text=aspect.full_text[:4000] if aspect.full_text else "",
|
||||
priority=1 if aspect.requirement_level.value == "MUSS" else (
|
||||
2 if aspect.requirement_level.value == "SOLL" else 3
|
||||
),
|
||||
)
|
||||
saved_count += 1
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to save aspect {aspect.aspect_id}: {e}")
|
||||
|
||||
db.commit()
|
||||
|
||||
# Convert aspects to response format
|
||||
aspect_responses = [
|
||||
BSIAspectResponse(
|
||||
aspect_id=a.aspect_id,
|
||||
title=a.title,
|
||||
full_text=a.full_text,
|
||||
category=a.category.value,
|
||||
page_number=a.page_number,
|
||||
section=a.section,
|
||||
requirement_level=a.requirement_level.value,
|
||||
source_document=a.source_document,
|
||||
)
|
||||
for a in aspects
|
||||
]
|
||||
|
||||
return PDFExtractionResponse(
|
||||
doc_code=doc_code,
|
||||
total_extracted=len(aspects),
|
||||
saved_to_db=saved_count,
|
||||
aspects=aspect_responses,
|
||||
)
|
||||
|
||||
|
||||
@router.get("/pdf/extraction-stats")
|
||||
async def get_extraction_stats(db: Session = Depends(get_db)):
|
||||
"""Get statistics about extracted PDF requirements."""
|
||||
from sqlalchemy import func
|
||||
|
||||
# Count requirements per BSI regulation
|
||||
stats = (
|
||||
db.query(
|
||||
RegulationDB.code,
|
||||
func.count(RequirementDB.id).label('count')
|
||||
)
|
||||
.join(RequirementDB, RequirementDB.regulation_id == RegulationDB.id)
|
||||
.filter(RegulationDB.code.like('BSI-%'))
|
||||
.group_by(RegulationDB.code)
|
||||
.all()
|
||||
)
|
||||
|
||||
return {
|
||||
"bsi_requirements": {code: count for code, count in stats},
|
||||
"total_bsi_requirements": sum(count for _, count in stats),
|
||||
}
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Automatic Control Mapping
|
||||
# ============================================================================
|
||||
|
||||
# Domain keyword mapping for automatic control assignment
|
||||
DOMAIN_KEYWORDS = {
|
||||
"priv": ["datenschutz", "dsgvo", "gdpr", "privacy", "personenbezogen", "einwilligung",
|
||||
"consent", "betroffenenrechte", "verarbeitungsverzeichnis", "pii", "auftragsverarbeitung"],
|
||||
"iam": ["authentifizierung", "auth", "login", "passwort", "password", "zugang", "access",
|
||||
"berechtigung", "session", "token", "jwt", "oauth", "sso", "mfa", "2fa", "rbac"],
|
||||
"crypto": ["verschlüsselung", "encryption", "kryptograph", "crypto", "hash", "schlüssel",
|
||||
"key", "tls", "ssl", "zertifikat", "signatur", "aes", "rsa"],
|
||||
"sdlc": ["entwicklung", "code", "software", "sast", "dast", "dependency", "vulnerable",
|
||||
"cve", "security scan", "semgrep", "trivy", "sbom", "ci/cd", "build"],
|
||||
"ops": ["monitoring", "logging", "log", "protokoll", "backup", "incident", "alert",
|
||||
"availability", "uptime", "patch", "update", "deployment"],
|
||||
"ai": ["künstliche intelligenz", "ki", "ai", "machine learning", "ml", "modell",
|
||||
"training", "inference", "bias", "ai act", "hochrisiko"],
|
||||
"cra": ["vulnerability", "schwachstelle", "disclosure", "patch", "eol", "end-of-life",
|
||||
"supply chain", "sbom", "cve", "update"],
|
||||
"gov": ["richtlinie", "policy", "governance", "verantwortlich", "raci", "dokumentation",
|
||||
"prozess", "awareness", "schulung", "training"],
|
||||
"aud": ["audit", "prüfung", "nachweis", "evidence", "traceability", "nachvollzieh",
|
||||
"protokoll", "export", "report"],
|
||||
}
|
||||
|
||||
|
||||
@router.post("/ai/auto-map-controls")
|
||||
async def auto_map_controls(
|
||||
requirement_id: str = Query(..., description="Requirement UUID"),
|
||||
save_to_db: bool = Query(True, description="Save mappings to database"),
|
||||
use_ai: bool = Query(False, description="Use AI for better matching (slower)"),
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""
|
||||
Automatically map controls to a requirement.
|
||||
|
||||
Uses keyword matching by default (fast) or AI for better accuracy (slower).
|
||||
"""
|
||||
from ..db.models import ControlMappingDB
|
||||
|
||||
# Get requirement
|
||||
req_repo = RequirementRepository(db)
|
||||
requirement = req_repo.get_by_id(requirement_id)
|
||||
if not requirement:
|
||||
raise HTTPException(status_code=404, detail=f"Requirement {requirement_id} not found")
|
||||
|
||||
# Get all controls
|
||||
ctrl_repo = ControlRepository(db)
|
||||
all_controls = ctrl_repo.get_all()
|
||||
|
||||
# Text to analyze
|
||||
text_to_analyze = f"{requirement.title} {requirement.requirement_text or ''} {requirement.description or ''}"
|
||||
text_lower = text_to_analyze.lower()
|
||||
|
||||
matched_controls = []
|
||||
|
||||
if use_ai:
|
||||
# Use AI for matching (slower but more accurate)
|
||||
from ..services.ai_compliance_assistant import get_ai_assistant
|
||||
assistant = get_ai_assistant()
|
||||
|
||||
reg_repo = RegulationRepository(db)
|
||||
regulation = reg_repo.get_by_id(requirement.regulation_id)
|
||||
|
||||
try:
|
||||
suggestions = await assistant.suggest_controls(
|
||||
requirement_title=requirement.title,
|
||||
requirement_text=requirement.requirement_text or "",
|
||||
regulation_name=regulation.name if regulation else "Unknown",
|
||||
affected_modules=[],
|
||||
)
|
||||
|
||||
# Match suggestions to existing controls by domain
|
||||
for suggestion in suggestions:
|
||||
domain = suggestion.domain.lower()
|
||||
domain_controls = [c for c in all_controls if c.domain and c.domain.value.lower() == domain]
|
||||
if domain_controls:
|
||||
# Take the first matching control from this domain
|
||||
matched_controls.append({
|
||||
"control": domain_controls[0],
|
||||
"coverage": "partial",
|
||||
"notes": f"AI suggested: {suggestion.title}",
|
||||
"confidence": suggestion.confidence_score,
|
||||
})
|
||||
except Exception as e:
|
||||
logger.warning(f"AI mapping failed, falling back to keyword matching: {e}")
|
||||
use_ai = False # Fall back to keyword matching
|
||||
|
||||
if not use_ai:
|
||||
# Keyword-based matching (fast)
|
||||
domain_scores = {}
|
||||
for domain, keywords in DOMAIN_KEYWORDS.items():
|
||||
score = sum(1 for kw in keywords if kw.lower() in text_lower)
|
||||
if score > 0:
|
||||
domain_scores[domain] = score
|
||||
|
||||
# Sort domains by score
|
||||
sorted_domains = sorted(domain_scores.items(), key=lambda x: x[1], reverse=True)
|
||||
|
||||
# Take top 3 domains
|
||||
for domain, score in sorted_domains[:3]:
|
||||
domain_controls = [c for c in all_controls if c.domain and c.domain.value.lower() == domain]
|
||||
for ctrl in domain_controls[:2]: # Max 2 controls per domain
|
||||
matched_controls.append({
|
||||
"control": ctrl,
|
||||
"coverage": "partial" if score < 3 else "full",
|
||||
"notes": f"Keyword match (score: {score})",
|
||||
"confidence": min(0.9, 0.5 + score * 0.1),
|
||||
})
|
||||
|
||||
# Save mappings to database if requested
|
||||
created_mappings = []
|
||||
if save_to_db and matched_controls:
|
||||
for match in matched_controls:
|
||||
ctrl = match["control"]
|
||||
|
||||
# Check if mapping already exists
|
||||
existing = db.query(ControlMappingDB).filter(
|
||||
ControlMappingDB.requirement_id == requirement_id,
|
||||
ControlMappingDB.control_id == ctrl.id,
|
||||
).first()
|
||||
|
||||
if not existing:
|
||||
mapping = ControlMappingDB(
|
||||
requirement_id=requirement_id,
|
||||
control_id=ctrl.id,
|
||||
coverage_level=match["coverage"],
|
||||
notes=match["notes"],
|
||||
)
|
||||
db.add(mapping)
|
||||
created_mappings.append({
|
||||
"control_id": ctrl.control_id,
|
||||
"domain": ctrl.domain.value if ctrl.domain else None,
|
||||
"title": ctrl.title,
|
||||
"coverage_level": match["coverage"],
|
||||
"notes": match["notes"],
|
||||
})
|
||||
|
||||
db.commit()
|
||||
|
||||
return {
|
||||
"requirement_id": requirement_id,
|
||||
"requirement_title": requirement.title,
|
||||
"matched_controls": len(matched_controls),
|
||||
"created_mappings": len(created_mappings),
|
||||
"mappings": created_mappings if save_to_db else [
|
||||
{
|
||||
"control_id": m["control"].control_id,
|
||||
"domain": m["control"].domain.value if m["control"].domain else None,
|
||||
"title": m["control"].title,
|
||||
"coverage_level": m["coverage"],
|
||||
"confidence": m.get("confidence", 0.7),
|
||||
}
|
||||
for m in matched_controls
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
@router.post("/ai/batch-map-controls")
|
||||
async def batch_map_controls(
|
||||
regulation_code: Optional[str] = Query(None, description="Filter by regulation code"),
|
||||
limit: int = Query(100, description="Max requirements to process"),
|
||||
use_ai: bool = Query(False, description="Use AI for matching (slower)"),
|
||||
background_tasks: BackgroundTasks = None,
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""
|
||||
Batch map controls to multiple requirements.
|
||||
|
||||
Processes requirements that don't have mappings yet.
|
||||
"""
|
||||
from ..db.models import ControlMappingDB
|
||||
|
||||
# Get requirements without mappings
|
||||
req_repo = RequirementRepository(db)
|
||||
|
||||
if regulation_code:
|
||||
reg_repo = RegulationRepository(db)
|
||||
regulation = reg_repo.get_by_code(regulation_code)
|
||||
if not regulation:
|
||||
raise HTTPException(status_code=404, detail=f"Regulation {regulation_code} not found")
|
||||
all_requirements = req_repo.get_by_regulation(regulation.id)
|
||||
else:
|
||||
all_requirements = req_repo.get_all()
|
||||
|
||||
# Filter to requirements without mappings
|
||||
requirements_without_mappings = []
|
||||
for req in all_requirements:
|
||||
existing = db.query(ControlMappingDB).filter(
|
||||
ControlMappingDB.requirement_id == req.id
|
||||
).first()
|
||||
if not existing:
|
||||
requirements_without_mappings.append(req)
|
||||
|
||||
# Limit processing
|
||||
to_process = requirements_without_mappings[:limit]
|
||||
|
||||
# Get all controls once
|
||||
ctrl_repo = ControlRepository(db)
|
||||
all_controls = ctrl_repo.get_all()
|
||||
|
||||
# Process each requirement
|
||||
results = []
|
||||
for req in to_process:
|
||||
try:
|
||||
text_to_analyze = f"{req.title} {req.requirement_text or ''} {req.description or ''}"
|
||||
text_lower = text_to_analyze.lower()
|
||||
|
||||
# Quick keyword matching
|
||||
domain_scores = {}
|
||||
for domain, keywords in DOMAIN_KEYWORDS.items():
|
||||
score = sum(1 for kw in keywords if kw.lower() in text_lower)
|
||||
if score > 0:
|
||||
domain_scores[domain] = score
|
||||
|
||||
if domain_scores:
|
||||
# Get top domain
|
||||
top_domain = max(domain_scores.items(), key=lambda x: x[1])[0]
|
||||
domain_controls = [c for c in all_controls if c.domain and c.domain.value.lower() == top_domain]
|
||||
|
||||
if domain_controls:
|
||||
ctrl = domain_controls[0]
|
||||
|
||||
# Create mapping
|
||||
mapping = ControlMappingDB(
|
||||
requirement_id=req.id,
|
||||
control_id=ctrl.id,
|
||||
coverage_level="partial",
|
||||
notes=f"Auto-mapped (domain: {top_domain})",
|
||||
)
|
||||
db.add(mapping)
|
||||
|
||||
results.append({
|
||||
"requirement_id": req.id,
|
||||
"requirement_title": req.title[:50],
|
||||
"control_id": ctrl.control_id,
|
||||
"domain": top_domain,
|
||||
})
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to map requirement {req.id}: {e}")
|
||||
|
||||
db.commit()
|
||||
|
||||
return {
|
||||
"processed": len(to_process),
|
||||
"mapped": len(results),
|
||||
"remaining": len(requirements_without_mappings) - len(to_process),
|
||||
"mappings": results[:20], # Only return first 20 for readability
|
||||
}
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# LLM Provider Switch Endpoints (Runtime Configuration)
|
||||
# ============================================================================
|
||||
|
||||
class ProviderSwitchRequest(BaseModel):
|
||||
"""Request to switch LLM provider at runtime."""
|
||||
provider: str # "anthropic" or "self_hosted"
|
||||
model: Optional[str] = None # Optional: override model
|
||||
url: Optional[str] = None # Optional: override URL for self-hosted
|
||||
|
||||
|
||||
class ProviderSwitchResponse(BaseModel):
|
||||
"""Response after switching LLM provider."""
|
||||
success: bool
|
||||
previous_provider: str
|
||||
new_provider: str
|
||||
model: str
|
||||
url: Optional[str] = None
|
||||
message: str
|
||||
|
||||
|
||||
@router.post("/ai/switch-provider", response_model=ProviderSwitchResponse)
|
||||
async def switch_llm_provider(request: ProviderSwitchRequest):
|
||||
"""
|
||||
Switch the LLM provider at runtime between Anthropic API and Self-Hosted (Ollama).
|
||||
|
||||
This allows developers to toggle between:
|
||||
- **anthropic**: Cloud-based Claude API (kostenpflichtig, Daten gehen zu Anthropic)
|
||||
- **self_hosted**: Self-hosted Ollama on Mac Mini (kostenlos, DSGVO-konform, Daten bleiben intern)
|
||||
|
||||
Note: This change is temporary for the current container session.
|
||||
For permanent changes, modify the docker-compose.yml environment variables.
|
||||
"""
|
||||
from ..services.llm_provider import (
|
||||
reset_shared_provider,
|
||||
get_shared_provider,
|
||||
LLMProviderType,
|
||||
)
|
||||
|
||||
try:
|
||||
# Get current provider info before switch
|
||||
old_provider = get_shared_provider()
|
||||
old_provider_name = old_provider.provider_name
|
||||
|
||||
# Map string to enum
|
||||
provider_map = {
|
||||
"anthropic": LLMProviderType.ANTHROPIC,
|
||||
"self_hosted": LLMProviderType.SELF_HOSTED,
|
||||
"mock": LLMProviderType.MOCK,
|
||||
}
|
||||
|
||||
if request.provider.lower() not in provider_map:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Invalid provider: {request.provider}. Use 'anthropic' or 'self_hosted'"
|
||||
)
|
||||
|
||||
# Update environment variables for the new provider
|
||||
os.environ["COMPLIANCE_LLM_PROVIDER"] = request.provider.lower()
|
||||
|
||||
if request.provider.lower() == "self_hosted":
|
||||
if request.url:
|
||||
os.environ["SELF_HOSTED_LLM_URL"] = request.url
|
||||
if request.model:
|
||||
os.environ["SELF_HOSTED_LLM_MODEL"] = request.model
|
||||
else:
|
||||
# Default to llama3.1:70b for compliance tasks
|
||||
os.environ["SELF_HOSTED_LLM_MODEL"] = os.environ.get(
|
||||
"SELF_HOSTED_LLM_MODEL", "llama3.1:70b"
|
||||
)
|
||||
elif request.provider.lower() == "anthropic":
|
||||
if request.model:
|
||||
os.environ["ANTHROPIC_MODEL"] = request.model
|
||||
|
||||
# Reset the shared provider to pick up new config
|
||||
reset_shared_provider()
|
||||
|
||||
# Get the new provider
|
||||
new_provider = get_shared_provider()
|
||||
|
||||
return ProviderSwitchResponse(
|
||||
success=True,
|
||||
previous_provider=old_provider_name,
|
||||
new_provider=new_provider.provider_name,
|
||||
model=new_provider.config.model,
|
||||
url=new_provider.config.base_url if hasattr(new_provider.config, 'base_url') else None,
|
||||
message=f"Successfully switched from {old_provider_name} to {new_provider.provider_name}",
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to switch LLM provider: {e}")
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
|
||||
|
||||
@router.get("/ai/providers")
|
||||
async def list_available_providers():
|
||||
"""
|
||||
List available LLM providers with their descriptions.
|
||||
|
||||
This helps developers understand which provider to use for which scenario.
|
||||
"""
|
||||
return {
|
||||
"providers": [
|
||||
{
|
||||
"id": "anthropic",
|
||||
"name": "Anthropic Claude API",
|
||||
"description_de": "Cloud-basierte KI von Anthropic. Kostenpflichtig (API-Credits). Daten werden zur Verarbeitung an Anthropic gesendet.",
|
||||
"description_en": "Cloud-based AI from Anthropic. Paid service (API credits). Data is sent to Anthropic for processing.",
|
||||
"gdpr_compliant": False,
|
||||
"data_location": "Anthropic Cloud (USA)",
|
||||
"cost": "Kostenpflichtig pro Token",
|
||||
"use_case": "Produktiv, wenn hohe Qualitaet benoetigt wird",
|
||||
"models": ["claude-sonnet-4-20250514", "claude-3-5-sonnet-20241022"],
|
||||
},
|
||||
{
|
||||
"id": "self_hosted",
|
||||
"name": "Self-Hosted Ollama",
|
||||
"description_de": "Lokales LLM auf dem Mac Mini M4 Pro (64GB RAM). Kostenlos. Alle Daten bleiben intern - DSGVO-konform!",
|
||||
"description_en": "Local LLM on Mac Mini M4 Pro (64GB RAM). Free. All data stays internal - GDPR compliant!",
|
||||
"gdpr_compliant": True,
|
||||
"data_location": "Lokal auf Mac Mini",
|
||||
"cost": "Kostenlos (Hardware bereits vorhanden)",
|
||||
"use_case": "Entwicklung, Testing, DSGVO-sensitive Dokumente",
|
||||
"models": ["llama3.1:70b", "llama3.2-vision", "mixtral:8x7b"],
|
||||
},
|
||||
],
|
||||
"current_provider": None, # Will be filled by get_ai_status
|
||||
"note_de": "Umschaltung erfolgt sofort, aber nur fuer diese Container-Session. Fuer permanente Aenderung docker-compose.yml anpassen.",
|
||||
"note_en": "Switch takes effect immediately but only for this container session. For permanent change, modify docker-compose.yml.",
|
||||
}
|
||||
637
backend/compliance/api/audit_routes.py
Normal file
637
backend/compliance/api/audit_routes.py
Normal file
@@ -0,0 +1,637 @@
|
||||
"""
|
||||
FastAPI routes for Audit Sessions & Sign-off functionality.
|
||||
|
||||
Sprint 3 Phase 3: Auditor-Verbesserungen
|
||||
|
||||
Endpoints:
|
||||
- /audit/sessions: Manage audit sessions
|
||||
- /audit/checklist: Audit checklist with sign-off
|
||||
"""
|
||||
|
||||
import logging
|
||||
from datetime import datetime
|
||||
from typing import Optional, List
|
||||
from uuid import uuid4
|
||||
import hashlib
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query
|
||||
from sqlalchemy.orm import Session
|
||||
from sqlalchemy import func
|
||||
|
||||
from classroom_engine.database import get_db
|
||||
|
||||
from ..db.models import (
|
||||
AuditSessionDB, AuditSignOffDB, AuditResultEnum, AuditSessionStatusEnum,
|
||||
RequirementDB, RegulationDB, ControlMappingDB
|
||||
)
|
||||
from .schemas import (
|
||||
CreateAuditSessionRequest, AuditSessionResponse, AuditSessionSummary, AuditSessionDetailResponse,
|
||||
AuditSessionListResponse, SignOffRequest, SignOffResponse,
|
||||
AuditChecklistItem, AuditChecklistResponse, AuditStatistics,
|
||||
PaginationMeta,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter(prefix="/audit", tags=["compliance-audit"])
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Audit Sessions
|
||||
# ============================================================================
|
||||
|
||||
@router.post("/sessions", response_model=AuditSessionResponse)
|
||||
async def create_audit_session(
|
||||
request: CreateAuditSessionRequest,
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""
|
||||
Create a new audit session for structured compliance reviews.
|
||||
|
||||
An audit session groups requirements for systematic review by an auditor.
|
||||
"""
|
||||
# Get total requirements count based on filters
|
||||
query = db.query(RequirementDB)
|
||||
if request.regulation_codes:
|
||||
reg_ids = db.query(RegulationDB.id).filter(
|
||||
RegulationDB.code.in_(request.regulation_codes)
|
||||
).all()
|
||||
reg_ids = [r[0] for r in reg_ids]
|
||||
query = query.filter(RequirementDB.regulation_id.in_(reg_ids))
|
||||
|
||||
total_items = query.count()
|
||||
|
||||
# Create the session
|
||||
session = AuditSessionDB(
|
||||
id=str(uuid4()),
|
||||
name=request.name,
|
||||
description=request.description,
|
||||
auditor_name=request.auditor_name,
|
||||
auditor_email=request.auditor_email,
|
||||
auditor_organization=request.auditor_organization,
|
||||
status=AuditSessionStatusEnum.DRAFT,
|
||||
regulation_ids=request.regulation_codes,
|
||||
total_items=total_items,
|
||||
completed_items=0,
|
||||
compliant_count=0,
|
||||
non_compliant_count=0,
|
||||
)
|
||||
|
||||
db.add(session)
|
||||
db.commit()
|
||||
db.refresh(session)
|
||||
|
||||
return AuditSessionResponse(
|
||||
id=session.id,
|
||||
name=session.name,
|
||||
description=session.description,
|
||||
auditor_name=session.auditor_name,
|
||||
auditor_email=session.auditor_email,
|
||||
auditor_organization=session.auditor_organization,
|
||||
status=session.status.value,
|
||||
regulation_ids=session.regulation_ids,
|
||||
total_items=session.total_items,
|
||||
completed_items=session.completed_items,
|
||||
compliant_count=session.compliant_count,
|
||||
non_compliant_count=session.non_compliant_count,
|
||||
completion_percentage=session.completion_percentage,
|
||||
created_at=session.created_at,
|
||||
started_at=session.started_at,
|
||||
completed_at=session.completed_at,
|
||||
)
|
||||
|
||||
|
||||
@router.get("/sessions", response_model=List[AuditSessionSummary])
|
||||
async def list_audit_sessions(
|
||||
status: Optional[str] = None,
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""
|
||||
List all audit sessions, optionally filtered by status.
|
||||
"""
|
||||
query = db.query(AuditSessionDB)
|
||||
|
||||
if status:
|
||||
try:
|
||||
status_enum = AuditSessionStatusEnum(status)
|
||||
query = query.filter(AuditSessionDB.status == status_enum)
|
||||
except ValueError:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Invalid status: {status}. Valid values: draft, in_progress, completed, archived"
|
||||
)
|
||||
|
||||
sessions = query.order_by(AuditSessionDB.created_at.desc()).all()
|
||||
|
||||
return [
|
||||
AuditSessionSummary(
|
||||
id=s.id,
|
||||
name=s.name,
|
||||
auditor_name=s.auditor_name,
|
||||
status=s.status.value,
|
||||
total_items=s.total_items,
|
||||
completed_items=s.completed_items,
|
||||
completion_percentage=s.completion_percentage,
|
||||
created_at=s.created_at,
|
||||
started_at=s.started_at,
|
||||
completed_at=s.completed_at,
|
||||
)
|
||||
for s in sessions
|
||||
]
|
||||
|
||||
|
||||
@router.get("/sessions/{session_id}", response_model=AuditSessionDetailResponse)
|
||||
async def get_audit_session(
|
||||
session_id: str,
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""
|
||||
Get detailed information about a specific audit session.
|
||||
"""
|
||||
session = db.query(AuditSessionDB).filter(AuditSessionDB.id == session_id).first()
|
||||
if not session:
|
||||
raise HTTPException(status_code=404, detail=f"Audit session {session_id} not found")
|
||||
|
||||
# Get sign-off statistics
|
||||
signoffs = db.query(AuditSignOffDB).filter(AuditSignOffDB.session_id == session_id).all()
|
||||
|
||||
stats = AuditStatistics(
|
||||
total=session.total_items,
|
||||
compliant=sum(1 for s in signoffs if s.result == AuditResultEnum.COMPLIANT),
|
||||
compliant_with_notes=sum(1 for s in signoffs if s.result == AuditResultEnum.COMPLIANT_WITH_NOTES),
|
||||
non_compliant=sum(1 for s in signoffs if s.result == AuditResultEnum.NON_COMPLIANT),
|
||||
not_applicable=sum(1 for s in signoffs if s.result == AuditResultEnum.NOT_APPLICABLE),
|
||||
pending=session.total_items - len(signoffs),
|
||||
completion_percentage=session.completion_percentage,
|
||||
)
|
||||
|
||||
return AuditSessionDetail(
|
||||
id=session.id,
|
||||
name=session.name,
|
||||
description=session.description,
|
||||
auditor_name=session.auditor_name,
|
||||
auditor_email=session.auditor_email,
|
||||
auditor_organization=session.auditor_organization,
|
||||
status=session.status.value,
|
||||
regulation_ids=session.regulation_ids,
|
||||
total_items=session.total_items,
|
||||
completed_items=session.completed_items,
|
||||
compliant_count=session.compliant_count,
|
||||
non_compliant_count=session.non_compliant_count,
|
||||
completion_percentage=session.completion_percentage,
|
||||
created_at=session.created_at,
|
||||
started_at=session.started_at,
|
||||
completed_at=session.completed_at,
|
||||
statistics=stats,
|
||||
)
|
||||
|
||||
|
||||
@router.put("/sessions/{session_id}/start")
|
||||
async def start_audit_session(
|
||||
session_id: str,
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""
|
||||
Start an audit session (change status from draft to in_progress).
|
||||
"""
|
||||
session = db.query(AuditSessionDB).filter(AuditSessionDB.id == session_id).first()
|
||||
if not session:
|
||||
raise HTTPException(status_code=404, detail=f"Audit session {session_id} not found")
|
||||
|
||||
if session.status != AuditSessionStatusEnum.DRAFT:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Session cannot be started. Current status: {session.status.value}"
|
||||
)
|
||||
|
||||
session.status = AuditSessionStatusEnum.IN_PROGRESS
|
||||
session.started_at = datetime.utcnow()
|
||||
db.commit()
|
||||
|
||||
return {"success": True, "message": "Audit session started", "status": "in_progress"}
|
||||
|
||||
|
||||
@router.put("/sessions/{session_id}/complete")
|
||||
async def complete_audit_session(
|
||||
session_id: str,
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""
|
||||
Complete an audit session (change status from in_progress to completed).
|
||||
"""
|
||||
session = db.query(AuditSessionDB).filter(AuditSessionDB.id == session_id).first()
|
||||
if not session:
|
||||
raise HTTPException(status_code=404, detail=f"Audit session {session_id} not found")
|
||||
|
||||
if session.status != AuditSessionStatusEnum.IN_PROGRESS:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Session cannot be completed. Current status: {session.status.value}"
|
||||
)
|
||||
|
||||
session.status = AuditSessionStatusEnum.COMPLETED
|
||||
session.completed_at = datetime.utcnow()
|
||||
db.commit()
|
||||
|
||||
return {"success": True, "message": "Audit session completed", "status": "completed"}
|
||||
|
||||
|
||||
@router.put("/sessions/{session_id}/archive")
|
||||
async def archive_audit_session(
|
||||
session_id: str,
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""
|
||||
Archive a completed audit session.
|
||||
"""
|
||||
session = db.query(AuditSessionDB).filter(AuditSessionDB.id == session_id).first()
|
||||
if not session:
|
||||
raise HTTPException(status_code=404, detail=f"Audit session {session_id} not found")
|
||||
|
||||
if session.status != AuditSessionStatusEnum.COMPLETED:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Only completed sessions can be archived. Current status: {session.status.value}"
|
||||
)
|
||||
|
||||
session.status = AuditSessionStatusEnum.ARCHIVED
|
||||
db.commit()
|
||||
|
||||
return {"success": True, "message": "Audit session archived", "status": "archived"}
|
||||
|
||||
|
||||
@router.delete("/sessions/{session_id}")
|
||||
async def delete_audit_session(
|
||||
session_id: str,
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""
|
||||
Delete an audit session and all its sign-offs.
|
||||
|
||||
Only draft sessions can be deleted.
|
||||
"""
|
||||
session = db.query(AuditSessionDB).filter(AuditSessionDB.id == session_id).first()
|
||||
if not session:
|
||||
raise HTTPException(status_code=404, detail=f"Audit session {session_id} not found")
|
||||
|
||||
if session.status not in [AuditSessionStatusEnum.DRAFT, AuditSessionStatusEnum.ARCHIVED]:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Cannot delete session with status: {session.status.value}. Archive it first."
|
||||
)
|
||||
|
||||
# Delete all sign-offs first (cascade should handle this, but be explicit)
|
||||
db.query(AuditSignOffDB).filter(AuditSignOffDB.session_id == session_id).delete()
|
||||
|
||||
# Delete the session
|
||||
db.delete(session)
|
||||
db.commit()
|
||||
|
||||
return {"success": True, "message": f"Audit session {session_id} deleted"}
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Audit Checklist & Sign-off
|
||||
# ============================================================================
|
||||
|
||||
@router.get("/checklist/{session_id}", response_model=AuditChecklistResponse)
|
||||
async def get_audit_checklist(
|
||||
session_id: str,
|
||||
page: int = Query(1, ge=1),
|
||||
page_size: int = Query(50, ge=1, le=200),
|
||||
status_filter: Optional[str] = None,
|
||||
regulation_filter: Optional[str] = None,
|
||||
search: Optional[str] = None,
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""
|
||||
Get the audit checklist for a session with pagination.
|
||||
|
||||
Returns requirements with their current sign-off status.
|
||||
"""
|
||||
# Get the session
|
||||
session = db.query(AuditSessionDB).filter(AuditSessionDB.id == session_id).first()
|
||||
if not session:
|
||||
raise HTTPException(status_code=404, detail=f"Audit session {session_id} not found")
|
||||
|
||||
# Build base query for requirements
|
||||
query = db.query(RequirementDB).join(RegulationDB)
|
||||
|
||||
# Apply session's regulation filter
|
||||
if session.regulation_ids:
|
||||
query = query.filter(RegulationDB.code.in_(session.regulation_ids))
|
||||
|
||||
# Apply additional filters
|
||||
if regulation_filter:
|
||||
query = query.filter(RegulationDB.code == regulation_filter)
|
||||
|
||||
if search:
|
||||
search_term = f"%{search}%"
|
||||
query = query.filter(
|
||||
(RequirementDB.title.ilike(search_term)) |
|
||||
(RequirementDB.article.ilike(search_term)) |
|
||||
(RequirementDB.description.ilike(search_term))
|
||||
)
|
||||
|
||||
# Get total count before pagination
|
||||
total_count = query.count()
|
||||
|
||||
# Apply pagination
|
||||
requirements = (
|
||||
query
|
||||
.order_by(RegulationDB.code, RequirementDB.article)
|
||||
.offset((page - 1) * page_size)
|
||||
.limit(page_size)
|
||||
.all()
|
||||
)
|
||||
|
||||
# Get existing sign-offs for these requirements
|
||||
req_ids = [r.id for r in requirements]
|
||||
signoffs = (
|
||||
db.query(AuditSignOffDB)
|
||||
.filter(AuditSignOffDB.session_id == session_id)
|
||||
.filter(AuditSignOffDB.requirement_id.in_(req_ids))
|
||||
.all()
|
||||
)
|
||||
signoff_map = {s.requirement_id: s for s in signoffs}
|
||||
|
||||
# Get control mappings counts
|
||||
mapping_counts = (
|
||||
db.query(ControlMappingDB.requirement_id, func.count(ControlMappingDB.id))
|
||||
.filter(ControlMappingDB.requirement_id.in_(req_ids))
|
||||
.group_by(ControlMappingDB.requirement_id)
|
||||
.all()
|
||||
)
|
||||
mapping_count_map = dict(mapping_counts)
|
||||
|
||||
# Build checklist items
|
||||
items = []
|
||||
for req in requirements:
|
||||
signoff = signoff_map.get(req.id)
|
||||
|
||||
# Apply status filter if specified
|
||||
if status_filter:
|
||||
if status_filter == "pending" and signoff is not None:
|
||||
continue
|
||||
elif status_filter != "pending" and (signoff is None or signoff.result.value != status_filter):
|
||||
continue
|
||||
|
||||
item = AuditChecklistItem(
|
||||
requirement_id=req.id,
|
||||
regulation_code=req.regulation.code,
|
||||
article=req.article,
|
||||
paragraph=req.paragraph,
|
||||
title=req.title,
|
||||
description=req.description,
|
||||
current_result=signoff.result.value if signoff else "pending",
|
||||
notes=signoff.notes if signoff else None,
|
||||
is_signed=signoff.signature_hash is not None if signoff else False,
|
||||
signed_at=signoff.signed_at if signoff else None,
|
||||
signed_by=signoff.signed_by if signoff else None,
|
||||
evidence_count=0, # TODO: Add evidence count
|
||||
controls_mapped=mapping_count_map.get(req.id, 0),
|
||||
implementation_status=req.implementation_status,
|
||||
priority=req.priority,
|
||||
)
|
||||
items.append(item)
|
||||
|
||||
# Calculate statistics
|
||||
all_signoffs = db.query(AuditSignOffDB).filter(AuditSignOffDB.session_id == session_id).all()
|
||||
stats = AuditStatistics(
|
||||
total=session.total_items,
|
||||
compliant=sum(1 for s in all_signoffs if s.result == AuditResultEnum.COMPLIANT),
|
||||
compliant_with_notes=sum(1 for s in all_signoffs if s.result == AuditResultEnum.COMPLIANT_WITH_NOTES),
|
||||
non_compliant=sum(1 for s in all_signoffs if s.result == AuditResultEnum.NON_COMPLIANT),
|
||||
not_applicable=sum(1 for s in all_signoffs if s.result == AuditResultEnum.NOT_APPLICABLE),
|
||||
pending=session.total_items - len(all_signoffs),
|
||||
completion_percentage=session.completion_percentage,
|
||||
)
|
||||
|
||||
return AuditChecklistResponse(
|
||||
session=AuditSessionSummary(
|
||||
id=session.id,
|
||||
name=session.name,
|
||||
auditor_name=session.auditor_name,
|
||||
status=session.status.value,
|
||||
total_items=session.total_items,
|
||||
completed_items=session.completed_items,
|
||||
completion_percentage=session.completion_percentage,
|
||||
created_at=session.created_at,
|
||||
started_at=session.started_at,
|
||||
completed_at=session.completed_at,
|
||||
),
|
||||
items=items,
|
||||
pagination=PaginationMeta(
|
||||
page=page,
|
||||
page_size=page_size,
|
||||
total=total_count,
|
||||
total_pages=(total_count + page_size - 1) // page_size,
|
||||
),
|
||||
statistics=stats,
|
||||
)
|
||||
|
||||
|
||||
@router.put("/checklist/{session_id}/items/{requirement_id}/sign-off", response_model=SignOffResponse)
|
||||
async def sign_off_item(
|
||||
session_id: str,
|
||||
requirement_id: str,
|
||||
request: SignOffRequest,
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""
|
||||
Sign off on a specific requirement in an audit session.
|
||||
|
||||
If sign=True, creates a digital signature (SHA-256 hash).
|
||||
"""
|
||||
# Validate session exists and is in progress
|
||||
session = db.query(AuditSessionDB).filter(AuditSessionDB.id == session_id).first()
|
||||
if not session:
|
||||
raise HTTPException(status_code=404, detail=f"Audit session {session_id} not found")
|
||||
|
||||
if session.status not in [AuditSessionStatusEnum.DRAFT, AuditSessionStatusEnum.IN_PROGRESS]:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Cannot sign off items in session with status: {session.status.value}"
|
||||
)
|
||||
|
||||
# Validate requirement exists
|
||||
requirement = db.query(RequirementDB).filter(RequirementDB.id == requirement_id).first()
|
||||
if not requirement:
|
||||
raise HTTPException(status_code=404, detail=f"Requirement {requirement_id} not found")
|
||||
|
||||
# Map string result to enum
|
||||
try:
|
||||
result_enum = AuditResultEnum(request.result)
|
||||
except ValueError:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Invalid result: {request.result}. Valid values: compliant, compliant_notes, non_compliant, not_applicable, pending"
|
||||
)
|
||||
|
||||
# Check if sign-off already exists
|
||||
signoff = (
|
||||
db.query(AuditSignOffDB)
|
||||
.filter(AuditSignOffDB.session_id == session_id)
|
||||
.filter(AuditSignOffDB.requirement_id == requirement_id)
|
||||
.first()
|
||||
)
|
||||
|
||||
was_new = signoff is None
|
||||
old_result = signoff.result if signoff else None
|
||||
|
||||
if signoff:
|
||||
# Update existing sign-off
|
||||
signoff.result = result_enum
|
||||
signoff.notes = request.notes
|
||||
signoff.updated_at = datetime.utcnow()
|
||||
else:
|
||||
# Create new sign-off
|
||||
signoff = AuditSignOffDB(
|
||||
id=str(uuid4()),
|
||||
session_id=session_id,
|
||||
requirement_id=requirement_id,
|
||||
result=result_enum,
|
||||
notes=request.notes,
|
||||
)
|
||||
db.add(signoff)
|
||||
|
||||
# Create digital signature if requested
|
||||
signature = None
|
||||
if request.sign:
|
||||
timestamp = datetime.utcnow().isoformat()
|
||||
data = f"{result_enum.value}|{requirement_id}|{session.auditor_name}|{timestamp}"
|
||||
signature = hashlib.sha256(data.encode()).hexdigest()
|
||||
signoff.signature_hash = signature
|
||||
signoff.signed_at = datetime.utcnow()
|
||||
signoff.signed_by = session.auditor_name
|
||||
|
||||
# Update session statistics
|
||||
if was_new:
|
||||
session.completed_items += 1
|
||||
|
||||
# Update compliant/non-compliant counts
|
||||
if old_result != result_enum:
|
||||
if old_result == AuditResultEnum.COMPLIANT or old_result == AuditResultEnum.COMPLIANT_WITH_NOTES:
|
||||
session.compliant_count = max(0, session.compliant_count - 1)
|
||||
elif old_result == AuditResultEnum.NON_COMPLIANT:
|
||||
session.non_compliant_count = max(0, session.non_compliant_count - 1)
|
||||
|
||||
if result_enum == AuditResultEnum.COMPLIANT or result_enum == AuditResultEnum.COMPLIANT_WITH_NOTES:
|
||||
session.compliant_count += 1
|
||||
elif result_enum == AuditResultEnum.NON_COMPLIANT:
|
||||
session.non_compliant_count += 1
|
||||
|
||||
# Auto-start session if this is the first sign-off
|
||||
if session.status == AuditSessionStatusEnum.DRAFT:
|
||||
session.status = AuditSessionStatusEnum.IN_PROGRESS
|
||||
session.started_at = datetime.utcnow()
|
||||
|
||||
db.commit()
|
||||
db.refresh(signoff)
|
||||
|
||||
return SignOffResponse(
|
||||
id=signoff.id,
|
||||
session_id=signoff.session_id,
|
||||
requirement_id=signoff.requirement_id,
|
||||
result=signoff.result.value,
|
||||
notes=signoff.notes,
|
||||
is_signed=signoff.signature_hash is not None,
|
||||
signature_hash=signoff.signature_hash,
|
||||
signed_at=signoff.signed_at,
|
||||
signed_by=signoff.signed_by,
|
||||
created_at=signoff.created_at,
|
||||
updated_at=signoff.updated_at,
|
||||
)
|
||||
|
||||
|
||||
@router.get("/checklist/{session_id}/items/{requirement_id}", response_model=SignOffResponse)
|
||||
async def get_sign_off(
|
||||
session_id: str,
|
||||
requirement_id: str,
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""
|
||||
Get the current sign-off status for a specific requirement.
|
||||
"""
|
||||
signoff = (
|
||||
db.query(AuditSignOffDB)
|
||||
.filter(AuditSignOffDB.session_id == session_id)
|
||||
.filter(AuditSignOffDB.requirement_id == requirement_id)
|
||||
.first()
|
||||
)
|
||||
|
||||
if not signoff:
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail=f"No sign-off found for requirement {requirement_id} in session {session_id}"
|
||||
)
|
||||
|
||||
return SignOffResponse(
|
||||
id=signoff.id,
|
||||
session_id=signoff.session_id,
|
||||
requirement_id=signoff.requirement_id,
|
||||
result=signoff.result.value,
|
||||
notes=signoff.notes,
|
||||
is_signed=signoff.signature_hash is not None,
|
||||
signature_hash=signoff.signature_hash,
|
||||
signed_at=signoff.signed_at,
|
||||
signed_by=signoff.signed_by,
|
||||
created_at=signoff.created_at,
|
||||
updated_at=signoff.updated_at,
|
||||
)
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# PDF Report Generation
|
||||
# ============================================================================
|
||||
|
||||
@router.get("/sessions/{session_id}/report/pdf")
|
||||
async def generate_audit_pdf_report(
|
||||
session_id: str,
|
||||
language: str = Query("de", regex="^(de|en)$"),
|
||||
include_signatures: bool = Query(True),
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""
|
||||
Generate a PDF report for an audit session.
|
||||
|
||||
Parameters:
|
||||
- session_id: The audit session ID
|
||||
- language: Output language ('de' or 'en'), default 'de'
|
||||
- include_signatures: Include digital signature verification section
|
||||
|
||||
Returns:
|
||||
- PDF file as streaming response
|
||||
"""
|
||||
from fastapi.responses import StreamingResponse
|
||||
import io
|
||||
from ..services.audit_pdf_generator import AuditPDFGenerator
|
||||
|
||||
# Validate session exists
|
||||
session = db.query(AuditSessionDB).filter(AuditSessionDB.id == session_id).first()
|
||||
if not session:
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail=f"Audit session {session_id} not found"
|
||||
)
|
||||
|
||||
try:
|
||||
generator = AuditPDFGenerator(db)
|
||||
pdf_bytes, filename = generator.generate(
|
||||
session_id=session_id,
|
||||
language=language,
|
||||
include_signatures=include_signatures,
|
||||
)
|
||||
|
||||
return StreamingResponse(
|
||||
io.BytesIO(pdf_bytes),
|
||||
media_type="application/pdf",
|
||||
headers={
|
||||
"Content-Disposition": f"attachment; filename={filename}",
|
||||
}
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to generate PDF report: {e}")
|
||||
raise HTTPException(
|
||||
status_code=500,
|
||||
detail=f"Failed to generate PDF report: {str(e)}"
|
||||
)
|
||||
384
backend/compliance/api/dashboard_routes.py
Normal file
384
backend/compliance/api/dashboard_routes.py
Normal file
@@ -0,0 +1,384 @@
|
||||
"""
|
||||
FastAPI routes for Dashboard, Executive Dashboard, and Reports.
|
||||
|
||||
Endpoints:
|
||||
- /dashboard: Main compliance dashboard
|
||||
- /dashboard/executive: Executive summary for managers
|
||||
- /dashboard/trend: Compliance score trend over time
|
||||
- /score: Quick compliance score
|
||||
- /reports: Report generation
|
||||
"""
|
||||
|
||||
import logging
|
||||
from datetime import datetime, timedelta
|
||||
from calendar import month_abbr
|
||||
from typing import Optional
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from classroom_engine.database import get_db
|
||||
|
||||
from ..db import (
|
||||
RegulationRepository,
|
||||
RequirementRepository,
|
||||
ControlRepository,
|
||||
EvidenceRepository,
|
||||
RiskRepository,
|
||||
)
|
||||
from .schemas import (
|
||||
DashboardResponse,
|
||||
ExecutiveDashboardResponse,
|
||||
TrendDataPoint,
|
||||
RiskSummary,
|
||||
DeadlineItem,
|
||||
TeamWorkloadItem,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter(tags=["compliance-dashboard"])
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Dashboard
|
||||
# ============================================================================
|
||||
|
||||
@router.get("/dashboard", response_model=DashboardResponse)
|
||||
async def get_dashboard(db: Session = Depends(get_db)):
|
||||
"""Get compliance dashboard statistics."""
|
||||
reg_repo = RegulationRepository(db)
|
||||
req_repo = RequirementRepository(db)
|
||||
ctrl_repo = ControlRepository(db)
|
||||
evidence_repo = EvidenceRepository(db)
|
||||
risk_repo = RiskRepository(db)
|
||||
|
||||
# Regulations
|
||||
regulations = reg_repo.get_active()
|
||||
requirements = req_repo.get_all()
|
||||
|
||||
# Controls statistics
|
||||
ctrl_stats = ctrl_repo.get_statistics()
|
||||
controls = ctrl_repo.get_all()
|
||||
|
||||
# Group controls by domain
|
||||
controls_by_domain = {}
|
||||
for ctrl in controls:
|
||||
domain = ctrl.domain.value if ctrl.domain else "unknown"
|
||||
if domain not in controls_by_domain:
|
||||
controls_by_domain[domain] = {"total": 0, "pass": 0, "partial": 0, "fail": 0, "planned": 0}
|
||||
controls_by_domain[domain]["total"] += 1
|
||||
status = ctrl.status.value if ctrl.status else "planned"
|
||||
if status in controls_by_domain[domain]:
|
||||
controls_by_domain[domain][status] += 1
|
||||
|
||||
# Evidence statistics
|
||||
evidence_stats = evidence_repo.get_statistics()
|
||||
|
||||
# Risk statistics
|
||||
risks = risk_repo.get_all()
|
||||
risks_by_level = {"low": 0, "medium": 0, "high": 0, "critical": 0}
|
||||
for risk in risks:
|
||||
level = risk.inherent_risk.value if risk.inherent_risk else "low"
|
||||
if level in risks_by_level:
|
||||
risks_by_level[level] += 1
|
||||
|
||||
# Calculate compliance score - use pre-calculated score from get_statistics()
|
||||
# or compute from by_status dict
|
||||
score = ctrl_stats.get("compliance_score", 0.0)
|
||||
|
||||
return DashboardResponse(
|
||||
compliance_score=round(score, 1),
|
||||
total_regulations=len(regulations),
|
||||
total_requirements=len(requirements),
|
||||
total_controls=ctrl_stats.get("total", 0),
|
||||
controls_by_status=ctrl_stats.get("by_status", {}),
|
||||
controls_by_domain=controls_by_domain,
|
||||
total_evidence=evidence_stats.get("total", 0),
|
||||
evidence_by_status=evidence_stats.get("by_status", {}),
|
||||
total_risks=len(risks),
|
||||
risks_by_level=risks_by_level,
|
||||
recent_activity=[],
|
||||
)
|
||||
|
||||
|
||||
@router.get("/score")
|
||||
async def get_compliance_score(db: Session = Depends(get_db)):
|
||||
"""Get just the compliance score."""
|
||||
ctrl_repo = ControlRepository(db)
|
||||
stats = ctrl_repo.get_statistics()
|
||||
|
||||
total = stats.get("total", 0)
|
||||
passing = stats.get("pass", 0)
|
||||
partial = stats.get("partial", 0)
|
||||
|
||||
if total > 0:
|
||||
score = ((passing + partial * 0.5) / total) * 100
|
||||
else:
|
||||
score = 0
|
||||
|
||||
return {
|
||||
"score": round(score, 1),
|
||||
"total_controls": total,
|
||||
"passing_controls": passing,
|
||||
"partial_controls": partial,
|
||||
}
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Executive Dashboard (Phase 3 - Sprint 1)
|
||||
# ============================================================================
|
||||
|
||||
@router.get("/dashboard/executive", response_model=ExecutiveDashboardResponse)
|
||||
async def get_executive_dashboard(db: Session = Depends(get_db)):
|
||||
"""
|
||||
Get executive dashboard for managers and decision makers.
|
||||
|
||||
Provides:
|
||||
- Traffic light status (green/yellow/red)
|
||||
- Overall compliance score with trend
|
||||
- Top 5 open risks
|
||||
- Upcoming deadlines (control reviews, evidence expiry)
|
||||
- Team workload distribution
|
||||
"""
|
||||
reg_repo = RegulationRepository(db)
|
||||
req_repo = RequirementRepository(db)
|
||||
ctrl_repo = ControlRepository(db)
|
||||
risk_repo = RiskRepository(db)
|
||||
|
||||
# Calculate compliance score
|
||||
ctrl_stats = ctrl_repo.get_statistics()
|
||||
total = ctrl_stats.get("total", 0)
|
||||
passing = ctrl_stats.get("pass", 0)
|
||||
partial = ctrl_stats.get("partial", 0)
|
||||
|
||||
if total > 0:
|
||||
score = ((passing + partial * 0.5) / total) * 100
|
||||
else:
|
||||
score = 0
|
||||
|
||||
# Determine traffic light status
|
||||
if score >= 80:
|
||||
traffic_light = "green"
|
||||
elif score >= 60:
|
||||
traffic_light = "yellow"
|
||||
else:
|
||||
traffic_light = "red"
|
||||
|
||||
# Generate trend data (last 12 months - simulated for now)
|
||||
trend_data = []
|
||||
now = datetime.utcnow()
|
||||
for i in range(11, -1, -1):
|
||||
month_date = now - timedelta(days=i * 30)
|
||||
trend_score = max(0, min(100, score - (11 - i) * 2 + (5 if i > 6 else 0)))
|
||||
trend_data.append(TrendDataPoint(
|
||||
date=month_date.strftime("%Y-%m-%d"),
|
||||
score=round(trend_score, 1),
|
||||
label=month_abbr[month_date.month][:3],
|
||||
))
|
||||
|
||||
# Get top 5 risks (sorted by severity)
|
||||
risks = risk_repo.get_all()
|
||||
risk_priority = {"critical": 4, "high": 3, "medium": 2, "low": 1}
|
||||
sorted_risks = sorted(
|
||||
[r for r in risks if r.status != "mitigated"],
|
||||
key=lambda r: (
|
||||
risk_priority.get(r.inherent_risk.value if r.inherent_risk else "low", 1),
|
||||
r.impact * r.likelihood
|
||||
),
|
||||
reverse=True
|
||||
)[:5]
|
||||
|
||||
top_risks = [
|
||||
RiskSummary(
|
||||
id=r.id,
|
||||
risk_id=r.risk_id,
|
||||
title=r.title,
|
||||
risk_level=r.inherent_risk.value if r.inherent_risk else "medium",
|
||||
owner=r.owner,
|
||||
status=r.status,
|
||||
category=r.category,
|
||||
impact=r.impact,
|
||||
likelihood=r.likelihood,
|
||||
)
|
||||
for r in sorted_risks
|
||||
]
|
||||
|
||||
# Get upcoming deadlines
|
||||
controls = ctrl_repo.get_all()
|
||||
upcoming_deadlines = []
|
||||
today = datetime.utcnow().date()
|
||||
|
||||
for ctrl in controls:
|
||||
if ctrl.next_review_at:
|
||||
review_date = ctrl.next_review_at.date() if hasattr(ctrl.next_review_at, 'date') else ctrl.next_review_at
|
||||
days_remaining = (review_date - today).days
|
||||
|
||||
if days_remaining <= 30:
|
||||
if days_remaining < 0:
|
||||
status = "overdue"
|
||||
elif days_remaining <= 7:
|
||||
status = "at_risk"
|
||||
else:
|
||||
status = "on_track"
|
||||
|
||||
upcoming_deadlines.append(DeadlineItem(
|
||||
id=ctrl.id,
|
||||
title=f"Review: {ctrl.control_id} - {ctrl.title[:30]}",
|
||||
deadline=review_date.isoformat(),
|
||||
days_remaining=days_remaining,
|
||||
type="control_review",
|
||||
status=status,
|
||||
owner=ctrl.owner,
|
||||
))
|
||||
|
||||
upcoming_deadlines.sort(key=lambda x: x.days_remaining)
|
||||
upcoming_deadlines = upcoming_deadlines[:10]
|
||||
|
||||
# Calculate team workload (by owner)
|
||||
owner_workload = {}
|
||||
for ctrl in controls:
|
||||
owner = ctrl.owner or "Unassigned"
|
||||
if owner not in owner_workload:
|
||||
owner_workload[owner] = {"pending": 0, "in_progress": 0, "completed": 0}
|
||||
|
||||
status = ctrl.status.value if ctrl.status else "planned"
|
||||
if status in ["pass"]:
|
||||
owner_workload[owner]["completed"] += 1
|
||||
elif status in ["partial"]:
|
||||
owner_workload[owner]["in_progress"] += 1
|
||||
else:
|
||||
owner_workload[owner]["pending"] += 1
|
||||
|
||||
team_workload = []
|
||||
for name, stats in owner_workload.items():
|
||||
total_tasks = stats["pending"] + stats["in_progress"] + stats["completed"]
|
||||
completion_rate = (stats["completed"] / total_tasks * 100) if total_tasks > 0 else 0
|
||||
team_workload.append(TeamWorkloadItem(
|
||||
name=name,
|
||||
pending_tasks=stats["pending"],
|
||||
in_progress_tasks=stats["in_progress"],
|
||||
completed_tasks=stats["completed"],
|
||||
total_tasks=total_tasks,
|
||||
completion_rate=round(completion_rate, 1),
|
||||
))
|
||||
|
||||
team_workload.sort(key=lambda x: x.total_tasks, reverse=True)
|
||||
|
||||
# Get counts
|
||||
regulations = reg_repo.get_active()
|
||||
requirements = req_repo.get_all()
|
||||
open_risks = len([r for r in risks if r.status != "mitigated"])
|
||||
|
||||
return ExecutiveDashboardResponse(
|
||||
traffic_light_status=traffic_light,
|
||||
overall_score=round(score, 1),
|
||||
score_trend=trend_data,
|
||||
previous_score=trend_data[-2].score if len(trend_data) >= 2 else None,
|
||||
score_change=round(score - trend_data[-2].score, 1) if len(trend_data) >= 2 else None,
|
||||
total_regulations=len(regulations),
|
||||
total_requirements=len(requirements),
|
||||
total_controls=total,
|
||||
open_risks=open_risks,
|
||||
top_risks=top_risks,
|
||||
upcoming_deadlines=upcoming_deadlines,
|
||||
team_workload=team_workload,
|
||||
last_updated=datetime.utcnow().isoformat(),
|
||||
)
|
||||
|
||||
|
||||
@router.get("/dashboard/trend")
|
||||
async def get_compliance_trend(
|
||||
months: int = Query(12, ge=1, le=24, description="Number of months to include"),
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""
|
||||
Get compliance score trend over time.
|
||||
|
||||
Returns monthly compliance scores for trend visualization.
|
||||
"""
|
||||
ctrl_repo = ControlRepository(db)
|
||||
stats = ctrl_repo.get_statistics()
|
||||
total = stats.get("total", 0)
|
||||
passing = stats.get("pass", 0)
|
||||
partial = stats.get("partial", 0)
|
||||
|
||||
current_score = ((passing + partial * 0.5) / total) * 100 if total > 0 else 0
|
||||
|
||||
# Generate simulated historical data
|
||||
trend_data = []
|
||||
now = datetime.utcnow()
|
||||
|
||||
for i in range(months - 1, -1, -1):
|
||||
month_date = now - timedelta(days=i * 30)
|
||||
variation = ((i * 7) % 5) - 2
|
||||
trend_score = max(0, min(100, current_score - (months - 1 - i) * 1.5 + variation))
|
||||
|
||||
trend_data.append({
|
||||
"date": month_date.strftime("%Y-%m-%d"),
|
||||
"score": round(trend_score, 1),
|
||||
"label": f"{month_abbr[month_date.month]} {month_date.year % 100}",
|
||||
"month": month_date.month,
|
||||
"year": month_date.year,
|
||||
})
|
||||
|
||||
return {
|
||||
"current_score": round(current_score, 1),
|
||||
"trend": trend_data,
|
||||
"period_months": months,
|
||||
"generated_at": datetime.utcnow().isoformat(),
|
||||
}
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Reports
|
||||
# ============================================================================
|
||||
|
||||
@router.get("/reports/summary")
|
||||
async def get_summary_report(db: Session = Depends(get_db)):
|
||||
"""Get a quick summary report for the dashboard."""
|
||||
from ..services.report_generator import ComplianceReportGenerator
|
||||
|
||||
generator = ComplianceReportGenerator(db)
|
||||
return generator.generate_summary_report()
|
||||
|
||||
|
||||
@router.get("/reports/{period}")
|
||||
async def generate_period_report(
|
||||
period: str = "monthly",
|
||||
as_of_date: Optional[str] = None,
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""
|
||||
Generate a compliance report for the specified period.
|
||||
|
||||
Args:
|
||||
period: One of 'weekly', 'monthly', 'quarterly', 'yearly'
|
||||
as_of_date: Report date (YYYY-MM-DD format, defaults to today)
|
||||
|
||||
Returns:
|
||||
Complete compliance report
|
||||
"""
|
||||
from ..services.report_generator import ComplianceReportGenerator, ReportPeriod
|
||||
|
||||
# Validate period
|
||||
try:
|
||||
report_period = ReportPeriod(period)
|
||||
except ValueError:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Invalid period '{period}'. Must be one of: weekly, monthly, quarterly, yearly"
|
||||
)
|
||||
|
||||
# Parse date
|
||||
report_date = None
|
||||
if as_of_date:
|
||||
try:
|
||||
report_date = datetime.strptime(as_of_date, "%Y-%m-%d").date()
|
||||
except ValueError:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail="Invalid date format. Use YYYY-MM-DD"
|
||||
)
|
||||
|
||||
generator = ComplianceReportGenerator(db)
|
||||
return generator.generate_report(report_period, report_date)
|
||||
530
backend/compliance/api/evidence_routes.py
Normal file
530
backend/compliance/api/evidence_routes.py
Normal file
@@ -0,0 +1,530 @@
|
||||
"""
|
||||
FastAPI routes for Evidence management.
|
||||
|
||||
Endpoints:
|
||||
- /evidence: Evidence listing and creation
|
||||
- /evidence/upload: Evidence file upload
|
||||
- /evidence/collect: CI/CD evidence collection
|
||||
- /evidence/ci-status: CI/CD evidence status
|
||||
"""
|
||||
|
||||
import logging
|
||||
import os
|
||||
from datetime import datetime, timedelta
|
||||
from typing import Optional
|
||||
from collections import defaultdict
|
||||
import uuid as uuid_module
|
||||
import hashlib
|
||||
import json
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, UploadFile, File, Query
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from classroom_engine.database import get_db
|
||||
|
||||
from ..db import (
|
||||
ControlRepository,
|
||||
EvidenceRepository,
|
||||
EvidenceStatusEnum,
|
||||
)
|
||||
from ..db.models import EvidenceDB, ControlDB
|
||||
from ..services.auto_risk_updater import AutoRiskUpdater
|
||||
from .schemas import (
|
||||
EvidenceCreate, EvidenceResponse, EvidenceListResponse,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter(tags=["compliance-evidence"])
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Evidence
|
||||
# ============================================================================
|
||||
|
||||
@router.get("/evidence", response_model=EvidenceListResponse)
|
||||
async def list_evidence(
|
||||
control_id: Optional[str] = None,
|
||||
evidence_type: Optional[str] = None,
|
||||
status: Optional[str] = None,
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""List evidence with optional filters."""
|
||||
repo = EvidenceRepository(db)
|
||||
|
||||
if control_id:
|
||||
# First get the control UUID
|
||||
ctrl_repo = ControlRepository(db)
|
||||
control = ctrl_repo.get_by_control_id(control_id)
|
||||
if not control:
|
||||
raise HTTPException(status_code=404, detail=f"Control {control_id} not found")
|
||||
evidence = repo.get_by_control(control.id)
|
||||
else:
|
||||
evidence = repo.get_all()
|
||||
|
||||
if evidence_type:
|
||||
evidence = [e for e in evidence if e.evidence_type == evidence_type]
|
||||
|
||||
if status:
|
||||
try:
|
||||
status_enum = EvidenceStatusEnum(status)
|
||||
evidence = [e for e in evidence if e.status == status_enum]
|
||||
except ValueError:
|
||||
pass
|
||||
|
||||
results = [
|
||||
EvidenceResponse(
|
||||
id=e.id,
|
||||
control_id=e.control_id,
|
||||
evidence_type=e.evidence_type,
|
||||
title=e.title,
|
||||
description=e.description,
|
||||
artifact_path=e.artifact_path,
|
||||
artifact_url=e.artifact_url,
|
||||
artifact_hash=e.artifact_hash,
|
||||
file_size_bytes=e.file_size_bytes,
|
||||
mime_type=e.mime_type,
|
||||
valid_from=e.valid_from,
|
||||
valid_until=e.valid_until,
|
||||
status=e.status.value if e.status else None,
|
||||
source=e.source,
|
||||
ci_job_id=e.ci_job_id,
|
||||
uploaded_by=e.uploaded_by,
|
||||
collected_at=e.collected_at,
|
||||
created_at=e.created_at,
|
||||
)
|
||||
for e in evidence
|
||||
]
|
||||
|
||||
return EvidenceListResponse(evidence=results, total=len(results))
|
||||
|
||||
|
||||
@router.post("/evidence", response_model=EvidenceResponse)
|
||||
async def create_evidence(
|
||||
evidence_data: EvidenceCreate,
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""Create new evidence record."""
|
||||
repo = EvidenceRepository(db)
|
||||
|
||||
# Get control UUID
|
||||
ctrl_repo = ControlRepository(db)
|
||||
control = ctrl_repo.get_by_control_id(evidence_data.control_id)
|
||||
if not control:
|
||||
raise HTTPException(status_code=404, detail=f"Control {evidence_data.control_id} not found")
|
||||
|
||||
evidence = repo.create(
|
||||
control_id=control.id,
|
||||
evidence_type=evidence_data.evidence_type,
|
||||
title=evidence_data.title,
|
||||
description=evidence_data.description,
|
||||
artifact_url=evidence_data.artifact_url,
|
||||
valid_from=evidence_data.valid_from,
|
||||
valid_until=evidence_data.valid_until,
|
||||
source=evidence_data.source or "api",
|
||||
ci_job_id=evidence_data.ci_job_id,
|
||||
)
|
||||
db.commit()
|
||||
|
||||
return EvidenceResponse(
|
||||
id=evidence.id,
|
||||
control_id=evidence.control_id,
|
||||
evidence_type=evidence.evidence_type,
|
||||
title=evidence.title,
|
||||
description=evidence.description,
|
||||
artifact_path=evidence.artifact_path,
|
||||
artifact_url=evidence.artifact_url,
|
||||
artifact_hash=evidence.artifact_hash,
|
||||
file_size_bytes=evidence.file_size_bytes,
|
||||
mime_type=evidence.mime_type,
|
||||
valid_from=evidence.valid_from,
|
||||
valid_until=evidence.valid_until,
|
||||
status=evidence.status.value if evidence.status else None,
|
||||
source=evidence.source,
|
||||
ci_job_id=evidence.ci_job_id,
|
||||
uploaded_by=evidence.uploaded_by,
|
||||
collected_at=evidence.collected_at,
|
||||
created_at=evidence.created_at,
|
||||
)
|
||||
|
||||
|
||||
@router.post("/evidence/upload")
|
||||
async def upload_evidence(
|
||||
control_id: str = Query(...),
|
||||
evidence_type: str = Query(...),
|
||||
title: str = Query(...),
|
||||
file: UploadFile = File(...),
|
||||
description: Optional[str] = Query(None),
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""Upload evidence file."""
|
||||
# Get control UUID
|
||||
ctrl_repo = ControlRepository(db)
|
||||
control = ctrl_repo.get_by_control_id(control_id)
|
||||
if not control:
|
||||
raise HTTPException(status_code=404, detail=f"Control {control_id} not found")
|
||||
|
||||
# Create upload directory
|
||||
upload_dir = f"/tmp/compliance_evidence/{control_id}"
|
||||
os.makedirs(upload_dir, exist_ok=True)
|
||||
|
||||
# Save file
|
||||
file_path = os.path.join(upload_dir, file.filename)
|
||||
content = await file.read()
|
||||
|
||||
with open(file_path, "wb") as f:
|
||||
f.write(content)
|
||||
|
||||
# Calculate hash
|
||||
file_hash = hashlib.sha256(content).hexdigest()
|
||||
|
||||
# Create evidence record
|
||||
repo = EvidenceRepository(db)
|
||||
evidence = repo.create(
|
||||
control_id=control.id,
|
||||
evidence_type=evidence_type,
|
||||
title=title,
|
||||
description=description,
|
||||
artifact_path=file_path,
|
||||
artifact_hash=file_hash,
|
||||
file_size_bytes=len(content),
|
||||
mime_type=file.content_type,
|
||||
source="upload",
|
||||
)
|
||||
db.commit()
|
||||
|
||||
return EvidenceResponse(
|
||||
id=evidence.id,
|
||||
control_id=evidence.control_id,
|
||||
evidence_type=evidence.evidence_type,
|
||||
title=evidence.title,
|
||||
description=evidence.description,
|
||||
artifact_path=evidence.artifact_path,
|
||||
artifact_url=evidence.artifact_url,
|
||||
artifact_hash=evidence.artifact_hash,
|
||||
file_size_bytes=evidence.file_size_bytes,
|
||||
mime_type=evidence.mime_type,
|
||||
valid_from=evidence.valid_from,
|
||||
valid_until=evidence.valid_until,
|
||||
status=evidence.status.value if evidence.status else None,
|
||||
source=evidence.source,
|
||||
ci_job_id=evidence.ci_job_id,
|
||||
uploaded_by=evidence.uploaded_by,
|
||||
collected_at=evidence.collected_at,
|
||||
created_at=evidence.created_at,
|
||||
)
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# CI/CD Evidence Collection
|
||||
# ============================================================================
|
||||
|
||||
@router.post("/evidence/collect")
|
||||
async def collect_ci_evidence(
|
||||
source: str = Query(..., description="Evidence source: sast, dependency_scan, sbom, container_scan, test_results"),
|
||||
ci_job_id: str = Query(None, description="CI/CD Job ID for traceability"),
|
||||
ci_job_url: str = Query(None, description="URL to CI/CD job"),
|
||||
report_data: dict = None,
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""
|
||||
Collect evidence from CI/CD pipeline.
|
||||
|
||||
This endpoint is designed to be called from CI/CD workflows (GitHub Actions,
|
||||
GitLab CI, Jenkins, etc.) to automatically collect compliance evidence.
|
||||
|
||||
Supported sources:
|
||||
- sast: Static Application Security Testing (Semgrep, SonarQube, etc.)
|
||||
- dependency_scan: Dependency vulnerability scanning (Trivy, Grype, Snyk)
|
||||
- sbom: Software Bill of Materials (CycloneDX, SPDX)
|
||||
- container_scan: Container image scanning (Trivy, Grype)
|
||||
- test_results: Test coverage and results
|
||||
- secret_scan: Secret detection (Gitleaks, TruffleHog)
|
||||
- code_review: Code review metrics
|
||||
"""
|
||||
# Map source to control_id
|
||||
SOURCE_CONTROL_MAP = {
|
||||
"sast": "SDLC-001",
|
||||
"dependency_scan": "SDLC-002",
|
||||
"secret_scan": "SDLC-003",
|
||||
"code_review": "SDLC-004",
|
||||
"sbom": "SDLC-005",
|
||||
"container_scan": "SDLC-006",
|
||||
"test_results": "AUD-001",
|
||||
}
|
||||
|
||||
if source not in SOURCE_CONTROL_MAP:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Unknown source '{source}'. Supported: {list(SOURCE_CONTROL_MAP.keys())}"
|
||||
)
|
||||
|
||||
control_id = SOURCE_CONTROL_MAP[source]
|
||||
|
||||
# Get control
|
||||
ctrl_repo = ControlRepository(db)
|
||||
control = ctrl_repo.get_by_control_id(control_id)
|
||||
if not control:
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail=f"Control {control_id} not found. Please seed the database first."
|
||||
)
|
||||
|
||||
# Parse and validate report data
|
||||
report_json = json.dumps(report_data) if report_data else "{}"
|
||||
report_hash = hashlib.sha256(report_json.encode()).hexdigest()
|
||||
|
||||
# Determine evidence status based on report content
|
||||
evidence_status = "valid"
|
||||
findings_count = 0
|
||||
critical_findings = 0
|
||||
|
||||
if report_data:
|
||||
# Try to extract findings from common report formats
|
||||
if isinstance(report_data, dict):
|
||||
# Semgrep format
|
||||
if "results" in report_data:
|
||||
findings_count = len(report_data.get("results", []))
|
||||
critical_findings = len([
|
||||
r for r in report_data.get("results", [])
|
||||
if r.get("extra", {}).get("severity", "").upper() in ["CRITICAL", "HIGH"]
|
||||
])
|
||||
|
||||
# Trivy format
|
||||
elif "Results" in report_data:
|
||||
for result in report_data.get("Results", []):
|
||||
vulns = result.get("Vulnerabilities", [])
|
||||
findings_count += len(vulns)
|
||||
critical_findings += len([
|
||||
v for v in vulns
|
||||
if v.get("Severity", "").upper() in ["CRITICAL", "HIGH"]
|
||||
])
|
||||
|
||||
# Generic findings array
|
||||
elif "findings" in report_data:
|
||||
findings_count = len(report_data.get("findings", []))
|
||||
|
||||
# SBOM format - just count components
|
||||
elif "components" in report_data:
|
||||
findings_count = len(report_data.get("components", []))
|
||||
|
||||
# If critical findings exist, mark as failed
|
||||
if critical_findings > 0:
|
||||
evidence_status = "failed"
|
||||
|
||||
# Create evidence title
|
||||
title = f"{source.upper()} Report - {datetime.now().strftime('%Y-%m-%d %H:%M')}"
|
||||
description = f"Automatically collected from CI/CD pipeline"
|
||||
if findings_count > 0:
|
||||
description += f"\n- Total findings: {findings_count}"
|
||||
if critical_findings > 0:
|
||||
description += f"\n- Critical/High findings: {critical_findings}"
|
||||
if ci_job_id:
|
||||
description += f"\n- CI Job ID: {ci_job_id}"
|
||||
if ci_job_url:
|
||||
description += f"\n- CI Job URL: {ci_job_url}"
|
||||
|
||||
# Store report file
|
||||
upload_dir = f"/tmp/compliance_evidence/ci/{source}"
|
||||
os.makedirs(upload_dir, exist_ok=True)
|
||||
file_name = f"{source}_{datetime.now().strftime('%Y%m%d_%H%M%S')}_{report_hash[:8]}.json"
|
||||
file_path = os.path.join(upload_dir, file_name)
|
||||
|
||||
with open(file_path, "w") as f:
|
||||
json.dump(report_data or {}, f, indent=2)
|
||||
|
||||
# Create evidence record directly
|
||||
evidence = EvidenceDB(
|
||||
id=str(uuid_module.uuid4()),
|
||||
control_id=control.id,
|
||||
evidence_type=f"ci_{source}",
|
||||
title=title,
|
||||
description=description,
|
||||
artifact_path=file_path,
|
||||
artifact_hash=report_hash,
|
||||
file_size_bytes=len(report_json),
|
||||
mime_type="application/json",
|
||||
source="ci_pipeline",
|
||||
ci_job_id=ci_job_id,
|
||||
valid_from=datetime.utcnow(),
|
||||
valid_until=datetime.utcnow() + timedelta(days=90),
|
||||
status=EvidenceStatusEnum(evidence_status),
|
||||
)
|
||||
db.add(evidence)
|
||||
db.commit()
|
||||
db.refresh(evidence)
|
||||
|
||||
# =========================================================================
|
||||
# AUTOMATIC RISK UPDATE
|
||||
# Update Control status and linked Risks based on findings
|
||||
# =========================================================================
|
||||
risk_update_result = None
|
||||
try:
|
||||
# Extract detailed findings for risk assessment
|
||||
findings_detail = {
|
||||
"critical": 0,
|
||||
"high": 0,
|
||||
"medium": 0,
|
||||
"low": 0,
|
||||
}
|
||||
|
||||
if report_data:
|
||||
# Semgrep format
|
||||
if "results" in report_data:
|
||||
for r in report_data.get("results", []):
|
||||
severity = r.get("extra", {}).get("severity", "").upper()
|
||||
if severity == "CRITICAL":
|
||||
findings_detail["critical"] += 1
|
||||
elif severity == "HIGH":
|
||||
findings_detail["high"] += 1
|
||||
elif severity == "MEDIUM":
|
||||
findings_detail["medium"] += 1
|
||||
elif severity in ["LOW", "INFO"]:
|
||||
findings_detail["low"] += 1
|
||||
|
||||
# Trivy format
|
||||
elif "Results" in report_data:
|
||||
for result in report_data.get("Results", []):
|
||||
for v in result.get("Vulnerabilities", []):
|
||||
severity = v.get("Severity", "").upper()
|
||||
if severity == "CRITICAL":
|
||||
findings_detail["critical"] += 1
|
||||
elif severity == "HIGH":
|
||||
findings_detail["high"] += 1
|
||||
elif severity == "MEDIUM":
|
||||
findings_detail["medium"] += 1
|
||||
elif severity == "LOW":
|
||||
findings_detail["low"] += 1
|
||||
|
||||
# Generic findings with severity
|
||||
elif "findings" in report_data:
|
||||
for f in report_data.get("findings", []):
|
||||
severity = f.get("severity", "").upper()
|
||||
if severity == "CRITICAL":
|
||||
findings_detail["critical"] += 1
|
||||
elif severity == "HIGH":
|
||||
findings_detail["high"] += 1
|
||||
elif severity == "MEDIUM":
|
||||
findings_detail["medium"] += 1
|
||||
else:
|
||||
findings_detail["low"] += 1
|
||||
|
||||
# Use AutoRiskUpdater to update Control status and Risks
|
||||
auto_updater = AutoRiskUpdater(db)
|
||||
risk_update_result = auto_updater.process_evidence_collect_request(
|
||||
tool=source,
|
||||
control_id=control_id,
|
||||
evidence_type=f"ci_{source}",
|
||||
timestamp=datetime.utcnow().isoformat(),
|
||||
commit_sha=report_data.get("commit_sha", "unknown") if report_data else "unknown",
|
||||
ci_job_id=ci_job_id,
|
||||
findings=findings_detail,
|
||||
)
|
||||
|
||||
logger.info(f"Auto-risk update completed for {control_id}: "
|
||||
f"control_updated={risk_update_result.control_updated}, "
|
||||
f"risks_affected={len(risk_update_result.risks_affected)}")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Auto-risk update failed for {control_id}: {str(e)}")
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"evidence_id": evidence.id,
|
||||
"control_id": control_id,
|
||||
"source": source,
|
||||
"status": evidence_status,
|
||||
"findings_count": findings_count,
|
||||
"critical_findings": critical_findings,
|
||||
"artifact_path": file_path,
|
||||
"message": f"Evidence collected successfully for control {control_id}",
|
||||
"auto_risk_update": {
|
||||
"enabled": True,
|
||||
"control_updated": risk_update_result.control_updated if risk_update_result else False,
|
||||
"old_status": risk_update_result.old_status if risk_update_result else None,
|
||||
"new_status": risk_update_result.new_status if risk_update_result else None,
|
||||
"risks_affected": risk_update_result.risks_affected if risk_update_result else [],
|
||||
"alerts_generated": risk_update_result.alerts_generated if risk_update_result else [],
|
||||
} if risk_update_result else {"enabled": False, "error": "Auto-update skipped"},
|
||||
}
|
||||
|
||||
|
||||
@router.get("/evidence/ci-status")
|
||||
async def get_ci_evidence_status(
|
||||
control_id: str = Query(None, description="Filter by control ID"),
|
||||
days: int = Query(30, description="Look back N days"),
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""
|
||||
Get CI/CD evidence collection status.
|
||||
|
||||
Returns overview of recent evidence collected from CI/CD pipelines,
|
||||
useful for dashboards and monitoring.
|
||||
"""
|
||||
cutoff_date = datetime.utcnow() - timedelta(days=days)
|
||||
|
||||
# Build query
|
||||
query = db.query(EvidenceDB).filter(
|
||||
EvidenceDB.source == "ci_pipeline",
|
||||
EvidenceDB.collected_at >= cutoff_date,
|
||||
)
|
||||
|
||||
if control_id:
|
||||
ctrl_repo = ControlRepository(db)
|
||||
control = ctrl_repo.get_by_control_id(control_id)
|
||||
if control:
|
||||
query = query.filter(EvidenceDB.control_id == control.id)
|
||||
|
||||
evidence_list = query.order_by(EvidenceDB.collected_at.desc()).limit(100).all()
|
||||
|
||||
# Group by control and calculate stats
|
||||
control_stats = defaultdict(lambda: {
|
||||
"total": 0,
|
||||
"valid": 0,
|
||||
"failed": 0,
|
||||
"last_collected": None,
|
||||
"evidence": [],
|
||||
})
|
||||
|
||||
for e in evidence_list:
|
||||
# Get control_id string
|
||||
control = db.query(ControlDB).filter(ControlDB.id == e.control_id).first()
|
||||
ctrl_id = control.control_id if control else "unknown"
|
||||
|
||||
stats = control_stats[ctrl_id]
|
||||
stats["total"] += 1
|
||||
if e.status:
|
||||
if e.status.value == "valid":
|
||||
stats["valid"] += 1
|
||||
elif e.status.value == "failed":
|
||||
stats["failed"] += 1
|
||||
if not stats["last_collected"] or e.collected_at > stats["last_collected"]:
|
||||
stats["last_collected"] = e.collected_at
|
||||
|
||||
# Add evidence summary
|
||||
stats["evidence"].append({
|
||||
"id": e.id,
|
||||
"type": e.evidence_type,
|
||||
"status": e.status.value if e.status else None,
|
||||
"collected_at": e.collected_at.isoformat() if e.collected_at else None,
|
||||
"ci_job_id": e.ci_job_id,
|
||||
})
|
||||
|
||||
# Convert to list and sort
|
||||
result = []
|
||||
for ctrl_id, stats in control_stats.items():
|
||||
result.append({
|
||||
"control_id": ctrl_id,
|
||||
"total_evidence": stats["total"],
|
||||
"valid_count": stats["valid"],
|
||||
"failed_count": stats["failed"],
|
||||
"last_collected": stats["last_collected"].isoformat() if stats["last_collected"] else None,
|
||||
"recent_evidence": stats["evidence"][:5],
|
||||
})
|
||||
|
||||
result.sort(key=lambda x: x["last_collected"] or "", reverse=True)
|
||||
|
||||
return {
|
||||
"period_days": days,
|
||||
"total_evidence": len(evidence_list),
|
||||
"controls": result,
|
||||
}
|
||||
1649
backend/compliance/api/isms_routes.py
Normal file
1649
backend/compliance/api/isms_routes.py
Normal file
File diff suppressed because it is too large
Load Diff
250
backend/compliance/api/module_routes.py
Normal file
250
backend/compliance/api/module_routes.py
Normal file
@@ -0,0 +1,250 @@
|
||||
"""
|
||||
FastAPI routes for Service Module Registry.
|
||||
|
||||
Endpoints:
|
||||
- /modules: Module listing and management
|
||||
- /modules/overview: Module compliance overview
|
||||
- /modules/{module_id}: Module details
|
||||
- /modules/seed: Seed modules from data
|
||||
- /modules/{module_id}/regulations: Add regulation mapping
|
||||
"""
|
||||
|
||||
import logging
|
||||
from typing import Optional
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from classroom_engine.database import get_db
|
||||
|
||||
from ..db import RegulationRepository
|
||||
from .schemas import (
|
||||
ServiceModuleResponse, ServiceModuleListResponse, ServiceModuleDetailResponse,
|
||||
ModuleRegulationMappingCreate, ModuleRegulationMappingResponse,
|
||||
ModuleSeedRequest, ModuleSeedResponse, ModuleComplianceOverview,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter(tags=["compliance-modules"])
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Service Module Registry
|
||||
# ============================================================================
|
||||
|
||||
@router.get("/modules", response_model=ServiceModuleListResponse)
|
||||
async def list_modules(
|
||||
service_type: Optional[str] = None,
|
||||
criticality: Optional[str] = None,
|
||||
processes_pii: Optional[bool] = None,
|
||||
ai_components: Optional[bool] = None,
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""List all service modules with optional filters."""
|
||||
from ..db.repository import ServiceModuleRepository
|
||||
|
||||
repo = ServiceModuleRepository(db)
|
||||
modules = repo.get_all(
|
||||
service_type=service_type,
|
||||
criticality=criticality,
|
||||
processes_pii=processes_pii,
|
||||
ai_components=ai_components,
|
||||
)
|
||||
|
||||
# Count regulations and risks for each module
|
||||
results = []
|
||||
for m in modules:
|
||||
reg_count = len(m.regulation_mappings) if m.regulation_mappings else 0
|
||||
risk_count = len(m.module_risks) if m.module_risks else 0
|
||||
|
||||
results.append(ServiceModuleResponse(
|
||||
id=m.id,
|
||||
name=m.name,
|
||||
display_name=m.display_name,
|
||||
description=m.description,
|
||||
service_type=m.service_type.value if m.service_type else None,
|
||||
port=m.port,
|
||||
technology_stack=m.technology_stack or [],
|
||||
repository_path=m.repository_path,
|
||||
docker_image=m.docker_image,
|
||||
data_categories=m.data_categories or [],
|
||||
processes_pii=m.processes_pii,
|
||||
processes_health_data=m.processes_health_data,
|
||||
ai_components=m.ai_components,
|
||||
criticality=m.criticality,
|
||||
owner_team=m.owner_team,
|
||||
owner_contact=m.owner_contact,
|
||||
is_active=m.is_active,
|
||||
compliance_score=m.compliance_score,
|
||||
last_compliance_check=m.last_compliance_check,
|
||||
created_at=m.created_at,
|
||||
updated_at=m.updated_at,
|
||||
regulation_count=reg_count,
|
||||
risk_count=risk_count,
|
||||
))
|
||||
|
||||
return ServiceModuleListResponse(modules=results, total=len(results))
|
||||
|
||||
|
||||
@router.get("/modules/overview", response_model=ModuleComplianceOverview)
|
||||
async def get_modules_overview(db: Session = Depends(get_db)):
|
||||
"""Get overview statistics for all modules."""
|
||||
from ..db.repository import ServiceModuleRepository
|
||||
|
||||
repo = ServiceModuleRepository(db)
|
||||
overview = repo.get_overview()
|
||||
|
||||
return ModuleComplianceOverview(**overview)
|
||||
|
||||
|
||||
@router.get("/modules/{module_id}", response_model=ServiceModuleDetailResponse)
|
||||
async def get_module(module_id: str, db: Session = Depends(get_db)):
|
||||
"""Get a specific module with its regulations and risks."""
|
||||
from ..db.repository import ServiceModuleRepository
|
||||
|
||||
repo = ServiceModuleRepository(db)
|
||||
module = repo.get_with_regulations(module_id)
|
||||
|
||||
if not module:
|
||||
# Try by name
|
||||
module = repo.get_by_name(module_id)
|
||||
if module:
|
||||
module = repo.get_with_regulations(module.id)
|
||||
|
||||
if not module:
|
||||
raise HTTPException(status_code=404, detail=f"Module {module_id} not found")
|
||||
|
||||
# Build regulation list
|
||||
regulations = []
|
||||
for mapping in (module.regulation_mappings or []):
|
||||
reg = mapping.regulation
|
||||
if reg:
|
||||
regulations.append({
|
||||
"code": reg.code,
|
||||
"name": reg.name,
|
||||
"relevance_level": mapping.relevance_level.value if mapping.relevance_level else "medium",
|
||||
"notes": mapping.notes,
|
||||
})
|
||||
|
||||
# Build risk list
|
||||
risks = []
|
||||
for mr in (module.module_risks or []):
|
||||
risk = mr.risk
|
||||
if risk:
|
||||
risks.append({
|
||||
"risk_id": risk.risk_id,
|
||||
"title": risk.title,
|
||||
"inherent_risk": risk.inherent_risk.value if risk.inherent_risk else None,
|
||||
"module_risk_level": mr.module_risk_level.value if mr.module_risk_level else None,
|
||||
})
|
||||
|
||||
return ServiceModuleDetailResponse(
|
||||
id=module.id,
|
||||
name=module.name,
|
||||
display_name=module.display_name,
|
||||
description=module.description,
|
||||
service_type=module.service_type.value if module.service_type else None,
|
||||
port=module.port,
|
||||
technology_stack=module.technology_stack or [],
|
||||
repository_path=module.repository_path,
|
||||
docker_image=module.docker_image,
|
||||
data_categories=module.data_categories or [],
|
||||
processes_pii=module.processes_pii,
|
||||
processes_health_data=module.processes_health_data,
|
||||
ai_components=module.ai_components,
|
||||
criticality=module.criticality,
|
||||
owner_team=module.owner_team,
|
||||
owner_contact=module.owner_contact,
|
||||
is_active=module.is_active,
|
||||
compliance_score=module.compliance_score,
|
||||
last_compliance_check=module.last_compliance_check,
|
||||
created_at=module.created_at,
|
||||
updated_at=module.updated_at,
|
||||
regulation_count=len(regulations),
|
||||
risk_count=len(risks),
|
||||
regulations=regulations,
|
||||
risks=risks,
|
||||
)
|
||||
|
||||
|
||||
@router.post("/modules/seed", response_model=ModuleSeedResponse)
|
||||
async def seed_modules(
|
||||
request: ModuleSeedRequest,
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""Seed service modules from predefined data."""
|
||||
from classroom_engine.database import engine
|
||||
from ..db.models import ServiceModuleDB, ModuleRegulationMappingDB, ModuleRiskDB
|
||||
from ..db.repository import ServiceModuleRepository
|
||||
from ..data.service_modules import BREAKPILOT_SERVICES
|
||||
|
||||
try:
|
||||
# Ensure tables exist
|
||||
ServiceModuleDB.__table__.create(engine, checkfirst=True)
|
||||
ModuleRegulationMappingDB.__table__.create(engine, checkfirst=True)
|
||||
ModuleRiskDB.__table__.create(engine, checkfirst=True)
|
||||
|
||||
repo = ServiceModuleRepository(db)
|
||||
result = repo.seed_from_data(BREAKPILOT_SERVICES, force=request.force)
|
||||
|
||||
return ModuleSeedResponse(
|
||||
success=True,
|
||||
message=f"Seeded {result['modules_created']} modules with {result['mappings_created']} regulation mappings",
|
||||
modules_created=result["modules_created"],
|
||||
mappings_created=result["mappings_created"],
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error(f"Module seeding failed: {e}")
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
|
||||
|
||||
@router.post("/modules/{module_id}/regulations", response_model=ModuleRegulationMappingResponse)
|
||||
async def add_module_regulation(
|
||||
module_id: str,
|
||||
mapping: ModuleRegulationMappingCreate,
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""Add a regulation mapping to a module."""
|
||||
from ..db.repository import ServiceModuleRepository
|
||||
|
||||
repo = ServiceModuleRepository(db)
|
||||
module = repo.get_by_id(module_id)
|
||||
|
||||
if not module:
|
||||
module = repo.get_by_name(module_id)
|
||||
|
||||
if not module:
|
||||
raise HTTPException(status_code=404, detail=f"Module {module_id} not found")
|
||||
|
||||
# Verify regulation exists
|
||||
reg_repo = RegulationRepository(db)
|
||||
regulation = reg_repo.get_by_id(mapping.regulation_id)
|
||||
if not regulation:
|
||||
regulation = reg_repo.get_by_code(mapping.regulation_id)
|
||||
if not regulation:
|
||||
raise HTTPException(status_code=404, detail=f"Regulation {mapping.regulation_id} not found")
|
||||
|
||||
try:
|
||||
new_mapping = repo.add_regulation_mapping(
|
||||
module_id=module.id,
|
||||
regulation_id=regulation.id,
|
||||
relevance_level=mapping.relevance_level,
|
||||
notes=mapping.notes,
|
||||
applicable_articles=mapping.applicable_articles,
|
||||
)
|
||||
|
||||
return ModuleRegulationMappingResponse(
|
||||
id=new_mapping.id,
|
||||
module_id=new_mapping.module_id,
|
||||
regulation_id=new_mapping.regulation_id,
|
||||
relevance_level=new_mapping.relevance_level.value if new_mapping.relevance_level else "medium",
|
||||
notes=new_mapping.notes,
|
||||
applicable_articles=new_mapping.applicable_articles,
|
||||
module_name=module.name,
|
||||
regulation_code=regulation.code,
|
||||
regulation_name=regulation.name,
|
||||
created_at=new_mapping.created_at,
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to add regulation mapping: {e}")
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
200
backend/compliance/api/risk_routes.py
Normal file
200
backend/compliance/api/risk_routes.py
Normal file
@@ -0,0 +1,200 @@
|
||||
"""
|
||||
FastAPI routes for Risk management.
|
||||
|
||||
Endpoints:
|
||||
- /risks: Risk listing and CRUD
|
||||
- /risks/matrix: Risk matrix visualization
|
||||
"""
|
||||
|
||||
import logging
|
||||
from typing import Optional
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from classroom_engine.database import get_db
|
||||
|
||||
from ..db import RiskRepository, RiskLevelEnum
|
||||
from .schemas import (
|
||||
RiskCreate, RiskUpdate, RiskResponse, RiskListResponse, RiskMatrixResponse,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter(tags=["compliance-risks"])
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Risks
|
||||
# ============================================================================
|
||||
|
||||
@router.get("/risks", response_model=RiskListResponse)
|
||||
async def list_risks(
|
||||
category: Optional[str] = None,
|
||||
status: Optional[str] = None,
|
||||
risk_level: Optional[str] = None,
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""List risks with optional filters."""
|
||||
repo = RiskRepository(db)
|
||||
risks = repo.get_all()
|
||||
|
||||
if category:
|
||||
risks = [r for r in risks if r.category == category]
|
||||
|
||||
if status:
|
||||
risks = [r for r in risks if r.status == status]
|
||||
|
||||
if risk_level:
|
||||
try:
|
||||
level = RiskLevelEnum(risk_level)
|
||||
risks = [r for r in risks if r.inherent_risk == level]
|
||||
except ValueError:
|
||||
pass
|
||||
|
||||
results = [
|
||||
RiskResponse(
|
||||
id=r.id,
|
||||
risk_id=r.risk_id,
|
||||
title=r.title,
|
||||
description=r.description,
|
||||
category=r.category,
|
||||
likelihood=r.likelihood,
|
||||
impact=r.impact,
|
||||
inherent_risk=r.inherent_risk.value if r.inherent_risk else None,
|
||||
mitigating_controls=r.mitigating_controls,
|
||||
residual_likelihood=r.residual_likelihood,
|
||||
residual_impact=r.residual_impact,
|
||||
residual_risk=r.residual_risk.value if r.residual_risk else None,
|
||||
owner=r.owner,
|
||||
status=r.status,
|
||||
treatment_plan=r.treatment_plan,
|
||||
identified_date=r.identified_date,
|
||||
review_date=r.review_date,
|
||||
last_assessed_at=r.last_assessed_at,
|
||||
created_at=r.created_at,
|
||||
updated_at=r.updated_at,
|
||||
)
|
||||
for r in risks
|
||||
]
|
||||
|
||||
return RiskListResponse(risks=results, total=len(results))
|
||||
|
||||
|
||||
@router.post("/risks", response_model=RiskResponse)
|
||||
async def create_risk(
|
||||
risk_data: RiskCreate,
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""Create a new risk."""
|
||||
repo = RiskRepository(db)
|
||||
risk = repo.create(
|
||||
risk_id=risk_data.risk_id,
|
||||
title=risk_data.title,
|
||||
description=risk_data.description,
|
||||
category=risk_data.category,
|
||||
likelihood=risk_data.likelihood,
|
||||
impact=risk_data.impact,
|
||||
mitigating_controls=risk_data.mitigating_controls,
|
||||
owner=risk_data.owner,
|
||||
treatment_plan=risk_data.treatment_plan,
|
||||
)
|
||||
db.commit()
|
||||
|
||||
return RiskResponse(
|
||||
id=risk.id,
|
||||
risk_id=risk.risk_id,
|
||||
title=risk.title,
|
||||
description=risk.description,
|
||||
category=risk.category,
|
||||
likelihood=risk.likelihood,
|
||||
impact=risk.impact,
|
||||
inherent_risk=risk.inherent_risk.value if risk.inherent_risk else None,
|
||||
mitigating_controls=risk.mitigating_controls,
|
||||
residual_likelihood=risk.residual_likelihood,
|
||||
residual_impact=risk.residual_impact,
|
||||
residual_risk=risk.residual_risk.value if risk.residual_risk else None,
|
||||
owner=risk.owner,
|
||||
status=risk.status,
|
||||
treatment_plan=risk.treatment_plan,
|
||||
identified_date=risk.identified_date,
|
||||
review_date=risk.review_date,
|
||||
last_assessed_at=risk.last_assessed_at,
|
||||
created_at=risk.created_at,
|
||||
updated_at=risk.updated_at,
|
||||
)
|
||||
|
||||
|
||||
@router.put("/risks/{risk_id}", response_model=RiskResponse)
|
||||
async def update_risk(
|
||||
risk_id: str,
|
||||
update: RiskUpdate,
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""Update a risk."""
|
||||
repo = RiskRepository(db)
|
||||
risk = repo.get_by_risk_id(risk_id)
|
||||
if not risk:
|
||||
raise HTTPException(status_code=404, detail=f"Risk {risk_id} not found")
|
||||
|
||||
update_data = update.model_dump(exclude_unset=True)
|
||||
updated = repo.update(risk.id, **update_data)
|
||||
db.commit()
|
||||
|
||||
return RiskResponse(
|
||||
id=updated.id,
|
||||
risk_id=updated.risk_id,
|
||||
title=updated.title,
|
||||
description=updated.description,
|
||||
category=updated.category,
|
||||
likelihood=updated.likelihood,
|
||||
impact=updated.impact,
|
||||
inherent_risk=updated.inherent_risk.value if updated.inherent_risk else None,
|
||||
mitigating_controls=updated.mitigating_controls,
|
||||
residual_likelihood=updated.residual_likelihood,
|
||||
residual_impact=updated.residual_impact,
|
||||
residual_risk=updated.residual_risk.value if updated.residual_risk else None,
|
||||
owner=updated.owner,
|
||||
status=updated.status,
|
||||
treatment_plan=updated.treatment_plan,
|
||||
identified_date=updated.identified_date,
|
||||
review_date=updated.review_date,
|
||||
last_assessed_at=updated.last_assessed_at,
|
||||
created_at=updated.created_at,
|
||||
updated_at=updated.updated_at,
|
||||
)
|
||||
|
||||
|
||||
@router.get("/risks/matrix", response_model=RiskMatrixResponse)
|
||||
async def get_risk_matrix(db: Session = Depends(get_db)):
|
||||
"""Get risk matrix data for visualization."""
|
||||
repo = RiskRepository(db)
|
||||
matrix_data = repo.get_risk_matrix()
|
||||
risks = repo.get_all()
|
||||
|
||||
risk_responses = [
|
||||
RiskResponse(
|
||||
id=r.id,
|
||||
risk_id=r.risk_id,
|
||||
title=r.title,
|
||||
description=r.description,
|
||||
category=r.category,
|
||||
likelihood=r.likelihood,
|
||||
impact=r.impact,
|
||||
inherent_risk=r.inherent_risk.value if r.inherent_risk else None,
|
||||
mitigating_controls=r.mitigating_controls,
|
||||
residual_likelihood=r.residual_likelihood,
|
||||
residual_impact=r.residual_impact,
|
||||
residual_risk=r.residual_risk.value if r.residual_risk else None,
|
||||
owner=r.owner,
|
||||
status=r.status,
|
||||
treatment_plan=r.treatment_plan,
|
||||
identified_date=r.identified_date,
|
||||
review_date=r.review_date,
|
||||
last_assessed_at=r.last_assessed_at,
|
||||
created_at=r.created_at,
|
||||
updated_at=r.updated_at,
|
||||
)
|
||||
for r in risks
|
||||
]
|
||||
|
||||
return RiskMatrixResponse(matrix=matrix_data, risks=risk_responses)
|
||||
914
backend/compliance/api/routes.py
Normal file
914
backend/compliance/api/routes.py
Normal file
@@ -0,0 +1,914 @@
|
||||
"""
|
||||
FastAPI routes for Compliance module.
|
||||
|
||||
Endpoints:
|
||||
- /regulations: Manage regulations
|
||||
- /requirements: Manage requirements
|
||||
- /controls: Manage controls
|
||||
- /mappings: Requirement-Control mappings
|
||||
- /evidence: Evidence management
|
||||
- /risks: Risk management
|
||||
- /dashboard: Dashboard statistics
|
||||
- /export: Audit export
|
||||
"""
|
||||
|
||||
import logging
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
import os
|
||||
from datetime import datetime, timedelta
|
||||
from typing import Optional, List
|
||||
|
||||
from pydantic import BaseModel
|
||||
from fastapi import APIRouter, Depends, HTTPException, UploadFile, File, Query, BackgroundTasks
|
||||
from fastapi.responses import FileResponse
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from classroom_engine.database import get_db
|
||||
|
||||
from ..db import (
|
||||
RegulationRepository,
|
||||
RequirementRepository,
|
||||
ControlRepository,
|
||||
EvidenceRepository,
|
||||
RiskRepository,
|
||||
AuditExportRepository,
|
||||
ControlStatusEnum,
|
||||
ControlDomainEnum,
|
||||
RiskLevelEnum,
|
||||
EvidenceStatusEnum,
|
||||
)
|
||||
from ..db.models import EvidenceDB, ControlDB
|
||||
from ..services.seeder import ComplianceSeeder
|
||||
from ..services.export_generator import AuditExportGenerator
|
||||
from ..services.auto_risk_updater import AutoRiskUpdater, ScanType
|
||||
from .schemas import (
|
||||
RegulationCreate, RegulationResponse, RegulationListResponse,
|
||||
RequirementCreate, RequirementResponse, RequirementListResponse,
|
||||
ControlCreate, ControlUpdate, ControlResponse, ControlListResponse, ControlReviewRequest,
|
||||
MappingCreate, MappingResponse, MappingListResponse,
|
||||
ExportRequest, ExportResponse, ExportListResponse,
|
||||
SeedRequest, SeedResponse,
|
||||
# Pagination schemas
|
||||
PaginationMeta, PaginatedRequirementResponse, PaginatedControlResponse,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter(prefix="/compliance", tags=["compliance"])
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Regulations
|
||||
# ============================================================================
|
||||
|
||||
@router.get("/regulations", response_model=RegulationListResponse)
|
||||
async def list_regulations(
|
||||
is_active: Optional[bool] = None,
|
||||
regulation_type: Optional[str] = None,
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""List all regulations."""
|
||||
repo = RegulationRepository(db)
|
||||
if is_active is not None:
|
||||
regulations = repo.get_active() if is_active else repo.get_all()
|
||||
else:
|
||||
regulations = repo.get_all()
|
||||
|
||||
if regulation_type:
|
||||
from ..db.models import RegulationTypeEnum
|
||||
try:
|
||||
reg_type = RegulationTypeEnum(regulation_type)
|
||||
regulations = [r for r in regulations if r.regulation_type == reg_type]
|
||||
except ValueError:
|
||||
pass
|
||||
|
||||
# Add requirement counts
|
||||
req_repo = RequirementRepository(db)
|
||||
results = []
|
||||
for reg in regulations:
|
||||
reqs = req_repo.get_by_regulation(reg.id)
|
||||
reg_dict = {
|
||||
"id": reg.id,
|
||||
"code": reg.code,
|
||||
"name": reg.name,
|
||||
"full_name": reg.full_name,
|
||||
"regulation_type": reg.regulation_type.value if reg.regulation_type else None,
|
||||
"source_url": reg.source_url,
|
||||
"local_pdf_path": reg.local_pdf_path,
|
||||
"effective_date": reg.effective_date,
|
||||
"description": reg.description,
|
||||
"is_active": reg.is_active,
|
||||
"created_at": reg.created_at,
|
||||
"updated_at": reg.updated_at,
|
||||
"requirement_count": len(reqs),
|
||||
}
|
||||
results.append(RegulationResponse(**reg_dict))
|
||||
|
||||
return RegulationListResponse(regulations=results, total=len(results))
|
||||
|
||||
|
||||
@router.get("/regulations/{code}", response_model=RegulationResponse)
|
||||
async def get_regulation(code: str, db: Session = Depends(get_db)):
|
||||
"""Get a specific regulation by code."""
|
||||
repo = RegulationRepository(db)
|
||||
regulation = repo.get_by_code(code)
|
||||
if not regulation:
|
||||
raise HTTPException(status_code=404, detail=f"Regulation {code} not found")
|
||||
|
||||
req_repo = RequirementRepository(db)
|
||||
reqs = req_repo.get_by_regulation(regulation.id)
|
||||
|
||||
return RegulationResponse(
|
||||
id=regulation.id,
|
||||
code=regulation.code,
|
||||
name=regulation.name,
|
||||
full_name=regulation.full_name,
|
||||
regulation_type=regulation.regulation_type.value if regulation.regulation_type else None,
|
||||
source_url=regulation.source_url,
|
||||
local_pdf_path=regulation.local_pdf_path,
|
||||
effective_date=regulation.effective_date,
|
||||
description=regulation.description,
|
||||
is_active=regulation.is_active,
|
||||
created_at=regulation.created_at,
|
||||
updated_at=regulation.updated_at,
|
||||
requirement_count=len(reqs),
|
||||
)
|
||||
|
||||
|
||||
@router.get("/regulations/{code}/requirements", response_model=RequirementListResponse)
|
||||
async def get_regulation_requirements(
|
||||
code: str,
|
||||
is_applicable: Optional[bool] = None,
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""Get requirements for a specific regulation."""
|
||||
reg_repo = RegulationRepository(db)
|
||||
regulation = reg_repo.get_by_code(code)
|
||||
if not regulation:
|
||||
raise HTTPException(status_code=404, detail=f"Regulation {code} not found")
|
||||
|
||||
req_repo = RequirementRepository(db)
|
||||
if is_applicable is not None:
|
||||
requirements = req_repo.get_applicable(regulation.id) if is_applicable else req_repo.get_by_regulation(regulation.id)
|
||||
else:
|
||||
requirements = req_repo.get_by_regulation(regulation.id)
|
||||
|
||||
results = [
|
||||
RequirementResponse(
|
||||
id=r.id,
|
||||
regulation_id=r.regulation_id,
|
||||
regulation_code=code,
|
||||
article=r.article,
|
||||
paragraph=r.paragraph,
|
||||
title=r.title,
|
||||
description=r.description,
|
||||
requirement_text=r.requirement_text,
|
||||
breakpilot_interpretation=r.breakpilot_interpretation,
|
||||
is_applicable=r.is_applicable,
|
||||
applicability_reason=r.applicability_reason,
|
||||
priority=r.priority,
|
||||
created_at=r.created_at,
|
||||
updated_at=r.updated_at,
|
||||
)
|
||||
for r in requirements
|
||||
]
|
||||
|
||||
return RequirementListResponse(requirements=results, total=len(results))
|
||||
|
||||
|
||||
@router.get("/requirements/{requirement_id}")
|
||||
async def get_requirement(requirement_id: str, db: Session = Depends(get_db)):
|
||||
"""Get a specific requirement by ID."""
|
||||
from ..db.models import RequirementDB, RegulationDB
|
||||
|
||||
requirement = db.query(RequirementDB).filter(RequirementDB.id == requirement_id).first()
|
||||
if not requirement:
|
||||
raise HTTPException(status_code=404, detail=f"Requirement {requirement_id} not found")
|
||||
|
||||
regulation = db.query(RegulationDB).filter(RegulationDB.id == requirement.regulation_id).first()
|
||||
|
||||
return {
|
||||
"id": requirement.id,
|
||||
"regulation_id": requirement.regulation_id,
|
||||
"regulation_code": regulation.code if regulation else None,
|
||||
"article": requirement.article,
|
||||
"paragraph": requirement.paragraph,
|
||||
"title": requirement.title,
|
||||
"description": requirement.description,
|
||||
"requirement_text": requirement.requirement_text,
|
||||
"breakpilot_interpretation": requirement.breakpilot_interpretation,
|
||||
"implementation_status": requirement.implementation_status or "not_started",
|
||||
"implementation_details": requirement.implementation_details,
|
||||
"code_references": requirement.code_references,
|
||||
"documentation_links": requirement.documentation_links,
|
||||
"evidence_description": requirement.evidence_description,
|
||||
"evidence_artifacts": requirement.evidence_artifacts,
|
||||
"auditor_notes": requirement.auditor_notes,
|
||||
"audit_status": requirement.audit_status or "pending",
|
||||
"last_audit_date": requirement.last_audit_date,
|
||||
"last_auditor": requirement.last_auditor,
|
||||
"is_applicable": requirement.is_applicable,
|
||||
"applicability_reason": requirement.applicability_reason,
|
||||
"priority": requirement.priority,
|
||||
"source_page": requirement.source_page,
|
||||
"source_section": requirement.source_section,
|
||||
}
|
||||
|
||||
|
||||
@router.get("/requirements", response_model=PaginatedRequirementResponse)
|
||||
async def list_requirements_paginated(
|
||||
page: int = Query(1, ge=1, description="Page number"),
|
||||
page_size: int = Query(50, ge=1, le=500, description="Items per page"),
|
||||
regulation_code: Optional[str] = Query(None, description="Filter by regulation code"),
|
||||
status: Optional[str] = Query(None, description="Filter by implementation status"),
|
||||
is_applicable: Optional[bool] = Query(None, description="Filter by applicability"),
|
||||
search: Optional[str] = Query(None, description="Search in title/description"),
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""
|
||||
List requirements with pagination and eager-loaded relationships.
|
||||
|
||||
This endpoint is optimized for large datasets (1000+ requirements) with:
|
||||
- Eager loading to prevent N+1 queries
|
||||
- Server-side pagination
|
||||
- Full-text search support
|
||||
"""
|
||||
req_repo = RequirementRepository(db)
|
||||
|
||||
# Use the new paginated method with eager loading
|
||||
requirements, total = req_repo.get_paginated(
|
||||
page=page,
|
||||
page_size=page_size,
|
||||
regulation_code=regulation_code,
|
||||
status=status,
|
||||
is_applicable=is_applicable,
|
||||
search=search,
|
||||
)
|
||||
|
||||
# Calculate pagination metadata
|
||||
total_pages = (total + page_size - 1) // page_size
|
||||
|
||||
results = [
|
||||
RequirementResponse(
|
||||
id=r.id,
|
||||
regulation_id=r.regulation_id,
|
||||
regulation_code=r.regulation.code if r.regulation else None,
|
||||
article=r.article,
|
||||
paragraph=r.paragraph,
|
||||
title=r.title,
|
||||
description=r.description,
|
||||
requirement_text=r.requirement_text,
|
||||
breakpilot_interpretation=r.breakpilot_interpretation,
|
||||
is_applicable=r.is_applicable,
|
||||
applicability_reason=r.applicability_reason,
|
||||
priority=r.priority,
|
||||
implementation_status=r.implementation_status or "not_started",
|
||||
implementation_details=r.implementation_details,
|
||||
code_references=r.code_references,
|
||||
documentation_links=r.documentation_links,
|
||||
evidence_description=r.evidence_description,
|
||||
evidence_artifacts=r.evidence_artifacts,
|
||||
auditor_notes=r.auditor_notes,
|
||||
audit_status=r.audit_status or "pending",
|
||||
last_audit_date=r.last_audit_date,
|
||||
last_auditor=r.last_auditor,
|
||||
source_page=r.source_page,
|
||||
source_section=r.source_section,
|
||||
created_at=r.created_at,
|
||||
updated_at=r.updated_at,
|
||||
)
|
||||
for r in requirements
|
||||
]
|
||||
|
||||
return PaginatedRequirementResponse(
|
||||
data=results,
|
||||
pagination=PaginationMeta(
|
||||
page=page,
|
||||
page_size=page_size,
|
||||
total=total,
|
||||
total_pages=total_pages,
|
||||
has_next=page < total_pages,
|
||||
has_prev=page > 1,
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
@router.put("/requirements/{requirement_id}")
|
||||
async def update_requirement(requirement_id: str, updates: dict, db: Session = Depends(get_db)):
|
||||
"""Update a requirement with implementation/audit details."""
|
||||
from ..db.models import RequirementDB
|
||||
from datetime import datetime
|
||||
|
||||
requirement = db.query(RequirementDB).filter(RequirementDB.id == requirement_id).first()
|
||||
if not requirement:
|
||||
raise HTTPException(status_code=404, detail=f"Requirement {requirement_id} not found")
|
||||
|
||||
# Allowed fields to update
|
||||
allowed_fields = [
|
||||
'implementation_status', 'implementation_details', 'code_references',
|
||||
'documentation_links', 'evidence_description', 'evidence_artifacts',
|
||||
'auditor_notes', 'audit_status', 'is_applicable', 'applicability_reason',
|
||||
'breakpilot_interpretation'
|
||||
]
|
||||
|
||||
for field in allowed_fields:
|
||||
if field in updates:
|
||||
setattr(requirement, field, updates[field])
|
||||
|
||||
# Track audit changes
|
||||
if 'audit_status' in updates:
|
||||
requirement.last_audit_date = datetime.utcnow()
|
||||
# TODO: Get auditor from auth
|
||||
requirement.last_auditor = updates.get('auditor_name', 'api_user')
|
||||
|
||||
requirement.updated_at = datetime.utcnow()
|
||||
db.commit()
|
||||
db.refresh(requirement)
|
||||
|
||||
return {"success": True, "message": "Requirement updated"}
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Controls
|
||||
# ============================================================================
|
||||
|
||||
@router.get("/controls", response_model=ControlListResponse)
|
||||
async def list_controls(
|
||||
domain: Optional[str] = None,
|
||||
status: Optional[str] = None,
|
||||
is_automated: Optional[bool] = None,
|
||||
search: Optional[str] = None,
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""List all controls with optional filters."""
|
||||
repo = ControlRepository(db)
|
||||
|
||||
if domain:
|
||||
try:
|
||||
domain_enum = ControlDomainEnum(domain)
|
||||
controls = repo.get_by_domain(domain_enum)
|
||||
except ValueError:
|
||||
raise HTTPException(status_code=400, detail=f"Invalid domain: {domain}")
|
||||
elif status:
|
||||
try:
|
||||
status_enum = ControlStatusEnum(status)
|
||||
controls = repo.get_by_status(status_enum)
|
||||
except ValueError:
|
||||
raise HTTPException(status_code=400, detail=f"Invalid status: {status}")
|
||||
else:
|
||||
controls = repo.get_all()
|
||||
|
||||
# Apply additional filters
|
||||
if is_automated is not None:
|
||||
controls = [c for c in controls if c.is_automated == is_automated]
|
||||
|
||||
if search:
|
||||
search_lower = search.lower()
|
||||
controls = [
|
||||
c for c in controls
|
||||
if search_lower in c.control_id.lower()
|
||||
or search_lower in c.title.lower()
|
||||
or (c.description and search_lower in c.description.lower())
|
||||
]
|
||||
|
||||
# Add counts
|
||||
evidence_repo = EvidenceRepository(db)
|
||||
results = []
|
||||
for ctrl in controls:
|
||||
evidence = evidence_repo.get_by_control(ctrl.id)
|
||||
results.append(ControlResponse(
|
||||
id=ctrl.id,
|
||||
control_id=ctrl.control_id,
|
||||
domain=ctrl.domain.value if ctrl.domain else None,
|
||||
control_type=ctrl.control_type.value if ctrl.control_type else None,
|
||||
title=ctrl.title,
|
||||
description=ctrl.description,
|
||||
pass_criteria=ctrl.pass_criteria,
|
||||
implementation_guidance=ctrl.implementation_guidance,
|
||||
code_reference=ctrl.code_reference,
|
||||
documentation_url=ctrl.documentation_url,
|
||||
is_automated=ctrl.is_automated,
|
||||
automation_tool=ctrl.automation_tool,
|
||||
automation_config=ctrl.automation_config,
|
||||
owner=ctrl.owner,
|
||||
review_frequency_days=ctrl.review_frequency_days,
|
||||
status=ctrl.status.value if ctrl.status else None,
|
||||
status_notes=ctrl.status_notes,
|
||||
last_reviewed_at=ctrl.last_reviewed_at,
|
||||
next_review_at=ctrl.next_review_at,
|
||||
created_at=ctrl.created_at,
|
||||
updated_at=ctrl.updated_at,
|
||||
evidence_count=len(evidence),
|
||||
))
|
||||
|
||||
return ControlListResponse(controls=results, total=len(results))
|
||||
|
||||
|
||||
@router.get("/controls/paginated", response_model=PaginatedControlResponse)
|
||||
async def list_controls_paginated(
|
||||
page: int = Query(1, ge=1, description="Page number"),
|
||||
page_size: int = Query(50, ge=1, le=500, description="Items per page"),
|
||||
domain: Optional[str] = Query(None, description="Filter by domain"),
|
||||
status: Optional[str] = Query(None, description="Filter by status"),
|
||||
is_automated: Optional[bool] = Query(None, description="Filter by automation"),
|
||||
search: Optional[str] = Query(None, description="Search in title/description"),
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""
|
||||
List controls with pagination and eager-loaded relationships.
|
||||
|
||||
This endpoint is optimized for large datasets with:
|
||||
- Eager loading to prevent N+1 queries
|
||||
- Server-side pagination
|
||||
- Full-text search support
|
||||
"""
|
||||
repo = ControlRepository(db)
|
||||
|
||||
# Convert domain/status to enums if provided
|
||||
domain_enum = None
|
||||
status_enum = None
|
||||
if domain:
|
||||
try:
|
||||
domain_enum = ControlDomainEnum(domain)
|
||||
except ValueError:
|
||||
raise HTTPException(status_code=400, detail=f"Invalid domain: {domain}")
|
||||
if status:
|
||||
try:
|
||||
status_enum = ControlStatusEnum(status)
|
||||
except ValueError:
|
||||
raise HTTPException(status_code=400, detail=f"Invalid status: {status}")
|
||||
|
||||
controls, total = repo.get_paginated(
|
||||
page=page,
|
||||
page_size=page_size,
|
||||
domain=domain_enum,
|
||||
status=status_enum,
|
||||
is_automated=is_automated,
|
||||
search=search,
|
||||
)
|
||||
|
||||
total_pages = (total + page_size - 1) // page_size
|
||||
|
||||
results = [
|
||||
ControlResponse(
|
||||
id=c.id,
|
||||
control_id=c.control_id,
|
||||
domain=c.domain.value if c.domain else None,
|
||||
control_type=c.control_type.value if c.control_type else None,
|
||||
title=c.title,
|
||||
description=c.description,
|
||||
pass_criteria=c.pass_criteria,
|
||||
implementation_guidance=c.implementation_guidance,
|
||||
code_reference=c.code_reference,
|
||||
documentation_url=c.documentation_url,
|
||||
is_automated=c.is_automated,
|
||||
automation_tool=c.automation_tool,
|
||||
automation_config=c.automation_config,
|
||||
owner=c.owner,
|
||||
review_frequency_days=c.review_frequency_days,
|
||||
status=c.status.value if c.status else None,
|
||||
status_notes=c.status_notes,
|
||||
last_reviewed_at=c.last_reviewed_at,
|
||||
next_review_at=c.next_review_at,
|
||||
created_at=c.created_at,
|
||||
updated_at=c.updated_at,
|
||||
evidence_count=len(c.evidence) if c.evidence else 0,
|
||||
)
|
||||
for c in controls
|
||||
]
|
||||
|
||||
return PaginatedControlResponse(
|
||||
data=results,
|
||||
pagination=PaginationMeta(
|
||||
page=page,
|
||||
page_size=page_size,
|
||||
total=total,
|
||||
total_pages=total_pages,
|
||||
has_next=page < total_pages,
|
||||
has_prev=page > 1,
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
@router.get("/controls/{control_id}", response_model=ControlResponse)
|
||||
async def get_control(control_id: str, db: Session = Depends(get_db)):
|
||||
"""Get a specific control by control_id."""
|
||||
repo = ControlRepository(db)
|
||||
control = repo.get_by_control_id(control_id)
|
||||
if not control:
|
||||
raise HTTPException(status_code=404, detail=f"Control {control_id} not found")
|
||||
|
||||
evidence_repo = EvidenceRepository(db)
|
||||
evidence = evidence_repo.get_by_control(control.id)
|
||||
|
||||
return ControlResponse(
|
||||
id=control.id,
|
||||
control_id=control.control_id,
|
||||
domain=control.domain.value if control.domain else None,
|
||||
control_type=control.control_type.value if control.control_type else None,
|
||||
title=control.title,
|
||||
description=control.description,
|
||||
pass_criteria=control.pass_criteria,
|
||||
implementation_guidance=control.implementation_guidance,
|
||||
code_reference=control.code_reference,
|
||||
documentation_url=control.documentation_url,
|
||||
is_automated=control.is_automated,
|
||||
automation_tool=control.automation_tool,
|
||||
automation_config=control.automation_config,
|
||||
owner=control.owner,
|
||||
review_frequency_days=control.review_frequency_days,
|
||||
status=control.status.value if control.status else None,
|
||||
status_notes=control.status_notes,
|
||||
last_reviewed_at=control.last_reviewed_at,
|
||||
next_review_at=control.next_review_at,
|
||||
created_at=control.created_at,
|
||||
updated_at=control.updated_at,
|
||||
evidence_count=len(evidence),
|
||||
)
|
||||
|
||||
|
||||
@router.put("/controls/{control_id}", response_model=ControlResponse)
|
||||
async def update_control(
|
||||
control_id: str,
|
||||
update: ControlUpdate,
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""Update a control."""
|
||||
repo = ControlRepository(db)
|
||||
control = repo.get_by_control_id(control_id)
|
||||
if not control:
|
||||
raise HTTPException(status_code=404, detail=f"Control {control_id} not found")
|
||||
|
||||
update_data = update.model_dump(exclude_unset=True)
|
||||
|
||||
# Convert status string to enum
|
||||
if "status" in update_data:
|
||||
try:
|
||||
update_data["status"] = ControlStatusEnum(update_data["status"])
|
||||
except ValueError:
|
||||
raise HTTPException(status_code=400, detail=f"Invalid status: {update_data['status']}")
|
||||
|
||||
updated = repo.update(control.id, **update_data)
|
||||
db.commit()
|
||||
|
||||
return ControlResponse(
|
||||
id=updated.id,
|
||||
control_id=updated.control_id,
|
||||
domain=updated.domain.value if updated.domain else None,
|
||||
control_type=updated.control_type.value if updated.control_type else None,
|
||||
title=updated.title,
|
||||
description=updated.description,
|
||||
pass_criteria=updated.pass_criteria,
|
||||
implementation_guidance=updated.implementation_guidance,
|
||||
code_reference=updated.code_reference,
|
||||
documentation_url=updated.documentation_url,
|
||||
is_automated=updated.is_automated,
|
||||
automation_tool=updated.automation_tool,
|
||||
automation_config=updated.automation_config,
|
||||
owner=updated.owner,
|
||||
review_frequency_days=updated.review_frequency_days,
|
||||
status=updated.status.value if updated.status else None,
|
||||
status_notes=updated.status_notes,
|
||||
last_reviewed_at=updated.last_reviewed_at,
|
||||
next_review_at=updated.next_review_at,
|
||||
created_at=updated.created_at,
|
||||
updated_at=updated.updated_at,
|
||||
)
|
||||
|
||||
|
||||
@router.put("/controls/{control_id}/review", response_model=ControlResponse)
|
||||
async def review_control(
|
||||
control_id: str,
|
||||
review: ControlReviewRequest,
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""Mark a control as reviewed with new status."""
|
||||
repo = ControlRepository(db)
|
||||
control = repo.get_by_control_id(control_id)
|
||||
if not control:
|
||||
raise HTTPException(status_code=404, detail=f"Control {control_id} not found")
|
||||
|
||||
try:
|
||||
status_enum = ControlStatusEnum(review.status)
|
||||
except ValueError:
|
||||
raise HTTPException(status_code=400, detail=f"Invalid status: {review.status}")
|
||||
|
||||
updated = repo.mark_reviewed(control.id, status_enum, review.status_notes)
|
||||
db.commit()
|
||||
|
||||
return ControlResponse(
|
||||
id=updated.id,
|
||||
control_id=updated.control_id,
|
||||
domain=updated.domain.value if updated.domain else None,
|
||||
control_type=updated.control_type.value if updated.control_type else None,
|
||||
title=updated.title,
|
||||
description=updated.description,
|
||||
pass_criteria=updated.pass_criteria,
|
||||
implementation_guidance=updated.implementation_guidance,
|
||||
code_reference=updated.code_reference,
|
||||
documentation_url=updated.documentation_url,
|
||||
is_automated=updated.is_automated,
|
||||
automation_tool=updated.automation_tool,
|
||||
automation_config=updated.automation_config,
|
||||
owner=updated.owner,
|
||||
review_frequency_days=updated.review_frequency_days,
|
||||
status=updated.status.value if updated.status else None,
|
||||
status_notes=updated.status_notes,
|
||||
last_reviewed_at=updated.last_reviewed_at,
|
||||
next_review_at=updated.next_review_at,
|
||||
created_at=updated.created_at,
|
||||
updated_at=updated.updated_at,
|
||||
)
|
||||
|
||||
|
||||
@router.get("/controls/by-domain/{domain}", response_model=ControlListResponse)
|
||||
async def get_controls_by_domain(domain: str, db: Session = Depends(get_db)):
|
||||
"""Get controls by domain."""
|
||||
try:
|
||||
domain_enum = ControlDomainEnum(domain)
|
||||
except ValueError:
|
||||
raise HTTPException(status_code=400, detail=f"Invalid domain: {domain}")
|
||||
|
||||
repo = ControlRepository(db)
|
||||
controls = repo.get_by_domain(domain_enum)
|
||||
|
||||
results = [
|
||||
ControlResponse(
|
||||
id=c.id,
|
||||
control_id=c.control_id,
|
||||
domain=c.domain.value if c.domain else None,
|
||||
control_type=c.control_type.value if c.control_type else None,
|
||||
title=c.title,
|
||||
description=c.description,
|
||||
pass_criteria=c.pass_criteria,
|
||||
implementation_guidance=c.implementation_guidance,
|
||||
code_reference=c.code_reference,
|
||||
documentation_url=c.documentation_url,
|
||||
is_automated=c.is_automated,
|
||||
automation_tool=c.automation_tool,
|
||||
automation_config=c.automation_config,
|
||||
owner=c.owner,
|
||||
review_frequency_days=c.review_frequency_days,
|
||||
status=c.status.value if c.status else None,
|
||||
status_notes=c.status_notes,
|
||||
last_reviewed_at=c.last_reviewed_at,
|
||||
next_review_at=c.next_review_at,
|
||||
created_at=c.created_at,
|
||||
updated_at=c.updated_at,
|
||||
)
|
||||
for c in controls
|
||||
]
|
||||
|
||||
return ControlListResponse(controls=results, total=len(results))
|
||||
|
||||
|
||||
@router.post("/export", response_model=ExportResponse)
|
||||
async def create_export(
|
||||
request: ExportRequest,
|
||||
background_tasks: BackgroundTasks,
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""Create a new audit export."""
|
||||
generator = AuditExportGenerator(db)
|
||||
export = generator.create_export(
|
||||
requested_by="api_user", # TODO: Get from auth
|
||||
export_type=request.export_type,
|
||||
included_regulations=request.included_regulations,
|
||||
included_domains=request.included_domains,
|
||||
date_range_start=request.date_range_start,
|
||||
date_range_end=request.date_range_end,
|
||||
)
|
||||
|
||||
return ExportResponse(
|
||||
id=export.id,
|
||||
export_type=export.export_type,
|
||||
export_name=export.export_name,
|
||||
status=export.status.value if export.status else None,
|
||||
requested_by=export.requested_by,
|
||||
requested_at=export.requested_at,
|
||||
completed_at=export.completed_at,
|
||||
file_path=export.file_path,
|
||||
file_hash=export.file_hash,
|
||||
file_size_bytes=export.file_size_bytes,
|
||||
total_controls=export.total_controls,
|
||||
total_evidence=export.total_evidence,
|
||||
compliance_score=export.compliance_score,
|
||||
error_message=export.error_message,
|
||||
)
|
||||
|
||||
|
||||
@router.get("/export/{export_id}", response_model=ExportResponse)
|
||||
async def get_export(export_id: str, db: Session = Depends(get_db)):
|
||||
"""Get export status."""
|
||||
generator = AuditExportGenerator(db)
|
||||
export = generator.get_export_status(export_id)
|
||||
if not export:
|
||||
raise HTTPException(status_code=404, detail=f"Export {export_id} not found")
|
||||
|
||||
return ExportResponse(
|
||||
id=export.id,
|
||||
export_type=export.export_type,
|
||||
export_name=export.export_name,
|
||||
status=export.status.value if export.status else None,
|
||||
requested_by=export.requested_by,
|
||||
requested_at=export.requested_at,
|
||||
completed_at=export.completed_at,
|
||||
file_path=export.file_path,
|
||||
file_hash=export.file_hash,
|
||||
file_size_bytes=export.file_size_bytes,
|
||||
total_controls=export.total_controls,
|
||||
total_evidence=export.total_evidence,
|
||||
compliance_score=export.compliance_score,
|
||||
error_message=export.error_message,
|
||||
)
|
||||
|
||||
|
||||
@router.get("/export/{export_id}/download")
|
||||
async def download_export(export_id: str, db: Session = Depends(get_db)):
|
||||
"""Download export file."""
|
||||
generator = AuditExportGenerator(db)
|
||||
export = generator.get_export_status(export_id)
|
||||
if not export:
|
||||
raise HTTPException(status_code=404, detail=f"Export {export_id} not found")
|
||||
|
||||
if export.status.value != "completed":
|
||||
raise HTTPException(status_code=400, detail="Export not completed")
|
||||
|
||||
if not export.file_path or not os.path.exists(export.file_path):
|
||||
raise HTTPException(status_code=404, detail="Export file not found")
|
||||
|
||||
return FileResponse(
|
||||
export.file_path,
|
||||
media_type="application/zip",
|
||||
filename=os.path.basename(export.file_path),
|
||||
)
|
||||
|
||||
|
||||
@router.get("/exports", response_model=ExportListResponse)
|
||||
async def list_exports(
|
||||
limit: int = 20,
|
||||
offset: int = 0,
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""List recent exports."""
|
||||
generator = AuditExportGenerator(db)
|
||||
exports = generator.list_exports(limit, offset)
|
||||
|
||||
results = [
|
||||
ExportResponse(
|
||||
id=e.id,
|
||||
export_type=e.export_type,
|
||||
export_name=e.export_name,
|
||||
status=e.status.value if e.status else None,
|
||||
requested_by=e.requested_by,
|
||||
requested_at=e.requested_at,
|
||||
completed_at=e.completed_at,
|
||||
file_path=e.file_path,
|
||||
file_hash=e.file_hash,
|
||||
file_size_bytes=e.file_size_bytes,
|
||||
total_controls=e.total_controls,
|
||||
total_evidence=e.total_evidence,
|
||||
compliance_score=e.compliance_score,
|
||||
error_message=e.error_message,
|
||||
)
|
||||
for e in exports
|
||||
]
|
||||
|
||||
return ExportListResponse(exports=results, total=len(results))
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Seeding
|
||||
# ============================================================================
|
||||
|
||||
@router.post("/init-tables")
|
||||
async def init_tables(db: Session = Depends(get_db)):
|
||||
"""Create compliance tables if they don't exist."""
|
||||
from classroom_engine.database import engine
|
||||
from ..db.models import (
|
||||
RegulationDB, RequirementDB, ControlDB, ControlMappingDB,
|
||||
EvidenceDB, RiskDB, AuditExportDB
|
||||
)
|
||||
|
||||
try:
|
||||
# Create all tables
|
||||
RegulationDB.__table__.create(engine, checkfirst=True)
|
||||
RequirementDB.__table__.create(engine, checkfirst=True)
|
||||
ControlDB.__table__.create(engine, checkfirst=True)
|
||||
ControlMappingDB.__table__.create(engine, checkfirst=True)
|
||||
EvidenceDB.__table__.create(engine, checkfirst=True)
|
||||
RiskDB.__table__.create(engine, checkfirst=True)
|
||||
AuditExportDB.__table__.create(engine, checkfirst=True)
|
||||
|
||||
return {"success": True, "message": "Tables created successfully"}
|
||||
except Exception as e:
|
||||
logger.error(f"Table creation failed: {e}")
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
|
||||
|
||||
@router.post("/create-indexes")
|
||||
async def create_performance_indexes(db: Session = Depends(get_db)):
|
||||
"""
|
||||
Create additional performance indexes for large datasets.
|
||||
|
||||
These indexes are optimized for:
|
||||
- Pagination queries (1000+ requirements)
|
||||
- Full-text search
|
||||
- Filtering by status/priority
|
||||
"""
|
||||
from sqlalchemy import text
|
||||
|
||||
indexes = [
|
||||
# Priority index for sorting (descending, as we want high priority first)
|
||||
("ix_req_priority_desc", "CREATE INDEX IF NOT EXISTS ix_req_priority_desc ON compliance_requirements (priority DESC)"),
|
||||
|
||||
# Compound index for common filtering patterns
|
||||
("ix_req_applicable_status", "CREATE INDEX IF NOT EXISTS ix_req_applicable_status ON compliance_requirements (is_applicable, implementation_status)"),
|
||||
|
||||
# Control status index
|
||||
("ix_ctrl_status", "CREATE INDEX IF NOT EXISTS ix_ctrl_status ON compliance_controls (status)"),
|
||||
|
||||
# Evidence collected_at for timeline queries
|
||||
("ix_evidence_collected", "CREATE INDEX IF NOT EXISTS ix_evidence_collected ON compliance_evidence (collected_at DESC)"),
|
||||
|
||||
# Risk inherent risk level
|
||||
("ix_risk_level", "CREATE INDEX IF NOT EXISTS ix_risk_level ON compliance_risks (inherent_risk)"),
|
||||
]
|
||||
|
||||
created = []
|
||||
errors = []
|
||||
|
||||
for idx_name, idx_sql in indexes:
|
||||
try:
|
||||
db.execute(text(idx_sql))
|
||||
db.commit()
|
||||
created.append(idx_name)
|
||||
except Exception as e:
|
||||
errors.append({"index": idx_name, "error": str(e)})
|
||||
logger.warning(f"Index creation failed for {idx_name}: {e}")
|
||||
|
||||
return {
|
||||
"success": len(errors) == 0,
|
||||
"created": created,
|
||||
"errors": errors,
|
||||
"message": f"Created {len(created)} indexes" + (f", {len(errors)} failed" if errors else ""),
|
||||
}
|
||||
|
||||
|
||||
@router.post("/seed-risks")
|
||||
async def seed_risks_only(db: Session = Depends(get_db)):
|
||||
"""Seed only risks (incremental update for existing databases)."""
|
||||
from classroom_engine.database import engine
|
||||
from ..db.models import RiskDB
|
||||
|
||||
try:
|
||||
# Ensure table exists
|
||||
RiskDB.__table__.create(engine, checkfirst=True)
|
||||
|
||||
seeder = ComplianceSeeder(db)
|
||||
count = seeder.seed_risks_only()
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"message": f"Successfully seeded {count} risks",
|
||||
"risks_seeded": count,
|
||||
}
|
||||
except Exception as e:
|
||||
logger.error(f"Risk seeding failed: {e}")
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
|
||||
|
||||
@router.post("/seed", response_model=SeedResponse)
|
||||
async def seed_database(
|
||||
request: SeedRequest,
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""Seed the compliance database with initial data."""
|
||||
from classroom_engine.database import engine
|
||||
from ..db.models import (
|
||||
RegulationDB, RequirementDB, ControlDB, ControlMappingDB,
|
||||
EvidenceDB, RiskDB, AuditExportDB
|
||||
)
|
||||
|
||||
try:
|
||||
# Ensure tables exist first
|
||||
RegulationDB.__table__.create(engine, checkfirst=True)
|
||||
RequirementDB.__table__.create(engine, checkfirst=True)
|
||||
ControlDB.__table__.create(engine, checkfirst=True)
|
||||
ControlMappingDB.__table__.create(engine, checkfirst=True)
|
||||
EvidenceDB.__table__.create(engine, checkfirst=True)
|
||||
RiskDB.__table__.create(engine, checkfirst=True)
|
||||
AuditExportDB.__table__.create(engine, checkfirst=True)
|
||||
|
||||
seeder = ComplianceSeeder(db)
|
||||
counts = seeder.seed_all(force=request.force)
|
||||
return SeedResponse(
|
||||
success=True,
|
||||
message="Database seeded successfully",
|
||||
counts=counts,
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error(f"Seeding failed: {e}")
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
|
||||
|
||||
2512
backend/compliance/api/routes.py.backup
Normal file
2512
backend/compliance/api/routes.py.backup
Normal file
File diff suppressed because it is too large
Load Diff
1805
backend/compliance/api/schemas.py
Normal file
1805
backend/compliance/api/schemas.py
Normal file
File diff suppressed because it is too large
Load Diff
296
backend/compliance/api/scraper_routes.py
Normal file
296
backend/compliance/api/scraper_routes.py
Normal file
@@ -0,0 +1,296 @@
|
||||
"""
|
||||
FastAPI routes for Regulation Scraper and PDF extraction.
|
||||
|
||||
Endpoints:
|
||||
- /scraper/status: Scraper status
|
||||
- /scraper/sources: Available sources
|
||||
- /scraper/scrape-all: Scrape all sources
|
||||
- /scraper/scrape/{code}: Scrape single source
|
||||
- /scraper/extract-bsi: Extract BSI requirements
|
||||
- /scraper/extract-pdf: Extract from PDF
|
||||
- /scraper/pdf-documents: List available PDFs
|
||||
"""
|
||||
|
||||
import logging
|
||||
import os
|
||||
import uuid
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query, BackgroundTasks
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from classroom_engine.database import get_db
|
||||
|
||||
from ..db import RegulationRepository, RequirementRepository
|
||||
from ..db.models import RequirementDB
|
||||
from .schemas import BSIAspectResponse, PDFExtractionResponse, PDFExtractionRequest
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter(tags=["compliance-scraper"])
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Regulation Scraper
|
||||
# ============================================================================
|
||||
|
||||
@router.get("/scraper/status")
|
||||
async def get_scraper_status(db: Session = Depends(get_db)):
|
||||
"""Get current scraper status."""
|
||||
from ..services.regulation_scraper import RegulationScraperService
|
||||
|
||||
scraper = RegulationScraperService(db)
|
||||
return await scraper.get_status()
|
||||
|
||||
|
||||
@router.get("/scraper/sources")
|
||||
async def get_scraper_sources(db: Session = Depends(get_db)):
|
||||
"""Get list of known regulation sources."""
|
||||
from ..services.regulation_scraper import RegulationScraperService
|
||||
|
||||
scraper = RegulationScraperService(db)
|
||||
return {
|
||||
"sources": scraper.get_known_sources(),
|
||||
"total": len(scraper.KNOWN_SOURCES),
|
||||
}
|
||||
|
||||
|
||||
@router.post("/scraper/scrape-all")
|
||||
async def scrape_all_sources(
|
||||
background_tasks: BackgroundTasks,
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""Start scraping all known regulation sources."""
|
||||
from ..services.regulation_scraper import RegulationScraperService
|
||||
|
||||
scraper = RegulationScraperService(db)
|
||||
results = await scraper.scrape_all()
|
||||
return {
|
||||
"status": "completed",
|
||||
"results": results,
|
||||
}
|
||||
|
||||
|
||||
@router.post("/scraper/scrape/{code}")
|
||||
async def scrape_single_source(
|
||||
code: str,
|
||||
force: bool = Query(False, description="Force re-scrape even if data exists"),
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""Scrape a specific regulation source."""
|
||||
from ..services.regulation_scraper import RegulationScraperService
|
||||
|
||||
scraper = RegulationScraperService(db)
|
||||
|
||||
try:
|
||||
result = await scraper.scrape_single(code, force=force)
|
||||
return result
|
||||
except ValueError as e:
|
||||
raise HTTPException(status_code=400, detail=str(e))
|
||||
except Exception as e:
|
||||
logger.error(f"Scraping {code} failed: {e}")
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
|
||||
|
||||
@router.post("/scraper/extract-bsi")
|
||||
async def extract_bsi_requirements(
|
||||
code: str = Query("BSI-TR-03161-2", description="BSI TR code"),
|
||||
force: bool = Query(False),
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""
|
||||
Extract requirements from BSI Technical Guidelines.
|
||||
|
||||
Uses pre-defined Pruefaspekte from BSI-TR-03161 documents.
|
||||
"""
|
||||
from ..services.regulation_scraper import RegulationScraperService
|
||||
|
||||
if not code.startswith("BSI"):
|
||||
raise HTTPException(status_code=400, detail="Only BSI codes are supported")
|
||||
|
||||
scraper = RegulationScraperService(db)
|
||||
|
||||
try:
|
||||
result = await scraper.scrape_single(code, force=force)
|
||||
return result
|
||||
except ValueError as e:
|
||||
raise HTTPException(status_code=400, detail=str(e))
|
||||
except Exception as e:
|
||||
logger.error(f"BSI extraction failed: {e}")
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
|
||||
|
||||
@router.post("/scraper/extract-pdf", response_model=PDFExtractionResponse)
|
||||
async def extract_pdf_requirements(
|
||||
request: PDFExtractionRequest,
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""
|
||||
Extract Pruefaspekte from BSI-TR PDF documents using PyMuPDF.
|
||||
|
||||
Supported documents:
|
||||
- BSI-TR-03161-1: General security requirements
|
||||
- BSI-TR-03161-2: Web application security (OAuth, Sessions, etc.)
|
||||
- BSI-TR-03161-3: Backend/server security
|
||||
"""
|
||||
from ..services.pdf_extractor import BSIPDFExtractor
|
||||
from ..db.models import RegulationTypeEnum
|
||||
|
||||
# Map document codes to file paths
|
||||
PDF_PATHS = {
|
||||
"BSI-TR-03161-1": "/app/docs/BSI-TR-03161-1.pdf",
|
||||
"BSI-TR-03161-2": "/app/docs/BSI-TR-03161-2.pdf",
|
||||
"BSI-TR-03161-3": "/app/docs/BSI-TR-03161-3.pdf",
|
||||
}
|
||||
|
||||
# Local development paths (fallback)
|
||||
LOCAL_PDF_PATHS = {
|
||||
"BSI-TR-03161-1": "/Users/benjaminadmin/Projekte/breakpilot-pwa/docs/BSI-TR-03161-1.pdf",
|
||||
"BSI-TR-03161-2": "/Users/benjaminadmin/Projekte/breakpilot-pwa/docs/BSI-TR-03161-2.pdf",
|
||||
"BSI-TR-03161-3": "/Users/benjaminadmin/Projekte/breakpilot-pwa/docs/BSI-TR-03161-3.pdf",
|
||||
}
|
||||
|
||||
doc_code = request.document_code.upper()
|
||||
if doc_code not in PDF_PATHS:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Unsupported document: {doc_code}. Supported: {list(PDF_PATHS.keys())}"
|
||||
)
|
||||
|
||||
# Try container path first, then local path
|
||||
pdf_path = PDF_PATHS[doc_code]
|
||||
if not os.path.exists(pdf_path):
|
||||
pdf_path = LOCAL_PDF_PATHS.get(doc_code)
|
||||
if not pdf_path or not os.path.exists(pdf_path):
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail=f"PDF file not found for {doc_code}"
|
||||
)
|
||||
|
||||
try:
|
||||
extractor = BSIPDFExtractor()
|
||||
aspects = extractor.extract_from_file(pdf_path, source_name=doc_code)
|
||||
stats = extractor.get_statistics(aspects)
|
||||
|
||||
# Convert to response format
|
||||
aspect_responses = [
|
||||
BSIAspectResponse(
|
||||
aspect_id=a.aspect_id,
|
||||
title=a.title,
|
||||
full_text=a.full_text[:2000],
|
||||
category=a.category.value,
|
||||
page_number=a.page_number,
|
||||
section=a.section,
|
||||
requirement_level=a.requirement_level.value,
|
||||
source_document=a.source_document,
|
||||
keywords=a.keywords,
|
||||
related_aspects=a.related_aspects,
|
||||
)
|
||||
for a in aspects
|
||||
]
|
||||
|
||||
requirements_created = 0
|
||||
|
||||
# Save to database if requested
|
||||
if request.save_to_db:
|
||||
# Get or create regulation
|
||||
reg_repo = RegulationRepository(db)
|
||||
regulation = reg_repo.get_by_code(doc_code)
|
||||
|
||||
if not regulation:
|
||||
regulation = reg_repo.create(
|
||||
code=doc_code,
|
||||
name=f"BSI TR {doc_code.split('-')[-1]}",
|
||||
full_name=f"BSI Technische Richtlinie {doc_code}",
|
||||
regulation_type=RegulationTypeEnum.BSI_STANDARD,
|
||||
local_pdf_path=pdf_path,
|
||||
)
|
||||
|
||||
# Create requirements from extracted aspects
|
||||
req_repo = RequirementRepository(db)
|
||||
existing_articles = {r.article for r in req_repo.get_by_regulation(regulation.id)}
|
||||
|
||||
for aspect in aspects:
|
||||
if aspect.aspect_id not in existing_articles or request.force:
|
||||
# Delete existing if force
|
||||
if request.force and aspect.aspect_id in existing_articles:
|
||||
existing = db.query(RequirementDB).filter(
|
||||
RequirementDB.regulation_id == regulation.id,
|
||||
RequirementDB.article == aspect.aspect_id
|
||||
).first()
|
||||
if existing:
|
||||
db.delete(existing)
|
||||
|
||||
# Determine priority based on requirement level
|
||||
priority_map = {"MUSS": 3, "SOLL": 2, "KANN": 1, "DARF NICHT": 3}
|
||||
priority = priority_map.get(aspect.requirement_level.value, 2)
|
||||
|
||||
requirement = RequirementDB(
|
||||
id=str(uuid.uuid4()),
|
||||
regulation_id=regulation.id,
|
||||
article=aspect.aspect_id,
|
||||
paragraph=aspect.section,
|
||||
title=aspect.title[:300],
|
||||
description=f"Kategorie: {aspect.category.value}",
|
||||
requirement_text=aspect.full_text[:4000],
|
||||
is_applicable=True,
|
||||
priority=priority,
|
||||
source_page=aspect.page_number,
|
||||
source_section=aspect.section,
|
||||
)
|
||||
db.add(requirement)
|
||||
requirements_created += 1
|
||||
|
||||
db.commit()
|
||||
|
||||
return PDFExtractionResponse(
|
||||
success=True,
|
||||
source_document=doc_code,
|
||||
total_aspects=len(aspects),
|
||||
aspects=aspect_responses,
|
||||
statistics=stats,
|
||||
requirements_created=requirements_created,
|
||||
)
|
||||
|
||||
except ImportError as e:
|
||||
raise HTTPException(
|
||||
status_code=500,
|
||||
detail=f"PyMuPDF not installed: {e}. Run: pip install PyMuPDF"
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error(f"PDF extraction failed for {doc_code}: {e}")
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
|
||||
|
||||
@router.get("/scraper/pdf-documents")
|
||||
async def list_pdf_documents():
|
||||
"""List available PDF documents for extraction."""
|
||||
PDF_DOCS = [
|
||||
{
|
||||
"code": "BSI-TR-03161-1",
|
||||
"name": "BSI TR 03161 Teil 1",
|
||||
"description": "Allgemeine Sicherheitsanforderungen für mobile Anwendungen",
|
||||
"expected_aspects": "~30",
|
||||
},
|
||||
{
|
||||
"code": "BSI-TR-03161-2",
|
||||
"name": "BSI TR 03161 Teil 2",
|
||||
"description": "Web-Anwendungssicherheit (OAuth, Sessions, Input Validation, etc.)",
|
||||
"expected_aspects": "~80-100",
|
||||
},
|
||||
{
|
||||
"code": "BSI-TR-03161-3",
|
||||
"name": "BSI TR 03161 Teil 3",
|
||||
"description": "Backend/Server-Sicherheit",
|
||||
"expected_aspects": "~40",
|
||||
},
|
||||
]
|
||||
|
||||
# Check which PDFs exist
|
||||
for doc in PDF_DOCS:
|
||||
local_path = f"/Users/benjaminadmin/Projekte/breakpilot-pwa/docs/{doc['code']}.pdf"
|
||||
container_path = f"/app/docs/{doc['code']}.pdf"
|
||||
doc["available"] = os.path.exists(local_path) or os.path.exists(container_path)
|
||||
|
||||
return {
|
||||
"documents": PDF_DOCS,
|
||||
"total": len(PDF_DOCS),
|
||||
}
|
||||
218
backend/compliance/data/README.md
Normal file
218
backend/compliance/data/README.md
Normal file
@@ -0,0 +1,218 @@
|
||||
# Compliance Data - Service Module Registry
|
||||
|
||||
Diese Dateien enthalten die Seed-Daten für das Compliance-Modul.
|
||||
|
||||
## Sprint 3: Service-Module Registry
|
||||
|
||||
### Dateien
|
||||
|
||||
- **`service_modules.py`**: Vollständige Registry aller 30+ Breakpilot Services mit:
|
||||
- Technische Details (Port, Stack, Repository)
|
||||
- Datenkategorien und PII-Verarbeitung
|
||||
- Anwendbare Regulierungen (GDPR, AI Act, BSI-TR, etc.)
|
||||
- Kritikalität und Ownership
|
||||
|
||||
### Service-Typen
|
||||
|
||||
| Typ | Beschreibung | Beispiele |
|
||||
|-----|--------------|-----------|
|
||||
| `backend` | API/Backend Services | consent-service, python-backend |
|
||||
| `database` | Datenbanken | PostgreSQL, Qdrant, Valkey |
|
||||
| `ai` | KI/ML Services | klausur-service, embedding-service |
|
||||
| `communication` | Chat/Video | Matrix, Jitsi |
|
||||
| `storage` | Speichersysteme | MinIO, DSMS |
|
||||
| `infrastructure` | Infrastruktur | Vault, Mailpit, Backup |
|
||||
| `monitoring` | Monitoring (geplant) | Loki, Grafana, Prometheus |
|
||||
| `security` | Sicherheit | Vault |
|
||||
|
||||
### Relevanz-Stufen
|
||||
|
||||
| Stufe | Bedeutung |
|
||||
|-------|-----------|
|
||||
| `critical` | Non-Compliance = Shutdown |
|
||||
| `high` | Hohes Risiko |
|
||||
| `medium` | Mittleres Risiko |
|
||||
| `low` | Geringes Risiko |
|
||||
|
||||
### Verwendung
|
||||
|
||||
```python
|
||||
from compliance.data.service_modules import (
|
||||
BREAKPILOT_SERVICES,
|
||||
get_service_count,
|
||||
get_services_by_type,
|
||||
get_services_processing_pii,
|
||||
get_services_with_ai,
|
||||
get_critical_services
|
||||
)
|
||||
|
||||
# Alle Services
|
||||
total = get_service_count()
|
||||
|
||||
# Backend Services
|
||||
backends = get_services_by_type("backend")
|
||||
|
||||
# PII-verarbeitende Services
|
||||
pii_services = get_services_processing_pii()
|
||||
|
||||
# KI-Services
|
||||
ai_services = get_services_with_ai()
|
||||
|
||||
# Kritische Services
|
||||
critical = get_critical_services()
|
||||
```
|
||||
|
||||
### Seeding
|
||||
|
||||
Services werden automatisch beim ersten Start geseedet:
|
||||
|
||||
```bash
|
||||
# Nur Service-Module seeden
|
||||
python -m compliance.scripts.seed_service_modules --mode modules
|
||||
|
||||
# Vollständige Compliance-DB seeden
|
||||
python -m compliance.scripts.seed_service_modules --mode all
|
||||
```
|
||||
|
||||
### Validierung
|
||||
|
||||
Vor dem Seeding können die Daten validiert werden:
|
||||
|
||||
```bash
|
||||
python -m compliance.scripts.validate_service_modules
|
||||
```
|
||||
|
||||
Prüft:
|
||||
- Pflichtfelder vorhanden
|
||||
- Keine Port-Konflikte
|
||||
- Regulierungen existieren
|
||||
- Datenkategorien bei PII-Services
|
||||
|
||||
## Service-Dokumentation
|
||||
|
||||
Jeder Service ist dokumentiert mit:
|
||||
|
||||
1. **Identifikation**
|
||||
- `name`: Technischer Name (Docker-Container)
|
||||
- `display_name`: Anzeigename
|
||||
- `description`: Kurzbeschreibung
|
||||
|
||||
2. **Technische Details**
|
||||
- `service_type`: Typ (backend, database, etc.)
|
||||
- `port`: Hauptport (falls vorhanden)
|
||||
- `technology_stack`: Verwendete Technologien
|
||||
- `repository_path`: Pfad im Repository
|
||||
- `docker_image`: Docker Image Name
|
||||
|
||||
3. **Datenschutz**
|
||||
- `data_categories`: Welche Datenkategorien werden verarbeitet
|
||||
- `processes_pii`: Verarbeitet personenbezogene Daten?
|
||||
- `processes_health_data`: Verarbeitet Gesundheitsdaten?
|
||||
- `ai_components`: Enthält KI-Komponenten?
|
||||
|
||||
4. **Compliance**
|
||||
- `criticality`: Kritikalität (critical, high, medium, low)
|
||||
- `owner_team`: Verantwortliches Team
|
||||
- `regulations`: Liste anwendbarer Regulierungen mit Relevanz
|
||||
|
||||
### Beispiel: consent-service
|
||||
|
||||
```python
|
||||
{
|
||||
"name": "consent-service",
|
||||
"display_name": "Go Consent Service",
|
||||
"description": "Kernlogik für Consent-Management, Einwilligungsverwaltung und Versionierung",
|
||||
"service_type": "backend",
|
||||
"port": 8081,
|
||||
"technology_stack": ["Go", "Gin", "GORM", "PostgreSQL"],
|
||||
"repository_path": "/consent-service",
|
||||
"docker_image": "breakpilot-pwa-consent-service",
|
||||
"data_categories": ["consent_records", "user_preferences", "audit_logs"],
|
||||
"processes_pii": True,
|
||||
"processes_health_data": False,
|
||||
"ai_components": False,
|
||||
"criticality": "critical",
|
||||
"owner_team": "Backend Team",
|
||||
"regulations": [
|
||||
{"code": "GDPR", "relevance": "critical", "notes": "Art. 7 Einwilligung, Art. 30 VVZ"},
|
||||
{"code": "TDDDG", "relevance": "critical", "notes": "§ 25 Cookie-Consent"},
|
||||
{"code": "BSI-TR-03161-2", "relevance": "high", "notes": "Session-Management"}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Datenbank-Schema
|
||||
|
||||
Die Service-Module werden in folgenden Tabellen gespeichert:
|
||||
|
||||
### `compliance_service_modules`
|
||||
|
||||
Haupttabelle für Services:
|
||||
- `id`, `name`, `display_name`, `description`
|
||||
- `service_type`, `port`, `technology_stack`
|
||||
- `data_categories`, `processes_pii`, `ai_components`
|
||||
- `criticality`, `owner_team`
|
||||
- `compliance_score` (berechnet)
|
||||
|
||||
### `compliance_module_regulations`
|
||||
|
||||
Mapping Service ↔ Regulation:
|
||||
- `module_id`, `regulation_id`
|
||||
- `relevance_level` (critical, high, medium, low)
|
||||
- `notes`
|
||||
- `applicable_articles` (JSON Liste)
|
||||
|
||||
### `compliance_module_risks`
|
||||
|
||||
Service-spezifische Risikobewertungen:
|
||||
- `module_id`, `risk_id`
|
||||
- `module_likelihood`, `module_impact`
|
||||
- `module_risk_level`
|
||||
- `assessment_notes`
|
||||
|
||||
## API Endpoints
|
||||
|
||||
Nach dem Seeding stehen folgende Endpoints zur Verfügung:
|
||||
|
||||
```
|
||||
GET /api/compliance/modules
|
||||
Liste aller Service-Module
|
||||
|
||||
GET /api/compliance/modules/{module_id}
|
||||
Details zu einem Service
|
||||
|
||||
GET /api/compliance/modules/{module_id}/regulations
|
||||
Anwendbare Regulierungen für einen Service
|
||||
|
||||
GET /api/compliance/modules/{module_id}/compliance-score
|
||||
Compliance-Score für einen Service
|
||||
```
|
||||
|
||||
## Erweiterung
|
||||
|
||||
Um einen neuen Service hinzuzufügen:
|
||||
|
||||
1. Service zu `BREAKPILOT_SERVICES` in `service_modules.py` hinzufügen
|
||||
2. Validierung ausführen: `python -m compliance.scripts.validate_service_modules`
|
||||
3. Seeding ausführen: `python -m compliance.scripts.seed_service_modules`
|
||||
|
||||
Oder über die API (wenn aktiviert):
|
||||
|
||||
```bash
|
||||
POST /api/compliance/modules
|
||||
{
|
||||
"name": "new-service",
|
||||
"display_name": "New Service",
|
||||
...
|
||||
}
|
||||
```
|
||||
|
||||
## Aktueller Stand (Sprint 3)
|
||||
|
||||
- ✅ 30+ Services dokumentiert
|
||||
- ✅ Alle docker-compose.yml Services erfasst
|
||||
- ✅ Regulation Mappings definiert
|
||||
- ✅ Seeder implementiert
|
||||
- ✅ Validierung verfügbar
|
||||
- 🔄 Compliance-Score Berechnung (geplant)
|
||||
- 🔄 Gap-Analyse pro Service (geplant)
|
||||
22
backend/compliance/data/__init__.py
Normal file
22
backend/compliance/data/__init__.py
Normal file
@@ -0,0 +1,22 @@
|
||||
"""
|
||||
Seed data for Compliance module.
|
||||
|
||||
Contains initial data for:
|
||||
- 16 EU Regulations + 3 BSI-TR documents
|
||||
- ~45 Controls across 9 domains
|
||||
- Key requirements from GDPR, AI Act, CRA
|
||||
- ISO 27001:2022 Annex A (93 Controls)
|
||||
"""
|
||||
|
||||
from .regulations import REGULATIONS_SEED
|
||||
from .controls import CONTROLS_SEED
|
||||
from .requirements import REQUIREMENTS_SEED
|
||||
from .iso27001_annex_a import ISO27001_ANNEX_A_CONTROLS, ANNEX_A_SUMMARY
|
||||
|
||||
__all__ = [
|
||||
"REGULATIONS_SEED",
|
||||
"CONTROLS_SEED",
|
||||
"REQUIREMENTS_SEED",
|
||||
"ISO27001_ANNEX_A_CONTROLS",
|
||||
"ANNEX_A_SUMMARY",
|
||||
]
|
||||
624
backend/compliance/data/controls.py
Normal file
624
backend/compliance/data/controls.py
Normal file
@@ -0,0 +1,624 @@
|
||||
"""
|
||||
Seed data for Controls.
|
||||
|
||||
~45 Controls across 9 domains:
|
||||
- GOV: Governance & Organisation
|
||||
- PRIV: Datenschutz & Privacy
|
||||
- IAM: Identity & Access Management
|
||||
- CRYPTO: Kryptografie
|
||||
- SDLC: Secure Development Lifecycle
|
||||
- OPS: Betrieb & Monitoring
|
||||
- AI: KI-spezifisch
|
||||
- CRA: CRA & Supply Chain
|
||||
- AUD: Audit & Nachvollziehbarkeit
|
||||
"""
|
||||
|
||||
CONTROLS_SEED = [
|
||||
# =========================================================================
|
||||
# GOV - Governance & Organisation
|
||||
# =========================================================================
|
||||
{
|
||||
"control_id": "GOV-001",
|
||||
"domain": "gov",
|
||||
"control_type": "preventive",
|
||||
"title": "ISMS Policy",
|
||||
"description": "Dokumentierte Informationssicherheits-Management-System Policy mit jährlicher Überprüfung.",
|
||||
"pass_criteria": "ISMS Policy vorhanden, aktuell (nicht älter als 12 Monate), von Management genehmigt.",
|
||||
"implementation_guidance": "Policy erstellen nach ISO 27001 Struktur, Scope definieren, Management-Commitment dokumentieren.",
|
||||
"is_automated": False,
|
||||
"owner": "Security Team",
|
||||
"review_frequency_days": 365,
|
||||
},
|
||||
{
|
||||
"control_id": "GOV-002",
|
||||
"domain": "gov",
|
||||
"control_type": "preventive",
|
||||
"title": "Rollen & Verantwortlichkeiten",
|
||||
"description": "RACI-Matrix für alle sicherheitsrelevanten Prozesse dokumentiert.",
|
||||
"pass_criteria": "RACI-Matrix vorhanden und aktuell, alle kritischen Rollen besetzt.",
|
||||
"implementation_guidance": "RACI-Matrix erstellen für: Incident Response, Vulnerability Management, Access Management, Change Management.",
|
||||
"is_automated": False,
|
||||
"owner": "Security Team",
|
||||
"review_frequency_days": 180,
|
||||
},
|
||||
{
|
||||
"control_id": "GOV-003",
|
||||
"domain": "gov",
|
||||
"control_type": "preventive",
|
||||
"title": "Security Awareness Training",
|
||||
"description": "Alle Mitarbeiter absolvieren jährlich Security Awareness Training.",
|
||||
"pass_criteria": "100% Completion Rate für alle aktiven Mitarbeiter, Nachweis nicht älter als 12 Monate.",
|
||||
"implementation_guidance": "Training-Plattform einrichten (z.B. KnowBe4), Pflichttraining für Onboarding, jährliche Auffrischung.",
|
||||
"is_automated": False,
|
||||
"owner": "HR / Security Team",
|
||||
"review_frequency_days": 365,
|
||||
},
|
||||
{
|
||||
"control_id": "GOV-004",
|
||||
"domain": "gov",
|
||||
"control_type": "preventive",
|
||||
"title": "Change Management",
|
||||
"description": "Alle Code-Änderungen erfolgen über Pull Requests mit Review.",
|
||||
"pass_criteria": "100% der Merges in main/master via PR, mindestens 1 Reviewer pro PR.",
|
||||
"implementation_guidance": "Branch Protection Rules in GitHub aktivieren, CODEOWNERS definieren.",
|
||||
"code_reference": ".github/CODEOWNERS",
|
||||
"is_automated": True,
|
||||
"automation_tool": "GitHub Branch Protection",
|
||||
"owner": "Engineering Lead",
|
||||
"review_frequency_days": 30,
|
||||
},
|
||||
{
|
||||
"control_id": "GOV-005",
|
||||
"domain": "gov",
|
||||
"control_type": "corrective",
|
||||
"title": "Incident Response Plan",
|
||||
"description": "Dokumentierter Incident Response Plan mit Eskalationspfaden und Kontakten.",
|
||||
"pass_criteria": "IRP vorhanden, getestet innerhalb der letzten 12 Monate, Kontaktdaten aktuell.",
|
||||
"implementation_guidance": "IRP nach NIST SP 800-61 erstellen, Tabletop-Übungen durchführen.",
|
||||
"is_automated": False,
|
||||
"owner": "Security Team",
|
||||
"review_frequency_days": 365,
|
||||
},
|
||||
|
||||
# =========================================================================
|
||||
# PRIV - Datenschutz & Privacy
|
||||
# =========================================================================
|
||||
{
|
||||
"control_id": "PRIV-001",
|
||||
"domain": "priv",
|
||||
"control_type": "preventive",
|
||||
"title": "Verarbeitungsverzeichnis (Art. 30)",
|
||||
"description": "Aktuelles Verzeichnis aller Verarbeitungstätigkeiten gemäß Art. 30 DSGVO.",
|
||||
"pass_criteria": "VVT vorhanden, vollständig (alle Kategorien), nicht älter als 6 Monate aktualisiert.",
|
||||
"implementation_guidance": "VVT mit allen erforderlichen Feldern: Zweck, Kategorien, Empfänger, Fristen, TOMs.",
|
||||
"is_automated": False,
|
||||
"owner": "Datenschutzbeauftragter",
|
||||
"review_frequency_days": 180,
|
||||
},
|
||||
{
|
||||
"control_id": "PRIV-002",
|
||||
"domain": "priv",
|
||||
"control_type": "preventive",
|
||||
"title": "DPIA durchgeführt (Art. 35)",
|
||||
"description": "Datenschutz-Folgenabschätzung für Hochrisiko-Verarbeitungen durchgeführt.",
|
||||
"pass_criteria": "DPIA für alle identifizierten Hochrisiko-Verarbeitungen vorhanden und dokumentiert.",
|
||||
"implementation_guidance": "DPIA nach Art. 35 Abs. 7 DSGVO: Beschreibung, Notwendigkeit, Risikobewertung, Maßnahmen.",
|
||||
"is_automated": False,
|
||||
"owner": "Datenschutzbeauftragter",
|
||||
"review_frequency_days": 365,
|
||||
},
|
||||
{
|
||||
"control_id": "PRIV-003",
|
||||
"domain": "priv",
|
||||
"control_type": "preventive",
|
||||
"title": "Privacy by Design (Art. 25)",
|
||||
"description": "Datenschutz durch Technikgestaltung und datenschutzfreundliche Voreinstellungen.",
|
||||
"pass_criteria": "PbD-Checkliste für alle neuen Features, Datensparsamkeit als Default.",
|
||||
"implementation_guidance": "PbD-Review in Feature-Development-Prozess integrieren, Minimaldatenerhebung als Standard.",
|
||||
"is_automated": False,
|
||||
"owner": "Engineering Lead / DPO",
|
||||
"review_frequency_days": 90,
|
||||
},
|
||||
{
|
||||
"control_id": "PRIV-004",
|
||||
"domain": "priv",
|
||||
"control_type": "corrective",
|
||||
"title": "Betroffenenrechte (Art. 15-22)",
|
||||
"description": "Prozess für Betroffenenrechte (Auskunft, Löschung, Portabilität) implementiert.",
|
||||
"pass_criteria": "DSR-Prozess dokumentiert, SLA < 30 Tage, Export-Funktion vorhanden.",
|
||||
"implementation_guidance": "Self-Service-Portal für DSR, automatisierte Löschfunktion, Export im maschinenlesbaren Format.",
|
||||
"code_reference": "backend/gdpr_api.py",
|
||||
"is_automated": True,
|
||||
"automation_tool": "Breakpilot GDPR Export",
|
||||
"owner": "Engineering Team",
|
||||
"review_frequency_days": 90,
|
||||
},
|
||||
{
|
||||
"control_id": "PRIV-005",
|
||||
"domain": "priv",
|
||||
"control_type": "preventive",
|
||||
"title": "AVV mit Auftragsverarbeitern",
|
||||
"description": "Auftragsverarbeitungsverträge mit allen Sub-Processors abgeschlossen.",
|
||||
"pass_criteria": "AVV für alle Auftragsverarbeiter vorhanden, Art. 28 Abs. 3 konform.",
|
||||
"implementation_guidance": "Liste aller Sub-Processors, AVV-Vorlagen nach Art. 28, jährliche Überprüfung.",
|
||||
"is_automated": False,
|
||||
"owner": "Legal / DPO",
|
||||
"review_frequency_days": 365,
|
||||
},
|
||||
{
|
||||
"control_id": "PRIV-006",
|
||||
"domain": "priv",
|
||||
"control_type": "preventive",
|
||||
"title": "TOMs dokumentiert (Art. 32)",
|
||||
"description": "Technische und organisatorische Maßnahmen gemäß Art. 32 DSGVO dokumentiert.",
|
||||
"pass_criteria": "TOM-Dokument vorhanden, alle Kategorien abgedeckt, aktuell.",
|
||||
"implementation_guidance": "TOMs nach Art. 32: Pseudonymisierung, Verschlüsselung, Wiederherstellung, regelmäßige Tests.",
|
||||
"is_automated": False,
|
||||
"owner": "Security Team / DPO",
|
||||
"review_frequency_days": 180,
|
||||
},
|
||||
{
|
||||
"control_id": "PRIV-007",
|
||||
"domain": "priv",
|
||||
"control_type": "preventive",
|
||||
"title": "PII-Logging verhindert",
|
||||
"description": "Personenbezogene Daten werden nicht in Logs geschrieben (PII Redaction).",
|
||||
"pass_criteria": "PII-Redactor aktiv, keine PII in Logs (stichprobenartige Prüfung).",
|
||||
"implementation_guidance": "PII-Redactor Middleware implementieren, regex-basierte Filterung für E-Mail, Namen, etc.",
|
||||
"code_reference": "backend/middleware/pii_redactor.py",
|
||||
"is_automated": True,
|
||||
"automation_tool": "PII Redactor Middleware",
|
||||
"owner": "Engineering Team",
|
||||
"review_frequency_days": 90,
|
||||
},
|
||||
|
||||
# =========================================================================
|
||||
# IAM - Identity & Access Management
|
||||
# =========================================================================
|
||||
{
|
||||
"control_id": "IAM-001",
|
||||
"domain": "iam",
|
||||
"control_type": "preventive",
|
||||
"title": "RBAC implementiert",
|
||||
"description": "Role-Based Access Control mit dokumentierten Rollen und Berechtigungen.",
|
||||
"pass_criteria": "RBAC-Modell dokumentiert, Rollen im Code enforced, keine Hardcoded-Berechtigungen.",
|
||||
"implementation_guidance": "Rollen definieren (user, admin, dpo), Middleware für Berechtigungsprüfung.",
|
||||
"code_reference": "consent-service/internal/middleware/auth.go",
|
||||
"is_automated": True,
|
||||
"automation_tool": "JWT Role Claims",
|
||||
"owner": "Engineering Team",
|
||||
"review_frequency_days": 90,
|
||||
},
|
||||
{
|
||||
"control_id": "IAM-002",
|
||||
"domain": "iam",
|
||||
"control_type": "preventive",
|
||||
"title": "MFA für Admin-Accounts",
|
||||
"description": "Multi-Faktor-Authentifizierung für alle Admin-Zugänge aktiviert.",
|
||||
"pass_criteria": "100% MFA-Abdeckung für Admin-Accounts, Enforcement-Policy aktiv.",
|
||||
"implementation_guidance": "MFA über Identity Provider (Auth0, Keycloak) oder TOTP-Integration.",
|
||||
"is_automated": False,
|
||||
"owner": "Security Team",
|
||||
"review_frequency_days": 90,
|
||||
},
|
||||
{
|
||||
"control_id": "IAM-003",
|
||||
"domain": "iam",
|
||||
"control_type": "preventive",
|
||||
"title": "Mandantentrennung",
|
||||
"description": "Strikte Tenant-Isolation zwischen verschiedenen Kunden/Schulen.",
|
||||
"pass_criteria": "Tenant-ID in allen Queries, keine Cross-Tenant-Datenzugriffe möglich.",
|
||||
"implementation_guidance": "Tenant-ID als Pflichtfeld, Row-Level-Security in Queries, Penetration-Test.",
|
||||
"code_reference": "consent-service/internal/handlers/handlers.go",
|
||||
"is_automated": True,
|
||||
"automation_tool": "Database Query Filter",
|
||||
"owner": "Engineering Team",
|
||||
"review_frequency_days": 90,
|
||||
},
|
||||
{
|
||||
"control_id": "IAM-004",
|
||||
"domain": "iam",
|
||||
"control_type": "preventive",
|
||||
"title": "Session Management",
|
||||
"description": "Sichere Session-Verwaltung mit Token-Expiry und Rotation.",
|
||||
"pass_criteria": "Token-Expiry < 24h, Refresh-Token-Rotation, Logout invalidiert Token.",
|
||||
"implementation_guidance": "JWT mit kurzer Expiry, Refresh-Token-Flow, Token-Blacklisting bei Logout.",
|
||||
"code_reference": "consent-service/internal/services/auth_service.go",
|
||||
"is_automated": True,
|
||||
"automation_tool": "JWT Token Management",
|
||||
"owner": "Engineering Team",
|
||||
"review_frequency_days": 90,
|
||||
},
|
||||
{
|
||||
"control_id": "IAM-005",
|
||||
"domain": "iam",
|
||||
"control_type": "detective",
|
||||
"title": "Least Privilege",
|
||||
"description": "Regelmäßige Access Reviews zur Sicherstellung minimaler Berechtigungen.",
|
||||
"pass_criteria": "Vierteljährliche Access Reviews durchgeführt, überflüssige Rechte entfernt.",
|
||||
"implementation_guidance": "Access Review Prozess etablieren, automatisierte Reports über Berechtigungen.",
|
||||
"is_automated": False,
|
||||
"owner": "Security Team",
|
||||
"review_frequency_days": 90,
|
||||
},
|
||||
|
||||
# =========================================================================
|
||||
# CRYPTO - Kryptografie
|
||||
# =========================================================================
|
||||
{
|
||||
"control_id": "CRYPTO-001",
|
||||
"domain": "crypto",
|
||||
"control_type": "preventive",
|
||||
"title": "Encryption at Rest",
|
||||
"description": "Sensible Daten sind im Ruhezustand verschlüsselt (AES-256).",
|
||||
"pass_criteria": "Datenbank-Verschlüsselung aktiv, Backup-Verschlüsselung aktiv.",
|
||||
"implementation_guidance": "PostgreSQL mit TDE oder pgcrypto, verschlüsselte Backups.",
|
||||
"is_automated": True,
|
||||
"automation_tool": "PostgreSQL Encryption",
|
||||
"owner": "Infrastructure Team",
|
||||
"review_frequency_days": 180,
|
||||
},
|
||||
{
|
||||
"control_id": "CRYPTO-002",
|
||||
"domain": "crypto",
|
||||
"control_type": "preventive",
|
||||
"title": "Encryption in Transit",
|
||||
"description": "Alle Datenübertragungen sind TLS 1.3 verschlüsselt.",
|
||||
"pass_criteria": "TLS 1.3 enforced, HSTS aktiv, keine unsicheren Cipher Suites.",
|
||||
"implementation_guidance": "Nginx/Traefik mit TLS 1.3 Mindestversion, HSTS Header, SSL Labs A+ Rating.",
|
||||
"is_automated": True,
|
||||
"automation_tool": "SSL Labs / testssl.sh",
|
||||
"owner": "Infrastructure Team",
|
||||
"review_frequency_days": 90,
|
||||
},
|
||||
{
|
||||
"control_id": "CRYPTO-003",
|
||||
"domain": "crypto",
|
||||
"control_type": "preventive",
|
||||
"title": "Key Management",
|
||||
"description": "Kryptografische Schlüssel sicher in Vault gespeichert mit Rotation.",
|
||||
"pass_criteria": "Keys in Vault, automatische Rotation, keine Hardcoded Secrets.",
|
||||
"implementation_guidance": "HashiCorp Vault oder AWS KMS, Key-Rotation alle 90 Tage.",
|
||||
"code_reference": "vault/",
|
||||
"is_automated": True,
|
||||
"automation_tool": "HashiCorp Vault",
|
||||
"owner": "Infrastructure Team",
|
||||
"review_frequency_days": 90,
|
||||
},
|
||||
{
|
||||
"control_id": "CRYPTO-004",
|
||||
"domain": "crypto",
|
||||
"control_type": "preventive",
|
||||
"title": "Password Hashing",
|
||||
"description": "Passwörter werden mit bcrypt oder Argon2 gehasht.",
|
||||
"pass_criteria": "bcrypt/Argon2 verwendet, Cost Factor angemessen, keine MD5/SHA1.",
|
||||
"implementation_guidance": "bcrypt mit Cost >= 10, keine eigenentwickelten Hash-Funktionen.",
|
||||
"code_reference": "consent-service/internal/services/auth_service.go",
|
||||
"is_automated": True,
|
||||
"automation_tool": "Semgrep Rule",
|
||||
"owner": "Engineering Team",
|
||||
"review_frequency_days": 180,
|
||||
},
|
||||
|
||||
# =========================================================================
|
||||
# SDLC - Secure Development Lifecycle
|
||||
# =========================================================================
|
||||
{
|
||||
"control_id": "SDLC-001",
|
||||
"domain": "sdlc",
|
||||
"control_type": "detective",
|
||||
"title": "SAST Scanning",
|
||||
"description": "Static Application Security Testing in CI Pipeline integriert.",
|
||||
"pass_criteria": "Semgrep in CI, 0 High/Critical Findings, Blocking bei neuen Findings.",
|
||||
"implementation_guidance": "Semgrep mit OWASP Top 10 Rules, GitHub Actions Integration.",
|
||||
"code_reference": ".github/workflows/security.yml",
|
||||
"is_automated": True,
|
||||
"automation_tool": "Semgrep",
|
||||
"owner": "Engineering Team",
|
||||
"review_frequency_days": 7,
|
||||
},
|
||||
{
|
||||
"control_id": "SDLC-002",
|
||||
"domain": "sdlc",
|
||||
"control_type": "detective",
|
||||
"title": "Dependency Scanning",
|
||||
"description": "Automatische Überprüfung auf bekannte Schwachstellen in Dependencies.",
|
||||
"pass_criteria": "Trivy/Grype in CI, keine kritischen CVEs in Produktion.",
|
||||
"implementation_guidance": "Trivy für Container + Dependencies, Dependabot für automatische Updates.",
|
||||
"code_reference": ".github/workflows/security.yml",
|
||||
"is_automated": True,
|
||||
"automation_tool": "Trivy / Dependabot",
|
||||
"owner": "Engineering Team",
|
||||
"review_frequency_days": 7,
|
||||
},
|
||||
{
|
||||
"control_id": "SDLC-003",
|
||||
"domain": "sdlc",
|
||||
"control_type": "detective",
|
||||
"title": "Secret Detection",
|
||||
"description": "Automatische Erkennung von Secrets in Code und Commits.",
|
||||
"pass_criteria": "Gitleaks in CI, Pre-Commit-Hook aktiv, 0 Findings.",
|
||||
"implementation_guidance": "Gitleaks als Pre-Commit-Hook und in CI, Custom-Rules für eigene Secrets.",
|
||||
"code_reference": ".github/workflows/security.yml",
|
||||
"is_automated": True,
|
||||
"automation_tool": "Gitleaks",
|
||||
"owner": "Engineering Team",
|
||||
"review_frequency_days": 7,
|
||||
},
|
||||
{
|
||||
"control_id": "SDLC-004",
|
||||
"domain": "sdlc",
|
||||
"control_type": "preventive",
|
||||
"title": "Code Review",
|
||||
"description": "Alle Code-Änderungen werden von mindestens einem anderen Entwickler reviewed.",
|
||||
"pass_criteria": "100% PR-Coverage, mindestens 1 Approval pro PR.",
|
||||
"implementation_guidance": "GitHub Branch Protection, CODEOWNERS für kritische Pfade.",
|
||||
"is_automated": True,
|
||||
"automation_tool": "GitHub Branch Protection",
|
||||
"owner": "Engineering Lead",
|
||||
"review_frequency_days": 30,
|
||||
},
|
||||
{
|
||||
"control_id": "SDLC-005",
|
||||
"domain": "sdlc",
|
||||
"control_type": "preventive",
|
||||
"title": "SBOM Generation",
|
||||
"description": "Software Bill of Materials wird automatisch generiert.",
|
||||
"pass_criteria": "CycloneDX SBOM vorhanden, bei jedem Release aktualisiert.",
|
||||
"implementation_guidance": "cyclonedx-cli in Release-Pipeline, SBOM in GitHub Releases.",
|
||||
"is_automated": True,
|
||||
"automation_tool": "CycloneDX",
|
||||
"owner": "Engineering Team",
|
||||
"review_frequency_days": 30,
|
||||
},
|
||||
{
|
||||
"control_id": "SDLC-006",
|
||||
"domain": "sdlc",
|
||||
"control_type": "detective",
|
||||
"title": "Container Scanning",
|
||||
"description": "Docker Images werden auf Schwachstellen gescannt.",
|
||||
"pass_criteria": "Trivy Image Scan in CI, keine Critical/High in Base Images.",
|
||||
"implementation_guidance": "Trivy Image Scan vor Push zu Registry, Slim Base Images verwenden.",
|
||||
"code_reference": ".github/workflows/security.yml",
|
||||
"is_automated": True,
|
||||
"automation_tool": "Trivy Image Scan",
|
||||
"owner": "Engineering Team",
|
||||
"review_frequency_days": 7,
|
||||
},
|
||||
|
||||
# =========================================================================
|
||||
# OPS - Betrieb & Monitoring
|
||||
# =========================================================================
|
||||
{
|
||||
"control_id": "OPS-001",
|
||||
"domain": "ops",
|
||||
"control_type": "detective",
|
||||
"title": "Audit Logging",
|
||||
"description": "Alle sicherheitsrelevanten Events werden geloggt.",
|
||||
"pass_criteria": "Login/Logout, Consent-Änderungen, Admin-Aktionen geloggt, Retention >= 1 Jahr.",
|
||||
"implementation_guidance": "Structured Logging mit Request-ID, zentrale Log-Aggregation.",
|
||||
"code_reference": "backend/audit_log.py",
|
||||
"is_automated": True,
|
||||
"automation_tool": "Structured Logging",
|
||||
"owner": "Engineering Team",
|
||||
"review_frequency_days": 90,
|
||||
},
|
||||
{
|
||||
"control_id": "OPS-002",
|
||||
"domain": "ops",
|
||||
"control_type": "corrective",
|
||||
"title": "Backup & Recovery",
|
||||
"description": "Tägliche Backups mit getesteter Wiederherstellung.",
|
||||
"pass_criteria": "Tägliche Backups, RTO < 4h, RPO < 24h, Recovery-Test vierteljährlich.",
|
||||
"implementation_guidance": "Automatisierte Backups, Offsite-Kopie, dokumentierter Recovery-Prozess.",
|
||||
"code_reference": "scripts/backup.sh",
|
||||
"is_automated": True,
|
||||
"automation_tool": "PostgreSQL pg_dump / Docker Volumes",
|
||||
"owner": "Infrastructure Team",
|
||||
"review_frequency_days": 90,
|
||||
},
|
||||
{
|
||||
"control_id": "OPS-003",
|
||||
"domain": "ops",
|
||||
"control_type": "detective",
|
||||
"title": "Incident Response",
|
||||
"description": "Mean Time to Detect (MTTD) für Security Incidents < 24h.",
|
||||
"pass_criteria": "Alerting konfiguriert, MTTD < 24h, dokumentierte Incidents.",
|
||||
"implementation_guidance": "Alert-Regeln für Anomalien, Pager-Rotation, Incident-Runbooks.",
|
||||
"is_automated": True,
|
||||
"automation_tool": "Prometheus / Alertmanager",
|
||||
"owner": "Engineering Team",
|
||||
"review_frequency_days": 30,
|
||||
},
|
||||
{
|
||||
"control_id": "OPS-004",
|
||||
"domain": "ops",
|
||||
"control_type": "corrective",
|
||||
"title": "Vulnerability Management",
|
||||
"description": "Definierte Patch-SLAs für Schwachstellen nach Severity.",
|
||||
"pass_criteria": "Critical < 7 Tage, High < 30 Tage, Medium < 90 Tage.",
|
||||
"implementation_guidance": "Vulnerability Tracking in Issues, SLA-Monitoring, Patch-Prozess.",
|
||||
"is_automated": False,
|
||||
"owner": "Engineering Team",
|
||||
"review_frequency_days": 30,
|
||||
},
|
||||
{
|
||||
"control_id": "OPS-005",
|
||||
"domain": "ops",
|
||||
"control_type": "detective",
|
||||
"title": "Monitoring & Alerting",
|
||||
"description": "Uptime Monitoring mit 99.9% Verfügbarkeitsziel.",
|
||||
"pass_criteria": "Uptime >= 99.9% (monatlich), Alerts bei Ausfällen < 5 Min.",
|
||||
"implementation_guidance": "Health-Checks, Uptime-Monitoring (Uptime Kuma), Status Page.",
|
||||
"is_automated": True,
|
||||
"automation_tool": "Uptime Kuma / Prometheus",
|
||||
"owner": "Infrastructure Team",
|
||||
"review_frequency_days": 30,
|
||||
},
|
||||
|
||||
# =========================================================================
|
||||
# AI - KI-spezifisch (AI Act)
|
||||
# =========================================================================
|
||||
{
|
||||
"control_id": "AI-001",
|
||||
"domain": "ai",
|
||||
"control_type": "preventive",
|
||||
"title": "Training Data Governance",
|
||||
"description": "Dokumentation aller Trainingsdatenquellen und deren Lizenzierung.",
|
||||
"pass_criteria": "Datenquellen inventarisiert, Lizenzen dokumentiert, keine unlizenzierte Daten.",
|
||||
"implementation_guidance": "Data Catalog mit Quellen, Lizenzen, Verarbeitungszwecken.",
|
||||
"is_automated": False,
|
||||
"owner": "ML Team",
|
||||
"review_frequency_days": 180,
|
||||
},
|
||||
{
|
||||
"control_id": "AI-002",
|
||||
"domain": "ai",
|
||||
"control_type": "detective",
|
||||
"title": "Model Logging",
|
||||
"description": "Alle KI-Inferenzen werden für Nachvollziehbarkeit geloggt.",
|
||||
"pass_criteria": "Input/Output Logging für KI-Aufrufe, Retention >= 6 Monate.",
|
||||
"implementation_guidance": "LLM-Gateway mit Request/Response Logging, Token-Tracking.",
|
||||
"code_reference": "backend/llm_client.py",
|
||||
"is_automated": True,
|
||||
"automation_tool": "LLM Gateway Logging",
|
||||
"owner": "ML Team",
|
||||
"review_frequency_days": 90,
|
||||
},
|
||||
{
|
||||
"control_id": "AI-003",
|
||||
"domain": "ai",
|
||||
"control_type": "preventive",
|
||||
"title": "Human-in-the-Loop",
|
||||
"description": "Review-Prozess für KI-generierte Inhalte vor Veröffentlichung.",
|
||||
"pass_criteria": "HITL-Prozess dokumentiert, keine automatische Veröffentlichung ohne Review.",
|
||||
"implementation_guidance": "Review-Queue für KI-Outputs, Freigabe-Workflow.",
|
||||
"is_automated": False,
|
||||
"owner": "Product Team",
|
||||
"review_frequency_days": 90,
|
||||
},
|
||||
{
|
||||
"control_id": "AI-004",
|
||||
"domain": "ai",
|
||||
"control_type": "detective",
|
||||
"title": "Bias Monitoring",
|
||||
"description": "Regelmäßige Überprüfung von KI-Outputs auf Bias.",
|
||||
"pass_criteria": "Bias-Metriken definiert, vierteljährliche Überprüfung, Findings dokumentiert.",
|
||||
"implementation_guidance": "Fairness-Metriken (Demographic Parity, Equalized Odds), Bias-Audits.",
|
||||
"is_automated": False,
|
||||
"owner": "ML Team",
|
||||
"review_frequency_days": 90,
|
||||
},
|
||||
{
|
||||
"control_id": "AI-005",
|
||||
"domain": "ai",
|
||||
"control_type": "preventive",
|
||||
"title": "AI Act Risk Classification",
|
||||
"description": "Risikoklassifizierung der KI-Systeme gemäß EU AI Act dokumentiert.",
|
||||
"pass_criteria": "Alle KI-Systeme klassifiziert (minimal/limited/high/unacceptable), Dokumentation aktuell.",
|
||||
"implementation_guidance": "AI Act Risk Assessment Framework, Klassifizierung pro Use Case.",
|
||||
"is_automated": False,
|
||||
"owner": "Legal / ML Team",
|
||||
"review_frequency_days": 365,
|
||||
},
|
||||
|
||||
# =========================================================================
|
||||
# CRA - CRA & Supply Chain
|
||||
# =========================================================================
|
||||
{
|
||||
"control_id": "CRA-001",
|
||||
"domain": "cra",
|
||||
"control_type": "preventive",
|
||||
"title": "SBOM vorhanden",
|
||||
"description": "Software Bill of Materials im CycloneDX oder SPDX Format.",
|
||||
"pass_criteria": "SBOM vorhanden, automatisch generiert, bei Release aktualisiert.",
|
||||
"implementation_guidance": "CycloneDX in CI, SBOM in GitHub Releases, automatische Updates.",
|
||||
"is_automated": True,
|
||||
"automation_tool": "CycloneDX / SPDX",
|
||||
"owner": "Engineering Team",
|
||||
"review_frequency_days": 30,
|
||||
},
|
||||
{
|
||||
"control_id": "CRA-002",
|
||||
"domain": "cra",
|
||||
"control_type": "corrective",
|
||||
"title": "Vulnerability Disclosure",
|
||||
"description": "Öffentliche Vulnerability Disclosure Policy (VDP) vorhanden.",
|
||||
"pass_criteria": "VDP veröffentlicht, Kontaktdaten aktuell, Prozess dokumentiert.",
|
||||
"implementation_guidance": "security.txt, SECURITY.md in Repository, Responsible Disclosure Policy.",
|
||||
"code_reference": "SECURITY.md",
|
||||
"is_automated": False,
|
||||
"owner": "Security Team",
|
||||
"review_frequency_days": 365,
|
||||
},
|
||||
{
|
||||
"control_id": "CRA-003",
|
||||
"domain": "cra",
|
||||
"control_type": "corrective",
|
||||
"title": "Patch-SLA",
|
||||
"description": "Dokumentierte und eingehaltene Patch-Zeiten für Schwachstellen.",
|
||||
"pass_criteria": "SLAs definiert und kommuniziert, Einhaltung >= 95%.",
|
||||
"implementation_guidance": "Patch-SLA: Critical < 7d, High < 30d, Medium < 90d.",
|
||||
"is_automated": False,
|
||||
"owner": "Engineering Team",
|
||||
"review_frequency_days": 30,
|
||||
},
|
||||
{
|
||||
"control_id": "CRA-004",
|
||||
"domain": "cra",
|
||||
"control_type": "preventive",
|
||||
"title": "End-of-Support Policy",
|
||||
"description": "EOL-Datum für Produktversionen kommuniziert.",
|
||||
"pass_criteria": "Support-Zeiträume dokumentiert, Kunden informiert, EOL >= 24 Monate vor Ende.",
|
||||
"implementation_guidance": "Support-Matrix veröffentlichen, EOL-Kommunikation an Kunden.",
|
||||
"is_automated": False,
|
||||
"owner": "Product Team",
|
||||
"review_frequency_days": 365,
|
||||
},
|
||||
|
||||
# =========================================================================
|
||||
# AUD - Audit & Nachvollziehbarkeit
|
||||
# =========================================================================
|
||||
{
|
||||
"control_id": "AUD-001",
|
||||
"domain": "aud",
|
||||
"control_type": "detective",
|
||||
"title": "Traceability",
|
||||
"description": "Request-ID durchgängig in allen Logs für Nachverfolgbarkeit.",
|
||||
"pass_criteria": "Request-ID in allen Service-Logs, korrelierbar über Services.",
|
||||
"implementation_guidance": "X-Request-ID Header, Propagation über alle Services.",
|
||||
"code_reference": "backend/middleware/request_id.py",
|
||||
"is_automated": True,
|
||||
"automation_tool": "Request ID Middleware",
|
||||
"owner": "Engineering Team",
|
||||
"review_frequency_days": 90,
|
||||
},
|
||||
{
|
||||
"control_id": "AUD-002",
|
||||
"domain": "aud",
|
||||
"control_type": "corrective",
|
||||
"title": "Audit Export",
|
||||
"description": "ZIP-Export-Funktion für externe Prüfer funktional.",
|
||||
"pass_criteria": "Export-Funktion verfügbar, alle relevanten Daten enthalten, signiert.",
|
||||
"implementation_guidance": "Export mit Controls, Evidence, Risks als ZIP, SHA-256 Hash.",
|
||||
"code_reference": "backend/compliance/services/export_generator.py",
|
||||
"is_automated": True,
|
||||
"automation_tool": "Compliance Export Service",
|
||||
"owner": "Engineering Team",
|
||||
"review_frequency_days": 180,
|
||||
},
|
||||
{
|
||||
"control_id": "AUD-003",
|
||||
"domain": "aud",
|
||||
"control_type": "detective",
|
||||
"title": "Compliance Dashboard",
|
||||
"description": "Echtzeit-Compliance-Score und Status-Übersicht.",
|
||||
"pass_criteria": "Dashboard verfügbar, Score automatisch berechnet, Drill-Down möglich.",
|
||||
"implementation_guidance": "Dashboard mit Score-Berechnung, Regulation-Coverage, Trend-Anzeige.",
|
||||
"code_reference": "website/app/admin/compliance/page.tsx",
|
||||
"is_automated": True,
|
||||
"automation_tool": "Compliance Dashboard",
|
||||
"owner": "Engineering Team",
|
||||
"review_frequency_days": 30,
|
||||
},
|
||||
]
|
||||
986
backend/compliance/data/iso27001_annex_a.py
Normal file
986
backend/compliance/data/iso27001_annex_a.py
Normal file
@@ -0,0 +1,986 @@
|
||||
"""
|
||||
ISO 27001:2022 Annex A Controls Seed Data.
|
||||
|
||||
Contains all 93 controls from ISO/IEC 27001:2022 Annex A, organized into 4 themes:
|
||||
- A.5: Organizational controls (37 controls)
|
||||
- A.6: People controls (8 controls)
|
||||
- A.7: Physical controls (14 controls)
|
||||
- A.8: Technological controls (34 controls)
|
||||
|
||||
This data is used to populate the Statement of Applicability (SoA),
|
||||
which is MANDATORY for ISO 27001 certification.
|
||||
"""
|
||||
|
||||
from typing import List, Dict, Any, Optional
|
||||
|
||||
# ISO 27001:2022 Annex A Controls
|
||||
ISO27001_ANNEX_A_CONTROLS: List[Dict[str, Any]] = [
|
||||
# ==========================================================================
|
||||
# A.5 ORGANIZATIONAL CONTROLS (37 controls)
|
||||
# ==========================================================================
|
||||
{
|
||||
"control_id": "A.5.1",
|
||||
"title": "Policies for information security",
|
||||
"category": "organizational",
|
||||
"description": "Information security policy and topic-specific policies shall be defined, approved by management, published, communicated to and acknowledged by relevant personnel and relevant interested parties, and reviewed at planned intervals and if significant changes occur.",
|
||||
"iso_chapter": "5.2",
|
||||
"breakpilot_controls": ["GOV-001", "GOV-002"],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "Create and maintain an ISMS Master Policy and supporting policies for key topics (access control, cryptography, etc.)."
|
||||
},
|
||||
{
|
||||
"control_id": "A.5.2",
|
||||
"title": "Information security roles and responsibilities",
|
||||
"category": "organizational",
|
||||
"description": "Information security roles and responsibilities shall be defined and allocated according to the organization needs.",
|
||||
"iso_chapter": "5.3",
|
||||
"breakpilot_controls": ["GOV-003"],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "Define RACI matrix for security responsibilities, appoint Information Security Officer."
|
||||
},
|
||||
{
|
||||
"control_id": "A.5.3",
|
||||
"title": "Segregation of duties",
|
||||
"category": "organizational",
|
||||
"description": "Conflicting duties and conflicting areas of responsibility shall be segregated.",
|
||||
"iso_chapter": "5.3",
|
||||
"breakpilot_controls": ["IAM-003"],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "Implement role-based access control, separate development/test/production environments."
|
||||
},
|
||||
{
|
||||
"control_id": "A.5.4",
|
||||
"title": "Management responsibilities",
|
||||
"category": "organizational",
|
||||
"description": "Management shall require all personnel to apply information security in accordance with the established information security policy and topic-specific policies and procedures of the organization.",
|
||||
"iso_chapter": "5.1",
|
||||
"breakpilot_controls": ["GOV-001"],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "Management commitment documented in ISMS policy, security training mandatory."
|
||||
},
|
||||
{
|
||||
"control_id": "A.5.5",
|
||||
"title": "Contact with authorities",
|
||||
"category": "organizational",
|
||||
"description": "The organization shall establish and maintain contact with relevant authorities.",
|
||||
"iso_chapter": "4.2",
|
||||
"breakpilot_controls": ["GOV-004"],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "Maintain contact list for BSI, Datenschutzbehörde, CERT-Bund, law enforcement."
|
||||
},
|
||||
{
|
||||
"control_id": "A.5.6",
|
||||
"title": "Contact with special interest groups",
|
||||
"category": "organizational",
|
||||
"description": "The organization shall establish and maintain contact with special interest groups or other specialist security forums and professional associations.",
|
||||
"iso_chapter": "4.2",
|
||||
"breakpilot_controls": [],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "Participate in ISACA, ISC2, industry security forums, BSI security advisories."
|
||||
},
|
||||
{
|
||||
"control_id": "A.5.7",
|
||||
"title": "Threat intelligence",
|
||||
"category": "organizational",
|
||||
"description": "Information relating to information security threats shall be collected and analysed to produce threat intelligence.",
|
||||
"iso_chapter": "6.1",
|
||||
"breakpilot_controls": ["OPS-006"],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "Subscribe to threat feeds (BSI, MITRE ATT&CK), integrate with SIEM."
|
||||
},
|
||||
{
|
||||
"control_id": "A.5.8",
|
||||
"title": "Information security in project management",
|
||||
"category": "organizational",
|
||||
"description": "Information security shall be integrated into project management.",
|
||||
"iso_chapter": "6.1",
|
||||
"breakpilot_controls": ["SDLC-001"],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "Security requirements in all project charters, security review gates."
|
||||
},
|
||||
{
|
||||
"control_id": "A.5.9",
|
||||
"title": "Inventory of information and other associated assets",
|
||||
"category": "organizational",
|
||||
"description": "An inventory of information and other associated assets, including owners, shall be developed and maintained.",
|
||||
"iso_chapter": "7.5",
|
||||
"breakpilot_controls": ["GOV-005"],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "Maintain asset register with classification, owner, location for all IT assets."
|
||||
},
|
||||
{
|
||||
"control_id": "A.5.10",
|
||||
"title": "Acceptable use of information and other associated assets",
|
||||
"category": "organizational",
|
||||
"description": "Rules for the acceptable use of information and other associated assets shall be identified, documented and implemented.",
|
||||
"iso_chapter": "5.2",
|
||||
"breakpilot_controls": ["GOV-002"],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "Create Acceptable Use Policy, communicate to all employees."
|
||||
},
|
||||
{
|
||||
"control_id": "A.5.11",
|
||||
"title": "Return of assets",
|
||||
"category": "organizational",
|
||||
"description": "Personnel and other interested parties as appropriate shall return all the organization's assets in their possession upon change or termination of their employment, contract or agreement.",
|
||||
"iso_chapter": "7.3",
|
||||
"breakpilot_controls": [],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "Offboarding checklist includes asset return, access revocation within 24h."
|
||||
},
|
||||
{
|
||||
"control_id": "A.5.12",
|
||||
"title": "Classification of information",
|
||||
"category": "organizational",
|
||||
"description": "Information shall be classified according to the information security needs of the organization based on confidentiality, integrity, availability and relevant interested party requirements.",
|
||||
"iso_chapter": "7.5",
|
||||
"breakpilot_controls": ["PRIV-001"],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "Define classification levels: Public, Internal, Confidential, Strictly Confidential."
|
||||
},
|
||||
{
|
||||
"control_id": "A.5.13",
|
||||
"title": "Labelling of information",
|
||||
"category": "organizational",
|
||||
"description": "An appropriate set of procedures for information labelling shall be developed and implemented in accordance with the information classification scheme adopted by the organization.",
|
||||
"iso_chapter": "7.5",
|
||||
"breakpilot_controls": ["PRIV-001"],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "Document headers/footers with classification, email subject prefixes."
|
||||
},
|
||||
{
|
||||
"control_id": "A.5.14",
|
||||
"title": "Information transfer",
|
||||
"category": "organizational",
|
||||
"description": "Information transfer rules, procedures, or agreements shall be in place for all types of transfer facilities within the organization and between the organization and other parties.",
|
||||
"iso_chapter": "7.5",
|
||||
"breakpilot_controls": ["CRYPTO-002"],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "Encrypted file transfer, secure email, NDA for external transfers."
|
||||
},
|
||||
{
|
||||
"control_id": "A.5.15",
|
||||
"title": "Access control",
|
||||
"category": "organizational",
|
||||
"description": "Rules to control physical and logical access to information and other associated assets shall be established and implemented based on business and information security requirements.",
|
||||
"iso_chapter": "7.5",
|
||||
"breakpilot_controls": ["IAM-001", "IAM-002"],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "Access Control Policy, least privilege principle, need-to-know basis."
|
||||
},
|
||||
{
|
||||
"control_id": "A.5.16",
|
||||
"title": "Identity management",
|
||||
"category": "organizational",
|
||||
"description": "The full life cycle of identities shall be managed.",
|
||||
"iso_chapter": "7.5",
|
||||
"breakpilot_controls": ["IAM-001"],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "Unique user IDs, no shared accounts, regular access reviews."
|
||||
},
|
||||
{
|
||||
"control_id": "A.5.17",
|
||||
"title": "Authentication information",
|
||||
"category": "organizational",
|
||||
"description": "Allocation and management of authentication information shall be controlled by a management process including advising personnel on appropriate handling of authentication information.",
|
||||
"iso_chapter": "7.5",
|
||||
"breakpilot_controls": ["IAM-002", "IAM-004"],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "Password policy, MFA enrollment process, credential management."
|
||||
},
|
||||
{
|
||||
"control_id": "A.5.18",
|
||||
"title": "Access rights",
|
||||
"category": "organizational",
|
||||
"description": "Access rights to information and other associated assets shall be provisioned, reviewed, modified and removed in accordance with the organization's topic-specific policy on and rules for access control.",
|
||||
"iso_chapter": "7.5",
|
||||
"breakpilot_controls": ["IAM-001", "IAM-003"],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "Access request workflow, quarterly access reviews, privileged access management."
|
||||
},
|
||||
{
|
||||
"control_id": "A.5.19",
|
||||
"title": "Information security in supplier relationships",
|
||||
"category": "organizational",
|
||||
"description": "Processes and procedures shall be defined and implemented to manage the information security risks associated with the use of supplier's products or services.",
|
||||
"iso_chapter": "8.1",
|
||||
"breakpilot_controls": ["PRIV-005"],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "Supplier security assessment, DPA for data processors, vendor risk management."
|
||||
},
|
||||
{
|
||||
"control_id": "A.5.20",
|
||||
"title": "Addressing information security within supplier agreements",
|
||||
"category": "organizational",
|
||||
"description": "Relevant information security requirements shall be established and agreed with each supplier based on the type of supplier relationship.",
|
||||
"iso_chapter": "8.1",
|
||||
"breakpilot_controls": ["PRIV-005"],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "Security clauses in contracts, audit rights, incident notification requirements."
|
||||
},
|
||||
{
|
||||
"control_id": "A.5.21",
|
||||
"title": "Managing information security in the ICT supply chain",
|
||||
"category": "organizational",
|
||||
"description": "Processes and procedures shall be defined and implemented to manage the information security risks associated with the ICT products and services supply chain.",
|
||||
"iso_chapter": "8.1",
|
||||
"breakpilot_controls": ["CRA-001", "SDLC-005"],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "SBOM management, dependency scanning, supply chain security assessment."
|
||||
},
|
||||
{
|
||||
"control_id": "A.5.22",
|
||||
"title": "Monitoring, review and change management of supplier services",
|
||||
"category": "organizational",
|
||||
"description": "The organization shall regularly monitor, review, evaluate and manage change in supplier information security practices and service delivery.",
|
||||
"iso_chapter": "8.1",
|
||||
"breakpilot_controls": [],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "Annual supplier security reviews, SLA monitoring, change notification process."
|
||||
},
|
||||
{
|
||||
"control_id": "A.5.23",
|
||||
"title": "Information security for use of cloud services",
|
||||
"category": "organizational",
|
||||
"description": "Processes for acquisition, use, management and exit from cloud services shall be established in accordance with the organization's information security requirements.",
|
||||
"iso_chapter": "8.1",
|
||||
"breakpilot_controls": ["OPS-001"],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "Cloud security policy, CSP due diligence, data residency requirements."
|
||||
},
|
||||
{
|
||||
"control_id": "A.5.24",
|
||||
"title": "Information security incident management planning and preparation",
|
||||
"category": "organizational",
|
||||
"description": "The organization shall plan and prepare for managing information security incidents by defining, establishing and communicating information security incident management processes, roles and responsibilities.",
|
||||
"iso_chapter": "10.2",
|
||||
"breakpilot_controls": ["OPS-003"],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "Incident Response Plan, IR team roles, communication templates."
|
||||
},
|
||||
{
|
||||
"control_id": "A.5.25",
|
||||
"title": "Assessment and decision on information security events",
|
||||
"category": "organizational",
|
||||
"description": "The organization shall assess information security events and decide if they are to be categorized as information security incidents.",
|
||||
"iso_chapter": "10.2",
|
||||
"breakpilot_controls": ["OPS-003"],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "Event triage procedure, severity classification matrix, escalation criteria."
|
||||
},
|
||||
{
|
||||
"control_id": "A.5.26",
|
||||
"title": "Response to information security incidents",
|
||||
"category": "organizational",
|
||||
"description": "Information security incidents shall be responded to in accordance with the documented procedures.",
|
||||
"iso_chapter": "10.2",
|
||||
"breakpilot_controls": ["OPS-003"],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "Playbooks for common incidents, containment procedures, communication plan."
|
||||
},
|
||||
{
|
||||
"control_id": "A.5.27",
|
||||
"title": "Learning from information security incidents",
|
||||
"category": "organizational",
|
||||
"description": "Knowledge gained from information security incidents shall be used to strengthen and improve the information security controls.",
|
||||
"iso_chapter": "10.2",
|
||||
"breakpilot_controls": ["OPS-003"],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "Post-incident reviews, lessons learned documentation, control improvements."
|
||||
},
|
||||
{
|
||||
"control_id": "A.5.28",
|
||||
"title": "Collection of evidence",
|
||||
"category": "organizational",
|
||||
"description": "The organization shall establish and implement procedures for the identification, collection, acquisition and preservation of evidence related to information security events.",
|
||||
"iso_chapter": "10.2",
|
||||
"breakpilot_controls": ["OPS-003"],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "Chain of custody procedures, forensic imaging, log preservation."
|
||||
},
|
||||
{
|
||||
"control_id": "A.5.29",
|
||||
"title": "Information security during disruption",
|
||||
"category": "organizational",
|
||||
"description": "The organization shall plan how to maintain information security at an appropriate level during disruption.",
|
||||
"iso_chapter": "8.1",
|
||||
"breakpilot_controls": ["OPS-004"],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "BCP/DRP with security considerations, alternate processing sites."
|
||||
},
|
||||
{
|
||||
"control_id": "A.5.30",
|
||||
"title": "ICT readiness for business continuity",
|
||||
"category": "organizational",
|
||||
"description": "ICT readiness shall be planned, implemented, maintained and tested based on business continuity objectives and ICT continuity requirements.",
|
||||
"iso_chapter": "8.1",
|
||||
"breakpilot_controls": ["OPS-004"],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "ICT continuity plan, RTO/RPO definitions, DR testing."
|
||||
},
|
||||
{
|
||||
"control_id": "A.5.31",
|
||||
"title": "Legal, statutory, regulatory and contractual requirements",
|
||||
"category": "organizational",
|
||||
"description": "Legal, statutory, regulatory and contractual requirements relevant to information security and the organization's approach to meet these requirements shall be identified, documented and kept up to date.",
|
||||
"iso_chapter": "4.2",
|
||||
"breakpilot_controls": ["GOV-004"],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "Compliance register (GDPR, AI Act, CRA, NIS2), legal requirements tracking."
|
||||
},
|
||||
{
|
||||
"control_id": "A.5.32",
|
||||
"title": "Intellectual property rights",
|
||||
"category": "organizational",
|
||||
"description": "The organization shall implement appropriate procedures to protect intellectual property rights.",
|
||||
"iso_chapter": "4.2",
|
||||
"breakpilot_controls": [],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "License management, software inventory, OSS compliance."
|
||||
},
|
||||
{
|
||||
"control_id": "A.5.33",
|
||||
"title": "Protection of records",
|
||||
"category": "organizational",
|
||||
"description": "Records shall be protected from loss, destruction, falsification, unauthorized access and unauthorized release in accordance with legal, statutory, regulatory and contractual requirements.",
|
||||
"iso_chapter": "7.5",
|
||||
"breakpilot_controls": ["PRIV-001"],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "Records retention policy, secure storage, access controls, audit trails."
|
||||
},
|
||||
{
|
||||
"control_id": "A.5.34",
|
||||
"title": "Privacy and protection of PII",
|
||||
"category": "organizational",
|
||||
"description": "The organization shall identify and meet the requirements regarding the preservation of privacy and protection of PII according to applicable laws and regulations and contractual requirements.",
|
||||
"iso_chapter": "4.2",
|
||||
"breakpilot_controls": ["PRIV-001", "PRIV-003", "PRIV-006", "PRIV-007"],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "GDPR compliance, privacy by design, DPIA, consent management."
|
||||
},
|
||||
{
|
||||
"control_id": "A.5.35",
|
||||
"title": "Independent review of information security",
|
||||
"category": "organizational",
|
||||
"description": "The organization's approach to managing information security and its implementation including people, processes and technologies shall be reviewed independently at planned intervals, or when significant changes occur.",
|
||||
"iso_chapter": "9.2",
|
||||
"breakpilot_controls": [],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "Annual internal audit, external penetration testing, ISO 27001 certification audit."
|
||||
},
|
||||
{
|
||||
"control_id": "A.5.36",
|
||||
"title": "Compliance with policies, rules and standards for information security",
|
||||
"category": "organizational",
|
||||
"description": "Compliance with the organization's information security policy, topic-specific policies, rules and standards shall be regularly reviewed.",
|
||||
"iso_chapter": "9.2",
|
||||
"breakpilot_controls": [],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "Policy compliance monitoring, automated configuration checks, exception management."
|
||||
},
|
||||
{
|
||||
"control_id": "A.5.37",
|
||||
"title": "Documented operating procedures",
|
||||
"category": "organizational",
|
||||
"description": "Operating procedures for information processing facilities shall be documented and made available to personnel who need them.",
|
||||
"iso_chapter": "7.5",
|
||||
"breakpilot_controls": ["OPS-001"],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "Runbooks, SOPs, operational documentation in wiki/Confluence."
|
||||
},
|
||||
|
||||
# ==========================================================================
|
||||
# A.6 PEOPLE CONTROLS (8 controls)
|
||||
# ==========================================================================
|
||||
{
|
||||
"control_id": "A.6.1",
|
||||
"title": "Screening",
|
||||
"category": "people",
|
||||
"description": "Background verification checks on all candidates to become personnel shall be carried out prior to joining the organization and on an ongoing basis taking into consideration applicable laws, regulations and ethics and be proportional to the business requirements, the classification of the information to be accessed and the perceived risks.",
|
||||
"iso_chapter": "7.2",
|
||||
"breakpilot_controls": [],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "Background checks for employees with access to sensitive data/systems."
|
||||
},
|
||||
{
|
||||
"control_id": "A.6.2",
|
||||
"title": "Terms and conditions of employment",
|
||||
"category": "people",
|
||||
"description": "The employment contractual agreements shall state the personnel's and the organization's responsibilities for information security.",
|
||||
"iso_chapter": "7.2",
|
||||
"breakpilot_controls": [],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "Security clauses in employment contracts, NDA, acceptable use acknowledgment."
|
||||
},
|
||||
{
|
||||
"control_id": "A.6.3",
|
||||
"title": "Information security awareness, education and training",
|
||||
"category": "people",
|
||||
"description": "Personnel of the organization and relevant interested parties shall receive appropriate information security awareness, education and training and regular updates of the organization's information security policy, topic-specific policies and procedures, as relevant for their job function.",
|
||||
"iso_chapter": "7.2",
|
||||
"breakpilot_controls": ["GOV-006"],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "Annual security training, phishing simulations, role-specific training."
|
||||
},
|
||||
{
|
||||
"control_id": "A.6.4",
|
||||
"title": "Disciplinary process",
|
||||
"category": "people",
|
||||
"description": "A disciplinary process shall be formalized and communicated to take actions against personnel and other relevant interested parties who have committed an information security policy violation.",
|
||||
"iso_chapter": "7.2",
|
||||
"breakpilot_controls": [],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "Security policy violation consequences documented, HR process defined."
|
||||
},
|
||||
{
|
||||
"control_id": "A.6.5",
|
||||
"title": "Responsibilities after termination or change of employment",
|
||||
"category": "people",
|
||||
"description": "Information security responsibilities and duties that remain valid after termination or change of employment shall be defined, enforced and communicated to relevant personnel and other interested parties.",
|
||||
"iso_chapter": "7.3",
|
||||
"breakpilot_controls": [],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "Exit interview, NDA reminder, continued confidentiality obligations."
|
||||
},
|
||||
{
|
||||
"control_id": "A.6.6",
|
||||
"title": "Confidentiality or non-disclosure agreements",
|
||||
"category": "people",
|
||||
"description": "Confidentiality or non-disclosure agreements reflecting the organization's needs for the protection of information shall be identified, documented, regularly reviewed and signed by personnel and other relevant interested parties.",
|
||||
"iso_chapter": "7.2",
|
||||
"breakpilot_controls": [],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "NDA for all employees and contractors, annual review."
|
||||
},
|
||||
{
|
||||
"control_id": "A.6.7",
|
||||
"title": "Remote working",
|
||||
"category": "people",
|
||||
"description": "Security measures shall be implemented when personnel are working remotely to protect information accessed, processed or stored outside the organization's premises.",
|
||||
"iso_chapter": "7.5",
|
||||
"breakpilot_controls": ["IAM-005"],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "VPN, endpoint protection, secure home office guidelines."
|
||||
},
|
||||
{
|
||||
"control_id": "A.6.8",
|
||||
"title": "Information security event reporting",
|
||||
"category": "people",
|
||||
"description": "The organization shall provide a mechanism for personnel to report observed or suspected information security events through appropriate channels in a timely manner.",
|
||||
"iso_chapter": "10.2",
|
||||
"breakpilot_controls": ["OPS-003"],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "Security incident reporting portal, hotline, no-blame culture."
|
||||
},
|
||||
|
||||
# ==========================================================================
|
||||
# A.7 PHYSICAL CONTROLS (14 controls)
|
||||
# ==========================================================================
|
||||
{
|
||||
"control_id": "A.7.1",
|
||||
"title": "Physical security perimeters",
|
||||
"category": "physical",
|
||||
"description": "Security perimeters shall be defined and used to protect areas that contain information and other associated assets.",
|
||||
"iso_chapter": "7.5",
|
||||
"breakpilot_controls": [],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "Define secure areas (server rooms, offices), physical barriers."
|
||||
},
|
||||
{
|
||||
"control_id": "A.7.2",
|
||||
"title": "Physical entry",
|
||||
"category": "physical",
|
||||
"description": "Secure areas shall be protected by appropriate entry controls and access points.",
|
||||
"iso_chapter": "7.5",
|
||||
"breakpilot_controls": [],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "Access cards, visitor management, entry logs."
|
||||
},
|
||||
{
|
||||
"control_id": "A.7.3",
|
||||
"title": "Securing offices, rooms and facilities",
|
||||
"category": "physical",
|
||||
"description": "Physical security for offices, rooms and facilities shall be designed and implemented.",
|
||||
"iso_chapter": "7.5",
|
||||
"breakpilot_controls": [],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "Secure server rooms, locked cabinets, clean desk policy."
|
||||
},
|
||||
{
|
||||
"control_id": "A.7.4",
|
||||
"title": "Physical security monitoring",
|
||||
"category": "physical",
|
||||
"description": "Premises shall be continuously monitored for unauthorized physical access.",
|
||||
"iso_chapter": "7.5",
|
||||
"breakpilot_controls": [],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "CCTV, intrusion detection, security guards for sensitive areas."
|
||||
},
|
||||
{
|
||||
"control_id": "A.7.5",
|
||||
"title": "Protecting against physical and environmental threats",
|
||||
"category": "physical",
|
||||
"description": "Protection against physical and environmental threats, such as natural disasters and other intentional or unintentional physical threats to infrastructure shall be designed and implemented.",
|
||||
"iso_chapter": "7.5",
|
||||
"breakpilot_controls": [],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "Fire suppression, UPS, climate control, flood protection."
|
||||
},
|
||||
{
|
||||
"control_id": "A.7.6",
|
||||
"title": "Working in secure areas",
|
||||
"category": "physical",
|
||||
"description": "Security measures for working in secure areas shall be designed and implemented.",
|
||||
"iso_chapter": "7.5",
|
||||
"breakpilot_controls": [],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "Access restrictions, supervision requirements, no photography policy."
|
||||
},
|
||||
{
|
||||
"control_id": "A.7.7",
|
||||
"title": "Clear desk and clear screen",
|
||||
"category": "physical",
|
||||
"description": "Clear desk rules for papers and removable storage media and clear screen rules for information processing facilities shall be defined and appropriately enforced.",
|
||||
"iso_chapter": "7.5",
|
||||
"breakpilot_controls": [],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "Clear desk policy, screen lock after inactivity, secure document disposal."
|
||||
},
|
||||
{
|
||||
"control_id": "A.7.8",
|
||||
"title": "Equipment siting and protection",
|
||||
"category": "physical",
|
||||
"description": "Equipment shall be sited securely and protected.",
|
||||
"iso_chapter": "7.5",
|
||||
"breakpilot_controls": [],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "Secure server room location, rack security, cable management."
|
||||
},
|
||||
{
|
||||
"control_id": "A.7.9",
|
||||
"title": "Security of assets off-premises",
|
||||
"category": "physical",
|
||||
"description": "Off-site assets shall be protected.",
|
||||
"iso_chapter": "7.5",
|
||||
"breakpilot_controls": [],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "Laptop encryption, mobile device policy, asset tracking."
|
||||
},
|
||||
{
|
||||
"control_id": "A.7.10",
|
||||
"title": "Storage media",
|
||||
"category": "physical",
|
||||
"description": "Storage media shall be managed through their life cycle of acquisition, use, transportation and disposal in accordance with the organization's classification scheme and handling requirements.",
|
||||
"iso_chapter": "7.5",
|
||||
"breakpilot_controls": ["CRYPTO-003"],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "Media inventory, secure transport, cryptographic erasure."
|
||||
},
|
||||
{
|
||||
"control_id": "A.7.11",
|
||||
"title": "Supporting utilities",
|
||||
"category": "physical",
|
||||
"description": "Information processing facilities shall be protected from power failures and other disruptions caused by failures in supporting utilities.",
|
||||
"iso_chapter": "7.5",
|
||||
"breakpilot_controls": [],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "UPS, redundant power, generator backup for critical systems."
|
||||
},
|
||||
{
|
||||
"control_id": "A.7.12",
|
||||
"title": "Cabling security",
|
||||
"category": "physical",
|
||||
"description": "Cables carrying power and data or supporting information services shall be protected from interception, interference or damage.",
|
||||
"iso_chapter": "7.5",
|
||||
"breakpilot_controls": [],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "Secure cable routing, conduits, labeled and documented cabling."
|
||||
},
|
||||
{
|
||||
"control_id": "A.7.13",
|
||||
"title": "Equipment maintenance",
|
||||
"category": "physical",
|
||||
"description": "Equipment shall be maintained correctly to ensure availability, integrity and confidentiality of information.",
|
||||
"iso_chapter": "8.1",
|
||||
"breakpilot_controls": [],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "Maintenance schedules, authorized service personnel, maintenance logs."
|
||||
},
|
||||
{
|
||||
"control_id": "A.7.14",
|
||||
"title": "Secure disposal or re-use of equipment",
|
||||
"category": "physical",
|
||||
"description": "Items of equipment containing storage media shall be verified to ensure that any sensitive data and licensed software has been removed or securely overwritten prior to disposal or re-use.",
|
||||
"iso_chapter": "7.5",
|
||||
"breakpilot_controls": ["CRYPTO-003"],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "Secure data destruction, certificates of destruction, verified erasure."
|
||||
},
|
||||
|
||||
# ==========================================================================
|
||||
# A.8 TECHNOLOGICAL CONTROLS (34 controls)
|
||||
# ==========================================================================
|
||||
{
|
||||
"control_id": "A.8.1",
|
||||
"title": "User endpoint devices",
|
||||
"category": "technological",
|
||||
"description": "Information stored on, processed by or accessible via user endpoint devices shall be protected.",
|
||||
"iso_chapter": "8.1",
|
||||
"breakpilot_controls": ["IAM-005"],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "MDM, endpoint protection, device encryption, remote wipe capability."
|
||||
},
|
||||
{
|
||||
"control_id": "A.8.2",
|
||||
"title": "Privileged access rights",
|
||||
"category": "technological",
|
||||
"description": "The allocation and use of privileged access rights shall be restricted and managed.",
|
||||
"iso_chapter": "7.5",
|
||||
"breakpilot_controls": ["IAM-003"],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "PAM solution, just-in-time access, admin account monitoring."
|
||||
},
|
||||
{
|
||||
"control_id": "A.8.3",
|
||||
"title": "Information access restriction",
|
||||
"category": "technological",
|
||||
"description": "Access to information and other associated assets shall be restricted in accordance with the established topic-specific policy on access control.",
|
||||
"iso_chapter": "7.5",
|
||||
"breakpilot_controls": ["IAM-001", "IAM-002"],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "RBAC implementation, need-to-know enforcement, data classification."
|
||||
},
|
||||
{
|
||||
"control_id": "A.8.4",
|
||||
"title": "Access to source code",
|
||||
"category": "technological",
|
||||
"description": "Read and write access to source code, development tools and software libraries shall be appropriately managed.",
|
||||
"iso_chapter": "8.1",
|
||||
"breakpilot_controls": ["SDLC-004"],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "Git access controls, branch protection, code review requirements."
|
||||
},
|
||||
{
|
||||
"control_id": "A.8.5",
|
||||
"title": "Secure authentication",
|
||||
"category": "technological",
|
||||
"description": "Secure authentication technologies and procedures shall be implemented based on information access restrictions and the topic-specific policy on access control.",
|
||||
"iso_chapter": "7.5",
|
||||
"breakpilot_controls": ["IAM-002", "IAM-004"],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "MFA, SSO, OAuth 2.0/OIDC, password hashing (Argon2)."
|
||||
},
|
||||
{
|
||||
"control_id": "A.8.6",
|
||||
"title": "Capacity management",
|
||||
"category": "technological",
|
||||
"description": "The use of resources shall be monitored and adjusted in line with current and expected capacity requirements.",
|
||||
"iso_chapter": "8.1",
|
||||
"breakpilot_controls": ["OPS-001"],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "Resource monitoring, capacity planning, auto-scaling policies."
|
||||
},
|
||||
{
|
||||
"control_id": "A.8.7",
|
||||
"title": "Protection against malware",
|
||||
"category": "technological",
|
||||
"description": "Protection against malware shall be implemented and supported by appropriate user awareness.",
|
||||
"iso_chapter": "8.1",
|
||||
"breakpilot_controls": ["OPS-002"],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "Antivirus/EDR, email filtering, sandboxing, user awareness training."
|
||||
},
|
||||
{
|
||||
"control_id": "A.8.8",
|
||||
"title": "Management of technical vulnerabilities",
|
||||
"category": "technological",
|
||||
"description": "Information about technical vulnerabilities of information systems in use shall be obtained, the organization's exposure to such vulnerabilities shall be evaluated and appropriate measures shall be taken.",
|
||||
"iso_chapter": "8.1",
|
||||
"breakpilot_controls": ["SDLC-003", "OPS-005"],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "Vulnerability scanning, patch management, CVE monitoring."
|
||||
},
|
||||
{
|
||||
"control_id": "A.8.9",
|
||||
"title": "Configuration management",
|
||||
"category": "technological",
|
||||
"description": "Configurations, including security configurations, of hardware, software, services and networks shall be established, documented, implemented, monitored and reviewed.",
|
||||
"iso_chapter": "8.1",
|
||||
"breakpilot_controls": ["OPS-001"],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "IaC, configuration baselines, drift detection, hardening guides."
|
||||
},
|
||||
{
|
||||
"control_id": "A.8.10",
|
||||
"title": "Information deletion",
|
||||
"category": "technological",
|
||||
"description": "Information stored in information systems, devices or in any other storage media shall be deleted when no longer required.",
|
||||
"iso_chapter": "7.5",
|
||||
"breakpilot_controls": ["PRIV-006"],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "Data retention policies, automated deletion, right to erasure compliance."
|
||||
},
|
||||
{
|
||||
"control_id": "A.8.11",
|
||||
"title": "Data masking",
|
||||
"category": "technological",
|
||||
"description": "Data masking shall be used in accordance with the organization's topic-specific policy on access control and other related topic-specific policies, and business requirements, taking applicable legislation into consideration.",
|
||||
"iso_chapter": "7.5",
|
||||
"breakpilot_controls": ["PRIV-007"],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "PII masking in logs, test data anonymization, dynamic data masking."
|
||||
},
|
||||
{
|
||||
"control_id": "A.8.12",
|
||||
"title": "Data leakage prevention",
|
||||
"category": "technological",
|
||||
"description": "Data leakage prevention measures shall be applied to systems, networks and any other devices that process, store or transmit sensitive information.",
|
||||
"iso_chapter": "8.1",
|
||||
"breakpilot_controls": ["PRIV-007"],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "DLP tools, email scanning, USB restrictions, cloud access security."
|
||||
},
|
||||
{
|
||||
"control_id": "A.8.13",
|
||||
"title": "Information backup",
|
||||
"category": "technological",
|
||||
"description": "Backup copies of information, software and systems shall be maintained and regularly tested in accordance with the agreed topic-specific policy on backup.",
|
||||
"iso_chapter": "8.1",
|
||||
"breakpilot_controls": ["OPS-004"],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "3-2-1 backup strategy, encrypted backups, regular restore testing."
|
||||
},
|
||||
{
|
||||
"control_id": "A.8.14",
|
||||
"title": "Redundancy of information processing facilities",
|
||||
"category": "technological",
|
||||
"description": "Information processing facilities shall be implemented with redundancy sufficient to meet availability requirements.",
|
||||
"iso_chapter": "8.1",
|
||||
"breakpilot_controls": ["OPS-004"],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "High availability clusters, multi-region deployment, load balancing."
|
||||
},
|
||||
{
|
||||
"control_id": "A.8.15",
|
||||
"title": "Logging",
|
||||
"category": "technological",
|
||||
"description": "Logs that record activities, exceptions, faults and other relevant events shall be produced, stored, protected and analysed.",
|
||||
"iso_chapter": "9.1",
|
||||
"breakpilot_controls": ["OPS-002"],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "Centralized logging, log retention, tamper protection, SIEM integration."
|
||||
},
|
||||
{
|
||||
"control_id": "A.8.16",
|
||||
"title": "Monitoring activities",
|
||||
"category": "technological",
|
||||
"description": "Networks, systems and applications shall be monitored for anomalous behaviour and appropriate actions taken to evaluate potential information security incidents.",
|
||||
"iso_chapter": "9.1",
|
||||
"breakpilot_controls": ["OPS-002", "OPS-006"],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "SIEM, IDS/IPS, application monitoring, alerting thresholds."
|
||||
},
|
||||
{
|
||||
"control_id": "A.8.17",
|
||||
"title": "Clock synchronization",
|
||||
"category": "technological",
|
||||
"description": "The clocks of information processing systems used by the organization shall be synchronized to approved time sources.",
|
||||
"iso_chapter": "8.1",
|
||||
"breakpilot_controls": [],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "NTP configuration, consistent timezone, GPS/atomic clock sources."
|
||||
},
|
||||
{
|
||||
"control_id": "A.8.18",
|
||||
"title": "Use of privileged utility programs",
|
||||
"category": "technological",
|
||||
"description": "The use of utility programs that might be capable of overriding system and application controls shall be restricted and tightly controlled.",
|
||||
"iso_chapter": "8.1",
|
||||
"breakpilot_controls": ["IAM-003"],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "Restricted admin tools, logging of privileged actions, approval workflow."
|
||||
},
|
||||
{
|
||||
"control_id": "A.8.19",
|
||||
"title": "Installation of software on operational systems",
|
||||
"category": "technological",
|
||||
"description": "Procedures and measures shall be implemented to securely manage software installation on operational systems.",
|
||||
"iso_chapter": "8.1",
|
||||
"breakpilot_controls": ["SDLC-002"],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "Approved software list, installation controls, change management."
|
||||
},
|
||||
{
|
||||
"control_id": "A.8.20",
|
||||
"title": "Networks security",
|
||||
"category": "technological",
|
||||
"description": "Networks and network devices shall be secured, managed and controlled to protect information in systems and applications.",
|
||||
"iso_chapter": "8.1",
|
||||
"breakpilot_controls": ["OPS-001"],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "Network segmentation, firewall rules, VPN, network monitoring."
|
||||
},
|
||||
{
|
||||
"control_id": "A.8.21",
|
||||
"title": "Security of network services",
|
||||
"category": "technological",
|
||||
"description": "Security mechanisms, service levels and service requirements of network services shall be identified, implemented and monitored.",
|
||||
"iso_chapter": "8.1",
|
||||
"breakpilot_controls": ["OPS-001"],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "SLA monitoring, network service security assessments, DDoS protection."
|
||||
},
|
||||
{
|
||||
"control_id": "A.8.22",
|
||||
"title": "Segregation of networks",
|
||||
"category": "technological",
|
||||
"description": "Groups of information services, users and information systems shall be segregated in the organization's networks.",
|
||||
"iso_chapter": "8.1",
|
||||
"breakpilot_controls": ["OPS-001"],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "VLANs, network zones, micro-segmentation, DMZ for public services."
|
||||
},
|
||||
{
|
||||
"control_id": "A.8.23",
|
||||
"title": "Web filtering",
|
||||
"category": "technological",
|
||||
"description": "Access to external websites shall be managed to reduce exposure to malicious content.",
|
||||
"iso_chapter": "8.1",
|
||||
"breakpilot_controls": [],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "URL filtering, category-based blocking, SSL inspection."
|
||||
},
|
||||
{
|
||||
"control_id": "A.8.24",
|
||||
"title": "Use of cryptography",
|
||||
"category": "technological",
|
||||
"description": "Rules for the effective use of cryptography, including cryptographic key management, shall be defined and implemented.",
|
||||
"iso_chapter": "8.1",
|
||||
"breakpilot_controls": ["CRYPTO-001", "CRYPTO-002", "CRYPTO-004"],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "Cryptography policy, approved algorithms, key management procedures."
|
||||
},
|
||||
{
|
||||
"control_id": "A.8.25",
|
||||
"title": "Secure development life cycle",
|
||||
"category": "technological",
|
||||
"description": "Rules for the secure development of software and systems shall be established and applied.",
|
||||
"iso_chapter": "8.1",
|
||||
"breakpilot_controls": ["SDLC-001", "SDLC-002"],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "SSDLC policy, secure coding guidelines, security requirements."
|
||||
},
|
||||
{
|
||||
"control_id": "A.8.26",
|
||||
"title": "Application security requirements",
|
||||
"category": "technological",
|
||||
"description": "Information security requirements shall be identified, specified and approved when developing or acquiring applications.",
|
||||
"iso_chapter": "8.1",
|
||||
"breakpilot_controls": ["SDLC-001"],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "Security requirements checklist, threat modeling, security acceptance criteria."
|
||||
},
|
||||
{
|
||||
"control_id": "A.8.27",
|
||||
"title": "Secure system architecture and engineering principles",
|
||||
"category": "technological",
|
||||
"description": "Principles for engineering secure systems shall be established, documented, maintained and applied to any information system development activities.",
|
||||
"iso_chapter": "8.1",
|
||||
"breakpilot_controls": ["SDLC-001", "GOV-001"],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "Security architecture principles, defense in depth, least privilege."
|
||||
},
|
||||
{
|
||||
"control_id": "A.8.28",
|
||||
"title": "Secure coding",
|
||||
"category": "technological",
|
||||
"description": "Secure coding principles shall be applied to software development.",
|
||||
"iso_chapter": "8.1",
|
||||
"breakpilot_controls": ["SDLC-001", "SDLC-006"],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "OWASP guidelines, secure coding training, code review checklists."
|
||||
},
|
||||
{
|
||||
"control_id": "A.8.29",
|
||||
"title": "Security testing in development and acceptance",
|
||||
"category": "technological",
|
||||
"description": "Security testing processes shall be defined and implemented in the development life cycle.",
|
||||
"iso_chapter": "8.1",
|
||||
"breakpilot_controls": ["SDLC-002", "SDLC-003"],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "SAST, DAST, penetration testing, security acceptance testing."
|
||||
},
|
||||
{
|
||||
"control_id": "A.8.30",
|
||||
"title": "Outsourced development",
|
||||
"category": "technological",
|
||||
"description": "The organization shall direct, monitor and review the activities related to outsourced system development.",
|
||||
"iso_chapter": "8.1",
|
||||
"breakpilot_controls": [],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "Vendor security requirements, code review rights, security testing."
|
||||
},
|
||||
{
|
||||
"control_id": "A.8.31",
|
||||
"title": "Separation of development, test and production environments",
|
||||
"category": "technological",
|
||||
"description": "Development, testing and production environments shall be separated and secured.",
|
||||
"iso_chapter": "8.1",
|
||||
"breakpilot_controls": ["SDLC-002"],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "Separate environments, access controls, data anonymization in test."
|
||||
},
|
||||
{
|
||||
"control_id": "A.8.32",
|
||||
"title": "Change management",
|
||||
"category": "technological",
|
||||
"description": "Changes to information processing facilities and information systems shall be subject to change management procedures.",
|
||||
"iso_chapter": "8.1",
|
||||
"breakpilot_controls": ["OPS-001"],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "Change advisory board, change request workflow, rollback procedures."
|
||||
},
|
||||
{
|
||||
"control_id": "A.8.33",
|
||||
"title": "Test information",
|
||||
"category": "technological",
|
||||
"description": "Test information shall be appropriately selected, protected and managed.",
|
||||
"iso_chapter": "8.1",
|
||||
"breakpilot_controls": ["PRIV-007"],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "Synthetic test data, PII removal, test data management policy."
|
||||
},
|
||||
{
|
||||
"control_id": "A.8.34",
|
||||
"title": "Protection of information systems during audit testing",
|
||||
"category": "technological",
|
||||
"description": "Audit tests and other assurance activities involving assessment of operational systems shall be planned and agreed between the tester and appropriate management.",
|
||||
"iso_chapter": "9.2",
|
||||
"breakpilot_controls": [],
|
||||
"default_applicable": True,
|
||||
"implementation_guidance": "Audit planning, system access controls during audits, audit trails."
|
||||
},
|
||||
]
|
||||
|
||||
|
||||
def get_annex_a_by_category(category: str) -> List[Dict[str, Any]]:
|
||||
"""Get Annex A controls filtered by category."""
|
||||
return [c for c in ISO27001_ANNEX_A_CONTROLS if c["category"] == category]
|
||||
|
||||
|
||||
def get_annex_a_control(control_id: str) -> Optional[Dict[str, Any]]:
|
||||
"""Get a specific Annex A control by ID."""
|
||||
for control in ISO27001_ANNEX_A_CONTROLS:
|
||||
if control["control_id"] == control_id:
|
||||
return control
|
||||
return None
|
||||
|
||||
|
||||
# Summary statistics
|
||||
ANNEX_A_SUMMARY = {
|
||||
"total_controls": len(ISO27001_ANNEX_A_CONTROLS),
|
||||
"organizational_controls": len([c for c in ISO27001_ANNEX_A_CONTROLS if c["category"] == "organizational"]),
|
||||
"people_controls": len([c for c in ISO27001_ANNEX_A_CONTROLS if c["category"] == "people"]),
|
||||
"physical_controls": len([c for c in ISO27001_ANNEX_A_CONTROLS if c["category"] == "physical"]),
|
||||
"technological_controls": len([c for c in ISO27001_ANNEX_A_CONTROLS if c["category"] == "technological"]),
|
||||
}
|
||||
247
backend/compliance/data/regulations.py
Normal file
247
backend/compliance/data/regulations.py
Normal file
@@ -0,0 +1,247 @@
|
||||
"""
|
||||
Seed data for Regulations.
|
||||
|
||||
16 EU Regulations + 3 BSI-TR documents covering:
|
||||
- A. Datenschutz & Datenübermittlung
|
||||
- B. KI-Regulierung
|
||||
- C. Cybersecurity & Produktsicherheit
|
||||
- D. Datenökonomie & Interoperabilität
|
||||
- E. Plattform-Pflichten
|
||||
- F. Barrierefreiheit
|
||||
- G. IP & Urheberrecht
|
||||
- H. Produkthaftung
|
||||
- I. BSI-Standards (Deutschland)
|
||||
"""
|
||||
|
||||
from datetime import date
|
||||
|
||||
REGULATIONS_SEED = [
|
||||
# =========================================================================
|
||||
# A. Datenschutz & Datenübermittlung
|
||||
# =========================================================================
|
||||
{
|
||||
"code": "GDPR",
|
||||
"name": "DSGVO",
|
||||
"full_name": "Verordnung (EU) 2016/679 - Datenschutz-Grundverordnung",
|
||||
"regulation_type": "eu_regulation",
|
||||
"source_url": "https://eur-lex.europa.eu/eli/reg/2016/679/oj/eng",
|
||||
"effective_date": date(2018, 5, 25),
|
||||
"description": "Grundverordnung zum Schutz natürlicher Personen bei der Verarbeitung personenbezogener Daten. Kernstück der EU-Datenschutzgesetzgebung.",
|
||||
"is_active": True,
|
||||
},
|
||||
{
|
||||
"code": "EPRIVACY",
|
||||
"name": "ePrivacy-Richtlinie",
|
||||
"full_name": "Richtlinie 2002/58/EG - Datenschutz in der elektronischen Kommunikation",
|
||||
"regulation_type": "eu_directive",
|
||||
"source_url": "https://eur-lex.europa.eu/eli/dir/2002/58/oj/eng",
|
||||
"effective_date": date(2002, 7, 31),
|
||||
"description": "Regelt den Datenschutz in der elektronischen Kommunikation, insbesondere Cookies, Tracking und elektronisches Marketing.",
|
||||
"is_active": True,
|
||||
},
|
||||
{
|
||||
"code": "TDDDG",
|
||||
"name": "TDDDG",
|
||||
"full_name": "Telekommunikation-Digitale-Dienste-Datenschutz-Gesetz (ehem. TTDSG)",
|
||||
"regulation_type": "de_law",
|
||||
"source_url": "https://www.gesetze-im-internet.de/ttdsg/",
|
||||
"effective_date": date(2021, 12, 1),
|
||||
"description": "Deutsche Umsetzung der ePrivacy-Richtlinie. Regelt Datenschutz bei Telemedien und Telekommunikation.",
|
||||
"is_active": True,
|
||||
},
|
||||
{
|
||||
"code": "SCC",
|
||||
"name": "Standardvertragsklauseln",
|
||||
"full_name": "Durchführungsbeschluss (EU) 2021/914 - Standardvertragsklauseln für Drittlandtransfers",
|
||||
"regulation_type": "eu_regulation",
|
||||
"source_url": "https://eur-lex.europa.eu/eli/dec_impl/2021/914/oj/eng",
|
||||
"effective_date": date(2021, 6, 27),
|
||||
"description": "Standardvertragsklauseln für die Übermittlung personenbezogener Daten an Drittländer gemäß DSGVO.",
|
||||
"is_active": True,
|
||||
},
|
||||
{
|
||||
"code": "DPF",
|
||||
"name": "EU-US Data Privacy Framework",
|
||||
"full_name": "Durchführungsbeschluss (EU) 2023/1795 - Angemessenheitsbeschluss EU-US",
|
||||
"regulation_type": "eu_regulation",
|
||||
"source_url": "https://eur-lex.europa.eu/eli/dec_impl/2023/1795/oj",
|
||||
"effective_date": date(2023, 7, 10),
|
||||
"description": "Angemessenheitsbeschluss für Datenübermittlungen in die USA unter dem EU-US Data Privacy Framework.",
|
||||
"is_active": True,
|
||||
},
|
||||
|
||||
# =========================================================================
|
||||
# B. KI-Regulierung
|
||||
# =========================================================================
|
||||
{
|
||||
"code": "AIACT",
|
||||
"name": "EU AI Act",
|
||||
"full_name": "Verordnung (EU) 2024/1689 - Verordnung zur Festlegung harmonisierter Vorschriften für KI",
|
||||
"regulation_type": "eu_regulation",
|
||||
"source_url": "https://eur-lex.europa.eu/eli/reg/2024/1689/oj/eng",
|
||||
"effective_date": date(2024, 8, 1),
|
||||
"description": "EU-Verordnung zur Regulierung von KI-Systemen. Klassifiziert KI-Systeme nach Risikostufen und definiert Anforderungen für Hochrisiko-KI.",
|
||||
"is_active": True,
|
||||
},
|
||||
|
||||
# =========================================================================
|
||||
# C. Cybersecurity & Produktsicherheit
|
||||
# =========================================================================
|
||||
{
|
||||
"code": "CRA",
|
||||
"name": "Cyber Resilience Act",
|
||||
"full_name": "Verordnung (EU) 2024/2847 - Horizontale Cybersicherheitsanforderungen für Produkte",
|
||||
"regulation_type": "eu_regulation",
|
||||
"source_url": "https://eur-lex.europa.eu/eli/reg/2024/2847/oj/eng",
|
||||
"effective_date": date(2024, 12, 10),
|
||||
"description": "Cybersicherheitsanforderungen für Produkte mit digitalen Elementen. Verpflichtet zu SBOM, Vulnerability Disclosure und Support-Zeiträumen.",
|
||||
"is_active": True,
|
||||
},
|
||||
{
|
||||
"code": "NIS2",
|
||||
"name": "NIS2-Richtlinie",
|
||||
"full_name": "Richtlinie (EU) 2022/2555 - Maßnahmen für hohes Cybersicherheitsniveau",
|
||||
"regulation_type": "eu_directive",
|
||||
"source_url": "https://eur-lex.europa.eu/eli/dir/2022/2555/oj/eng",
|
||||
"effective_date": date(2024, 10, 17),
|
||||
"description": "Richtlinie zur Stärkung der Cybersicherheit wesentlicher und wichtiger Einrichtungen in der EU.",
|
||||
"is_active": True,
|
||||
},
|
||||
{
|
||||
"code": "EUCSA",
|
||||
"name": "EU Cybersecurity Act",
|
||||
"full_name": "Verordnung (EU) 2019/881 - ENISA und Cybersicherheitszertifizierung",
|
||||
"regulation_type": "eu_regulation",
|
||||
"source_url": "https://eur-lex.europa.eu/eli/reg/2019/881/oj/eng",
|
||||
"effective_date": date(2019, 6, 27),
|
||||
"description": "Stärkt ENISA und etabliert einen EU-weiten Rahmen für Cybersicherheitszertifizierung von IKT-Produkten.",
|
||||
"is_active": True,
|
||||
},
|
||||
|
||||
# =========================================================================
|
||||
# D. Datenökonomie & Interoperabilität
|
||||
# =========================================================================
|
||||
{
|
||||
"code": "DATAACT",
|
||||
"name": "Data Act",
|
||||
"full_name": "Verordnung (EU) 2023/2854 - Harmonisierte Vorschriften für fairen Datenzugang",
|
||||
"regulation_type": "eu_regulation",
|
||||
"source_url": "https://eur-lex.europa.eu/eli/reg/2023/2854/oj/eng",
|
||||
"effective_date": date(2025, 9, 12),
|
||||
"description": "Regelt den fairen Zugang zu und die Nutzung von Daten. Betrifft IoT-Daten, Cloud-Wechsel und B2B-Datenfreigabe.",
|
||||
"is_active": True,
|
||||
},
|
||||
{
|
||||
"code": "DGA",
|
||||
"name": "Data Governance Act",
|
||||
"full_name": "Verordnung (EU) 2022/868 - Europäische Daten-Governance",
|
||||
"regulation_type": "eu_regulation",
|
||||
"source_url": "https://eur-lex.europa.eu/eli/reg/2022/868/oj/eng",
|
||||
"effective_date": date(2023, 9, 24),
|
||||
"description": "Rahmenwerk für die Weiterverwendung öffentlicher Daten und Datenvermittlungsdienste.",
|
||||
"is_active": True,
|
||||
},
|
||||
|
||||
# =========================================================================
|
||||
# E. Plattform-Pflichten
|
||||
# =========================================================================
|
||||
{
|
||||
"code": "DSA",
|
||||
"name": "Digital Services Act",
|
||||
"full_name": "Verordnung (EU) 2022/2065 - Binnenmarkt für digitale Dienste",
|
||||
"regulation_type": "eu_regulation",
|
||||
"source_url": "https://eur-lex.europa.eu/eli/reg/2022/2065/oj/eng",
|
||||
"effective_date": date(2024, 2, 17),
|
||||
"description": "Reguliert digitale Dienste und Plattformen. Transparenzpflichten, Content Moderation, Beschwerdemechanismen.",
|
||||
"is_active": True,
|
||||
},
|
||||
|
||||
# =========================================================================
|
||||
# F. Barrierefreiheit
|
||||
# =========================================================================
|
||||
{
|
||||
"code": "EAA",
|
||||
"name": "European Accessibility Act",
|
||||
"full_name": "Richtlinie (EU) 2019/882 - Barrierefreiheitsanforderungen für Produkte und Dienstleistungen",
|
||||
"regulation_type": "eu_directive",
|
||||
"source_url": "https://eur-lex.europa.eu/eli/dir/2019/882/oj/eng",
|
||||
"effective_date": date(2025, 6, 28),
|
||||
"description": "Barrierefreiheitsanforderungen für digitale Produkte und Dienstleistungen. Relevant für Web, Mobile Apps, E-Commerce.",
|
||||
"is_active": True,
|
||||
},
|
||||
|
||||
# =========================================================================
|
||||
# G. IP & Urheberrecht
|
||||
# =========================================================================
|
||||
{
|
||||
"code": "DSM",
|
||||
"name": "DSM-Urheberrechtsrichtlinie",
|
||||
"full_name": "Richtlinie (EU) 2019/790 - Urheberrecht im digitalen Binnenmarkt",
|
||||
"regulation_type": "eu_directive",
|
||||
"source_url": "https://eur-lex.europa.eu/eli/dir/2019/790/oj/eng",
|
||||
"effective_date": date(2021, 6, 7),
|
||||
"description": "Modernisiert das Urheberrecht für den digitalen Binnenmarkt. Betrifft Text- und Data-Mining, Upload-Filter.",
|
||||
"is_active": True,
|
||||
},
|
||||
|
||||
# =========================================================================
|
||||
# H. Produkthaftung
|
||||
# =========================================================================
|
||||
{
|
||||
"code": "PLD",
|
||||
"name": "Produkthaftungsrichtlinie",
|
||||
"full_name": "Richtlinie (EU) 2024/2853 - Haftung für fehlerhafte Produkte",
|
||||
"regulation_type": "eu_directive",
|
||||
"source_url": "https://eur-lex.europa.eu/eli/dir/2024/2853/oj/eng",
|
||||
"effective_date": date(2026, 12, 9),
|
||||
"description": "Neue Produkthaftungsrichtlinie. Erweitert Haftung auf Software und KI-Systeme.",
|
||||
"is_active": True,
|
||||
},
|
||||
{
|
||||
"code": "GPSR",
|
||||
"name": "General Product Safety Regulation",
|
||||
"full_name": "Verordnung (EU) 2023/988 - Allgemeine Produktsicherheit",
|
||||
"regulation_type": "eu_regulation",
|
||||
"source_url": "https://eur-lex.europa.eu/eli/reg/2023/988/oj/eng",
|
||||
"effective_date": date(2024, 12, 13),
|
||||
"description": "Allgemeine Produktsicherheitsverordnung. Sicherheitsanforderungen für Verbraucherprodukte inkl. digitaler Produkte.",
|
||||
"is_active": True,
|
||||
},
|
||||
|
||||
# =========================================================================
|
||||
# I. BSI-Standards (Deutschland)
|
||||
# =========================================================================
|
||||
{
|
||||
"code": "BSI-TR-03161-1",
|
||||
"name": "BSI-TR-03161 Teil 1",
|
||||
"full_name": "BSI Technische Richtlinie - Anforderungen an Anwendungen im Gesundheitswesen - Teil 1: Mobile Anwendungen",
|
||||
"regulation_type": "bsi_standard",
|
||||
"source_url": "https://www.bsi.bund.de/SharedDocs/Downloads/DE/BSI/Publikationen/TechnischeRichtlinien/TR03161/BSI-TR-03161-1.html",
|
||||
"local_pdf_path": "/docs/BSI-TR-03161-1.pdf",
|
||||
"effective_date": date(2022, 1, 1),
|
||||
"description": "BSI Richtlinie für mobile Anwendungen. Teil 1: Allgemeine Sicherheitsanforderungen.",
|
||||
"is_active": True,
|
||||
},
|
||||
{
|
||||
"code": "BSI-TR-03161-2",
|
||||
"name": "BSI-TR-03161 Teil 2",
|
||||
"full_name": "BSI Technische Richtlinie - Anforderungen an Anwendungen im Gesundheitswesen - Teil 2: Web-Anwendungen",
|
||||
"regulation_type": "bsi_standard",
|
||||
"source_url": "https://www.bsi.bund.de/SharedDocs/Downloads/DE/BSI/Publikationen/TechnischeRichtlinien/TR03161/BSI-TR-03161-2.html",
|
||||
"local_pdf_path": "/docs/BSI-TR-03161-2.pdf",
|
||||
"effective_date": date(2022, 1, 1),
|
||||
"description": "BSI Richtlinie für Web-Anwendungen. Teil 2: Sicherheitsanforderungen für Web-Frontends und APIs.",
|
||||
"is_active": True,
|
||||
},
|
||||
{
|
||||
"code": "BSI-TR-03161-3",
|
||||
"name": "BSI-TR-03161 Teil 3",
|
||||
"full_name": "BSI Technische Richtlinie - Anforderungen an Anwendungen im Gesundheitswesen - Teil 3: Hintergrundsysteme",
|
||||
"regulation_type": "bsi_standard",
|
||||
"source_url": "https://www.bsi.bund.de/SharedDocs/Downloads/DE/BSI/Publikationen/TechnischeRichtlinien/TR03161/BSI-TR-03161-3.html",
|
||||
"local_pdf_path": "/docs/BSI-TR-03161-3.pdf",
|
||||
"effective_date": date(2022, 1, 1),
|
||||
"description": "BSI Richtlinie für Hintergrundsysteme. Teil 3: Anforderungen an Backend-Systeme und Infrastruktur.",
|
||||
"is_active": True,
|
||||
},
|
||||
]
|
||||
391
backend/compliance/data/requirements.py
Normal file
391
backend/compliance/data/requirements.py
Normal file
@@ -0,0 +1,391 @@
|
||||
"""
|
||||
Seed data for Requirements.
|
||||
|
||||
Key requirements from:
|
||||
- GDPR: Art. 5, 25, 28, 30, 32, 35 (Core Articles)
|
||||
- AI Act: Art. 6, 9, 13, 14, 15 (High-Risk Requirements)
|
||||
- CRA: Art. 10-15 (Vulnerability Handling)
|
||||
- BSI-TR-03161: Security Requirements
|
||||
"""
|
||||
|
||||
REQUIREMENTS_SEED = [
|
||||
# =========================================================================
|
||||
# GDPR - Datenschutz-Grundverordnung
|
||||
# =========================================================================
|
||||
{
|
||||
"regulation_code": "GDPR",
|
||||
"article": "Art. 5",
|
||||
"paragraph": "(1)(a)",
|
||||
"title": "Rechtmäßigkeit, Verarbeitung nach Treu und Glauben, Transparenz",
|
||||
"description": "Personenbezogene Daten müssen rechtmäßig, nach Treu und Glauben und transparent verarbeitet werden.",
|
||||
"requirement_text": "Personenbezogene Daten müssen auf rechtmäßige Weise, nach Treu und Glauben und in einer für die betroffene Person nachvollziehbaren Weise verarbeitet werden.",
|
||||
"breakpilot_interpretation": "Breakpilot verarbeitet Daten nur mit gültiger Rechtsgrundlage (Einwilligung, Vertrag). Transparente Datenschutzerklärung und Consent-Management.",
|
||||
"is_applicable": True,
|
||||
"priority": 1,
|
||||
},
|
||||
{
|
||||
"regulation_code": "GDPR",
|
||||
"article": "Art. 5",
|
||||
"paragraph": "(1)(b)",
|
||||
"title": "Zweckbindung",
|
||||
"description": "Daten dürfen nur für festgelegte, eindeutige und legitime Zwecke erhoben werden.",
|
||||
"requirement_text": "Personenbezogene Daten müssen für festgelegte, eindeutige und legitime Zwecke erhoben werden und dürfen nicht in einer mit diesen Zwecken nicht zu vereinbarenden Weise weiterverarbeitet werden.",
|
||||
"breakpilot_interpretation": "Jeder Verarbeitungszweck ist im Consent-System klar definiert. Keine Zweckänderung ohne neue Einwilligung.",
|
||||
"is_applicable": True,
|
||||
"priority": 1,
|
||||
},
|
||||
{
|
||||
"regulation_code": "GDPR",
|
||||
"article": "Art. 5",
|
||||
"paragraph": "(1)(c)",
|
||||
"title": "Datenminimierung",
|
||||
"description": "Datenerhebung muss auf das notwendige Maß beschränkt sein.",
|
||||
"requirement_text": "Personenbezogene Daten müssen dem Zweck angemessen und erheblich sowie auf das für die Zwecke der Verarbeitung notwendige Maß beschränkt sein.",
|
||||
"breakpilot_interpretation": "Privacy by Design: Nur erforderliche Daten werden erhoben. Keine überschüssigen Profilfelder.",
|
||||
"is_applicable": True,
|
||||
"priority": 1,
|
||||
},
|
||||
{
|
||||
"regulation_code": "GDPR",
|
||||
"article": "Art. 5",
|
||||
"paragraph": "(1)(f)",
|
||||
"title": "Integrität und Vertraulichkeit",
|
||||
"description": "Daten müssen vor unbefugter Verarbeitung und Verlust geschützt sein.",
|
||||
"requirement_text": "Personenbezogene Daten müssen in einer Weise verarbeitet werden, die eine angemessene Sicherheit gewährleistet, einschließlich Schutz vor unbefugter oder unrechtmäßiger Verarbeitung und vor unbeabsichtigtem Verlust, Zerstörung oder Schädigung.",
|
||||
"breakpilot_interpretation": "Verschlüsselung at Rest und in Transit, RBAC, Audit Logging, regelmäßige Backups.",
|
||||
"is_applicable": True,
|
||||
"priority": 1,
|
||||
},
|
||||
{
|
||||
"regulation_code": "GDPR",
|
||||
"article": "Art. 25",
|
||||
"paragraph": "(1)",
|
||||
"title": "Datenschutz durch Technikgestaltung",
|
||||
"description": "Privacy by Design - Datenschutz muss in die Entwicklung eingebaut werden.",
|
||||
"requirement_text": "Der Verantwortliche trifft sowohl zum Zeitpunkt der Festlegung der Mittel als auch zum Zeitpunkt der Verarbeitung geeignete technische und organisatorische Maßnahmen.",
|
||||
"breakpilot_interpretation": "PbD-Checkliste für neue Features, Datenschutz-Review im Development-Prozess, Standard-Datenschutzeinstellungen.",
|
||||
"is_applicable": True,
|
||||
"priority": 1,
|
||||
},
|
||||
{
|
||||
"regulation_code": "GDPR",
|
||||
"article": "Art. 25",
|
||||
"paragraph": "(2)",
|
||||
"title": "Datenschutzfreundliche Voreinstellungen",
|
||||
"description": "Privacy by Default - Standardeinstellungen müssen datenschutzfreundlich sein.",
|
||||
"requirement_text": "Der Verantwortliche trifft geeignete Maßnahmen, die sicherstellen, dass durch Voreinstellung nur personenbezogene Daten verarbeitet werden, deren Verarbeitung für den jeweiligen bestimmten Verarbeitungszweck erforderlich ist.",
|
||||
"breakpilot_interpretation": "Opt-In statt Opt-Out, minimale Default-Datenerhebung, Consent granular einholbar.",
|
||||
"is_applicable": True,
|
||||
"priority": 1,
|
||||
},
|
||||
{
|
||||
"regulation_code": "GDPR",
|
||||
"article": "Art. 28",
|
||||
"paragraph": "(1)",
|
||||
"title": "Auftragsverarbeiter",
|
||||
"description": "Nur Auftragsverarbeiter mit hinreichenden Garantien dürfen beauftragt werden.",
|
||||
"requirement_text": "Erfolgt eine Verarbeitung im Auftrag eines Verantwortlichen, so arbeitet dieser nur mit Auftragsverarbeitern, die hinreichend Garantien dafür bieten.",
|
||||
"breakpilot_interpretation": "AVV mit allen Sub-Processors, regelmäßige Überprüfung der Garantien, Sub-Processor-Liste gepflegt.",
|
||||
"is_applicable": True,
|
||||
"priority": 2,
|
||||
},
|
||||
{
|
||||
"regulation_code": "GDPR",
|
||||
"article": "Art. 28",
|
||||
"paragraph": "(3)",
|
||||
"title": "AVV-Pflichtinhalte",
|
||||
"description": "Auftragsverarbeitungsverträge müssen bestimmte Mindestinhalte haben.",
|
||||
"requirement_text": "Die Verarbeitung durch einen Auftragsverarbeiter erfolgt auf der Grundlage eines Vertrags, der Gegenstand, Dauer, Art und Zweck der Verarbeitung, Art der Daten und Kategorien betroffener Personen festlegt.",
|
||||
"breakpilot_interpretation": "AVV-Template nach Art. 28 Abs. 3, alle Pflichtklauseln enthalten, rechtliche Prüfung.",
|
||||
"is_applicable": True,
|
||||
"priority": 2,
|
||||
},
|
||||
{
|
||||
"regulation_code": "GDPR",
|
||||
"article": "Art. 30",
|
||||
"paragraph": "(1)",
|
||||
"title": "Verarbeitungsverzeichnis",
|
||||
"description": "Führung eines Verzeichnisses aller Verarbeitungstätigkeiten.",
|
||||
"requirement_text": "Jeder Verantwortliche führt ein Verzeichnis aller Verarbeitungstätigkeiten, die seiner Zuständigkeit unterliegen.",
|
||||
"breakpilot_interpretation": "VVT gepflegt mit allen Verarbeitungen, regelmäßige Aktualisierung, alle Pflichtangaben enthalten.",
|
||||
"is_applicable": True,
|
||||
"priority": 1,
|
||||
},
|
||||
{
|
||||
"regulation_code": "GDPR",
|
||||
"article": "Art. 32",
|
||||
"paragraph": "(1)(a)",
|
||||
"title": "Pseudonymisierung und Verschlüsselung",
|
||||
"description": "Technische Maßnahmen zur Pseudonymisierung und Verschlüsselung.",
|
||||
"requirement_text": "Die Pseudonymisierung und Verschlüsselung personenbezogener Daten.",
|
||||
"breakpilot_interpretation": "AES-256 Encryption at Rest, TLS 1.3 in Transit, Pseudonymisierung wo möglich.",
|
||||
"is_applicable": True,
|
||||
"priority": 1,
|
||||
},
|
||||
{
|
||||
"regulation_code": "GDPR",
|
||||
"article": "Art. 32",
|
||||
"paragraph": "(1)(b)",
|
||||
"title": "Vertraulichkeit und Integrität der Systeme",
|
||||
"description": "Fähigkeit, Vertraulichkeit, Integrität und Verfügbarkeit sicherzustellen.",
|
||||
"requirement_text": "Die Fähigkeit, die Vertraulichkeit, Integrität, Verfügbarkeit und Belastbarkeit der Systeme und Dienste im Zusammenhang mit der Verarbeitung auf Dauer sicherzustellen.",
|
||||
"breakpilot_interpretation": "RBAC, Audit Logging, DDoS-Schutz, Monitoring & Alerting, redundante Infrastruktur.",
|
||||
"is_applicable": True,
|
||||
"priority": 1,
|
||||
},
|
||||
{
|
||||
"regulation_code": "GDPR",
|
||||
"article": "Art. 32",
|
||||
"paragraph": "(1)(c)",
|
||||
"title": "Wiederherstellbarkeit",
|
||||
"description": "Fähigkeit zur raschen Wiederherstellung nach Zwischenfällen.",
|
||||
"requirement_text": "Die Fähigkeit, die Verfügbarkeit der personenbezogenen Daten und den Zugang zu ihnen bei einem physischen oder technischen Zwischenfall rasch wiederherzustellen.",
|
||||
"breakpilot_interpretation": "Tägliche Backups, dokumentierter Recovery-Plan, RTO < 4h, getestete Wiederherstellung.",
|
||||
"is_applicable": True,
|
||||
"priority": 1,
|
||||
},
|
||||
{
|
||||
"regulation_code": "GDPR",
|
||||
"article": "Art. 32",
|
||||
"paragraph": "(1)(d)",
|
||||
"title": "Regelmäßige Überprüfung",
|
||||
"description": "Regelmäßige Überprüfung und Bewertung der Wirksamkeit der TOMs.",
|
||||
"requirement_text": "Ein Verfahren zur regelmäßigen Überprüfung, Bewertung und Evaluierung der Wirksamkeit der technischen und organisatorischen Maßnahmen.",
|
||||
"breakpilot_interpretation": "Jährliche Sicherheitsaudits, Penetration Tests, Control Reviews, Compliance Dashboard.",
|
||||
"is_applicable": True,
|
||||
"priority": 2,
|
||||
},
|
||||
{
|
||||
"regulation_code": "GDPR",
|
||||
"article": "Art. 35",
|
||||
"paragraph": "(1)",
|
||||
"title": "Datenschutz-Folgenabschätzung",
|
||||
"description": "DPIA bei voraussichtlich hohem Risiko für Betroffene.",
|
||||
"requirement_text": "Hat eine Form der Verarbeitung voraussichtlich ein hohes Risiko für die Rechte und Freiheiten natürlicher Personen zur Folge, so führt der Verantwortliche vorab eine Abschätzung der Folgen durch.",
|
||||
"breakpilot_interpretation": "DPIA für KI-Verarbeitung und Schülerdaten durchgeführt, Risiken bewertet, Maßnahmen dokumentiert.",
|
||||
"is_applicable": True,
|
||||
"priority": 1,
|
||||
},
|
||||
|
||||
# =========================================================================
|
||||
# AI Act - KI-Verordnung
|
||||
# =========================================================================
|
||||
{
|
||||
"regulation_code": "AIACT",
|
||||
"article": "Art. 6",
|
||||
"paragraph": "(1)",
|
||||
"title": "Klassifizierungsregeln für Hochrisiko-KI-Systeme",
|
||||
"description": "KI-Systeme müssen nach Risiko klassifiziert werden.",
|
||||
"requirement_text": "Ein KI-System wird als Hochrisiko-KI-System eingestuft, wenn es als Sicherheitskomponente eines Produkts verwendet wird oder selbst ein Produkt ist, das unter bestimmte Harmonisierungsrechtsvorschriften fällt.",
|
||||
"breakpilot_interpretation": "Breakpilot KI-Systeme klassifiziert als Limited Risk (Bildungsunterstützung). Keine High-Risk-Klassifizierung, da keine Bewertung/Prüfung mit rechtlicher Wirkung.",
|
||||
"is_applicable": True,
|
||||
"priority": 2,
|
||||
},
|
||||
{
|
||||
"regulation_code": "AIACT",
|
||||
"article": "Art. 9",
|
||||
"paragraph": "(1)",
|
||||
"title": "Risikomanagement für High-Risk-KI",
|
||||
"description": "Risikomanagement-System für Hochrisiko-KI etablieren.",
|
||||
"requirement_text": "Für Hochrisiko-KI-Systeme wird ein Risikomanagementsystem eingerichtet, umgesetzt, dokumentiert und aufrechterhalten.",
|
||||
"breakpilot_interpretation": "Obwohl nicht High-Risk: Risikobewertung für KI-Use-Cases durchgeführt, Mitigationsmaßnahmen dokumentiert.",
|
||||
"is_applicable": True,
|
||||
"applicability_reason": "Best Practice auch für Limited Risk KI",
|
||||
"priority": 2,
|
||||
},
|
||||
{
|
||||
"regulation_code": "AIACT",
|
||||
"article": "Art. 13",
|
||||
"paragraph": "(1)",
|
||||
"title": "Transparenz",
|
||||
"description": "KI-Systeme müssen so konzipiert sein, dass Nutzer sie verstehen können.",
|
||||
"requirement_text": "Hochrisiko-KI-Systeme werden so konzipiert und entwickelt, dass ihr Betrieb hinreichend transparent ist, damit die Nutzer die Ausgaben des Systems interpretieren und angemessen nutzen können.",
|
||||
"breakpilot_interpretation": "KI-generierte Inhalte sind als solche gekennzeichnet. Erklärbare KI-Outputs wo möglich.",
|
||||
"is_applicable": True,
|
||||
"priority": 2,
|
||||
},
|
||||
{
|
||||
"regulation_code": "AIACT",
|
||||
"article": "Art. 14",
|
||||
"paragraph": "(1)",
|
||||
"title": "Menschliche Aufsicht",
|
||||
"description": "Hochrisiko-KI-Systeme müssen menschliche Aufsicht ermöglichen.",
|
||||
"requirement_text": "Hochrisiko-KI-Systeme werden so konzipiert und entwickelt, dass sie während der Zeit ihrer Nutzung wirksam von natürlichen Personen beaufsichtigt werden können.",
|
||||
"breakpilot_interpretation": "Human-in-the-Loop für KI-generierte Arbeitsblätter. Lehrer können KI-Vorschläge prüfen und anpassen.",
|
||||
"is_applicable": True,
|
||||
"priority": 2,
|
||||
},
|
||||
{
|
||||
"regulation_code": "AIACT",
|
||||
"article": "Art. 15",
|
||||
"paragraph": "(1)",
|
||||
"title": "Genauigkeit, Robustheit und Cybersicherheit",
|
||||
"description": "KI-Systeme müssen ein angemessenes Maß an Genauigkeit erreichen.",
|
||||
"requirement_text": "Hochrisiko-KI-Systeme werden so konzipiert und entwickelt, dass sie in Bezug auf ihre Zweckbestimmung ein angemessenes Maß an Genauigkeit, Robustheit und Cybersicherheit erreichen.",
|
||||
"breakpilot_interpretation": "LLM-Outputs werden auf Qualität geprüft. Feedback-Loop für kontinuierliche Verbesserung.",
|
||||
"is_applicable": True,
|
||||
"priority": 2,
|
||||
},
|
||||
{
|
||||
"regulation_code": "AIACT",
|
||||
"article": "Art. 50",
|
||||
"paragraph": "(1)",
|
||||
"title": "Kennzeichnungspflicht für KI-generierte Inhalte",
|
||||
"description": "Nutzer müssen informiert werden, dass sie mit KI interagieren.",
|
||||
"requirement_text": "Anbieter stellen sicher, dass KI-Systeme, die für die Interaktion mit natürlichen Personen bestimmt sind, so konzipiert werden, dass natürliche Personen darüber informiert werden, dass sie mit einem KI-System interagieren.",
|
||||
"breakpilot_interpretation": "KI-Features sind klar als solche gekennzeichnet. KI-Icon und Hinweistexte implementiert.",
|
||||
"is_applicable": True,
|
||||
"priority": 1,
|
||||
},
|
||||
|
||||
# =========================================================================
|
||||
# CRA - Cyber Resilience Act
|
||||
# =========================================================================
|
||||
{
|
||||
"regulation_code": "CRA",
|
||||
"article": "Art. 10",
|
||||
"paragraph": "(1)",
|
||||
"title": "Wesentliche Cybersicherheitsanforderungen",
|
||||
"description": "Produkte müssen ohne bekannte ausnutzbare Schwachstellen ausgeliefert werden.",
|
||||
"requirement_text": "Produkte mit digitalen Elementen werden so konzipiert, entwickelt und hergestellt, dass sie ein angemessenes Cybersicherheitsniveau gewährleisten.",
|
||||
"breakpilot_interpretation": "Secure-by-Design Entwicklung, SAST/DAST in CI, keine bekannten Critical/High CVEs bei Release.",
|
||||
"is_applicable": True,
|
||||
"priority": 1,
|
||||
},
|
||||
{
|
||||
"regulation_code": "CRA",
|
||||
"article": "Art. 11",
|
||||
"paragraph": "(1)",
|
||||
"title": "Meldung ausgenutzter Schwachstellen",
|
||||
"description": "Aktiv ausgenutzte Schwachstellen müssen innerhalb von 24h gemeldet werden.",
|
||||
"requirement_text": "Der Hersteller meldet dem CSIRT und der ENISA jede aktiv ausgenutzte Schwachstelle innerhalb von 24 Stunden nach Kenntnisnahme.",
|
||||
"breakpilot_interpretation": "Incident Response Plan enthält 24h-Meldepflicht. Kontakt zu BSI/CERT-Bund etabliert.",
|
||||
"is_applicable": True,
|
||||
"priority": 1,
|
||||
},
|
||||
{
|
||||
"regulation_code": "CRA",
|
||||
"article": "Art. 13",
|
||||
"paragraph": "(1)",
|
||||
"title": "Software Bill of Materials",
|
||||
"description": "SBOM muss für alle Produkte erstellt werden.",
|
||||
"requirement_text": "Hersteller ermitteln und dokumentieren Komponenten, die in dem Produkt enthalten sind, unter anderem durch Erstellung einer Software-Stückliste.",
|
||||
"breakpilot_interpretation": "CycloneDX SBOM wird automatisch bei jedem Release generiert und veröffentlicht.",
|
||||
"is_applicable": True,
|
||||
"priority": 1,
|
||||
},
|
||||
{
|
||||
"regulation_code": "CRA",
|
||||
"article": "Art. 14",
|
||||
"paragraph": "(1)",
|
||||
"title": "Sicherheitsupdates",
|
||||
"description": "Sicherheitsupdates müssen kostenlos und zeitnah bereitgestellt werden.",
|
||||
"requirement_text": "Hersteller stellen sicher, dass Schwachstellen durch kostenlose Sicherheitsupdates behoben werden können, die unverzüglich bereitgestellt werden.",
|
||||
"breakpilot_interpretation": "Patch-SLA: Critical < 7 Tage, High < 30 Tage. Updates automatisch verteilt.",
|
||||
"is_applicable": True,
|
||||
"priority": 1,
|
||||
},
|
||||
{
|
||||
"regulation_code": "CRA",
|
||||
"article": "Art. 15",
|
||||
"paragraph": "(1)",
|
||||
"title": "Support-Zeitraum",
|
||||
"description": "Mindest-Support-Zeitraum für Sicherheitsupdates.",
|
||||
"requirement_text": "Der Support-Zeitraum beträgt mindestens fünf Jahre, es sei denn, die Lebensdauer des Produkts ist kürzer.",
|
||||
"breakpilot_interpretation": "Breakpilot garantiert 5 Jahre Sicherheitsupdates ab Produktversion. EOL-Policy kommuniziert.",
|
||||
"is_applicable": True,
|
||||
"priority": 2,
|
||||
},
|
||||
|
||||
# =========================================================================
|
||||
# BSI-TR-03161 - Mobile Application Security
|
||||
# =========================================================================
|
||||
{
|
||||
"regulation_code": "BSI-TR-03161-1",
|
||||
"article": "O.Arch_1",
|
||||
"paragraph": None,
|
||||
"title": "Sichere Architektur",
|
||||
"description": "Anwendung muss nach Prinzipien sicherer Architektur entwickelt werden.",
|
||||
"requirement_text": "Die Architektur der Anwendung MUSS nach anerkannten Prinzipien sicherer Software-Architektur entwickelt werden.",
|
||||
"breakpilot_interpretation": "Defense in Depth, Least Privilege, Fail Secure implementiert in Backend und Mobile App.",
|
||||
"is_applicable": True,
|
||||
"priority": 1,
|
||||
},
|
||||
{
|
||||
"regulation_code": "BSI-TR-03161-1",
|
||||
"article": "O.Auth_1",
|
||||
"paragraph": None,
|
||||
"title": "Starke Authentisierung",
|
||||
"description": "Sichere Authentisierungsmechanismen müssen implementiert sein.",
|
||||
"requirement_text": "Die Anwendung MUSS sichere Authentisierungsmechanismen implementieren.",
|
||||
"breakpilot_interpretation": "JWT-basierte Authentifizierung, MFA für Admin-Accounts, sichere Session-Verwaltung.",
|
||||
"is_applicable": True,
|
||||
"priority": 1,
|
||||
},
|
||||
{
|
||||
"regulation_code": "BSI-TR-03161-1",
|
||||
"article": "O.Cryp_1",
|
||||
"paragraph": None,
|
||||
"title": "Sichere Kryptographie",
|
||||
"description": "Nur sichere kryptographische Verfahren dürfen verwendet werden.",
|
||||
"requirement_text": "Die Anwendung MUSS ausschließlich als sicher anerkannte kryptographische Verfahren verwenden.",
|
||||
"breakpilot_interpretation": "TLS 1.3, AES-256, bcrypt für Passwörter. Keine schwachen Algorithmen (MD5, SHA1, DES).",
|
||||
"is_applicable": True,
|
||||
"priority": 1,
|
||||
},
|
||||
{
|
||||
"regulation_code": "BSI-TR-03161-1",
|
||||
"article": "O.Data_1",
|
||||
"paragraph": None,
|
||||
"title": "Datensicherheit",
|
||||
"description": "Sensible Daten müssen angemessen geschützt werden.",
|
||||
"requirement_text": "Die Anwendung MUSS sensible Daten sowohl bei der Übertragung als auch bei der Speicherung angemessen schützen.",
|
||||
"breakpilot_interpretation": "Encryption at Rest und in Transit, keine PII in Logs, sichere Key-Speicherung in Vault.",
|
||||
"is_applicable": True,
|
||||
"priority": 1,
|
||||
},
|
||||
{
|
||||
"regulation_code": "BSI-TR-03161-2",
|
||||
"article": "O.Auth_2",
|
||||
"paragraph": None,
|
||||
"title": "Session Management",
|
||||
"description": "Sichere Session-Verwaltung für Web-Anwendungen.",
|
||||
"requirement_text": "Web-Anwendungen MÜSSEN ein sicheres Session-Management implementieren.",
|
||||
"breakpilot_interpretation": "JWT mit kurzer Expiry, Refresh-Token-Rotation, CSRF-Schutz, Secure/HttpOnly Cookies.",
|
||||
"is_applicable": True,
|
||||
"priority": 1,
|
||||
},
|
||||
{
|
||||
"regulation_code": "BSI-TR-03161-2",
|
||||
"article": "O.Source_1",
|
||||
"paragraph": None,
|
||||
"title": "Input-Validierung",
|
||||
"description": "Alle Eingaben müssen validiert werden.",
|
||||
"requirement_text": "Alle Eingaben MÜSSEN vor der Verarbeitung auf Gültigkeit geprüft werden.",
|
||||
"breakpilot_interpretation": "Server-side Validation für alle Inputs, Sanitization, Protection gegen Injection-Angriffe.",
|
||||
"is_applicable": True,
|
||||
"priority": 1,
|
||||
},
|
||||
{
|
||||
"regulation_code": "BSI-TR-03161-3",
|
||||
"article": "O.Back_1",
|
||||
"paragraph": None,
|
||||
"title": "Sichere Backend-Kommunikation",
|
||||
"description": "Kommunikation zwischen Komponenten muss abgesichert sein.",
|
||||
"requirement_text": "Die Kommunikation zwischen Frontend und Backend MUSS über sichere Kanäle erfolgen.",
|
||||
"breakpilot_interpretation": "TLS 1.3 für alle internen Verbindungen, mTLS für Service-to-Service wo möglich.",
|
||||
"is_applicable": True,
|
||||
"priority": 1,
|
||||
},
|
||||
{
|
||||
"regulation_code": "BSI-TR-03161-3",
|
||||
"article": "O.Ops_1",
|
||||
"paragraph": None,
|
||||
"title": "Sichere Konfiguration",
|
||||
"description": "Backend-Systeme müssen sicher konfiguriert sein.",
|
||||
"requirement_text": "Backend-Systeme MÜSSEN nach Security-Best-Practices konfiguriert werden.",
|
||||
"breakpilot_interpretation": "Hardened Container Images, keine Default-Credentials, Secrets in Vault, minimale Ports.",
|
||||
"is_applicable": True,
|
||||
"priority": 1,
|
||||
},
|
||||
]
|
||||
309
backend/compliance/data/risks.py
Normal file
309
backend/compliance/data/risks.py
Normal file
@@ -0,0 +1,309 @@
|
||||
"""
|
||||
Compliance Risks Seed Data.
|
||||
|
||||
Contains potential risks for Breakpilot PWA based on regulatory requirements.
|
||||
Each risk is assessed with likelihood (1-5) and impact (1-5).
|
||||
|
||||
Risk Categories:
|
||||
- data_breach: Potential data breaches or unauthorized access
|
||||
- compliance_gap: Non-compliance with regulations
|
||||
- vendor_risk: Third-party/vendor related risks
|
||||
- operational: Operational and availability risks
|
||||
- legal: Legal and contractual risks
|
||||
- reputational: Reputation and trust risks
|
||||
"""
|
||||
|
||||
from typing import List, Dict, Any
|
||||
|
||||
# Likelihood Scale:
|
||||
# 1 = Very Unlikely (< 5% chance per year)
|
||||
# 2 = Unlikely (5-20% chance per year)
|
||||
# 3 = Possible (20-50% chance per year)
|
||||
# 4 = Likely (50-80% chance per year)
|
||||
# 5 = Very Likely (> 80% chance per year)
|
||||
|
||||
# Impact Scale:
|
||||
# 1 = Negligible (< 1.000 EUR, no operational impact)
|
||||
# 2 = Minor (1.000-10.000 EUR, minor disruption)
|
||||
# 3 = Moderate (10.000-100.000 EUR, significant disruption)
|
||||
# 4 = Major (100.000-1.000.000 EUR, severe impact)
|
||||
# 5 = Critical (> 1.000.000 EUR, existential threat)
|
||||
|
||||
RISKS_SEED: List[Dict[str, Any]] = [
|
||||
# ========================================================================
|
||||
# Datenschutz-Risiken (DSGVO)
|
||||
# ========================================================================
|
||||
{
|
||||
"risk_id": "RISK-001",
|
||||
"title": "Unbefugter Zugriff auf Schueler-PII",
|
||||
"description": "Angreifer oder unbefugte Mitarbeiter koennten auf personenbezogene Daten von Schuelern zugreifen (Namen, Noten, Lernfortschritt). Dies wuerde eine meldepflichtige Datenpanne nach Art. 33 DSGVO darstellen.",
|
||||
"category": "data_breach",
|
||||
"likelihood": 2,
|
||||
"impact": 4,
|
||||
"mitigating_controls": ["IAM-001", "IAM-003", "CRYPTO-001", "OPS-001"],
|
||||
"owner": "Datenschutzbeauftragter",
|
||||
"treatment_plan": "RBAC strikt umsetzen, Mandantentrennung pruefen, regelmaessige Access Reviews durchfuehren, Logging aller Zugriffe auf PII.",
|
||||
"related_regulations": ["GDPR", "BDSG"],
|
||||
},
|
||||
{
|
||||
"risk_id": "RISK-002",
|
||||
"title": "Fehlende oder ungueltige Einwilligungen",
|
||||
"description": "Verarbeitung von Daten ohne gueltige Einwilligung oder Rechtsgrundlage. Insbesondere bei minderjaehrigen Schuelern ist die Einwilligung der Erziehungsberechtigten erforderlich.",
|
||||
"category": "compliance_gap",
|
||||
"likelihood": 3,
|
||||
"impact": 3,
|
||||
"mitigating_controls": ["PRIV-001", "PRIV-003", "GOV-001"],
|
||||
"owner": "Datenschutzbeauftragter",
|
||||
"treatment_plan": "Consent-Management-System implementieren, Altersverifikation, Double-Opt-In fuer Eltern, Dokumentation aller Einwilligungen.",
|
||||
"related_regulations": ["GDPR", "TDDDG"],
|
||||
},
|
||||
{
|
||||
"risk_id": "RISK-003",
|
||||
"title": "Unvollstaendige Betroffenenrechte-Umsetzung",
|
||||
"description": "Art. 15-22 DSGVO Anfragen (Auskunft, Loeschung, Berichtigung, Portabilitaet) werden nicht fristgerecht oder unvollstaendig beantwortet.",
|
||||
"category": "compliance_gap",
|
||||
"likelihood": 2,
|
||||
"impact": 3,
|
||||
"mitigating_controls": ["PRIV-004", "GOV-005"],
|
||||
"owner": "Datenschutzbeauftragter",
|
||||
"treatment_plan": "DSR-Workflow automatisieren, Fristen-Tracking implementieren, Export-Funktion fuer alle Nutzerdaten bereitstellen.",
|
||||
"related_regulations": ["GDPR"],
|
||||
},
|
||||
{
|
||||
"risk_id": "RISK-004",
|
||||
"title": "PII in Logs und Fehlerberichten",
|
||||
"description": "Personenbezogene Daten werden versehentlich in Logs, Fehlerberichten oder Analytics-Daten erfasst und koennten durch Dritte eingesehen werden.",
|
||||
"category": "data_breach",
|
||||
"likelihood": 3,
|
||||
"impact": 2,
|
||||
"mitigating_controls": ["PRIV-007", "OPS-001", "SDLC-001"],
|
||||
"owner": "Engineering Lead",
|
||||
"treatment_plan": "PII-Redactor in allen Logging-Pipelines, SAST-Regeln fuer PII-Leaks, Log-Retention-Policy (max 30 Tage).",
|
||||
"related_regulations": ["GDPR"],
|
||||
},
|
||||
|
||||
# ========================================================================
|
||||
# KI-Risiken (AI Act)
|
||||
# ========================================================================
|
||||
{
|
||||
"risk_id": "RISK-005",
|
||||
"title": "Bias in KI-generierten Lerninhalten",
|
||||
"description": "KI-Modelle koennten verzerrte oder diskriminierende Inhalte generieren, die bestimmte Schuelergruppen benachteiligen.",
|
||||
"category": "compliance_gap",
|
||||
"likelihood": 3,
|
||||
"impact": 3,
|
||||
"mitigating_controls": ["AI-001", "AI-003", "AI-004"],
|
||||
"owner": "AI/ML Lead",
|
||||
"treatment_plan": "Bias-Monitoring implementieren, Human-in-the-Loop fuer alle KI-Outputs, regelmaessige Audits der Trainingsdaten.",
|
||||
"related_regulations": ["AIACT"],
|
||||
},
|
||||
{
|
||||
"risk_id": "RISK-006",
|
||||
"title": "Fehlende KI-Transparenz gegenueber Nutzern",
|
||||
"description": "Nutzer werden nicht ausreichend darueber informiert, wenn KI-Systeme Entscheidungen treffen oder Inhalte generieren (Art. 13 AI Act).",
|
||||
"category": "compliance_gap",
|
||||
"likelihood": 2,
|
||||
"impact": 3,
|
||||
"mitigating_controls": ["AI-002", "AI-005", "GOV-001"],
|
||||
"owner": "Product Owner",
|
||||
"treatment_plan": "KI-Disclosure in UI implementieren, Model Cards erstellen, Transparenzbericht veroeffentlichen.",
|
||||
"related_regulations": ["AIACT"],
|
||||
},
|
||||
{
|
||||
"risk_id": "RISK-007",
|
||||
"title": "Unzureichende KI-Risikobewertung",
|
||||
"description": "Bildungs-KI koennte als High-Risk nach AI Act klassifiziert werden, ohne dass die entsprechenden Anforderungen (Art. 9-15) erfuellt sind.",
|
||||
"category": "compliance_gap",
|
||||
"likelihood": 3,
|
||||
"impact": 4,
|
||||
"mitigating_controls": ["AI-005", "GOV-001", "AUD-001"],
|
||||
"owner": "Compliance Officer",
|
||||
"treatment_plan": "AI Act Impact Assessment durchfuehren, Risikoklassifizierung dokumentieren, Konformitaetsbewertung vorbereiten.",
|
||||
"related_regulations": ["AIACT"],
|
||||
},
|
||||
|
||||
# ========================================================================
|
||||
# Cybersecurity-Risiken (CRA, NIS2)
|
||||
# ========================================================================
|
||||
{
|
||||
"risk_id": "RISK-008",
|
||||
"title": "Schwachstellen in Abhaengigkeiten",
|
||||
"description": "Bekannte CVEs in Third-Party-Libraries werden nicht zeitnah gepatcht und koennten ausgenutzt werden.",
|
||||
"category": "operational",
|
||||
"likelihood": 4,
|
||||
"impact": 4,
|
||||
"mitigating_controls": ["SDLC-002", "SDLC-005", "OPS-004"],
|
||||
"owner": "Engineering Lead",
|
||||
"treatment_plan": "Trivy/Grype in CI/CD, automatische Dependency-Updates (Dependabot), SLA: Critical CVEs < 7 Tage patchen.",
|
||||
"related_regulations": ["CRA", "NIS2"],
|
||||
},
|
||||
{
|
||||
"risk_id": "RISK-009",
|
||||
"title": "Fehlende SBOM fuer CRA-Compliance",
|
||||
"description": "Ohne Software Bill of Materials (SBOM) koennen Schwachstellen nicht effektiv verfolgt werden. CRA verlangt SBOM ab 2027.",
|
||||
"category": "compliance_gap",
|
||||
"likelihood": 2,
|
||||
"impact": 3,
|
||||
"mitigating_controls": ["CRA-001", "SDLC-005"],
|
||||
"owner": "Engineering Lead",
|
||||
"treatment_plan": "CycloneDX SBOM in CI generieren, SBOM-Repository aufbauen, VEX (Vulnerability Exploitability eXchange) implementieren.",
|
||||
"related_regulations": ["CRA"],
|
||||
},
|
||||
{
|
||||
"risk_id": "RISK-010",
|
||||
"title": "Unzureichende Incident Response",
|
||||
"description": "Bei einem Sicherheitsvorfall fehlen klare Prozesse, was zu verzoegerter Reaktion und erhoehtem Schaden fuehrt.",
|
||||
"category": "operational",
|
||||
"likelihood": 2,
|
||||
"impact": 4,
|
||||
"mitigating_controls": ["GOV-005", "OPS-003"],
|
||||
"owner": "Security Lead",
|
||||
"treatment_plan": "Incident Response Plan dokumentieren, Runbooks erstellen, Tabletop-Uebungen durchfuehren, Notfall-Kontakte pflegen.",
|
||||
"related_regulations": ["NIS2", "GDPR"],
|
||||
},
|
||||
{
|
||||
"risk_id": "RISK-011",
|
||||
"title": "Secrets in Code oder Logs",
|
||||
"description": "API-Keys, Passwoerter oder andere Secrets werden versehentlich ins Repository committed oder in Logs geschrieben.",
|
||||
"category": "data_breach",
|
||||
"likelihood": 3,
|
||||
"impact": 4,
|
||||
"mitigating_controls": ["SDLC-003", "CRYPTO-003"],
|
||||
"owner": "Engineering Lead",
|
||||
"treatment_plan": "Gitleaks/TruffleHog in Pre-Commit-Hooks, Vault fuer Secrets, automatische Key-Rotation.",
|
||||
"related_regulations": ["CRA"],
|
||||
},
|
||||
|
||||
# ========================================================================
|
||||
# Vendor & Supply Chain Risiken
|
||||
# ========================================================================
|
||||
{
|
||||
"risk_id": "RISK-012",
|
||||
"title": "Drittanbieter-Datenweitergabe ohne AVV",
|
||||
"description": "Personenbezogene Daten werden an Cloud-Provider oder Subunternehmer uebermittelt, ohne dass ein Auftragsverarbeitungsvertrag (AVV) vorliegt.",
|
||||
"category": "vendor_risk",
|
||||
"likelihood": 2,
|
||||
"impact": 3,
|
||||
"mitigating_controls": ["PRIV-005"],
|
||||
"owner": "Datenschutzbeauftragter",
|
||||
"treatment_plan": "Vendor-Register fuehren, AVVs mit allen Auftragsverarbeitern abschliessen, regelmaessige Ueberpruefung.",
|
||||
"related_regulations": ["GDPR"],
|
||||
},
|
||||
{
|
||||
"risk_id": "RISK-013",
|
||||
"title": "US-Cloud-Dienste ohne angemessene Garantien",
|
||||
"description": "Nutzung von US-Cloud-Diensten ohne EU-US Data Privacy Framework Zertifizierung oder SCCs, was nach Schrems II problematisch ist.",
|
||||
"category": "vendor_risk",
|
||||
"likelihood": 2,
|
||||
"impact": 4,
|
||||
"mitigating_controls": ["PRIV-005", "GOV-001"],
|
||||
"owner": "Datenschutzbeauftragter",
|
||||
"treatment_plan": "Cloud-Provider auf DPF-Zertifizierung pruefen, Transfer Impact Assessment durchfuehren, EU-Hosting bevorzugen.",
|
||||
"related_regulations": ["GDPR", "SCC", "DPF"],
|
||||
},
|
||||
{
|
||||
"risk_id": "RISK-014",
|
||||
"title": "LLM-Provider Datennutzung fuer Training",
|
||||
"description": "LLM-Anbieter (OpenAI, Anthropic) koennten Nutzerdaten zum Training verwenden, was ohne Einwilligung problematisch waere.",
|
||||
"category": "vendor_risk",
|
||||
"likelihood": 2,
|
||||
"impact": 3,
|
||||
"mitigating_controls": ["PRIV-005", "AI-001"],
|
||||
"owner": "AI/ML Lead",
|
||||
"treatment_plan": "Opt-Out fuer Training bei allen LLM-Providern, Enterprise-Agreements mit No-Training-Klausel, PII-Filtering vor API-Calls.",
|
||||
"related_regulations": ["GDPR", "AIACT"],
|
||||
},
|
||||
|
||||
# ========================================================================
|
||||
# Betriebliche Risiken
|
||||
# ========================================================================
|
||||
{
|
||||
"risk_id": "RISK-015",
|
||||
"title": "Datenverlust durch fehlende Backups",
|
||||
"description": "Kritische Daten gehen durch Hardware-Ausfall, Ransomware oder menschliches Versagen verloren.",
|
||||
"category": "operational",
|
||||
"likelihood": 2,
|
||||
"impact": 5,
|
||||
"mitigating_controls": ["OPS-002"],
|
||||
"owner": "DevOps Lead",
|
||||
"treatment_plan": "Taegliche automatische Backups, geografisch redundante Speicherung, regelmaessige Restore-Tests (quartalsweise).",
|
||||
"related_regulations": ["GDPR"],
|
||||
},
|
||||
{
|
||||
"risk_id": "RISK-016",
|
||||
"title": "Verfuegbarkeitsausfall waehrend Unterricht",
|
||||
"description": "System ist waehrend des Unterrichts nicht verfuegbar, was den Lehrbetrieb stoert und Vertrauen beschaedigt.",
|
||||
"category": "operational",
|
||||
"likelihood": 3,
|
||||
"impact": 3,
|
||||
"mitigating_controls": ["OPS-005"],
|
||||
"owner": "DevOps Lead",
|
||||
"treatment_plan": "99.9% SLA definieren, Monitoring mit Alerting, Runbooks fuer haeufige Ausfaelle, Status-Page einrichten.",
|
||||
"related_regulations": [],
|
||||
},
|
||||
|
||||
# ========================================================================
|
||||
# Rechtliche Risiken
|
||||
# ========================================================================
|
||||
{
|
||||
"risk_id": "RISK-017",
|
||||
"title": "DSGVO-Bussgeld wegen Compliance-Verstoss",
|
||||
"description": "Aufsichtsbehoerde verhaengt Bussgeld wegen Datenschutzverstoss (bis zu 20 Mio EUR oder 4% Jahresumsatz).",
|
||||
"category": "legal",
|
||||
"likelihood": 1,
|
||||
"impact": 5,
|
||||
"mitigating_controls": ["PRIV-001", "PRIV-002", "PRIV-006", "GOV-001"],
|
||||
"owner": "Geschaeftsfuehrung",
|
||||
"treatment_plan": "Datenschutz-Audit jaehrlich, DPIA fuer neue Verarbeitungen, enge Zusammenarbeit mit DSB.",
|
||||
"related_regulations": ["GDPR"],
|
||||
},
|
||||
{
|
||||
"risk_id": "RISK-018",
|
||||
"title": "Haftung bei KI-verursachten Schaeden",
|
||||
"description": "Neue Produkthaftungsrichtlinie macht Hersteller fuer KI-verursachte Schaeden haftbar (fehlerhafte Lerninhalte, falsche Bewertungen).",
|
||||
"category": "legal",
|
||||
"likelihood": 2,
|
||||
"impact": 4,
|
||||
"mitigating_controls": ["AI-003", "AI-004", "AUD-001"],
|
||||
"owner": "Geschaeftsfuehrung",
|
||||
"treatment_plan": "Human-in-the-Loop fuer kritische Entscheidungen, Disclaimer in UI, Haftpflichtversicherung pruefen.",
|
||||
"related_regulations": ["PLD", "AIACT"],
|
||||
},
|
||||
|
||||
# ========================================================================
|
||||
# Barrierefreiheit
|
||||
# ========================================================================
|
||||
{
|
||||
"risk_id": "RISK-019",
|
||||
"title": "Nicht barrierefreie Anwendung (EAA)",
|
||||
"description": "European Accessibility Act verlangt ab 2025 Barrierefreiheit. Nicht-konforme Software kann vom Markt ausgeschlossen werden.",
|
||||
"category": "compliance_gap",
|
||||
"likelihood": 3,
|
||||
"impact": 3,
|
||||
"mitigating_controls": ["GOV-001"],
|
||||
"owner": "Product Owner",
|
||||
"treatment_plan": "WCAG 2.1 AA Audit durchfuehren, Accessibility-Tests in CI, Screenreader-Kompatibilitaet sicherstellen.",
|
||||
"related_regulations": ["EAA"],
|
||||
},
|
||||
|
||||
# ========================================================================
|
||||
# Reputationsrisiken
|
||||
# ========================================================================
|
||||
{
|
||||
"risk_id": "RISK-020",
|
||||
"title": "Reputationsschaden durch Datenpanne",
|
||||
"description": "Oeffentlich bekannt gewordene Datenpanne fuehrt zu Vertrauensverlust bei Schulen, Eltern und Schuelern.",
|
||||
"category": "reputational",
|
||||
"likelihood": 2,
|
||||
"impact": 4,
|
||||
"mitigating_controls": ["GOV-005", "OPS-003"],
|
||||
"owner": "Geschaeftsfuehrung",
|
||||
"treatment_plan": "Kommunikationsplan fuer Krisenfaelle, transparente Kommunikation, schnelle Behebung und Information Betroffener.",
|
||||
"related_regulations": ["GDPR"],
|
||||
},
|
||||
]
|
||||
|
||||
|
||||
def get_risks_for_seeding() -> List[Dict[str, Any]]:
|
||||
"""Return all risks for database seeding."""
|
||||
return RISKS_SEED
|
||||
834
backend/compliance/data/service_modules.py
Normal file
834
backend/compliance/data/service_modules.py
Normal file
@@ -0,0 +1,834 @@
|
||||
"""
|
||||
Breakpilot Service Module Registry - Seed Data
|
||||
|
||||
Contains all 51+ Breakpilot services with:
|
||||
- Technical details (port, stack, repository)
|
||||
- Data categories processed
|
||||
- Applicable regulations
|
||||
"""
|
||||
|
||||
from typing import Dict, List, Any
|
||||
|
||||
# Service Type Constants
|
||||
BACKEND = "backend"
|
||||
DATABASE = "database"
|
||||
AI = "ai"
|
||||
COMMUNICATION = "communication"
|
||||
STORAGE = "storage"
|
||||
INFRASTRUCTURE = "infrastructure"
|
||||
MONITORING = "monitoring"
|
||||
SECURITY = "security"
|
||||
|
||||
# Relevance Level Constants
|
||||
CRITICAL = "critical"
|
||||
HIGH = "high"
|
||||
MEDIUM = "medium"
|
||||
LOW = "low"
|
||||
|
||||
|
||||
BREAKPILOT_SERVICES: List[Dict[str, Any]] = [
|
||||
# =========================================================================
|
||||
# CORE BACKEND SERVICES
|
||||
# =========================================================================
|
||||
{
|
||||
"name": "python-backend",
|
||||
"display_name": "Python Backend (FastAPI)",
|
||||
"description": "Hauptbackend für API, Frontend-Serving, GDPR-Export und alle Core-Funktionen",
|
||||
"service_type": BACKEND,
|
||||
"port": 8000,
|
||||
"technology_stack": ["Python", "FastAPI", "SQLAlchemy", "PostgreSQL"],
|
||||
"repository_path": "/backend",
|
||||
"docker_image": "breakpilot-pwa-backend",
|
||||
"data_categories": ["user_data", "consent_records", "documents", "learning_data"],
|
||||
"processes_pii": True,
|
||||
"processes_health_data": False,
|
||||
"ai_components": False,
|
||||
"criticality": "critical",
|
||||
"owner_team": "Backend Team",
|
||||
"regulations": [
|
||||
{"code": "GDPR", "relevance": CRITICAL, "notes": "Verarbeitet alle personenbezogenen Daten"},
|
||||
{"code": "AIACT", "relevance": HIGH, "notes": "Orchestriert KI-Services"},
|
||||
{"code": "DSA", "relevance": MEDIUM, "notes": "Content-Moderation"},
|
||||
{"code": "NIS2", "relevance": HIGH, "notes": "Kritische Infrastruktur"},
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "consent-service",
|
||||
"display_name": "Go Consent Service",
|
||||
"description": "Kernlogik für Consent-Management, Einwilligungsverwaltung und Versionierung",
|
||||
"service_type": BACKEND,
|
||||
"port": 8081,
|
||||
"technology_stack": ["Go", "Gin", "GORM", "PostgreSQL"],
|
||||
"repository_path": "/consent-service",
|
||||
"docker_image": "breakpilot-pwa-consent-service",
|
||||
"data_categories": ["consent_records", "user_preferences", "audit_logs"],
|
||||
"processes_pii": True,
|
||||
"processes_health_data": False,
|
||||
"ai_components": False,
|
||||
"criticality": "critical",
|
||||
"owner_team": "Backend Team",
|
||||
"regulations": [
|
||||
{"code": "GDPR", "relevance": CRITICAL, "notes": "Art. 7 Einwilligung, Art. 30 VVZ"},
|
||||
{"code": "TDDDG", "relevance": CRITICAL, "notes": "§ 25 Cookie-Consent"},
|
||||
{"code": "BSI-TR-03161-2", "relevance": HIGH, "notes": "Session-Management"},
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "billing-service",
|
||||
"display_name": "Billing Service",
|
||||
"description": "Zahlungsabwicklung, Abonnements und Rechnungsstellung",
|
||||
"service_type": BACKEND,
|
||||
"port": 8083,
|
||||
"technology_stack": ["Python", "FastAPI", "Stripe API"],
|
||||
"repository_path": "/billing-service",
|
||||
"docker_image": "breakpilot-pwa-billing",
|
||||
"data_categories": ["payment_data", "subscriptions", "invoices"],
|
||||
"processes_pii": True,
|
||||
"processes_health_data": False,
|
||||
"ai_components": False,
|
||||
"criticality": "critical",
|
||||
"owner_team": "Backend Team",
|
||||
"regulations": [
|
||||
{"code": "GDPR", "relevance": CRITICAL, "notes": "Zahlungsdaten = besonders schützenswert"},
|
||||
{"code": "DSA", "relevance": LOW, "notes": "Transparenz bei Gebühren"},
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "school-service",
|
||||
"display_name": "School Service",
|
||||
"description": "Schulverwaltung, Klassen, Noten und Zeugnisse",
|
||||
"service_type": BACKEND,
|
||||
"port": 8084,
|
||||
"technology_stack": ["Python", "FastAPI", "PostgreSQL"],
|
||||
"repository_path": "/school-service",
|
||||
"docker_image": "breakpilot-pwa-school-service",
|
||||
"data_categories": ["student_data", "grades", "certificates", "class_data"],
|
||||
"processes_pii": True,
|
||||
"processes_health_data": False,
|
||||
"ai_components": False,
|
||||
"criticality": "high",
|
||||
"owner_team": "Education Team",
|
||||
"regulations": [
|
||||
{"code": "GDPR", "relevance": CRITICAL, "notes": "Schülerdaten = besonderer Schutz"},
|
||||
{"code": "BSI-TR-03161-1", "relevance": HIGH, "notes": "Sicherheit für Bildungsanwendungen"},
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "calendar-service",
|
||||
"display_name": "Calendar Service",
|
||||
"description": "Kalender, Termine und Stundenplanung",
|
||||
"service_type": BACKEND,
|
||||
"port": 8085,
|
||||
"technology_stack": ["Python", "FastAPI", "PostgreSQL"],
|
||||
"repository_path": "/calendar-service",
|
||||
"docker_image": "breakpilot-pwa-calendar",
|
||||
"data_categories": ["schedule_data", "appointments"],
|
||||
"processes_pii": True,
|
||||
"processes_health_data": False,
|
||||
"ai_components": False,
|
||||
"criticality": "medium",
|
||||
"owner_team": "Backend Team",
|
||||
"regulations": [
|
||||
{"code": "GDPR", "relevance": MEDIUM, "notes": "Terminbezogene Daten"},
|
||||
]
|
||||
},
|
||||
|
||||
# =========================================================================
|
||||
# AI / ML SERVICES
|
||||
# =========================================================================
|
||||
{
|
||||
"name": "klausur-service",
|
||||
"display_name": "Klausur Service (AI Correction)",
|
||||
"description": "KI-gestützte Klausurbewertung, PDF-Analyse und Feedback-Generierung",
|
||||
"service_type": AI,
|
||||
"port": 8086,
|
||||
"technology_stack": ["Python", "FastAPI", "Claude API", "PyMuPDF"],
|
||||
"repository_path": "/klausur-service",
|
||||
"docker_image": "breakpilot-pwa-klausur-service",
|
||||
"data_categories": ["exam_papers", "corrections", "student_submissions"],
|
||||
"processes_pii": True,
|
||||
"processes_health_data": False,
|
||||
"ai_components": True,
|
||||
"criticality": "high",
|
||||
"owner_team": "AI Team",
|
||||
"regulations": [
|
||||
{"code": "AIACT", "relevance": CRITICAL, "notes": "High-Risk KI im Bildungsbereich Art. 6"},
|
||||
{"code": "GDPR", "relevance": CRITICAL, "notes": "Automatisierte Entscheidung Art. 22"},
|
||||
{"code": "BSI-TR-03161-2", "relevance": HIGH, "notes": "Input-Validierung für Uploads"},
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "embedding-service",
|
||||
"display_name": "Embedding Service",
|
||||
"description": "Vektor-Embeddings für semantische Suche und RAG",
|
||||
"service_type": AI,
|
||||
"port": 8087,
|
||||
"technology_stack": ["Python", "FastAPI", "SentenceTransformers", "Qdrant"],
|
||||
"repository_path": "/embedding-service",
|
||||
"docker_image": "breakpilot-pwa-embedding-service",
|
||||
"data_categories": ["document_embeddings", "search_queries"],
|
||||
"processes_pii": False,
|
||||
"processes_health_data": False,
|
||||
"ai_components": True,
|
||||
"criticality": "medium",
|
||||
"owner_team": "AI Team",
|
||||
"regulations": [
|
||||
{"code": "AIACT", "relevance": MEDIUM, "notes": "General-Purpose AI System"},
|
||||
{"code": "GDPR", "relevance": LOW, "notes": "Keine direkten personenbezogenen Daten"},
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "transcription-worker",
|
||||
"display_name": "Transcription Worker",
|
||||
"description": "Whisper-basierte Audio-Transkription für Meetings und Videos",
|
||||
"service_type": AI,
|
||||
"port": None,
|
||||
"technology_stack": ["Python", "Whisper", "FFmpeg"],
|
||||
"repository_path": "/transcription-service",
|
||||
"docker_image": "breakpilot-pwa-transcription",
|
||||
"data_categories": ["audio_recordings", "transcripts"],
|
||||
"processes_pii": True,
|
||||
"processes_health_data": False,
|
||||
"ai_components": True,
|
||||
"criticality": "medium",
|
||||
"owner_team": "AI Team",
|
||||
"regulations": [
|
||||
{"code": "AIACT", "relevance": MEDIUM, "notes": "Audio-Analyse"},
|
||||
{"code": "GDPR", "relevance": HIGH, "notes": "Sprachaufnahmen = biometrische Daten"},
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "llm-gateway",
|
||||
"display_name": "LLM Gateway",
|
||||
"description": "Zentraler Gateway für alle LLM-Anfragen (Claude, OpenAI, Self-Hosted)",
|
||||
"service_type": AI,
|
||||
"port": 8088,
|
||||
"technology_stack": ["Python", "FastAPI", "LiteLLM"],
|
||||
"repository_path": "/llm-gateway",
|
||||
"docker_image": "breakpilot-pwa-llm-gateway",
|
||||
"data_categories": ["llm_prompts", "llm_responses"],
|
||||
"processes_pii": True,
|
||||
"processes_health_data": False,
|
||||
"ai_components": True,
|
||||
"criticality": "high",
|
||||
"owner_team": "AI Team",
|
||||
"regulations": [
|
||||
{"code": "AIACT", "relevance": CRITICAL, "notes": "Orchestrierung von KI-Systemen"},
|
||||
{"code": "GDPR", "relevance": HIGH, "notes": "Daten an externe APIs"},
|
||||
]
|
||||
},
|
||||
|
||||
# =========================================================================
|
||||
# DATABASES
|
||||
# =========================================================================
|
||||
{
|
||||
"name": "postgresql",
|
||||
"display_name": "PostgreSQL Database",
|
||||
"description": "Primäre relationale Datenbank für alle persistenten Daten",
|
||||
"service_type": DATABASE,
|
||||
"port": 5432,
|
||||
"technology_stack": ["PostgreSQL 15"],
|
||||
"repository_path": None,
|
||||
"docker_image": "postgres:15",
|
||||
"data_categories": ["all_persistent_data"],
|
||||
"processes_pii": True,
|
||||
"processes_health_data": False,
|
||||
"ai_components": False,
|
||||
"criticality": "critical",
|
||||
"owner_team": "Infrastructure",
|
||||
"regulations": [
|
||||
{"code": "GDPR", "relevance": CRITICAL, "notes": "Art. 32 Sicherheit der Verarbeitung"},
|
||||
{"code": "BSI-TR-03161-3", "relevance": CRITICAL, "notes": "Datenbank-Sicherheit"},
|
||||
{"code": "NIS2", "relevance": HIGH, "notes": "Kritische Infrastruktur"},
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "qdrant",
|
||||
"display_name": "Qdrant Vector DB",
|
||||
"description": "Vektordatenbank für Embeddings und semantische Suche",
|
||||
"service_type": DATABASE,
|
||||
"port": 6333,
|
||||
"technology_stack": ["Qdrant"],
|
||||
"repository_path": None,
|
||||
"docker_image": "qdrant/qdrant",
|
||||
"data_categories": ["vector_embeddings", "document_metadata"],
|
||||
"processes_pii": False,
|
||||
"processes_health_data": False,
|
||||
"ai_components": False,
|
||||
"criticality": "medium",
|
||||
"owner_team": "AI Team",
|
||||
"regulations": [
|
||||
{"code": "GDPR", "relevance": LOW, "notes": "Keine direkten PII"},
|
||||
{"code": "BSI-TR-03161-3", "relevance": MEDIUM, "notes": "Datenbank-Sicherheit"},
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "valkey",
|
||||
"display_name": "Valkey (Redis Fork)",
|
||||
"description": "In-Memory Cache und Message Queue",
|
||||
"service_type": DATABASE,
|
||||
"port": 6379,
|
||||
"technology_stack": ["Valkey"],
|
||||
"repository_path": None,
|
||||
"docker_image": "valkey/valkey",
|
||||
"data_categories": ["session_data", "cache_data", "job_queues"],
|
||||
"processes_pii": True,
|
||||
"processes_health_data": False,
|
||||
"ai_components": False,
|
||||
"criticality": "high",
|
||||
"owner_team": "Infrastructure",
|
||||
"regulations": [
|
||||
{"code": "GDPR", "relevance": MEDIUM, "notes": "Session-Daten"},
|
||||
{"code": "BSI-TR-03161-2", "relevance": HIGH, "notes": "Session-Management"},
|
||||
]
|
||||
},
|
||||
|
||||
# =========================================================================
|
||||
# STORAGE
|
||||
# =========================================================================
|
||||
{
|
||||
"name": "minio",
|
||||
"display_name": "MinIO Object Storage",
|
||||
"description": "S3-kompatibler Object Storage für Dateien, Bilder und Backups",
|
||||
"service_type": STORAGE,
|
||||
"port": 9000,
|
||||
"technology_stack": ["MinIO"],
|
||||
"repository_path": None,
|
||||
"docker_image": "minio/minio",
|
||||
"data_categories": ["uploaded_files", "recordings", "backups", "exports"],
|
||||
"processes_pii": True,
|
||||
"processes_health_data": False,
|
||||
"ai_components": False,
|
||||
"criticality": "critical",
|
||||
"owner_team": "Infrastructure",
|
||||
"regulations": [
|
||||
{"code": "GDPR", "relevance": CRITICAL, "notes": "Speicherung von Nutzerdaten"},
|
||||
{"code": "BSI-TR-03161-3", "relevance": HIGH, "notes": "Speichersicherheit"},
|
||||
]
|
||||
},
|
||||
|
||||
# =========================================================================
|
||||
# COMMUNICATION SERVICES
|
||||
# =========================================================================
|
||||
{
|
||||
"name": "matrix-synapse",
|
||||
"display_name": "Matrix Synapse",
|
||||
"description": "Dezentraler Chat-Server für Messaging",
|
||||
"service_type": COMMUNICATION,
|
||||
"port": 8008,
|
||||
"technology_stack": ["Python", "Matrix Protocol", "PostgreSQL"],
|
||||
"repository_path": None,
|
||||
"docker_image": "matrixdotorg/synapse",
|
||||
"data_categories": ["messages", "chat_history", "user_presence"],
|
||||
"processes_pii": True,
|
||||
"processes_health_data": False,
|
||||
"ai_components": False,
|
||||
"criticality": "high",
|
||||
"owner_team": "Communication Team",
|
||||
"regulations": [
|
||||
{"code": "GDPR", "relevance": CRITICAL, "notes": "Chat-Inhalte"},
|
||||
{"code": "DSA", "relevance": HIGH, "notes": "Content-Moderation"},
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "jitsi-meet",
|
||||
"display_name": "Jitsi Meet",
|
||||
"description": "WebRTC-basierte Videokonferenzen",
|
||||
"service_type": COMMUNICATION,
|
||||
"port": 8443,
|
||||
"technology_stack": ["JavaScript", "WebRTC", "Prosody"],
|
||||
"repository_path": None,
|
||||
"docker_image": "jitsi/web",
|
||||
"data_categories": ["video_streams", "audio_streams", "screen_shares"],
|
||||
"processes_pii": True,
|
||||
"processes_health_data": False,
|
||||
"ai_components": False,
|
||||
"criticality": "high",
|
||||
"owner_team": "Communication Team",
|
||||
"regulations": [
|
||||
{"code": "GDPR", "relevance": CRITICAL, "notes": "Video-/Audiodaten"},
|
||||
{"code": "BSI-TR-03161-2", "relevance": HIGH, "notes": "WebRTC-Sicherheit"},
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "jitsi-prosody",
|
||||
"display_name": "Jitsi Prosody (XMPP)",
|
||||
"description": "XMPP-Server für Jitsi Signaling",
|
||||
"service_type": COMMUNICATION,
|
||||
"port": 5222,
|
||||
"technology_stack": ["Lua", "Prosody", "XMPP"],
|
||||
"repository_path": None,
|
||||
"docker_image": "jitsi/prosody",
|
||||
"data_categories": ["signaling_data", "presence"],
|
||||
"processes_pii": True,
|
||||
"processes_health_data": False,
|
||||
"ai_components": False,
|
||||
"criticality": "high",
|
||||
"owner_team": "Communication Team",
|
||||
"regulations": [
|
||||
{"code": "GDPR", "relevance": MEDIUM, "notes": "Signaling-Metadaten"},
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "jitsi-jicofo",
|
||||
"display_name": "Jitsi Jicofo",
|
||||
"description": "Jitsi Focus Component für Konferenzkoordination",
|
||||
"service_type": COMMUNICATION,
|
||||
"port": None,
|
||||
"technology_stack": ["Java"],
|
||||
"repository_path": None,
|
||||
"docker_image": "jitsi/jicofo",
|
||||
"data_categories": ["conference_metadata"],
|
||||
"processes_pii": False,
|
||||
"processes_health_data": False,
|
||||
"ai_components": False,
|
||||
"criticality": "medium",
|
||||
"owner_team": "Communication Team",
|
||||
"regulations": [
|
||||
{"code": "GDPR", "relevance": LOW, "notes": "Nur Metadaten"},
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "jitsi-jvb",
|
||||
"display_name": "Jitsi JVB (Video Bridge)",
|
||||
"description": "Video Bridge für Multi-Party Konferenzen",
|
||||
"service_type": COMMUNICATION,
|
||||
"port": 10000,
|
||||
"technology_stack": ["Java", "WebRTC"],
|
||||
"repository_path": None,
|
||||
"docker_image": "jitsi/jvb",
|
||||
"data_categories": ["video_streams"],
|
||||
"processes_pii": True,
|
||||
"processes_health_data": False,
|
||||
"ai_components": False,
|
||||
"criticality": "high",
|
||||
"owner_team": "Communication Team",
|
||||
"regulations": [
|
||||
{"code": "GDPR", "relevance": HIGH, "notes": "Video-Routing"},
|
||||
{"code": "BSI-TR-03161-2", "relevance": MEDIUM, "notes": "WebRTC-Sicherheit"},
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "jibri",
|
||||
"display_name": "Jitsi Jibri (Recording)",
|
||||
"description": "Meeting-Aufzeichnung und Streaming",
|
||||
"service_type": COMMUNICATION,
|
||||
"port": None,
|
||||
"technology_stack": ["Java", "FFmpeg", "Chrome"],
|
||||
"repository_path": None,
|
||||
"docker_image": "jitsi/jibri",
|
||||
"data_categories": ["recordings", "video_files"],
|
||||
"processes_pii": True,
|
||||
"processes_health_data": False,
|
||||
"ai_components": False,
|
||||
"criticality": "high",
|
||||
"owner_team": "Communication Team",
|
||||
"regulations": [
|
||||
{"code": "GDPR", "relevance": CRITICAL, "notes": "Video-Aufzeichnungen"},
|
||||
]
|
||||
},
|
||||
|
||||
# =========================================================================
|
||||
# CONTENT SERVICES
|
||||
# =========================================================================
|
||||
{
|
||||
"name": "h5p-service",
|
||||
"display_name": "H5P Content Service",
|
||||
"description": "Interaktive Lerninhalte (H5P)",
|
||||
"service_type": BACKEND,
|
||||
"port": 8082,
|
||||
"technology_stack": ["PHP", "H5P Framework"],
|
||||
"repository_path": "/h5p-service",
|
||||
"docker_image": "breakpilot-pwa-h5p",
|
||||
"data_categories": ["learning_content", "user_progress"],
|
||||
"processes_pii": True,
|
||||
"processes_health_data": False,
|
||||
"ai_components": False,
|
||||
"criticality": "medium",
|
||||
"owner_team": "Education Team",
|
||||
"regulations": [
|
||||
{"code": "GDPR", "relevance": MEDIUM, "notes": "Lernfortschritt"},
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "content-db",
|
||||
"display_name": "Content Database",
|
||||
"description": "Dedizierte DB für Content-Services",
|
||||
"service_type": DATABASE,
|
||||
"port": 5433,
|
||||
"technology_stack": ["PostgreSQL 15"],
|
||||
"repository_path": None,
|
||||
"docker_image": "postgres:15",
|
||||
"data_categories": ["content_metadata"],
|
||||
"processes_pii": False,
|
||||
"processes_health_data": False,
|
||||
"ai_components": False,
|
||||
"criticality": "medium",
|
||||
"owner_team": "Infrastructure",
|
||||
"regulations": [
|
||||
{"code": "BSI-TR-03161-3", "relevance": MEDIUM, "notes": "Datenbank-Sicherheit"},
|
||||
]
|
||||
},
|
||||
|
||||
# =========================================================================
|
||||
# SECURITY SERVICES
|
||||
# =========================================================================
|
||||
{
|
||||
"name": "vault",
|
||||
"display_name": "HashiCorp Vault",
|
||||
"description": "Secrets Management und Encryption as a Service",
|
||||
"service_type": SECURITY,
|
||||
"port": 8200,
|
||||
"technology_stack": ["Vault"],
|
||||
"repository_path": "/vault",
|
||||
"docker_image": "hashicorp/vault",
|
||||
"data_categories": ["secrets", "encryption_keys", "api_credentials"],
|
||||
"processes_pii": False,
|
||||
"processes_health_data": False,
|
||||
"ai_components": False,
|
||||
"criticality": "critical",
|
||||
"owner_team": "Security Team",
|
||||
"regulations": [
|
||||
{"code": "GDPR", "relevance": HIGH, "notes": "Art. 32 Verschlüsselung"},
|
||||
{"code": "BSI-TR-03161-1", "relevance": CRITICAL, "notes": "Schlüsselverwaltung"},
|
||||
{"code": "BSI-TR-03161-3", "relevance": CRITICAL, "notes": "O.Cryp Prüfaspekte"},
|
||||
]
|
||||
},
|
||||
|
||||
# =========================================================================
|
||||
# INFRASTRUCTURE
|
||||
# =========================================================================
|
||||
{
|
||||
"name": "traefik",
|
||||
"display_name": "Traefik Reverse Proxy",
|
||||
"description": "Reverse Proxy, Load Balancer und TLS Termination",
|
||||
"service_type": INFRASTRUCTURE,
|
||||
"port": 443,
|
||||
"technology_stack": ["Traefik", "Let's Encrypt"],
|
||||
"repository_path": None,
|
||||
"docker_image": "traefik",
|
||||
"data_categories": ["access_logs", "request_metadata"],
|
||||
"processes_pii": True,
|
||||
"processes_health_data": False,
|
||||
"ai_components": False,
|
||||
"criticality": "critical",
|
||||
"owner_team": "Infrastructure",
|
||||
"regulations": [
|
||||
{"code": "NIS2", "relevance": HIGH, "notes": "Netzwerksicherheit"},
|
||||
{"code": "BSI-TR-03161-2", "relevance": HIGH, "notes": "TLS-Konfiguration"},
|
||||
]
|
||||
},
|
||||
|
||||
# =========================================================================
|
||||
# MONITORING
|
||||
# =========================================================================
|
||||
{
|
||||
"name": "loki",
|
||||
"display_name": "Grafana Loki",
|
||||
"description": "Log-Aggregation und -Analyse",
|
||||
"service_type": MONITORING,
|
||||
"port": 3100,
|
||||
"technology_stack": ["Loki", "Grafana"],
|
||||
"repository_path": None,
|
||||
"docker_image": "grafana/loki",
|
||||
"data_categories": ["logs", "audit_trails"],
|
||||
"processes_pii": True,
|
||||
"processes_health_data": False,
|
||||
"ai_components": False,
|
||||
"criticality": "high",
|
||||
"owner_team": "Infrastructure",
|
||||
"regulations": [
|
||||
{"code": "GDPR", "relevance": MEDIUM, "notes": "Log-Retention"},
|
||||
{"code": "BSI-TR-03161-3", "relevance": HIGH, "notes": "O.Log Prüfaspekte"},
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "grafana",
|
||||
"display_name": "Grafana",
|
||||
"description": "Dashboards und Visualisierung",
|
||||
"service_type": MONITORING,
|
||||
"port": 3000,
|
||||
"technology_stack": ["Grafana"],
|
||||
"repository_path": None,
|
||||
"docker_image": "grafana/grafana",
|
||||
"data_categories": ["metrics", "dashboards"],
|
||||
"processes_pii": False,
|
||||
"processes_health_data": False,
|
||||
"ai_components": False,
|
||||
"criticality": "medium",
|
||||
"owner_team": "Infrastructure",
|
||||
"regulations": [
|
||||
{"code": "BSI-TR-03161-3", "relevance": MEDIUM, "notes": "Monitoring"},
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "prometheus",
|
||||
"display_name": "Prometheus",
|
||||
"description": "Metrics Collection und Alerting",
|
||||
"service_type": MONITORING,
|
||||
"port": 9090,
|
||||
"technology_stack": ["Prometheus"],
|
||||
"repository_path": None,
|
||||
"docker_image": "prom/prometheus",
|
||||
"data_categories": ["metrics", "alerts"],
|
||||
"processes_pii": False,
|
||||
"processes_health_data": False,
|
||||
"ai_components": False,
|
||||
"criticality": "medium",
|
||||
"owner_team": "Infrastructure",
|
||||
"regulations": [
|
||||
{"code": "NIS2", "relevance": MEDIUM, "notes": "Incident Detection"},
|
||||
]
|
||||
},
|
||||
|
||||
# =========================================================================
|
||||
# WEBSITE / FRONTEND
|
||||
# =========================================================================
|
||||
{
|
||||
"name": "website",
|
||||
"display_name": "Next.js Website",
|
||||
"description": "Frontend-Anwendung für Nutzer und Admin-Panel",
|
||||
"service_type": BACKEND,
|
||||
"port": 3000,
|
||||
"technology_stack": ["Next.js", "React", "TypeScript", "TailwindCSS"],
|
||||
"repository_path": "/website",
|
||||
"docker_image": "breakpilot-pwa-website",
|
||||
"data_categories": ["frontend_state", "ui_preferences"],
|
||||
"processes_pii": True,
|
||||
"processes_health_data": False,
|
||||
"ai_components": False,
|
||||
"criticality": "high",
|
||||
"owner_team": "Frontend Team",
|
||||
"regulations": [
|
||||
{"code": "GDPR", "relevance": HIGH, "notes": "Cookie-Consent UI"},
|
||||
{"code": "TDDDG", "relevance": CRITICAL, "notes": "Cookie-Banner"},
|
||||
{"code": "DSA", "relevance": MEDIUM, "notes": "Transparenz-Anforderungen"},
|
||||
{"code": "BSI-TR-03161-2", "relevance": HIGH, "notes": "XSS-Prävention, CSRF"},
|
||||
]
|
||||
},
|
||||
|
||||
# =========================================================================
|
||||
# ERP / BUSINESS
|
||||
# =========================================================================
|
||||
{
|
||||
"name": "erpnext",
|
||||
"display_name": "ERPNext",
|
||||
"description": "Enterprise Resource Planning für Schulverwaltung",
|
||||
"service_type": BACKEND,
|
||||
"port": 8080,
|
||||
"technology_stack": ["Python", "Frappe", "MariaDB"],
|
||||
"repository_path": None,
|
||||
"docker_image": "frappe/erpnext",
|
||||
"data_categories": ["business_data", "employee_data", "financial_data"],
|
||||
"processes_pii": True,
|
||||
"processes_health_data": False,
|
||||
"ai_components": False,
|
||||
"criticality": "high",
|
||||
"owner_team": "Business Team",
|
||||
"regulations": [
|
||||
{"code": "GDPR", "relevance": CRITICAL, "notes": "Mitarbeiterdaten"},
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "erpnext-db",
|
||||
"display_name": "ERPNext Database (MariaDB)",
|
||||
"description": "Dedizierte MariaDB für ERPNext",
|
||||
"service_type": DATABASE,
|
||||
"port": 3306,
|
||||
"technology_stack": ["MariaDB"],
|
||||
"repository_path": None,
|
||||
"docker_image": "mariadb",
|
||||
"data_categories": ["erp_data"],
|
||||
"processes_pii": True,
|
||||
"processes_health_data": False,
|
||||
"ai_components": False,
|
||||
"criticality": "high",
|
||||
"owner_team": "Infrastructure",
|
||||
"regulations": [
|
||||
{"code": "GDPR", "relevance": HIGH, "notes": "ERP-Daten"},
|
||||
{"code": "BSI-TR-03161-3", "relevance": HIGH, "notes": "Datenbank-Sicherheit"},
|
||||
]
|
||||
},
|
||||
|
||||
# =========================================================================
|
||||
# COMPLIANCE SERVICE (Self-Reference)
|
||||
# =========================================================================
|
||||
{
|
||||
"name": "compliance-module",
|
||||
"display_name": "Compliance & Audit Module",
|
||||
"description": "Dieses Modul - Compliance-Management, Audit-Vorbereitung, Risiko-Tracking",
|
||||
"service_type": BACKEND,
|
||||
"port": None,
|
||||
"technology_stack": ["Python", "FastAPI", "SQLAlchemy"],
|
||||
"repository_path": "/backend/compliance",
|
||||
"docker_image": None,
|
||||
"data_categories": ["compliance_data", "audit_records", "risk_assessments"],
|
||||
"processes_pii": False,
|
||||
"processes_health_data": False,
|
||||
"ai_components": True,
|
||||
"criticality": "high",
|
||||
"owner_team": "Compliance Team",
|
||||
"regulations": [
|
||||
{"code": "GDPR", "relevance": HIGH, "notes": "Art. 30 VVZ, Art. 35 DPIA"},
|
||||
{"code": "AIACT", "relevance": MEDIUM, "notes": "KI-Interpretations-Feature"},
|
||||
]
|
||||
},
|
||||
|
||||
# =========================================================================
|
||||
# DSMS - Dezentrales Speichersystem (Private IPFS)
|
||||
# =========================================================================
|
||||
{
|
||||
"name": "dsms-node",
|
||||
"display_name": "DSMS Node (IPFS)",
|
||||
"description": "Dezentraler IPFS-Node für verteiltes Speichersystem",
|
||||
"service_type": STORAGE,
|
||||
"port": 5001,
|
||||
"technology_stack": ["IPFS", "Go"],
|
||||
"repository_path": "/dsms-node",
|
||||
"docker_image": "breakpilot-pwa-dsms-node",
|
||||
"data_categories": ["distributed_files", "content_hashes"],
|
||||
"processes_pii": True,
|
||||
"processes_health_data": False,
|
||||
"ai_components": False,
|
||||
"criticality": "medium",
|
||||
"owner_team": "Infrastructure",
|
||||
"regulations": [
|
||||
{"code": "GDPR", "relevance": HIGH, "notes": "Dezentrale Datenspeicherung"},
|
||||
{"code": "BSI-TR-03161-3", "relevance": MEDIUM, "notes": "Speichersicherheit"},
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "dsms-gateway",
|
||||
"display_name": "DSMS Gateway",
|
||||
"description": "REST API Gateway für DSMS/IPFS Zugriff",
|
||||
"service_type": BACKEND,
|
||||
"port": 8082,
|
||||
"technology_stack": ["Python", "FastAPI"],
|
||||
"repository_path": "/dsms-gateway",
|
||||
"docker_image": "breakpilot-pwa-dsms-gateway",
|
||||
"data_categories": ["file_metadata", "access_logs"],
|
||||
"processes_pii": True,
|
||||
"processes_health_data": False,
|
||||
"ai_components": False,
|
||||
"criticality": "medium",
|
||||
"owner_team": "Backend Team",
|
||||
"regulations": [
|
||||
{"code": "GDPR", "relevance": MEDIUM, "notes": "API für Dateizugriff"},
|
||||
]
|
||||
},
|
||||
|
||||
# =========================================================================
|
||||
# ADDITIONAL INFRASTRUCTURE
|
||||
# =========================================================================
|
||||
{
|
||||
"name": "mailpit",
|
||||
"display_name": "Mailpit (Development Mail Server)",
|
||||
"description": "Lokaler E-Mail-Server für Entwicklung und Testing",
|
||||
"service_type": INFRASTRUCTURE,
|
||||
"port": 8025,
|
||||
"technology_stack": ["Go"],
|
||||
"repository_path": None,
|
||||
"docker_image": "axllent/mailpit",
|
||||
"data_categories": ["test_emails"],
|
||||
"processes_pii": True,
|
||||
"processes_health_data": False,
|
||||
"ai_components": False,
|
||||
"criticality": "low",
|
||||
"owner_team": "Infrastructure",
|
||||
"regulations": [
|
||||
{"code": "GDPR", "relevance": LOW, "notes": "Nur für Entwicklung"},
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "backup",
|
||||
"display_name": "Database Backup Service",
|
||||
"description": "Automatisches PostgreSQL Backup (täglich 2 Uhr)",
|
||||
"service_type": INFRASTRUCTURE,
|
||||
"port": None,
|
||||
"technology_stack": ["PostgreSQL Tools"],
|
||||
"repository_path": None,
|
||||
"docker_image": "postgres:16-alpine",
|
||||
"data_categories": ["database_backups"],
|
||||
"processes_pii": True,
|
||||
"processes_health_data": False,
|
||||
"ai_components": False,
|
||||
"criticality": "critical",
|
||||
"owner_team": "Infrastructure",
|
||||
"regulations": [
|
||||
{"code": "GDPR", "relevance": CRITICAL, "notes": "Art. 32 Backup-Pflicht"},
|
||||
{"code": "BSI-TR-03161-3", "relevance": CRITICAL, "notes": "O.Back_1 Datensicherung"},
|
||||
]
|
||||
},
|
||||
|
||||
# =========================================================================
|
||||
# BREAKPILOT DRIVE - Unity WebGL Lernspiel
|
||||
# =========================================================================
|
||||
{
|
||||
"name": "breakpilot-drive",
|
||||
"display_name": "Breakpilot Drive (Unity Game)",
|
||||
"description": "Unity WebGL Lernspiel mit LLM-Integration",
|
||||
"service_type": BACKEND,
|
||||
"port": 3001,
|
||||
"technology_stack": ["Unity", "WebGL", "Nginx"],
|
||||
"repository_path": "/breakpilot-drive",
|
||||
"docker_image": "breakpilot-pwa-drive",
|
||||
"data_categories": ["game_progress", "player_data", "leaderboards"],
|
||||
"processes_pii": True,
|
||||
"processes_health_data": False,
|
||||
"ai_components": True,
|
||||
"criticality": "medium",
|
||||
"owner_team": "Education Team",
|
||||
"regulations": [
|
||||
{"code": "GDPR", "relevance": MEDIUM, "notes": "Spieldaten und Fortschritt"},
|
||||
{"code": "AIACT", "relevance": MEDIUM, "notes": "LLM-Integration"},
|
||||
]
|
||||
},
|
||||
|
||||
# =========================================================================
|
||||
# CAMUNDA - BPMN Workflow Engine
|
||||
# =========================================================================
|
||||
{
|
||||
"name": "camunda",
|
||||
"display_name": "Camunda BPMN Platform",
|
||||
"description": "Workflow Engine für Business Process Automation",
|
||||
"service_type": BACKEND,
|
||||
"port": 8089,
|
||||
"technology_stack": ["Java", "Camunda", "PostgreSQL"],
|
||||
"repository_path": None,
|
||||
"docker_image": "camunda/camunda-bpm-platform",
|
||||
"data_categories": ["workflow_instances", "process_variables"],
|
||||
"processes_pii": True,
|
||||
"processes_health_data": False,
|
||||
"ai_components": False,
|
||||
"criticality": "medium",
|
||||
"owner_team": "Backend Team",
|
||||
"regulations": [
|
||||
{"code": "GDPR", "relevance": MEDIUM, "notes": "Workflow-Daten können PII enthalten"},
|
||||
]
|
||||
},
|
||||
]
|
||||
|
||||
|
||||
def get_service_count() -> int:
|
||||
"""Returns the number of registered services."""
|
||||
return len(BREAKPILOT_SERVICES)
|
||||
|
||||
|
||||
def get_services_by_type(service_type: str) -> List[Dict[str, Any]]:
|
||||
"""Returns all services of a specific type."""
|
||||
return [s for s in BREAKPILOT_SERVICES if s["service_type"] == service_type]
|
||||
|
||||
|
||||
def get_services_processing_pii() -> List[Dict[str, Any]]:
|
||||
"""Returns all services that process PII."""
|
||||
return [s for s in BREAKPILOT_SERVICES if s["processes_pii"]]
|
||||
|
||||
|
||||
def get_services_with_ai() -> List[Dict[str, Any]]:
|
||||
"""Returns all services with AI components."""
|
||||
return [s for s in BREAKPILOT_SERVICES if s["ai_components"]]
|
||||
|
||||
|
||||
def get_critical_services() -> List[Dict[str, Any]]:
|
||||
"""Returns all critical services."""
|
||||
return [s for s in BREAKPILOT_SERVICES if s["criticality"] == "critical"]
|
||||
50
backend/compliance/db/__init__.py
Normal file
50
backend/compliance/db/__init__.py
Normal file
@@ -0,0 +1,50 @@
|
||||
"""Database models and repository for Compliance module."""
|
||||
|
||||
from .models import (
|
||||
RegulationDB,
|
||||
RequirementDB,
|
||||
ControlDB,
|
||||
ControlMappingDB,
|
||||
EvidenceDB,
|
||||
RiskDB,
|
||||
AuditExportDB,
|
||||
RegulationTypeEnum,
|
||||
ControlTypeEnum,
|
||||
ControlDomainEnum,
|
||||
RiskLevelEnum,
|
||||
EvidenceStatusEnum,
|
||||
ControlStatusEnum,
|
||||
)
|
||||
from .repository import (
|
||||
RegulationRepository,
|
||||
RequirementRepository,
|
||||
ControlRepository,
|
||||
EvidenceRepository,
|
||||
RiskRepository,
|
||||
AuditExportRepository,
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
# Models
|
||||
"RegulationDB",
|
||||
"RequirementDB",
|
||||
"ControlDB",
|
||||
"ControlMappingDB",
|
||||
"EvidenceDB",
|
||||
"RiskDB",
|
||||
"AuditExportDB",
|
||||
# Enums
|
||||
"RegulationTypeEnum",
|
||||
"ControlTypeEnum",
|
||||
"ControlDomainEnum",
|
||||
"RiskLevelEnum",
|
||||
"EvidenceStatusEnum",
|
||||
"ControlStatusEnum",
|
||||
# Repositories
|
||||
"RegulationRepository",
|
||||
"RequirementRepository",
|
||||
"ControlRepository",
|
||||
"EvidenceRepository",
|
||||
"RiskRepository",
|
||||
"AuditExportRepository",
|
||||
]
|
||||
839
backend/compliance/db/isms_repository.py
Normal file
839
backend/compliance/db/isms_repository.py
Normal file
@@ -0,0 +1,839 @@
|
||||
"""
|
||||
Repository layer for ISMS (Information Security Management System) entities.
|
||||
|
||||
Provides CRUD operations for ISO 27001 certification-related entities:
|
||||
- ISMS Scope & Context
|
||||
- Policies & Objectives
|
||||
- Statement of Applicability (SoA)
|
||||
- Audit Findings & CAPA
|
||||
- Management Reviews & Internal Audits
|
||||
"""
|
||||
|
||||
import uuid
|
||||
from datetime import datetime, date
|
||||
from typing import List, Optional, Dict, Any, Tuple
|
||||
|
||||
from sqlalchemy.orm import Session as DBSession
|
||||
from sqlalchemy import func, and_, or_
|
||||
|
||||
from .models import (
|
||||
ISMSScopeDB, ISMSContextDB, ISMSPolicyDB, SecurityObjectiveDB,
|
||||
StatementOfApplicabilityDB, AuditFindingDB, CorrectiveActionDB,
|
||||
ManagementReviewDB, InternalAuditDB, AuditTrailDB, ISMSReadinessCheckDB,
|
||||
ApprovalStatusEnum, FindingTypeEnum, FindingStatusEnum, CAPATypeEnum
|
||||
)
|
||||
|
||||
|
||||
class ISMSScopeRepository:
|
||||
"""Repository for ISMS Scope (ISO 27001 Chapter 4.3)."""
|
||||
|
||||
def __init__(self, db: DBSession):
|
||||
self.db = db
|
||||
|
||||
def create(
|
||||
self,
|
||||
scope_statement: str,
|
||||
created_by: str,
|
||||
included_locations: Optional[List[str]] = None,
|
||||
included_processes: Optional[List[str]] = None,
|
||||
included_services: Optional[List[str]] = None,
|
||||
excluded_items: Optional[List[str]] = None,
|
||||
exclusion_justification: Optional[str] = None,
|
||||
organizational_boundary: Optional[str] = None,
|
||||
physical_boundary: Optional[str] = None,
|
||||
technical_boundary: Optional[str] = None,
|
||||
) -> ISMSScopeDB:
|
||||
"""Create a new ISMS scope definition."""
|
||||
# Supersede existing scopes
|
||||
existing = self.db.query(ISMSScopeDB).filter(
|
||||
ISMSScopeDB.status != ApprovalStatusEnum.SUPERSEDED
|
||||
).all()
|
||||
for s in existing:
|
||||
s.status = ApprovalStatusEnum.SUPERSEDED
|
||||
|
||||
scope = ISMSScopeDB(
|
||||
id=str(uuid.uuid4()),
|
||||
scope_statement=scope_statement,
|
||||
included_locations=included_locations,
|
||||
included_processes=included_processes,
|
||||
included_services=included_services,
|
||||
excluded_items=excluded_items,
|
||||
exclusion_justification=exclusion_justification,
|
||||
organizational_boundary=organizational_boundary,
|
||||
physical_boundary=physical_boundary,
|
||||
technical_boundary=technical_boundary,
|
||||
status=ApprovalStatusEnum.DRAFT,
|
||||
created_by=created_by,
|
||||
)
|
||||
self.db.add(scope)
|
||||
self.db.commit()
|
||||
self.db.refresh(scope)
|
||||
return scope
|
||||
|
||||
def get_current(self) -> Optional[ISMSScopeDB]:
|
||||
"""Get the current (non-superseded) ISMS scope."""
|
||||
return self.db.query(ISMSScopeDB).filter(
|
||||
ISMSScopeDB.status != ApprovalStatusEnum.SUPERSEDED
|
||||
).order_by(ISMSScopeDB.created_at.desc()).first()
|
||||
|
||||
def get_by_id(self, scope_id: str) -> Optional[ISMSScopeDB]:
|
||||
"""Get scope by ID."""
|
||||
return self.db.query(ISMSScopeDB).filter(ISMSScopeDB.id == scope_id).first()
|
||||
|
||||
def approve(
|
||||
self,
|
||||
scope_id: str,
|
||||
approved_by: str,
|
||||
effective_date: date,
|
||||
review_date: date,
|
||||
) -> Optional[ISMSScopeDB]:
|
||||
"""Approve the ISMS scope."""
|
||||
scope = self.get_by_id(scope_id)
|
||||
if not scope:
|
||||
return None
|
||||
|
||||
import hashlib
|
||||
scope.status = ApprovalStatusEnum.APPROVED
|
||||
scope.approved_by = approved_by
|
||||
scope.approved_at = datetime.utcnow()
|
||||
scope.effective_date = effective_date
|
||||
scope.review_date = review_date
|
||||
scope.approval_signature = hashlib.sha256(
|
||||
f"{scope.scope_statement}|{approved_by}|{datetime.utcnow().isoformat()}".encode()
|
||||
).hexdigest()
|
||||
|
||||
self.db.commit()
|
||||
self.db.refresh(scope)
|
||||
return scope
|
||||
|
||||
|
||||
class ISMSPolicyRepository:
|
||||
"""Repository for ISMS Policies (ISO 27001 Chapter 5.2)."""
|
||||
|
||||
def __init__(self, db: DBSession):
|
||||
self.db = db
|
||||
|
||||
def create(
|
||||
self,
|
||||
policy_id: str,
|
||||
title: str,
|
||||
policy_type: str,
|
||||
authored_by: str,
|
||||
description: Optional[str] = None,
|
||||
policy_text: Optional[str] = None,
|
||||
applies_to: Optional[List[str]] = None,
|
||||
review_frequency_months: int = 12,
|
||||
related_controls: Optional[List[str]] = None,
|
||||
) -> ISMSPolicyDB:
|
||||
"""Create a new ISMS policy."""
|
||||
policy = ISMSPolicyDB(
|
||||
id=str(uuid.uuid4()),
|
||||
policy_id=policy_id,
|
||||
title=title,
|
||||
policy_type=policy_type,
|
||||
description=description,
|
||||
policy_text=policy_text,
|
||||
applies_to=applies_to,
|
||||
review_frequency_months=review_frequency_months,
|
||||
related_controls=related_controls,
|
||||
authored_by=authored_by,
|
||||
status=ApprovalStatusEnum.DRAFT,
|
||||
)
|
||||
self.db.add(policy)
|
||||
self.db.commit()
|
||||
self.db.refresh(policy)
|
||||
return policy
|
||||
|
||||
def get_by_id(self, policy_id: str) -> Optional[ISMSPolicyDB]:
|
||||
"""Get policy by UUID or policy_id."""
|
||||
return self.db.query(ISMSPolicyDB).filter(
|
||||
(ISMSPolicyDB.id == policy_id) | (ISMSPolicyDB.policy_id == policy_id)
|
||||
).first()
|
||||
|
||||
def get_all(
|
||||
self,
|
||||
policy_type: Optional[str] = None,
|
||||
status: Optional[ApprovalStatusEnum] = None,
|
||||
) -> List[ISMSPolicyDB]:
|
||||
"""Get all policies with optional filters."""
|
||||
query = self.db.query(ISMSPolicyDB)
|
||||
if policy_type:
|
||||
query = query.filter(ISMSPolicyDB.policy_type == policy_type)
|
||||
if status:
|
||||
query = query.filter(ISMSPolicyDB.status == status)
|
||||
return query.order_by(ISMSPolicyDB.policy_id).all()
|
||||
|
||||
def get_master_policy(self) -> Optional[ISMSPolicyDB]:
|
||||
"""Get the approved master ISMS policy."""
|
||||
return self.db.query(ISMSPolicyDB).filter(
|
||||
ISMSPolicyDB.policy_type == "master",
|
||||
ISMSPolicyDB.status == ApprovalStatusEnum.APPROVED
|
||||
).first()
|
||||
|
||||
def approve(
|
||||
self,
|
||||
policy_id: str,
|
||||
approved_by: str,
|
||||
reviewed_by: str,
|
||||
effective_date: date,
|
||||
) -> Optional[ISMSPolicyDB]:
|
||||
"""Approve a policy."""
|
||||
policy = self.get_by_id(policy_id)
|
||||
if not policy:
|
||||
return None
|
||||
|
||||
import hashlib
|
||||
policy.status = ApprovalStatusEnum.APPROVED
|
||||
policy.reviewed_by = reviewed_by
|
||||
policy.approved_by = approved_by
|
||||
policy.approved_at = datetime.utcnow()
|
||||
policy.effective_date = effective_date
|
||||
policy.next_review_date = date(
|
||||
effective_date.year + (policy.review_frequency_months // 12),
|
||||
effective_date.month,
|
||||
effective_date.day
|
||||
)
|
||||
policy.approval_signature = hashlib.sha256(
|
||||
f"{policy.policy_id}|{approved_by}|{datetime.utcnow().isoformat()}".encode()
|
||||
).hexdigest()
|
||||
|
||||
self.db.commit()
|
||||
self.db.refresh(policy)
|
||||
return policy
|
||||
|
||||
|
||||
class SecurityObjectiveRepository:
|
||||
"""Repository for Security Objectives (ISO 27001 Chapter 6.2)."""
|
||||
|
||||
def __init__(self, db: DBSession):
|
||||
self.db = db
|
||||
|
||||
def create(
|
||||
self,
|
||||
objective_id: str,
|
||||
title: str,
|
||||
description: str,
|
||||
category: str,
|
||||
owner: str,
|
||||
kpi_name: Optional[str] = None,
|
||||
kpi_target: Optional[float] = None,
|
||||
kpi_unit: Optional[str] = None,
|
||||
target_date: Optional[date] = None,
|
||||
related_controls: Optional[List[str]] = None,
|
||||
) -> SecurityObjectiveDB:
|
||||
"""Create a new security objective."""
|
||||
objective = SecurityObjectiveDB(
|
||||
id=str(uuid.uuid4()),
|
||||
objective_id=objective_id,
|
||||
title=title,
|
||||
description=description,
|
||||
category=category,
|
||||
kpi_name=kpi_name,
|
||||
kpi_target=kpi_target,
|
||||
kpi_unit=kpi_unit,
|
||||
owner=owner,
|
||||
target_date=target_date,
|
||||
related_controls=related_controls,
|
||||
status="active",
|
||||
)
|
||||
self.db.add(objective)
|
||||
self.db.commit()
|
||||
self.db.refresh(objective)
|
||||
return objective
|
||||
|
||||
def get_by_id(self, objective_id: str) -> Optional[SecurityObjectiveDB]:
|
||||
"""Get objective by UUID or objective_id."""
|
||||
return self.db.query(SecurityObjectiveDB).filter(
|
||||
(SecurityObjectiveDB.id == objective_id) |
|
||||
(SecurityObjectiveDB.objective_id == objective_id)
|
||||
).first()
|
||||
|
||||
def get_all(
|
||||
self,
|
||||
category: Optional[str] = None,
|
||||
status: Optional[str] = None,
|
||||
) -> List[SecurityObjectiveDB]:
|
||||
"""Get all objectives with optional filters."""
|
||||
query = self.db.query(SecurityObjectiveDB)
|
||||
if category:
|
||||
query = query.filter(SecurityObjectiveDB.category == category)
|
||||
if status:
|
||||
query = query.filter(SecurityObjectiveDB.status == status)
|
||||
return query.order_by(SecurityObjectiveDB.objective_id).all()
|
||||
|
||||
def update_progress(
|
||||
self,
|
||||
objective_id: str,
|
||||
kpi_current: float,
|
||||
) -> Optional[SecurityObjectiveDB]:
|
||||
"""Update objective progress."""
|
||||
objective = self.get_by_id(objective_id)
|
||||
if not objective:
|
||||
return None
|
||||
|
||||
objective.kpi_current = kpi_current
|
||||
if objective.kpi_target:
|
||||
objective.progress_percentage = min(100, (kpi_current / objective.kpi_target) * 100)
|
||||
|
||||
# Auto-mark as achieved if 100%
|
||||
if objective.progress_percentage >= 100 and objective.status == "active":
|
||||
objective.status = "achieved"
|
||||
objective.achieved_date = date.today()
|
||||
|
||||
self.db.commit()
|
||||
self.db.refresh(objective)
|
||||
return objective
|
||||
|
||||
|
||||
class StatementOfApplicabilityRepository:
|
||||
"""Repository for Statement of Applicability (SoA)."""
|
||||
|
||||
def __init__(self, db: DBSession):
|
||||
self.db = db
|
||||
|
||||
def create(
|
||||
self,
|
||||
annex_a_control: str,
|
||||
annex_a_title: str,
|
||||
annex_a_category: str,
|
||||
is_applicable: bool = True,
|
||||
applicability_justification: Optional[str] = None,
|
||||
implementation_status: str = "planned",
|
||||
breakpilot_control_ids: Optional[List[str]] = None,
|
||||
) -> StatementOfApplicabilityDB:
|
||||
"""Create a new SoA entry."""
|
||||
entry = StatementOfApplicabilityDB(
|
||||
id=str(uuid.uuid4()),
|
||||
annex_a_control=annex_a_control,
|
||||
annex_a_title=annex_a_title,
|
||||
annex_a_category=annex_a_category,
|
||||
is_applicable=is_applicable,
|
||||
applicability_justification=applicability_justification,
|
||||
implementation_status=implementation_status,
|
||||
breakpilot_control_ids=breakpilot_control_ids or [],
|
||||
)
|
||||
self.db.add(entry)
|
||||
self.db.commit()
|
||||
self.db.refresh(entry)
|
||||
return entry
|
||||
|
||||
def get_by_control(self, annex_a_control: str) -> Optional[StatementOfApplicabilityDB]:
|
||||
"""Get SoA entry by Annex A control ID (e.g., 'A.5.1')."""
|
||||
return self.db.query(StatementOfApplicabilityDB).filter(
|
||||
StatementOfApplicabilityDB.annex_a_control == annex_a_control
|
||||
).first()
|
||||
|
||||
def get_all(
|
||||
self,
|
||||
is_applicable: Optional[bool] = None,
|
||||
implementation_status: Optional[str] = None,
|
||||
category: Optional[str] = None,
|
||||
) -> List[StatementOfApplicabilityDB]:
|
||||
"""Get all SoA entries with optional filters."""
|
||||
query = self.db.query(StatementOfApplicabilityDB)
|
||||
if is_applicable is not None:
|
||||
query = query.filter(StatementOfApplicabilityDB.is_applicable == is_applicable)
|
||||
if implementation_status:
|
||||
query = query.filter(StatementOfApplicabilityDB.implementation_status == implementation_status)
|
||||
if category:
|
||||
query = query.filter(StatementOfApplicabilityDB.annex_a_category == category)
|
||||
return query.order_by(StatementOfApplicabilityDB.annex_a_control).all()
|
||||
|
||||
def get_statistics(self) -> Dict[str, Any]:
|
||||
"""Get SoA statistics."""
|
||||
entries = self.get_all()
|
||||
total = len(entries)
|
||||
applicable = sum(1 for e in entries if e.is_applicable)
|
||||
implemented = sum(1 for e in entries if e.implementation_status == "implemented")
|
||||
approved = sum(1 for e in entries if e.approved_at)
|
||||
|
||||
return {
|
||||
"total": total,
|
||||
"applicable": applicable,
|
||||
"not_applicable": total - applicable,
|
||||
"implemented": implemented,
|
||||
"planned": sum(1 for e in entries if e.implementation_status == "planned"),
|
||||
"approved": approved,
|
||||
"pending_approval": total - approved,
|
||||
"implementation_rate": round((implemented / applicable * 100) if applicable > 0 else 0, 1),
|
||||
}
|
||||
|
||||
|
||||
class AuditFindingRepository:
|
||||
"""Repository for Audit Findings (Major/Minor/OFI)."""
|
||||
|
||||
def __init__(self, db: DBSession):
|
||||
self.db = db
|
||||
|
||||
def create(
|
||||
self,
|
||||
finding_type: FindingTypeEnum,
|
||||
title: str,
|
||||
description: str,
|
||||
auditor: str,
|
||||
iso_chapter: Optional[str] = None,
|
||||
annex_a_control: Optional[str] = None,
|
||||
objective_evidence: Optional[str] = None,
|
||||
owner: Optional[str] = None,
|
||||
due_date: Optional[date] = None,
|
||||
internal_audit_id: Optional[str] = None,
|
||||
) -> AuditFindingDB:
|
||||
"""Create a new audit finding."""
|
||||
# Generate finding ID
|
||||
year = date.today().year
|
||||
existing_count = self.db.query(AuditFindingDB).filter(
|
||||
AuditFindingDB.finding_id.like(f"FIND-{year}-%")
|
||||
).count()
|
||||
finding_id = f"FIND-{year}-{existing_count + 1:03d}"
|
||||
|
||||
finding = AuditFindingDB(
|
||||
id=str(uuid.uuid4()),
|
||||
finding_id=finding_id,
|
||||
finding_type=finding_type,
|
||||
iso_chapter=iso_chapter,
|
||||
annex_a_control=annex_a_control,
|
||||
title=title,
|
||||
description=description,
|
||||
objective_evidence=objective_evidence,
|
||||
owner=owner,
|
||||
auditor=auditor,
|
||||
due_date=due_date,
|
||||
internal_audit_id=internal_audit_id,
|
||||
status=FindingStatusEnum.OPEN,
|
||||
)
|
||||
self.db.add(finding)
|
||||
self.db.commit()
|
||||
self.db.refresh(finding)
|
||||
return finding
|
||||
|
||||
def get_by_id(self, finding_id: str) -> Optional[AuditFindingDB]:
|
||||
"""Get finding by UUID or finding_id."""
|
||||
return self.db.query(AuditFindingDB).filter(
|
||||
(AuditFindingDB.id == finding_id) | (AuditFindingDB.finding_id == finding_id)
|
||||
).first()
|
||||
|
||||
def get_all(
|
||||
self,
|
||||
finding_type: Optional[FindingTypeEnum] = None,
|
||||
status: Optional[FindingStatusEnum] = None,
|
||||
internal_audit_id: Optional[str] = None,
|
||||
) -> List[AuditFindingDB]:
|
||||
"""Get all findings with optional filters."""
|
||||
query = self.db.query(AuditFindingDB)
|
||||
if finding_type:
|
||||
query = query.filter(AuditFindingDB.finding_type == finding_type)
|
||||
if status:
|
||||
query = query.filter(AuditFindingDB.status == status)
|
||||
if internal_audit_id:
|
||||
query = query.filter(AuditFindingDB.internal_audit_id == internal_audit_id)
|
||||
return query.order_by(AuditFindingDB.identified_date.desc()).all()
|
||||
|
||||
def get_open_majors(self) -> List[AuditFindingDB]:
|
||||
"""Get all open major findings (blocking certification)."""
|
||||
return self.db.query(AuditFindingDB).filter(
|
||||
AuditFindingDB.finding_type == FindingTypeEnum.MAJOR,
|
||||
AuditFindingDB.status != FindingStatusEnum.CLOSED
|
||||
).all()
|
||||
|
||||
def get_statistics(self) -> Dict[str, Any]:
|
||||
"""Get finding statistics."""
|
||||
findings = self.get_all()
|
||||
|
||||
return {
|
||||
"total": len(findings),
|
||||
"major": sum(1 for f in findings if f.finding_type == FindingTypeEnum.MAJOR),
|
||||
"minor": sum(1 for f in findings if f.finding_type == FindingTypeEnum.MINOR),
|
||||
"ofi": sum(1 for f in findings if f.finding_type == FindingTypeEnum.OFI),
|
||||
"positive": sum(1 for f in findings if f.finding_type == FindingTypeEnum.POSITIVE),
|
||||
"open": sum(1 for f in findings if f.status != FindingStatusEnum.CLOSED),
|
||||
"closed": sum(1 for f in findings if f.status == FindingStatusEnum.CLOSED),
|
||||
"blocking_certification": sum(
|
||||
1 for f in findings
|
||||
if f.finding_type == FindingTypeEnum.MAJOR and f.status != FindingStatusEnum.CLOSED
|
||||
),
|
||||
}
|
||||
|
||||
def close(
|
||||
self,
|
||||
finding_id: str,
|
||||
closed_by: str,
|
||||
closure_notes: str,
|
||||
verification_method: Optional[str] = None,
|
||||
verification_evidence: Optional[str] = None,
|
||||
) -> Optional[AuditFindingDB]:
|
||||
"""Close a finding after verification."""
|
||||
finding = self.get_by_id(finding_id)
|
||||
if not finding:
|
||||
return None
|
||||
|
||||
finding.status = FindingStatusEnum.CLOSED
|
||||
finding.closed_date = date.today()
|
||||
finding.closed_by = closed_by
|
||||
finding.closure_notes = closure_notes
|
||||
finding.verification_method = verification_method
|
||||
finding.verification_evidence = verification_evidence
|
||||
finding.verified_by = closed_by
|
||||
finding.verified_at = datetime.utcnow()
|
||||
|
||||
self.db.commit()
|
||||
self.db.refresh(finding)
|
||||
return finding
|
||||
|
||||
|
||||
class CorrectiveActionRepository:
|
||||
"""Repository for Corrective/Preventive Actions (CAPA)."""
|
||||
|
||||
def __init__(self, db: DBSession):
|
||||
self.db = db
|
||||
|
||||
def create(
|
||||
self,
|
||||
finding_id: str,
|
||||
capa_type: CAPATypeEnum,
|
||||
title: str,
|
||||
description: str,
|
||||
assigned_to: str,
|
||||
planned_completion: date,
|
||||
expected_outcome: Optional[str] = None,
|
||||
effectiveness_criteria: Optional[str] = None,
|
||||
) -> CorrectiveActionDB:
|
||||
"""Create a new CAPA."""
|
||||
# Generate CAPA ID
|
||||
year = date.today().year
|
||||
existing_count = self.db.query(CorrectiveActionDB).filter(
|
||||
CorrectiveActionDB.capa_id.like(f"CAPA-{year}-%")
|
||||
).count()
|
||||
capa_id = f"CAPA-{year}-{existing_count + 1:03d}"
|
||||
|
||||
capa = CorrectiveActionDB(
|
||||
id=str(uuid.uuid4()),
|
||||
capa_id=capa_id,
|
||||
finding_id=finding_id,
|
||||
capa_type=capa_type,
|
||||
title=title,
|
||||
description=description,
|
||||
expected_outcome=expected_outcome,
|
||||
assigned_to=assigned_to,
|
||||
planned_completion=planned_completion,
|
||||
effectiveness_criteria=effectiveness_criteria,
|
||||
status="planned",
|
||||
)
|
||||
self.db.add(capa)
|
||||
|
||||
# Update finding status
|
||||
finding = self.db.query(AuditFindingDB).filter(AuditFindingDB.id == finding_id).first()
|
||||
if finding:
|
||||
finding.status = FindingStatusEnum.CORRECTIVE_ACTION_PENDING
|
||||
|
||||
self.db.commit()
|
||||
self.db.refresh(capa)
|
||||
return capa
|
||||
|
||||
def get_by_id(self, capa_id: str) -> Optional[CorrectiveActionDB]:
|
||||
"""Get CAPA by UUID or capa_id."""
|
||||
return self.db.query(CorrectiveActionDB).filter(
|
||||
(CorrectiveActionDB.id == capa_id) | (CorrectiveActionDB.capa_id == capa_id)
|
||||
).first()
|
||||
|
||||
def get_by_finding(self, finding_id: str) -> List[CorrectiveActionDB]:
|
||||
"""Get all CAPAs for a finding."""
|
||||
return self.db.query(CorrectiveActionDB).filter(
|
||||
CorrectiveActionDB.finding_id == finding_id
|
||||
).order_by(CorrectiveActionDB.planned_completion).all()
|
||||
|
||||
def verify(
|
||||
self,
|
||||
capa_id: str,
|
||||
verified_by: str,
|
||||
is_effective: bool,
|
||||
effectiveness_notes: Optional[str] = None,
|
||||
) -> Optional[CorrectiveActionDB]:
|
||||
"""Verify a completed CAPA."""
|
||||
capa = self.get_by_id(capa_id)
|
||||
if not capa:
|
||||
return None
|
||||
|
||||
capa.effectiveness_verified = is_effective
|
||||
capa.effectiveness_verification_date = date.today()
|
||||
capa.effectiveness_notes = effectiveness_notes
|
||||
capa.status = "verified" if is_effective else "completed"
|
||||
|
||||
# If verified, check if all CAPAs for finding are verified
|
||||
if is_effective:
|
||||
finding = self.db.query(AuditFindingDB).filter(
|
||||
AuditFindingDB.id == capa.finding_id
|
||||
).first()
|
||||
if finding:
|
||||
unverified = self.db.query(CorrectiveActionDB).filter(
|
||||
CorrectiveActionDB.finding_id == finding.id,
|
||||
CorrectiveActionDB.id != capa.id,
|
||||
CorrectiveActionDB.status != "verified"
|
||||
).count()
|
||||
if unverified == 0:
|
||||
finding.status = FindingStatusEnum.VERIFICATION_PENDING
|
||||
|
||||
self.db.commit()
|
||||
self.db.refresh(capa)
|
||||
return capa
|
||||
|
||||
|
||||
class ManagementReviewRepository:
|
||||
"""Repository for Management Reviews (ISO 27001 Chapter 9.3)."""
|
||||
|
||||
def __init__(self, db: DBSession):
|
||||
self.db = db
|
||||
|
||||
def create(
|
||||
self,
|
||||
title: str,
|
||||
review_date: date,
|
||||
chairperson: str,
|
||||
review_period_start: Optional[date] = None,
|
||||
review_period_end: Optional[date] = None,
|
||||
) -> ManagementReviewDB:
|
||||
"""Create a new management review."""
|
||||
# Generate review ID
|
||||
year = review_date.year
|
||||
quarter = (review_date.month - 1) // 3 + 1
|
||||
review_id = f"MR-{year}-Q{quarter}"
|
||||
|
||||
# Check for duplicate
|
||||
existing = self.db.query(ManagementReviewDB).filter(
|
||||
ManagementReviewDB.review_id == review_id
|
||||
).first()
|
||||
if existing:
|
||||
review_id = f"{review_id}-{str(uuid.uuid4())[:4]}"
|
||||
|
||||
review = ManagementReviewDB(
|
||||
id=str(uuid.uuid4()),
|
||||
review_id=review_id,
|
||||
title=title,
|
||||
review_date=review_date,
|
||||
review_period_start=review_period_start,
|
||||
review_period_end=review_period_end,
|
||||
chairperson=chairperson,
|
||||
status="draft",
|
||||
)
|
||||
self.db.add(review)
|
||||
self.db.commit()
|
||||
self.db.refresh(review)
|
||||
return review
|
||||
|
||||
def get_by_id(self, review_id: str) -> Optional[ManagementReviewDB]:
|
||||
"""Get review by UUID or review_id."""
|
||||
return self.db.query(ManagementReviewDB).filter(
|
||||
(ManagementReviewDB.id == review_id) | (ManagementReviewDB.review_id == review_id)
|
||||
).first()
|
||||
|
||||
def get_latest_approved(self) -> Optional[ManagementReviewDB]:
|
||||
"""Get the most recent approved management review."""
|
||||
return self.db.query(ManagementReviewDB).filter(
|
||||
ManagementReviewDB.status == "approved"
|
||||
).order_by(ManagementReviewDB.review_date.desc()).first()
|
||||
|
||||
def approve(
|
||||
self,
|
||||
review_id: str,
|
||||
approved_by: str,
|
||||
next_review_date: date,
|
||||
minutes_document_path: Optional[str] = None,
|
||||
) -> Optional[ManagementReviewDB]:
|
||||
"""Approve a management review."""
|
||||
review = self.get_by_id(review_id)
|
||||
if not review:
|
||||
return None
|
||||
|
||||
review.status = "approved"
|
||||
review.approved_by = approved_by
|
||||
review.approved_at = datetime.utcnow()
|
||||
review.next_review_date = next_review_date
|
||||
review.minutes_document_path = minutes_document_path
|
||||
|
||||
self.db.commit()
|
||||
self.db.refresh(review)
|
||||
return review
|
||||
|
||||
|
||||
class InternalAuditRepository:
|
||||
"""Repository for Internal Audits (ISO 27001 Chapter 9.2)."""
|
||||
|
||||
def __init__(self, db: DBSession):
|
||||
self.db = db
|
||||
|
||||
def create(
|
||||
self,
|
||||
title: str,
|
||||
audit_type: str,
|
||||
planned_date: date,
|
||||
lead_auditor: str,
|
||||
scope_description: Optional[str] = None,
|
||||
iso_chapters_covered: Optional[List[str]] = None,
|
||||
annex_a_controls_covered: Optional[List[str]] = None,
|
||||
) -> InternalAuditDB:
|
||||
"""Create a new internal audit."""
|
||||
# Generate audit ID
|
||||
year = planned_date.year
|
||||
existing_count = self.db.query(InternalAuditDB).filter(
|
||||
InternalAuditDB.audit_id.like(f"IA-{year}-%")
|
||||
).count()
|
||||
audit_id = f"IA-{year}-{existing_count + 1:03d}"
|
||||
|
||||
audit = InternalAuditDB(
|
||||
id=str(uuid.uuid4()),
|
||||
audit_id=audit_id,
|
||||
title=title,
|
||||
audit_type=audit_type,
|
||||
scope_description=scope_description,
|
||||
iso_chapters_covered=iso_chapters_covered,
|
||||
annex_a_controls_covered=annex_a_controls_covered,
|
||||
planned_date=planned_date,
|
||||
lead_auditor=lead_auditor,
|
||||
status="planned",
|
||||
)
|
||||
self.db.add(audit)
|
||||
self.db.commit()
|
||||
self.db.refresh(audit)
|
||||
return audit
|
||||
|
||||
def get_by_id(self, audit_id: str) -> Optional[InternalAuditDB]:
|
||||
"""Get audit by UUID or audit_id."""
|
||||
return self.db.query(InternalAuditDB).filter(
|
||||
(InternalAuditDB.id == audit_id) | (InternalAuditDB.audit_id == audit_id)
|
||||
).first()
|
||||
|
||||
def get_latest_completed(self) -> Optional[InternalAuditDB]:
|
||||
"""Get the most recent completed internal audit."""
|
||||
return self.db.query(InternalAuditDB).filter(
|
||||
InternalAuditDB.status == "completed"
|
||||
).order_by(InternalAuditDB.actual_end_date.desc()).first()
|
||||
|
||||
def complete(
|
||||
self,
|
||||
audit_id: str,
|
||||
audit_conclusion: str,
|
||||
overall_assessment: str,
|
||||
follow_up_audit_required: bool = False,
|
||||
) -> Optional[InternalAuditDB]:
|
||||
"""Complete an internal audit."""
|
||||
audit = self.get_by_id(audit_id)
|
||||
if not audit:
|
||||
return None
|
||||
|
||||
audit.status = "completed"
|
||||
audit.actual_end_date = date.today()
|
||||
audit.report_date = date.today()
|
||||
audit.audit_conclusion = audit_conclusion
|
||||
audit.overall_assessment = overall_assessment
|
||||
audit.follow_up_audit_required = follow_up_audit_required
|
||||
|
||||
self.db.commit()
|
||||
self.db.refresh(audit)
|
||||
return audit
|
||||
|
||||
|
||||
class AuditTrailRepository:
|
||||
"""Repository for Audit Trail entries."""
|
||||
|
||||
def __init__(self, db: DBSession):
|
||||
self.db = db
|
||||
|
||||
def log(
|
||||
self,
|
||||
entity_type: str,
|
||||
entity_id: str,
|
||||
entity_name: str,
|
||||
action: str,
|
||||
performed_by: str,
|
||||
field_changed: Optional[str] = None,
|
||||
old_value: Optional[str] = None,
|
||||
new_value: Optional[str] = None,
|
||||
change_summary: Optional[str] = None,
|
||||
) -> AuditTrailDB:
|
||||
"""Log an audit trail entry."""
|
||||
import hashlib
|
||||
entry = AuditTrailDB(
|
||||
id=str(uuid.uuid4()),
|
||||
entity_type=entity_type,
|
||||
entity_id=entity_id,
|
||||
entity_name=entity_name,
|
||||
action=action,
|
||||
field_changed=field_changed,
|
||||
old_value=old_value,
|
||||
new_value=new_value,
|
||||
change_summary=change_summary,
|
||||
performed_by=performed_by,
|
||||
performed_at=datetime.utcnow(),
|
||||
checksum=hashlib.sha256(
|
||||
f"{entity_type}|{entity_id}|{action}|{performed_by}".encode()
|
||||
).hexdigest(),
|
||||
)
|
||||
self.db.add(entry)
|
||||
self.db.commit()
|
||||
self.db.refresh(entry)
|
||||
return entry
|
||||
|
||||
def get_by_entity(
|
||||
self,
|
||||
entity_type: str,
|
||||
entity_id: str,
|
||||
limit: int = 100,
|
||||
) -> List[AuditTrailDB]:
|
||||
"""Get audit trail for a specific entity."""
|
||||
return self.db.query(AuditTrailDB).filter(
|
||||
AuditTrailDB.entity_type == entity_type,
|
||||
AuditTrailDB.entity_id == entity_id
|
||||
).order_by(AuditTrailDB.performed_at.desc()).limit(limit).all()
|
||||
|
||||
def get_paginated(
|
||||
self,
|
||||
page: int = 1,
|
||||
page_size: int = 50,
|
||||
entity_type: Optional[str] = None,
|
||||
entity_id: Optional[str] = None,
|
||||
performed_by: Optional[str] = None,
|
||||
action: Optional[str] = None,
|
||||
) -> Tuple[List[AuditTrailDB], int]:
|
||||
"""Get paginated audit trail with filters."""
|
||||
query = self.db.query(AuditTrailDB)
|
||||
|
||||
if entity_type:
|
||||
query = query.filter(AuditTrailDB.entity_type == entity_type)
|
||||
if entity_id:
|
||||
query = query.filter(AuditTrailDB.entity_id == entity_id)
|
||||
if performed_by:
|
||||
query = query.filter(AuditTrailDB.performed_by == performed_by)
|
||||
if action:
|
||||
query = query.filter(AuditTrailDB.action == action)
|
||||
|
||||
total = query.count()
|
||||
entries = query.order_by(AuditTrailDB.performed_at.desc()).offset(
|
||||
(page - 1) * page_size
|
||||
).limit(page_size).all()
|
||||
|
||||
return entries, total
|
||||
|
||||
|
||||
class ISMSReadinessCheckRepository:
|
||||
"""Repository for ISMS Readiness Check results."""
|
||||
|
||||
def __init__(self, db: DBSession):
|
||||
self.db = db
|
||||
|
||||
def save(self, check: ISMSReadinessCheckDB) -> ISMSReadinessCheckDB:
|
||||
"""Save a readiness check result."""
|
||||
self.db.add(check)
|
||||
self.db.commit()
|
||||
self.db.refresh(check)
|
||||
return check
|
||||
|
||||
def get_latest(self) -> Optional[ISMSReadinessCheckDB]:
|
||||
"""Get the most recent readiness check."""
|
||||
return self.db.query(ISMSReadinessCheckDB).order_by(
|
||||
ISMSReadinessCheckDB.check_date.desc()
|
||||
).first()
|
||||
|
||||
def get_history(self, limit: int = 10) -> List[ISMSReadinessCheckDB]:
|
||||
"""Get readiness check history."""
|
||||
return self.db.query(ISMSReadinessCheckDB).order_by(
|
||||
ISMSReadinessCheckDB.check_date.desc()
|
||||
).limit(limit).all()
|
||||
1413
backend/compliance/db/models.py
Normal file
1413
backend/compliance/db/models.py
Normal file
File diff suppressed because it is too large
Load Diff
1538
backend/compliance/db/repository.py
Normal file
1538
backend/compliance/db/repository.py
Normal file
File diff suppressed because it is too large
Load Diff
3
backend/compliance/scripts/__init__.py
Normal file
3
backend/compliance/scripts/__init__.py
Normal file
@@ -0,0 +1,3 @@
|
||||
"""
|
||||
Compliance module scripts for seeding and validation.
|
||||
"""
|
||||
97
backend/compliance/scripts/seed_service_modules.py
Normal file
97
backend/compliance/scripts/seed_service_modules.py
Normal file
@@ -0,0 +1,97 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Script to seed service modules into the compliance database.
|
||||
|
||||
Usage:
|
||||
python -m compliance.scripts.seed_service_modules
|
||||
|
||||
This script can be run standalone or imported for use in migrations.
|
||||
"""
|
||||
|
||||
import sys
|
||||
import logging
|
||||
from pathlib import Path
|
||||
|
||||
# Add parent directory to path to allow imports
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent.parent))
|
||||
|
||||
from classroom_engine.database import SessionLocal
|
||||
from compliance.services.seeder import ComplianceSeeder
|
||||
from compliance.data.service_modules import get_service_count
|
||||
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
|
||||
)
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def seed_service_modules():
|
||||
"""Seed service modules and their regulation mappings."""
|
||||
logger.info("Starting service module seeding...")
|
||||
|
||||
db = SessionLocal()
|
||||
try:
|
||||
seeder = ComplianceSeeder(db)
|
||||
|
||||
# Get total services available
|
||||
total_services = get_service_count()
|
||||
logger.info(f"Found {total_services} services in BREAKPILOT_SERVICES")
|
||||
|
||||
# Seed service modules
|
||||
result = seeder.seed_service_modules_only()
|
||||
|
||||
logger.info(f"✓ Successfully seeded {result} items (modules + regulation mappings)")
|
||||
return result
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"✗ Seeding failed: {e}", exc_info=True)
|
||||
raise
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
|
||||
def seed_all_compliance_data():
|
||||
"""Seed all compliance data including service modules."""
|
||||
logger.info("Starting full compliance database seeding...")
|
||||
|
||||
db = SessionLocal()
|
||||
try:
|
||||
seeder = ComplianceSeeder(db)
|
||||
results = seeder.seed_all(force=False)
|
||||
|
||||
logger.info("✓ Seeding completed successfully!")
|
||||
logger.info(f" - Regulations: {results['regulations']}")
|
||||
logger.info(f" - Controls: {results['controls']}")
|
||||
logger.info(f" - Requirements: {results['requirements']}")
|
||||
logger.info(f" - Control Mappings: {results['mappings']}")
|
||||
logger.info(f" - Risks: {results['risks']}")
|
||||
logger.info(f" - Service Modules: {results['service_modules']}")
|
||||
logger.info(f" - Module-Regulation Mappings: {results['module_regulation_mappings']}")
|
||||
|
||||
return results
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"✗ Seeding failed: {e}", exc_info=True)
|
||||
raise
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
import argparse
|
||||
|
||||
parser = argparse.ArgumentParser(description="Seed compliance database")
|
||||
parser.add_argument(
|
||||
"--mode",
|
||||
choices=["modules", "all"],
|
||||
default="modules",
|
||||
help="Seeding mode: 'modules' (service modules only) or 'all' (complete database)"
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.mode == "modules":
|
||||
seed_service_modules()
|
||||
else:
|
||||
seed_all_compliance_data()
|
||||
207
backend/compliance/scripts/validate_service_modules.py
Normal file
207
backend/compliance/scripts/validate_service_modules.py
Normal file
@@ -0,0 +1,207 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Validate service modules configuration.
|
||||
|
||||
Checks:
|
||||
- All services have required fields
|
||||
- Port conflicts
|
||||
- Technology stack completeness
|
||||
- Regulation mappings validity
|
||||
"""
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from collections import defaultdict
|
||||
from typing import Dict, List, Set
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent.parent))
|
||||
|
||||
from compliance.data.service_modules import (
|
||||
BREAKPILOT_SERVICES,
|
||||
get_service_count,
|
||||
get_services_by_type,
|
||||
get_services_processing_pii,
|
||||
get_services_with_ai,
|
||||
get_critical_services
|
||||
)
|
||||
from compliance.data.regulations import REGULATIONS_SEED
|
||||
|
||||
# Color codes for output
|
||||
GREEN = '\033[92m'
|
||||
YELLOW = '\033[93m'
|
||||
RED = '\033[91m'
|
||||
RESET = '\033[0m'
|
||||
|
||||
|
||||
def validate_required_fields():
|
||||
"""Validate that all services have required fields."""
|
||||
print(f"\n{GREEN}=== Validating Required Fields ==={RESET}")
|
||||
errors = []
|
||||
|
||||
required_fields = ["name", "display_name", "service_type", "technology_stack"]
|
||||
|
||||
for service in BREAKPILOT_SERVICES:
|
||||
for field in required_fields:
|
||||
if field not in service or not service[field]:
|
||||
errors.append(f" {RED}✗{RESET} Service '{service.get('name', 'UNKNOWN')}' missing required field: {field}")
|
||||
|
||||
if errors:
|
||||
print(f"\n{RED}Found {len(errors)} errors:{RESET}")
|
||||
for error in errors:
|
||||
print(error)
|
||||
return False
|
||||
else:
|
||||
print(f"{GREEN}✓ All services have required fields{RESET}")
|
||||
return True
|
||||
|
||||
|
||||
def validate_port_conflicts():
|
||||
"""Check for port conflicts."""
|
||||
print(f"\n{GREEN}=== Checking Port Conflicts ==={RESET}")
|
||||
port_map: Dict[int, List[str]] = defaultdict(list)
|
||||
warnings = []
|
||||
|
||||
for service in BREAKPILOT_SERVICES:
|
||||
port = service.get("port")
|
||||
if port:
|
||||
port_map[port].append(service["name"])
|
||||
|
||||
for port, services in port_map.items():
|
||||
if len(services) > 1:
|
||||
warnings.append(f" {YELLOW}⚠{RESET} Port {port} used by multiple services: {', '.join(services)}")
|
||||
|
||||
if warnings:
|
||||
print(f"\n{YELLOW}Found {len(warnings)} port conflicts:{RESET}")
|
||||
for warning in warnings:
|
||||
print(warning)
|
||||
return False
|
||||
else:
|
||||
print(f"{GREEN}✓ No port conflicts detected{RESET}")
|
||||
return True
|
||||
|
||||
|
||||
def validate_regulation_mappings():
|
||||
"""Validate that all regulation codes exist."""
|
||||
print(f"\n{GREEN}=== Validating Regulation Mappings ==={RESET}")
|
||||
valid_regulation_codes = {reg["code"] for reg in REGULATIONS_SEED}
|
||||
errors = []
|
||||
|
||||
for service in BREAKPILOT_SERVICES:
|
||||
regulations = service.get("regulations", [])
|
||||
for reg_mapping in regulations:
|
||||
reg_code = reg_mapping.get("code")
|
||||
if reg_code not in valid_regulation_codes:
|
||||
errors.append(
|
||||
f" {RED}✗{RESET} Service '{service['name']}' references unknown regulation: {reg_code}"
|
||||
)
|
||||
|
||||
# Check for relevance field
|
||||
if "relevance" not in reg_mapping:
|
||||
errors.append(
|
||||
f" {RED}✗{RESET} Service '{service['name']}' regulation mapping for {reg_code} missing 'relevance'"
|
||||
)
|
||||
|
||||
if errors:
|
||||
print(f"\n{RED}Found {len(errors)} regulation mapping errors:{RESET}")
|
||||
for error in errors:
|
||||
print(error)
|
||||
return False
|
||||
else:
|
||||
print(f"{GREEN}✓ All regulation mappings are valid{RESET}")
|
||||
return True
|
||||
|
||||
|
||||
def print_statistics():
|
||||
"""Print statistics about the service registry."""
|
||||
print(f"\n{GREEN}=== Service Module Statistics ==={RESET}")
|
||||
|
||||
total = get_service_count()
|
||||
print(f"\nTotal Services: {total}")
|
||||
|
||||
# By type
|
||||
print(f"\n{GREEN}By Type:{RESET}")
|
||||
service_types = ["backend", "database", "ai", "communication", "storage", "infrastructure", "monitoring", "security"]
|
||||
for stype in service_types:
|
||||
count = len(get_services_by_type(stype))
|
||||
if count > 0:
|
||||
print(f" - {stype.capitalize()}: {count}")
|
||||
|
||||
# PII processing
|
||||
pii_count = len(get_services_processing_pii())
|
||||
print(f"\n{GREEN}Data Processing:{RESET}")
|
||||
print(f" - Processing PII: {pii_count}")
|
||||
print(f" - AI Components: {len(get_services_with_ai())}")
|
||||
|
||||
# Criticality
|
||||
critical_count = len(get_critical_services())
|
||||
print(f"\n{GREEN}Criticality:{RESET}")
|
||||
print(f" - Critical Services: {critical_count}")
|
||||
|
||||
# Ports
|
||||
services_with_ports = [s for s in BREAKPILOT_SERVICES if s.get("port")]
|
||||
print(f"\n{GREEN}Network:{RESET}")
|
||||
print(f" - Services with exposed ports: {len(services_with_ports)}")
|
||||
|
||||
# Docker images
|
||||
docker_services = [s for s in BREAKPILOT_SERVICES if s.get("docker_image")]
|
||||
print(f"\n{GREEN}Docker:{RESET}")
|
||||
print(f" - Services with Docker images: {len(docker_services)}")
|
||||
|
||||
# Most common technologies
|
||||
tech_count: Dict[str, int] = defaultdict(int)
|
||||
for service in BREAKPILOT_SERVICES:
|
||||
for tech in service.get("technology_stack", []):
|
||||
tech_count[tech] += 1
|
||||
|
||||
print(f"\n{GREEN}Top Technologies:{RESET}")
|
||||
for tech, count in sorted(tech_count.items(), key=lambda x: x[1], reverse=True)[:10]:
|
||||
print(f" - {tech}: {count}")
|
||||
|
||||
|
||||
def validate_data_categories():
|
||||
"""Check that PII-processing services have data_categories defined."""
|
||||
print(f"\n{GREEN}=== Validating Data Categories ==={RESET}")
|
||||
warnings = []
|
||||
|
||||
for service in BREAKPILOT_SERVICES:
|
||||
if service.get("processes_pii") and not service.get("data_categories"):
|
||||
warnings.append(
|
||||
f" {YELLOW}⚠{RESET} Service '{service['name']}' processes PII but has no data_categories defined"
|
||||
)
|
||||
|
||||
if warnings:
|
||||
print(f"\n{YELLOW}Found {len(warnings)} warnings:{RESET}")
|
||||
for warning in warnings:
|
||||
print(warning)
|
||||
return False
|
||||
else:
|
||||
print(f"{GREEN}✓ All PII-processing services have data categories{RESET}")
|
||||
return True
|
||||
|
||||
|
||||
def main():
|
||||
"""Run all validations."""
|
||||
print(f"{GREEN}{'='*60}")
|
||||
print(f" Breakpilot Service Module Validation")
|
||||
print(f"{'='*60}{RESET}")
|
||||
|
||||
all_passed = True
|
||||
|
||||
all_passed &= validate_required_fields()
|
||||
all_passed &= validate_port_conflicts()
|
||||
all_passed &= validate_regulation_mappings()
|
||||
all_passed &= validate_data_categories()
|
||||
|
||||
print_statistics()
|
||||
|
||||
print(f"\n{GREEN}{'='*60}{RESET}")
|
||||
if all_passed:
|
||||
print(f"{GREEN}✓ All validations passed!{RESET}")
|
||||
return 0
|
||||
else:
|
||||
print(f"{YELLOW}⚠ Some validations failed or have warnings{RESET}")
|
||||
return 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
17
backend/compliance/services/__init__.py
Normal file
17
backend/compliance/services/__init__.py
Normal file
@@ -0,0 +1,17 @@
|
||||
"""
|
||||
Compliance Services Module.
|
||||
|
||||
Contains business logic services for the compliance module:
|
||||
- PDF extraction from BSI-TR and EU regulations
|
||||
- LLM-based requirement interpretation
|
||||
- Export generation
|
||||
"""
|
||||
|
||||
from .pdf_extractor import BSIPDFExtractor, BSIAspect, EURegulationExtractor, EUArticle
|
||||
|
||||
__all__ = [
|
||||
"BSIPDFExtractor",
|
||||
"BSIAspect",
|
||||
"EURegulationExtractor",
|
||||
"EUArticle",
|
||||
]
|
||||
500
backend/compliance/services/ai_compliance_assistant.py
Normal file
500
backend/compliance/services/ai_compliance_assistant.py
Normal file
@@ -0,0 +1,500 @@
|
||||
"""
|
||||
AI Compliance Assistant for Breakpilot.
|
||||
|
||||
Provides AI-powered features for:
|
||||
- Requirement interpretation (translating legal text to technical guidance)
|
||||
- Control suggestions (recommending controls for requirements)
|
||||
- Risk assessment (evaluating compliance risks)
|
||||
- Gap analysis (identifying missing controls)
|
||||
"""
|
||||
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
from dataclasses import dataclass, field
|
||||
from typing import List, Optional, Dict, Any
|
||||
from enum import Enum
|
||||
|
||||
from .llm_provider import LLMProvider, get_shared_provider, LLMResponse
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class InterpretationSection(str, Enum):
|
||||
"""Sections in a requirement interpretation."""
|
||||
SUMMARY = "summary"
|
||||
APPLICABILITY = "applicability"
|
||||
TECHNICAL_MEASURES = "technical_measures"
|
||||
AFFECTED_MODULES = "affected_modules"
|
||||
RISK_LEVEL = "risk_level"
|
||||
IMPLEMENTATION_HINTS = "implementation_hints"
|
||||
|
||||
|
||||
@dataclass
|
||||
class RequirementInterpretation:
|
||||
"""AI-generated interpretation of a regulatory requirement."""
|
||||
requirement_id: str
|
||||
summary: str
|
||||
applicability: str
|
||||
technical_measures: List[str]
|
||||
affected_modules: List[str]
|
||||
risk_level: str # low, medium, high, critical
|
||||
implementation_hints: List[str]
|
||||
confidence_score: float # 0.0 - 1.0
|
||||
raw_response: Optional[str] = None
|
||||
error: Optional[str] = None
|
||||
|
||||
|
||||
@dataclass
|
||||
class ControlSuggestion:
|
||||
"""AI-suggested control for a requirement."""
|
||||
control_id: str # Suggested ID like "PRIV-XXX"
|
||||
domain: str # Control domain (priv, sdlc, iam, etc.)
|
||||
title: str
|
||||
description: str
|
||||
pass_criteria: str
|
||||
implementation_guidance: str
|
||||
is_automated: bool
|
||||
automation_tool: Optional[str] = None
|
||||
priority: str = "medium" # low, medium, high, critical
|
||||
confidence_score: float = 0.0
|
||||
|
||||
|
||||
@dataclass
|
||||
class RiskAssessment:
|
||||
"""AI-generated risk assessment for a module."""
|
||||
module_name: str
|
||||
overall_risk: str # low, medium, high, critical
|
||||
risk_factors: List[Dict[str, Any]]
|
||||
recommendations: List[str]
|
||||
compliance_gaps: List[str]
|
||||
confidence_score: float = 0.0
|
||||
|
||||
|
||||
@dataclass
|
||||
class GapAnalysis:
|
||||
"""Gap analysis result for requirement-control mapping."""
|
||||
requirement_id: str
|
||||
requirement_title: str
|
||||
coverage_level: str # full, partial, none
|
||||
existing_controls: List[str]
|
||||
missing_coverage: List[str]
|
||||
suggested_actions: List[str]
|
||||
|
||||
|
||||
class AIComplianceAssistant:
|
||||
"""
|
||||
AI-powered compliance assistant using LLM providers.
|
||||
|
||||
Supports both Claude API and self-hosted LLMs through the
|
||||
abstracted LLMProvider interface.
|
||||
"""
|
||||
|
||||
# System prompts for different tasks
|
||||
SYSTEM_PROMPT_BASE = """Du bist ein Compliance-Experte für die Breakpilot Bildungsplattform.
|
||||
Breakpilot ist ein EdTech SaaS-System mit folgenden Eigenschaften:
|
||||
- KI-gestützte Klausurkorrektur und Feedback
|
||||
- Videokonferenzen (Jitsi) und Chat (Matrix)
|
||||
- Schulverwaltung mit Noten und Zeugnissen
|
||||
- Consent-Management und DSGVO-Compliance
|
||||
- Self-Hosted in Deutschland
|
||||
|
||||
Du analysierst regulatorische Anforderungen und gibst konkrete technische Empfehlungen."""
|
||||
|
||||
INTERPRETATION_PROMPT = """Analysiere folgende regulatorische Anforderung für Breakpilot:
|
||||
|
||||
Verordnung: {regulation_name} ({regulation_code})
|
||||
Artikel: {article}
|
||||
Titel: {title}
|
||||
Originaltext: {requirement_text}
|
||||
|
||||
Erstelle eine strukturierte Analyse im JSON-Format:
|
||||
{{
|
||||
"summary": "Kurze Zusammenfassung in 2-3 Sätzen",
|
||||
"applicability": "Erklärung wie dies auf Breakpilot anwendbar ist",
|
||||
"technical_measures": ["Liste konkreter technischer Maßnahmen"],
|
||||
"affected_modules": ["Liste betroffener Breakpilot-Module (z.B. consent-service, klausur-service, matrix-synapse)"],
|
||||
"risk_level": "low|medium|high|critical",
|
||||
"implementation_hints": ["Konkrete Implementierungshinweise"]
|
||||
}}
|
||||
|
||||
Gib NUR das JSON zurück, keine zusätzlichen Erklärungen."""
|
||||
|
||||
CONTROL_SUGGESTION_PROMPT = """Basierend auf folgender Anforderung, schlage passende Controls vor:
|
||||
|
||||
Verordnung: {regulation_name}
|
||||
Anforderung: {requirement_title}
|
||||
Beschreibung: {requirement_text}
|
||||
Betroffene Module: {affected_modules}
|
||||
|
||||
Schlage 1-3 Controls im JSON-Format vor:
|
||||
{{
|
||||
"controls": [
|
||||
{{
|
||||
"control_id": "DOMAIN-XXX",
|
||||
"domain": "priv|iam|sdlc|crypto|ops|ai|cra|gov|aud",
|
||||
"title": "Kurzer Titel",
|
||||
"description": "Beschreibung des Controls",
|
||||
"pass_criteria": "Messbare Erfolgskriterien",
|
||||
"implementation_guidance": "Wie implementieren",
|
||||
"is_automated": true|false,
|
||||
"automation_tool": "Tool-Name oder null",
|
||||
"priority": "low|medium|high|critical"
|
||||
}}
|
||||
]
|
||||
}}
|
||||
|
||||
Domains:
|
||||
- priv: Datenschutz & Privacy (DSGVO)
|
||||
- iam: Identity & Access Management
|
||||
- sdlc: Secure Development Lifecycle
|
||||
- crypto: Kryptografie
|
||||
- ops: Betrieb & Monitoring
|
||||
- ai: KI-spezifisch (AI Act)
|
||||
- cra: Cyber Resilience Act
|
||||
- gov: Governance
|
||||
- aud: Audit & Nachvollziehbarkeit
|
||||
|
||||
Gib NUR das JSON zurück."""
|
||||
|
||||
RISK_ASSESSMENT_PROMPT = """Bewerte das Compliance-Risiko für folgendes Breakpilot-Modul:
|
||||
|
||||
Modul: {module_name}
|
||||
Typ: {service_type}
|
||||
Beschreibung: {description}
|
||||
Verarbeitet PII: {processes_pii}
|
||||
KI-Komponenten: {ai_components}
|
||||
Kritikalität: {criticality}
|
||||
Daten-Kategorien: {data_categories}
|
||||
Zugeordnete Verordnungen: {regulations}
|
||||
|
||||
Erstelle eine Risikobewertung im JSON-Format:
|
||||
{{
|
||||
"overall_risk": "low|medium|high|critical",
|
||||
"risk_factors": [
|
||||
{{"factor": "Beschreibung", "severity": "low|medium|high", "likelihood": "low|medium|high"}}
|
||||
],
|
||||
"recommendations": ["Empfehlungen zur Risikominderung"],
|
||||
"compliance_gaps": ["Identifizierte Compliance-Lücken"]
|
||||
}}
|
||||
|
||||
Gib NUR das JSON zurück."""
|
||||
|
||||
GAP_ANALYSIS_PROMPT = """Analysiere die Control-Abdeckung für folgende Anforderung:
|
||||
|
||||
Anforderung: {requirement_title}
|
||||
Verordnung: {regulation_code}
|
||||
Beschreibung: {requirement_text}
|
||||
|
||||
Existierende Controls:
|
||||
{existing_controls}
|
||||
|
||||
Bewerte die Abdeckung und identifiziere Lücken im JSON-Format:
|
||||
{{
|
||||
"coverage_level": "full|partial|none",
|
||||
"covered_aspects": ["Was ist bereits abgedeckt"],
|
||||
"missing_coverage": ["Was fehlt noch"],
|
||||
"suggested_actions": ["Empfohlene Maßnahmen"]
|
||||
}}
|
||||
|
||||
Gib NUR das JSON zurück."""
|
||||
|
||||
def __init__(self, llm_provider: Optional[LLMProvider] = None):
|
||||
"""Initialize the assistant with an LLM provider."""
|
||||
self.llm = llm_provider or get_shared_provider()
|
||||
|
||||
async def interpret_requirement(
|
||||
self,
|
||||
requirement_id: str,
|
||||
article: str,
|
||||
title: str,
|
||||
requirement_text: str,
|
||||
regulation_code: str,
|
||||
regulation_name: str
|
||||
) -> RequirementInterpretation:
|
||||
"""
|
||||
Generate an interpretation for a regulatory requirement.
|
||||
|
||||
Translates legal text into practical technical guidance
|
||||
for the Breakpilot development team.
|
||||
"""
|
||||
prompt = self.INTERPRETATION_PROMPT.format(
|
||||
regulation_name=regulation_name,
|
||||
regulation_code=regulation_code,
|
||||
article=article,
|
||||
title=title,
|
||||
requirement_text=requirement_text or "Kein Text verfügbar"
|
||||
)
|
||||
|
||||
try:
|
||||
response = await self.llm.complete(
|
||||
prompt=prompt,
|
||||
system_prompt=self.SYSTEM_PROMPT_BASE,
|
||||
max_tokens=2000,
|
||||
temperature=0.3
|
||||
)
|
||||
|
||||
# Parse JSON response
|
||||
data = self._parse_json_response(response.content)
|
||||
|
||||
return RequirementInterpretation(
|
||||
requirement_id=requirement_id,
|
||||
summary=data.get("summary", ""),
|
||||
applicability=data.get("applicability", ""),
|
||||
technical_measures=data.get("technical_measures", []),
|
||||
affected_modules=data.get("affected_modules", []),
|
||||
risk_level=data.get("risk_level", "medium"),
|
||||
implementation_hints=data.get("implementation_hints", []),
|
||||
confidence_score=0.85, # Based on model quality
|
||||
raw_response=response.content
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to interpret requirement {requirement_id}: {e}")
|
||||
return RequirementInterpretation(
|
||||
requirement_id=requirement_id,
|
||||
summary="",
|
||||
applicability="",
|
||||
technical_measures=[],
|
||||
affected_modules=[],
|
||||
risk_level="medium",
|
||||
implementation_hints=[],
|
||||
confidence_score=0.0,
|
||||
error=str(e)
|
||||
)
|
||||
|
||||
async def suggest_controls(
|
||||
self,
|
||||
requirement_title: str,
|
||||
requirement_text: str,
|
||||
regulation_name: str,
|
||||
affected_modules: List[str]
|
||||
) -> List[ControlSuggestion]:
|
||||
"""
|
||||
Suggest controls for a given requirement.
|
||||
|
||||
Returns a list of control suggestions with implementation guidance.
|
||||
"""
|
||||
prompt = self.CONTROL_SUGGESTION_PROMPT.format(
|
||||
regulation_name=regulation_name,
|
||||
requirement_title=requirement_title,
|
||||
requirement_text=requirement_text or "Keine Beschreibung",
|
||||
affected_modules=", ".join(affected_modules) if affected_modules else "Alle Module"
|
||||
)
|
||||
|
||||
try:
|
||||
response = await self.llm.complete(
|
||||
prompt=prompt,
|
||||
system_prompt=self.SYSTEM_PROMPT_BASE,
|
||||
max_tokens=2000,
|
||||
temperature=0.4
|
||||
)
|
||||
|
||||
data = self._parse_json_response(response.content)
|
||||
controls = data.get("controls", [])
|
||||
|
||||
return [
|
||||
ControlSuggestion(
|
||||
control_id=c.get("control_id", "NEW-001"),
|
||||
domain=c.get("domain", "gov"),
|
||||
title=c.get("title", ""),
|
||||
description=c.get("description", ""),
|
||||
pass_criteria=c.get("pass_criteria", ""),
|
||||
implementation_guidance=c.get("implementation_guidance", ""),
|
||||
is_automated=c.get("is_automated", False),
|
||||
automation_tool=c.get("automation_tool"),
|
||||
priority=c.get("priority", "medium"),
|
||||
confidence_score=0.75
|
||||
)
|
||||
for c in controls
|
||||
]
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to suggest controls: {e}")
|
||||
return []
|
||||
|
||||
async def assess_module_risk(
|
||||
self,
|
||||
module_name: str,
|
||||
service_type: str,
|
||||
description: str,
|
||||
processes_pii: bool,
|
||||
ai_components: bool,
|
||||
criticality: str,
|
||||
data_categories: List[str],
|
||||
regulations: List[Dict[str, str]]
|
||||
) -> RiskAssessment:
|
||||
"""
|
||||
Assess the compliance risk for a service module.
|
||||
"""
|
||||
prompt = self.RISK_ASSESSMENT_PROMPT.format(
|
||||
module_name=module_name,
|
||||
service_type=service_type,
|
||||
description=description or "Keine Beschreibung",
|
||||
processes_pii="Ja" if processes_pii else "Nein",
|
||||
ai_components="Ja" if ai_components else "Nein",
|
||||
criticality=criticality,
|
||||
data_categories=", ".join(data_categories) if data_categories else "Keine",
|
||||
regulations=", ".join([f"{r['code']} ({r.get('relevance', 'medium')})" for r in regulations]) if regulations else "Keine"
|
||||
)
|
||||
|
||||
try:
|
||||
response = await self.llm.complete(
|
||||
prompt=prompt,
|
||||
system_prompt=self.SYSTEM_PROMPT_BASE,
|
||||
max_tokens=1500,
|
||||
temperature=0.3
|
||||
)
|
||||
|
||||
data = self._parse_json_response(response.content)
|
||||
|
||||
return RiskAssessment(
|
||||
module_name=module_name,
|
||||
overall_risk=data.get("overall_risk", "medium"),
|
||||
risk_factors=data.get("risk_factors", []),
|
||||
recommendations=data.get("recommendations", []),
|
||||
compliance_gaps=data.get("compliance_gaps", []),
|
||||
confidence_score=0.8
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to assess risk for {module_name}: {e}")
|
||||
return RiskAssessment(
|
||||
module_name=module_name,
|
||||
overall_risk="unknown",
|
||||
risk_factors=[],
|
||||
recommendations=[],
|
||||
compliance_gaps=[],
|
||||
confidence_score=0.0
|
||||
)
|
||||
|
||||
async def analyze_gap(
|
||||
self,
|
||||
requirement_id: str,
|
||||
requirement_title: str,
|
||||
requirement_text: str,
|
||||
regulation_code: str,
|
||||
existing_controls: List[Dict[str, str]]
|
||||
) -> GapAnalysis:
|
||||
"""
|
||||
Analyze gaps between requirements and existing controls.
|
||||
"""
|
||||
controls_text = "\n".join([
|
||||
f"- {c.get('control_id', 'N/A')}: {c.get('title', 'N/A')} - {c.get('status', 'N/A')}"
|
||||
for c in existing_controls
|
||||
]) if existing_controls else "Keine Controls zugeordnet"
|
||||
|
||||
prompt = self.GAP_ANALYSIS_PROMPT.format(
|
||||
requirement_title=requirement_title,
|
||||
regulation_code=regulation_code,
|
||||
requirement_text=requirement_text or "Keine Beschreibung",
|
||||
existing_controls=controls_text
|
||||
)
|
||||
|
||||
try:
|
||||
response = await self.llm.complete(
|
||||
prompt=prompt,
|
||||
system_prompt=self.SYSTEM_PROMPT_BASE,
|
||||
max_tokens=1500,
|
||||
temperature=0.3
|
||||
)
|
||||
|
||||
data = self._parse_json_response(response.content)
|
||||
|
||||
return GapAnalysis(
|
||||
requirement_id=requirement_id,
|
||||
requirement_title=requirement_title,
|
||||
coverage_level=data.get("coverage_level", "none"),
|
||||
existing_controls=[c.get("control_id", "") for c in existing_controls],
|
||||
missing_coverage=data.get("missing_coverage", []),
|
||||
suggested_actions=data.get("suggested_actions", [])
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to analyze gap for {requirement_id}: {e}")
|
||||
return GapAnalysis(
|
||||
requirement_id=requirement_id,
|
||||
requirement_title=requirement_title,
|
||||
coverage_level="unknown",
|
||||
existing_controls=[],
|
||||
missing_coverage=[],
|
||||
suggested_actions=[]
|
||||
)
|
||||
|
||||
async def batch_interpret_requirements(
|
||||
self,
|
||||
requirements: List[Dict[str, Any]],
|
||||
rate_limit: float = 1.0
|
||||
) -> List[RequirementInterpretation]:
|
||||
"""
|
||||
Process multiple requirements with rate limiting.
|
||||
|
||||
Useful for bulk processing of regulations.
|
||||
"""
|
||||
results = []
|
||||
|
||||
for i, req in enumerate(requirements):
|
||||
if i > 0:
|
||||
import asyncio
|
||||
await asyncio.sleep(rate_limit)
|
||||
|
||||
result = await self.interpret_requirement(
|
||||
requirement_id=req.get("id", str(i)),
|
||||
article=req.get("article", ""),
|
||||
title=req.get("title", ""),
|
||||
requirement_text=req.get("requirement_text", ""),
|
||||
regulation_code=req.get("regulation_code", ""),
|
||||
regulation_name=req.get("regulation_name", "")
|
||||
)
|
||||
results.append(result)
|
||||
|
||||
logger.info(f"Processed requirement {i+1}/{len(requirements)}: {req.get('title', 'N/A')}")
|
||||
|
||||
return results
|
||||
|
||||
def _parse_json_response(self, content: str) -> Dict[str, Any]:
|
||||
"""
|
||||
Parse JSON from LLM response, handling common formatting issues.
|
||||
"""
|
||||
# Try to extract JSON from the response
|
||||
content = content.strip()
|
||||
|
||||
# Remove markdown code blocks if present
|
||||
if content.startswith("```json"):
|
||||
content = content[7:]
|
||||
elif content.startswith("```"):
|
||||
content = content[3:]
|
||||
if content.endswith("```"):
|
||||
content = content[:-3]
|
||||
|
||||
content = content.strip()
|
||||
|
||||
# Find JSON object in the response
|
||||
json_match = re.search(r'\{[\s\S]*\}', content)
|
||||
if json_match:
|
||||
content = json_match.group(0)
|
||||
|
||||
try:
|
||||
return json.loads(content)
|
||||
except json.JSONDecodeError as e:
|
||||
logger.warning(f"Failed to parse JSON response: {e}")
|
||||
logger.debug(f"Raw content: {content[:500]}")
|
||||
return {}
|
||||
|
||||
|
||||
# Singleton instance
|
||||
_assistant_instance: Optional[AIComplianceAssistant] = None
|
||||
|
||||
|
||||
def get_ai_assistant() -> AIComplianceAssistant:
|
||||
"""Get the shared AI compliance assistant instance."""
|
||||
global _assistant_instance
|
||||
if _assistant_instance is None:
|
||||
_assistant_instance = AIComplianceAssistant()
|
||||
return _assistant_instance
|
||||
|
||||
|
||||
def reset_ai_assistant():
|
||||
"""Reset the shared assistant instance (useful for testing)."""
|
||||
global _assistant_instance
|
||||
_assistant_instance = None
|
||||
880
backend/compliance/services/audit_pdf_generator.py
Normal file
880
backend/compliance/services/audit_pdf_generator.py
Normal file
@@ -0,0 +1,880 @@
|
||||
"""
|
||||
Audit Session PDF Report Generator.
|
||||
|
||||
Sprint 3 Phase 4: Generates PDF reports for completed audit sessions.
|
||||
|
||||
Features:
|
||||
- Cover page with audit session metadata
|
||||
- Executive summary with traffic light status
|
||||
- Statistics pie chart (compliant/non-compliant/pending)
|
||||
- Detailed checklist with sign-off status
|
||||
- Digital signature verification
|
||||
- Appendix with non-compliant items
|
||||
|
||||
Uses reportlab for PDF generation (lightweight, no external dependencies).
|
||||
"""
|
||||
|
||||
import io
|
||||
import logging
|
||||
from datetime import datetime
|
||||
from typing import Dict, List, Any, Optional, Tuple
|
||||
from uuid import uuid4
|
||||
import hashlib
|
||||
|
||||
from sqlalchemy.orm import Session, selectinload
|
||||
|
||||
from reportlab.lib import colors
|
||||
from reportlab.lib.pagesizes import A4
|
||||
from reportlab.lib.styles import getSampleStyleSheet, ParagraphStyle
|
||||
from reportlab.lib.units import mm, cm
|
||||
from reportlab.lib.enums import TA_CENTER, TA_LEFT, TA_RIGHT, TA_JUSTIFY
|
||||
from reportlab.platypus import (
|
||||
SimpleDocTemplate, Paragraph, Spacer, Table, TableStyle,
|
||||
PageBreak, Image, ListFlowable, ListItem, KeepTogether,
|
||||
HRFlowable
|
||||
)
|
||||
from reportlab.graphics.shapes import Drawing, Rect, String
|
||||
from reportlab.graphics.charts.piecharts import Pie
|
||||
|
||||
from ..db.models import (
|
||||
AuditSessionDB, AuditSignOffDB, AuditResultEnum, AuditSessionStatusEnum,
|
||||
RequirementDB, RegulationDB
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Color Definitions
|
||||
# =============================================================================
|
||||
|
||||
COLORS = {
|
||||
'primary': colors.HexColor('#1a365d'), # Dark blue
|
||||
'secondary': colors.HexColor('#2c5282'), # Medium blue
|
||||
'accent': colors.HexColor('#3182ce'), # Light blue
|
||||
'success': colors.HexColor('#38a169'), # Green
|
||||
'warning': colors.HexColor('#d69e2e'), # Yellow/Orange
|
||||
'danger': colors.HexColor('#e53e3e'), # Red
|
||||
'muted': colors.HexColor('#718096'), # Gray
|
||||
'light': colors.HexColor('#f7fafc'), # Light gray
|
||||
'white': colors.white,
|
||||
'black': colors.black,
|
||||
}
|
||||
|
||||
RESULT_COLORS = {
|
||||
'compliant': COLORS['success'],
|
||||
'compliant_notes': colors.HexColor('#68d391'), # Light green
|
||||
'non_compliant': COLORS['danger'],
|
||||
'not_applicable': COLORS['muted'],
|
||||
'pending': COLORS['warning'],
|
||||
}
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Custom Styles
|
||||
# =============================================================================
|
||||
|
||||
def get_custom_styles() -> Dict[str, ParagraphStyle]:
|
||||
"""Create custom paragraph styles for the audit report."""
|
||||
styles = getSampleStyleSheet()
|
||||
|
||||
custom = {
|
||||
'Title': ParagraphStyle(
|
||||
'AuditTitle',
|
||||
parent=styles['Title'],
|
||||
fontSize=24,
|
||||
textColor=COLORS['primary'],
|
||||
spaceAfter=12*mm,
|
||||
alignment=TA_CENTER,
|
||||
),
|
||||
'Subtitle': ParagraphStyle(
|
||||
'AuditSubtitle',
|
||||
parent=styles['Normal'],
|
||||
fontSize=14,
|
||||
textColor=COLORS['secondary'],
|
||||
spaceAfter=6*mm,
|
||||
alignment=TA_CENTER,
|
||||
),
|
||||
'Heading1': ParagraphStyle(
|
||||
'AuditH1',
|
||||
parent=styles['Heading1'],
|
||||
fontSize=18,
|
||||
textColor=COLORS['primary'],
|
||||
spaceBefore=12*mm,
|
||||
spaceAfter=6*mm,
|
||||
borderPadding=3*mm,
|
||||
),
|
||||
'Heading2': ParagraphStyle(
|
||||
'AuditH2',
|
||||
parent=styles['Heading2'],
|
||||
fontSize=14,
|
||||
textColor=COLORS['secondary'],
|
||||
spaceBefore=8*mm,
|
||||
spaceAfter=4*mm,
|
||||
),
|
||||
'Heading3': ParagraphStyle(
|
||||
'AuditH3',
|
||||
parent=styles['Heading3'],
|
||||
fontSize=12,
|
||||
textColor=COLORS['accent'],
|
||||
spaceBefore=6*mm,
|
||||
spaceAfter=3*mm,
|
||||
),
|
||||
'Normal': ParagraphStyle(
|
||||
'AuditNormal',
|
||||
parent=styles['Normal'],
|
||||
fontSize=10,
|
||||
textColor=COLORS['black'],
|
||||
spaceAfter=3*mm,
|
||||
alignment=TA_JUSTIFY,
|
||||
),
|
||||
'Small': ParagraphStyle(
|
||||
'AuditSmall',
|
||||
parent=styles['Normal'],
|
||||
fontSize=8,
|
||||
textColor=COLORS['muted'],
|
||||
spaceAfter=2*mm,
|
||||
),
|
||||
'Footer': ParagraphStyle(
|
||||
'AuditFooter',
|
||||
parent=styles['Normal'],
|
||||
fontSize=8,
|
||||
textColor=COLORS['muted'],
|
||||
alignment=TA_CENTER,
|
||||
),
|
||||
'Success': ParagraphStyle(
|
||||
'AuditSuccess',
|
||||
parent=styles['Normal'],
|
||||
fontSize=10,
|
||||
textColor=COLORS['success'],
|
||||
),
|
||||
'Warning': ParagraphStyle(
|
||||
'AuditWarning',
|
||||
parent=styles['Normal'],
|
||||
fontSize=10,
|
||||
textColor=COLORS['warning'],
|
||||
),
|
||||
'Danger': ParagraphStyle(
|
||||
'AuditDanger',
|
||||
parent=styles['Normal'],
|
||||
fontSize=10,
|
||||
textColor=COLORS['danger'],
|
||||
),
|
||||
}
|
||||
|
||||
return custom
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# PDF Generator Class
|
||||
# =============================================================================
|
||||
|
||||
class AuditPDFGenerator:
|
||||
"""Generates PDF reports for audit sessions."""
|
||||
|
||||
def __init__(self, db: Session):
|
||||
self.db = db
|
||||
self.styles = get_custom_styles()
|
||||
self.page_width, self.page_height = A4
|
||||
self.margin = 20 * mm
|
||||
|
||||
def generate(
|
||||
self,
|
||||
session_id: str,
|
||||
language: str = 'de',
|
||||
include_signatures: bool = True,
|
||||
) -> Tuple[bytes, str]:
|
||||
"""
|
||||
Generate a PDF report for an audit session.
|
||||
|
||||
Args:
|
||||
session_id: The audit session ID
|
||||
language: Report language ('de' or 'en')
|
||||
include_signatures: Whether to include digital signature info
|
||||
|
||||
Returns:
|
||||
Tuple of (PDF bytes, filename)
|
||||
"""
|
||||
# Load session with all related data
|
||||
session = self._load_session(session_id)
|
||||
if not session:
|
||||
raise ValueError(f"Audit session {session_id} not found")
|
||||
|
||||
# Load all sign-offs
|
||||
signoffs = self._load_signoffs(session_id)
|
||||
signoff_map = {s.requirement_id: s for s in signoffs}
|
||||
|
||||
# Load requirements for this session
|
||||
requirements = self._load_requirements(session)
|
||||
|
||||
# Calculate statistics
|
||||
stats = self._calculate_statistics(session, signoffs)
|
||||
|
||||
# Generate PDF
|
||||
buffer = io.BytesIO()
|
||||
doc = SimpleDocTemplate(
|
||||
buffer,
|
||||
pagesize=A4,
|
||||
leftMargin=self.margin,
|
||||
rightMargin=self.margin,
|
||||
topMargin=self.margin,
|
||||
bottomMargin=self.margin,
|
||||
)
|
||||
|
||||
# Build story (content)
|
||||
story = []
|
||||
|
||||
# 1. Cover page
|
||||
story.extend(self._build_cover_page(session, language))
|
||||
story.append(PageBreak())
|
||||
|
||||
# 2. Executive summary
|
||||
story.extend(self._build_executive_summary(session, stats, language))
|
||||
story.append(PageBreak())
|
||||
|
||||
# 3. Statistics overview
|
||||
story.extend(self._build_statistics_section(stats, language))
|
||||
|
||||
# 4. Detailed checklist
|
||||
story.extend(self._build_checklist_section(
|
||||
session, requirements, signoff_map, language
|
||||
))
|
||||
|
||||
# 5. Non-compliant items appendix (if any)
|
||||
non_compliant = [s for s in signoffs if s.result == AuditResultEnum.NON_COMPLIANT]
|
||||
if non_compliant:
|
||||
story.append(PageBreak())
|
||||
story.extend(self._build_non_compliant_appendix(
|
||||
non_compliant, requirements, language
|
||||
))
|
||||
|
||||
# 6. Signature verification (if requested)
|
||||
if include_signatures:
|
||||
signed_items = [s for s in signoffs if s.signature_hash]
|
||||
if signed_items:
|
||||
story.append(PageBreak())
|
||||
story.extend(self._build_signature_section(signed_items, language))
|
||||
|
||||
# Build the PDF
|
||||
doc.build(story)
|
||||
|
||||
# Generate filename
|
||||
date_str = datetime.utcnow().strftime('%Y%m%d')
|
||||
filename = f"audit_report_{session.name.replace(' ', '_')}_{date_str}.pdf"
|
||||
|
||||
return buffer.getvalue(), filename
|
||||
|
||||
def _load_session(self, session_id: str) -> Optional[AuditSessionDB]:
|
||||
"""Load an audit session by ID."""
|
||||
return self.db.query(AuditSessionDB).filter(
|
||||
AuditSessionDB.id == session_id
|
||||
).first()
|
||||
|
||||
def _load_signoffs(self, session_id: str) -> List[AuditSignOffDB]:
|
||||
"""Load all sign-offs for a session."""
|
||||
return (
|
||||
self.db.query(AuditSignOffDB)
|
||||
.filter(AuditSignOffDB.session_id == session_id)
|
||||
.all()
|
||||
)
|
||||
|
||||
def _load_requirements(self, session: AuditSessionDB) -> List[RequirementDB]:
|
||||
"""Load requirements for a session based on filters."""
|
||||
query = self.db.query(RequirementDB).join(RegulationDB)
|
||||
|
||||
if session.regulation_ids:
|
||||
query = query.filter(RegulationDB.code.in_(session.regulation_ids))
|
||||
|
||||
return query.order_by(RegulationDB.code, RequirementDB.article).all()
|
||||
|
||||
def _calculate_statistics(
|
||||
self,
|
||||
session: AuditSessionDB,
|
||||
signoffs: List[AuditSignOffDB],
|
||||
) -> Dict[str, Any]:
|
||||
"""Calculate audit statistics."""
|
||||
total = session.total_items
|
||||
completed = len(signoffs)
|
||||
|
||||
compliant = sum(1 for s in signoffs if s.result == AuditResultEnum.COMPLIANT)
|
||||
compliant_notes = sum(1 for s in signoffs if s.result == AuditResultEnum.COMPLIANT_WITH_NOTES)
|
||||
non_compliant = sum(1 for s in signoffs if s.result == AuditResultEnum.NON_COMPLIANT)
|
||||
not_applicable = sum(1 for s in signoffs if s.result == AuditResultEnum.NOT_APPLICABLE)
|
||||
pending = total - completed
|
||||
|
||||
# Calculate compliance rate (excluding N/A and pending)
|
||||
applicable = compliant + compliant_notes + non_compliant
|
||||
compliance_rate = ((compliant + compliant_notes) / applicable * 100) if applicable > 0 else 0
|
||||
|
||||
return {
|
||||
'total': total,
|
||||
'completed': completed,
|
||||
'pending': pending,
|
||||
'compliant': compliant,
|
||||
'compliant_notes': compliant_notes,
|
||||
'non_compliant': non_compliant,
|
||||
'not_applicable': not_applicable,
|
||||
'completion_percentage': round((completed / total * 100) if total > 0 else 0, 1),
|
||||
'compliance_rate': round(compliance_rate, 1),
|
||||
'traffic_light': self._determine_traffic_light(compliance_rate, pending, total),
|
||||
}
|
||||
|
||||
def _determine_traffic_light(
|
||||
self,
|
||||
compliance_rate: float,
|
||||
pending: int,
|
||||
total: int,
|
||||
) -> str:
|
||||
"""Determine traffic light status."""
|
||||
pending_ratio = pending / total if total > 0 else 0
|
||||
|
||||
if pending_ratio > 0.3:
|
||||
return 'yellow' # Too many pending items
|
||||
elif compliance_rate >= 90:
|
||||
return 'green'
|
||||
elif compliance_rate >= 70:
|
||||
return 'yellow'
|
||||
else:
|
||||
return 'red'
|
||||
|
||||
# =========================================================================
|
||||
# Build Page Sections
|
||||
# =========================================================================
|
||||
|
||||
def _build_cover_page(
|
||||
self,
|
||||
session: AuditSessionDB,
|
||||
language: str,
|
||||
) -> List:
|
||||
"""Build the cover page."""
|
||||
story = []
|
||||
|
||||
# Title
|
||||
title = 'AUDIT-BERICHT' if language == 'de' else 'AUDIT REPORT'
|
||||
story.append(Spacer(1, 30*mm))
|
||||
story.append(Paragraph(title, self.styles['Title']))
|
||||
|
||||
# Session name
|
||||
story.append(Paragraph(session.name, self.styles['Subtitle']))
|
||||
story.append(Spacer(1, 15*mm))
|
||||
|
||||
# Horizontal rule
|
||||
story.append(HRFlowable(
|
||||
width="80%",
|
||||
thickness=1,
|
||||
color=COLORS['accent'],
|
||||
spaceAfter=15*mm,
|
||||
))
|
||||
|
||||
# Metadata table
|
||||
labels = {
|
||||
'de': {
|
||||
'auditor': 'Auditor',
|
||||
'organization': 'Organisation',
|
||||
'status': 'Status',
|
||||
'created': 'Erstellt am',
|
||||
'started': 'Gestartet am',
|
||||
'completed': 'Abgeschlossen am',
|
||||
'regulations': 'Verordnungen',
|
||||
},
|
||||
'en': {
|
||||
'auditor': 'Auditor',
|
||||
'organization': 'Organization',
|
||||
'status': 'Status',
|
||||
'created': 'Created',
|
||||
'started': 'Started',
|
||||
'completed': 'Completed',
|
||||
'regulations': 'Regulations',
|
||||
},
|
||||
}
|
||||
l = labels.get(language, labels['de'])
|
||||
|
||||
status_map = {
|
||||
'draft': 'Entwurf' if language == 'de' else 'Draft',
|
||||
'in_progress': 'In Bearbeitung' if language == 'de' else 'In Progress',
|
||||
'completed': 'Abgeschlossen' if language == 'de' else 'Completed',
|
||||
'archived': 'Archiviert' if language == 'de' else 'Archived',
|
||||
}
|
||||
|
||||
data = [
|
||||
[l['auditor'], session.auditor_name],
|
||||
[l['organization'], session.auditor_organization or '-'],
|
||||
[l['status'], status_map.get(session.status.value, session.status.value)],
|
||||
[l['created'], session.created_at.strftime('%d.%m.%Y %H:%M') if session.created_at else '-'],
|
||||
[l['started'], session.started_at.strftime('%d.%m.%Y %H:%M') if session.started_at else '-'],
|
||||
[l['completed'], session.completed_at.strftime('%d.%m.%Y %H:%M') if session.completed_at else '-'],
|
||||
[l['regulations'], ', '.join(session.regulation_ids) if session.regulation_ids else 'Alle'],
|
||||
]
|
||||
|
||||
table = Table(data, colWidths=[50*mm, 100*mm])
|
||||
table.setStyle(TableStyle([
|
||||
('FONTNAME', (0, 0), (0, -1), 'Helvetica-Bold'),
|
||||
('FONTNAME', (1, 0), (1, -1), 'Helvetica'),
|
||||
('FONTSIZE', (0, 0), (-1, -1), 11),
|
||||
('TEXTCOLOR', (0, 0), (0, -1), COLORS['secondary']),
|
||||
('TEXTCOLOR', (1, 0), (1, -1), COLORS['black']),
|
||||
('BOTTOMPADDING', (0, 0), (-1, -1), 8),
|
||||
('TOPPADDING', (0, 0), (-1, -1), 8),
|
||||
('ALIGN', (0, 0), (0, -1), 'RIGHT'),
|
||||
('ALIGN', (1, 0), (1, -1), 'LEFT'),
|
||||
('VALIGN', (0, 0), (-1, -1), 'MIDDLE'),
|
||||
]))
|
||||
|
||||
story.append(table)
|
||||
story.append(Spacer(1, 20*mm))
|
||||
|
||||
# Description if available
|
||||
if session.description:
|
||||
desc_label = 'Beschreibung' if language == 'de' else 'Description'
|
||||
story.append(Paragraph(f"<b>{desc_label}:</b>", self.styles['Normal']))
|
||||
story.append(Paragraph(session.description, self.styles['Normal']))
|
||||
|
||||
# Generation timestamp
|
||||
story.append(Spacer(1, 30*mm))
|
||||
gen_label = 'Generiert am' if language == 'de' else 'Generated on'
|
||||
story.append(Paragraph(
|
||||
f"{gen_label}: {datetime.utcnow().strftime('%d.%m.%Y %H:%M')} UTC",
|
||||
self.styles['Footer']
|
||||
))
|
||||
|
||||
return story
|
||||
|
||||
def _build_executive_summary(
|
||||
self,
|
||||
session: AuditSessionDB,
|
||||
stats: Dict[str, Any],
|
||||
language: str,
|
||||
) -> List:
|
||||
"""Build the executive summary section."""
|
||||
story = []
|
||||
|
||||
title = 'ZUSAMMENFASSUNG' if language == 'de' else 'EXECUTIVE SUMMARY'
|
||||
story.append(Paragraph(title, self.styles['Heading1']))
|
||||
|
||||
# Traffic light status
|
||||
traffic_light = stats['traffic_light']
|
||||
tl_colors = {
|
||||
'green': COLORS['success'],
|
||||
'yellow': COLORS['warning'],
|
||||
'red': COLORS['danger'],
|
||||
}
|
||||
tl_labels = {
|
||||
'de': {'green': 'GUT', 'yellow': 'AUFMERKSAMKEIT', 'red': 'KRITISCH'},
|
||||
'en': {'green': 'GOOD', 'yellow': 'ATTENTION', 'red': 'CRITICAL'},
|
||||
}
|
||||
|
||||
# Create traffic light indicator
|
||||
tl_table = Table(
|
||||
[[tl_labels[language][traffic_light]]],
|
||||
colWidths=[60*mm],
|
||||
rowHeights=[15*mm],
|
||||
)
|
||||
tl_table.setStyle(TableStyle([
|
||||
('BACKGROUND', (0, 0), (0, 0), tl_colors[traffic_light]),
|
||||
('TEXTCOLOR', (0, 0), (0, 0), COLORS['white']),
|
||||
('FONTNAME', (0, 0), (0, 0), 'Helvetica-Bold'),
|
||||
('FONTSIZE', (0, 0), (0, 0), 16),
|
||||
('ALIGN', (0, 0), (0, 0), 'CENTER'),
|
||||
('VALIGN', (0, 0), (0, 0), 'MIDDLE'),
|
||||
('ROUNDEDCORNERS', [3, 3, 3, 3]),
|
||||
]))
|
||||
|
||||
story.append(tl_table)
|
||||
story.append(Spacer(1, 10*mm))
|
||||
|
||||
# Key metrics
|
||||
labels = {
|
||||
'de': {
|
||||
'completion': 'Abschlussrate',
|
||||
'compliance': 'Konformitaetsrate',
|
||||
'total': 'Gesamtanforderungen',
|
||||
'non_compliant': 'Nicht konform',
|
||||
'pending': 'Ausstehend',
|
||||
},
|
||||
'en': {
|
||||
'completion': 'Completion Rate',
|
||||
'compliance': 'Compliance Rate',
|
||||
'total': 'Total Requirements',
|
||||
'non_compliant': 'Non-Compliant',
|
||||
'pending': 'Pending',
|
||||
},
|
||||
}
|
||||
l = labels.get(language, labels['de'])
|
||||
|
||||
metrics_data = [
|
||||
[l['completion'], f"{stats['completion_percentage']}%"],
|
||||
[l['compliance'], f"{stats['compliance_rate']}%"],
|
||||
[l['total'], str(stats['total'])],
|
||||
[l['non_compliant'], str(stats['non_compliant'])],
|
||||
[l['pending'], str(stats['pending'])],
|
||||
]
|
||||
|
||||
metrics_table = Table(metrics_data, colWidths=[60*mm, 40*mm])
|
||||
metrics_table.setStyle(TableStyle([
|
||||
('FONTNAME', (0, 0), (0, -1), 'Helvetica-Bold'),
|
||||
('FONTNAME', (1, 0), (1, -1), 'Helvetica'),
|
||||
('FONTSIZE', (0, 0), (-1, -1), 12),
|
||||
('TEXTCOLOR', (0, 0), (0, -1), COLORS['secondary']),
|
||||
('TEXTCOLOR', (1, 0), (1, -1), COLORS['black']),
|
||||
('BOTTOMPADDING', (0, 0), (-1, -1), 6),
|
||||
('TOPPADDING', (0, 0), (-1, -1), 6),
|
||||
('ALIGN', (0, 0), (0, -1), 'LEFT'),
|
||||
('ALIGN', (1, 0), (1, -1), 'RIGHT'),
|
||||
('LINEABOVE', (0, 0), (-1, 0), 1, COLORS['light']),
|
||||
('LINEBELOW', (0, -1), (-1, -1), 1, COLORS['light']),
|
||||
]))
|
||||
|
||||
story.append(metrics_table)
|
||||
story.append(Spacer(1, 10*mm))
|
||||
|
||||
# Key findings
|
||||
findings_title = 'Wichtige Erkenntnisse' if language == 'de' else 'Key Findings'
|
||||
story.append(Paragraph(f"<b>{findings_title}:</b>", self.styles['Heading3']))
|
||||
|
||||
findings = self._generate_findings(stats, language)
|
||||
for finding in findings:
|
||||
story.append(Paragraph(f"• {finding}", self.styles['Normal']))
|
||||
|
||||
return story
|
||||
|
||||
def _generate_findings(self, stats: Dict[str, Any], language: str) -> List[str]:
|
||||
"""Generate key findings based on statistics."""
|
||||
findings = []
|
||||
|
||||
if language == 'de':
|
||||
if stats['non_compliant'] > 0:
|
||||
findings.append(
|
||||
f"{stats['non_compliant']} Anforderungen sind nicht konform und "
|
||||
f"erfordern Massnahmen."
|
||||
)
|
||||
if stats['pending'] > 0:
|
||||
findings.append(
|
||||
f"{stats['pending']} Anforderungen wurden noch nicht geprueft."
|
||||
)
|
||||
if stats['compliance_rate'] >= 90:
|
||||
findings.append(
|
||||
"Hohe Konformitaetsrate erreicht. Weiter so!"
|
||||
)
|
||||
elif stats['compliance_rate'] < 70:
|
||||
findings.append(
|
||||
"Konformitaetsrate unter 70%. Priorisierte Massnahmen erforderlich."
|
||||
)
|
||||
if stats['compliant_notes'] > 0:
|
||||
findings.append(
|
||||
f"{stats['compliant_notes']} Anforderungen sind konform mit Anmerkungen. "
|
||||
f"Verbesserungspotenzial identifiziert."
|
||||
)
|
||||
if not findings:
|
||||
findings.append("Audit vollstaendig abgeschlossen ohne kritische Befunde.")
|
||||
else:
|
||||
if stats['non_compliant'] > 0:
|
||||
findings.append(
|
||||
f"{stats['non_compliant']} requirements are non-compliant and "
|
||||
f"require action."
|
||||
)
|
||||
if stats['pending'] > 0:
|
||||
findings.append(
|
||||
f"{stats['pending']} requirements have not been reviewed yet."
|
||||
)
|
||||
if stats['compliance_rate'] >= 90:
|
||||
findings.append(
|
||||
"High compliance rate achieved. Keep up the good work!"
|
||||
)
|
||||
elif stats['compliance_rate'] < 70:
|
||||
findings.append(
|
||||
"Compliance rate below 70%. Prioritized actions required."
|
||||
)
|
||||
if stats['compliant_notes'] > 0:
|
||||
findings.append(
|
||||
f"{stats['compliant_notes']} requirements are compliant with notes. "
|
||||
f"Improvement potential identified."
|
||||
)
|
||||
if not findings:
|
||||
findings.append("Audit completed without critical findings.")
|
||||
|
||||
return findings
|
||||
|
||||
def _build_statistics_section(
|
||||
self,
|
||||
stats: Dict[str, Any],
|
||||
language: str,
|
||||
) -> List:
|
||||
"""Build the statistics overview section with pie chart."""
|
||||
story = []
|
||||
|
||||
title = 'STATISTIK-UEBERSICHT' if language == 'de' else 'STATISTICS OVERVIEW'
|
||||
story.append(Paragraph(title, self.styles['Heading1']))
|
||||
|
||||
# Create pie chart
|
||||
drawing = Drawing(200, 200)
|
||||
pie = Pie()
|
||||
pie.x = 50
|
||||
pie.y = 25
|
||||
pie.width = 100
|
||||
pie.height = 100
|
||||
|
||||
# Data for pie chart
|
||||
data = [
|
||||
stats['compliant'],
|
||||
stats['compliant_notes'],
|
||||
stats['non_compliant'],
|
||||
stats['not_applicable'],
|
||||
stats['pending'],
|
||||
]
|
||||
|
||||
# Only include non-zero values
|
||||
labels_de = ['Konform', 'Konform (Anm.)', 'Nicht konform', 'N/A', 'Ausstehend']
|
||||
labels_en = ['Compliant', 'Compliant (Notes)', 'Non-Compliant', 'N/A', 'Pending']
|
||||
labels = labels_de if language == 'de' else labels_en
|
||||
|
||||
pie_colors = [
|
||||
COLORS['success'],
|
||||
colors.HexColor('#68d391'),
|
||||
COLORS['danger'],
|
||||
COLORS['muted'],
|
||||
COLORS['warning'],
|
||||
]
|
||||
|
||||
# Filter out zero values
|
||||
filtered_data = []
|
||||
filtered_labels = []
|
||||
filtered_colors = []
|
||||
for i, val in enumerate(data):
|
||||
if val > 0:
|
||||
filtered_data.append(val)
|
||||
filtered_labels.append(labels[i])
|
||||
filtered_colors.append(pie_colors[i])
|
||||
|
||||
if filtered_data:
|
||||
pie.data = filtered_data
|
||||
pie.labels = filtered_labels
|
||||
pie.slices.strokeWidth = 0.5
|
||||
|
||||
for i, col in enumerate(filtered_colors):
|
||||
pie.slices[i].fillColor = col
|
||||
|
||||
drawing.add(pie)
|
||||
story.append(drawing)
|
||||
else:
|
||||
no_data = 'Keine Daten verfuegbar' if language == 'de' else 'No data available'
|
||||
story.append(Paragraph(no_data, self.styles['Normal']))
|
||||
|
||||
story.append(Spacer(1, 10*mm))
|
||||
|
||||
# Legend table
|
||||
legend_data = []
|
||||
for i, label in enumerate(labels):
|
||||
if data[i] > 0:
|
||||
count = data[i]
|
||||
pct = round(count / stats['total'] * 100, 1) if stats['total'] > 0 else 0
|
||||
legend_data.append([label, str(count), f"{pct}%"])
|
||||
|
||||
if legend_data:
|
||||
header = ['Status', 'Anzahl', '%'] if language == 'de' else ['Status', 'Count', '%']
|
||||
legend_table = Table([header] + legend_data, colWidths=[50*mm, 25*mm, 25*mm])
|
||||
legend_table.setStyle(TableStyle([
|
||||
('FONTNAME', (0, 0), (-1, 0), 'Helvetica-Bold'),
|
||||
('FONTNAME', (0, 1), (-1, -1), 'Helvetica'),
|
||||
('FONTSIZE', (0, 0), (-1, -1), 10),
|
||||
('BACKGROUND', (0, 0), (-1, 0), COLORS['light']),
|
||||
('ALIGN', (1, 0), (-1, -1), 'RIGHT'),
|
||||
('BOTTOMPADDING', (0, 0), (-1, -1), 5),
|
||||
('TOPPADDING', (0, 0), (-1, -1), 5),
|
||||
('GRID', (0, 0), (-1, -1), 0.5, COLORS['muted']),
|
||||
]))
|
||||
story.append(legend_table)
|
||||
|
||||
return story
|
||||
|
||||
def _build_checklist_section(
|
||||
self,
|
||||
session: AuditSessionDB,
|
||||
requirements: List[RequirementDB],
|
||||
signoff_map: Dict[str, AuditSignOffDB],
|
||||
language: str,
|
||||
) -> List:
|
||||
"""Build the detailed checklist section."""
|
||||
story = []
|
||||
|
||||
story.append(PageBreak())
|
||||
title = 'PRUEFUNGSCHECKLISTE' if language == 'de' else 'AUDIT CHECKLIST'
|
||||
story.append(Paragraph(title, self.styles['Heading1']))
|
||||
|
||||
# Group by regulation
|
||||
by_regulation = {}
|
||||
for req in requirements:
|
||||
reg_code = req.regulation.code if req.regulation else 'OTHER'
|
||||
if reg_code not in by_regulation:
|
||||
by_regulation[reg_code] = []
|
||||
by_regulation[reg_code].append(req)
|
||||
|
||||
result_labels = {
|
||||
'de': {
|
||||
'compliant': 'Konform',
|
||||
'compliant_notes': 'Konform (Anm.)',
|
||||
'non_compliant': 'Nicht konform',
|
||||
'not_applicable': 'N/A',
|
||||
'pending': 'Ausstehend',
|
||||
},
|
||||
'en': {
|
||||
'compliant': 'Compliant',
|
||||
'compliant_notes': 'Compliant (Notes)',
|
||||
'non_compliant': 'Non-Compliant',
|
||||
'not_applicable': 'N/A',
|
||||
'pending': 'Pending',
|
||||
},
|
||||
}
|
||||
labels = result_labels.get(language, result_labels['de'])
|
||||
|
||||
for reg_code, reqs in sorted(by_regulation.items()):
|
||||
story.append(Paragraph(reg_code, self.styles['Heading2']))
|
||||
|
||||
# Build table data
|
||||
header = ['Art.', 'Titel', 'Ergebnis', 'Signiert'] if language == 'de' else \
|
||||
['Art.', 'Title', 'Result', 'Signed']
|
||||
table_data = [header]
|
||||
|
||||
for req in reqs:
|
||||
signoff = signoff_map.get(req.id)
|
||||
result = signoff.result.value if signoff else 'pending'
|
||||
result_label = labels.get(result, result)
|
||||
signed = 'Ja' if (signoff and signoff.signature_hash) else '-'
|
||||
if language == 'en':
|
||||
signed = 'Yes' if (signoff and signoff.signature_hash) else '-'
|
||||
|
||||
# Truncate title if too long
|
||||
title_text = req.title[:50] + '...' if len(req.title) > 50 else req.title
|
||||
|
||||
table_data.append([
|
||||
req.article or '-',
|
||||
title_text,
|
||||
result_label,
|
||||
signed,
|
||||
])
|
||||
|
||||
table = Table(table_data, colWidths=[20*mm, 80*mm, 35*mm, 20*mm])
|
||||
|
||||
# Style rows based on result
|
||||
style_commands = [
|
||||
('FONTNAME', (0, 0), (-1, 0), 'Helvetica-Bold'),
|
||||
('FONTNAME', (0, 1), (-1, -1), 'Helvetica'),
|
||||
('FONTSIZE', (0, 0), (-1, -1), 9),
|
||||
('BACKGROUND', (0, 0), (-1, 0), COLORS['light']),
|
||||
('ALIGN', (0, 0), (-1, -1), 'LEFT'),
|
||||
('ALIGN', (2, 0), (3, -1), 'CENTER'),
|
||||
('BOTTOMPADDING', (0, 0), (-1, -1), 4),
|
||||
('TOPPADDING', (0, 0), (-1, -1), 4),
|
||||
('GRID', (0, 0), (-1, -1), 0.5, COLORS['muted']),
|
||||
('VALIGN', (0, 0), (-1, -1), 'TOP'),
|
||||
]
|
||||
|
||||
# Color code results
|
||||
for i, req in enumerate(reqs, start=1):
|
||||
signoff = signoff_map.get(req.id)
|
||||
if signoff:
|
||||
result = signoff.result.value
|
||||
if result == 'compliant':
|
||||
style_commands.append(('TEXTCOLOR', (2, i), (2, i), COLORS['success']))
|
||||
elif result == 'compliant_notes':
|
||||
style_commands.append(('TEXTCOLOR', (2, i), (2, i), colors.HexColor('#2f855a')))
|
||||
elif result == 'non_compliant':
|
||||
style_commands.append(('TEXTCOLOR', (2, i), (2, i), COLORS['danger']))
|
||||
else:
|
||||
style_commands.append(('TEXTCOLOR', (2, i), (2, i), COLORS['warning']))
|
||||
|
||||
table.setStyle(TableStyle(style_commands))
|
||||
story.append(table)
|
||||
story.append(Spacer(1, 5*mm))
|
||||
|
||||
return story
|
||||
|
||||
def _build_non_compliant_appendix(
|
||||
self,
|
||||
non_compliant: List[AuditSignOffDB],
|
||||
requirements: List[RequirementDB],
|
||||
language: str,
|
||||
) -> List:
|
||||
"""Build appendix with non-compliant items detail."""
|
||||
story = []
|
||||
|
||||
title = 'ANHANG: NICHT KONFORME ANFORDERUNGEN' if language == 'de' else \
|
||||
'APPENDIX: NON-COMPLIANT REQUIREMENTS'
|
||||
story.append(Paragraph(title, self.styles['Heading1']))
|
||||
|
||||
req_map = {r.id: r for r in requirements}
|
||||
|
||||
for i, signoff in enumerate(non_compliant, start=1):
|
||||
req = req_map.get(signoff.requirement_id)
|
||||
if not req:
|
||||
continue
|
||||
|
||||
# Requirement header
|
||||
story.append(Paragraph(
|
||||
f"<b>{i}. {req.regulation.code if req.regulation else ''} {req.article}</b>",
|
||||
self.styles['Heading3']
|
||||
))
|
||||
|
||||
story.append(Paragraph(f"<b>{req.title}</b>", self.styles['Normal']))
|
||||
|
||||
if req.description:
|
||||
desc = req.description[:500] + '...' if len(req.description) > 500 else req.description
|
||||
story.append(Paragraph(desc, self.styles['Small']))
|
||||
|
||||
# Notes from auditor
|
||||
if signoff.notes:
|
||||
notes_label = 'Auditor-Anmerkungen' if language == 'de' else 'Auditor Notes'
|
||||
story.append(Paragraph(f"<b>{notes_label}:</b>", self.styles['Normal']))
|
||||
story.append(Paragraph(signoff.notes, self.styles['Normal']))
|
||||
|
||||
story.append(Spacer(1, 5*mm))
|
||||
|
||||
return story
|
||||
|
||||
def _build_signature_section(
|
||||
self,
|
||||
signed_items: List[AuditSignOffDB],
|
||||
language: str,
|
||||
) -> List:
|
||||
"""Build section with digital signature verification."""
|
||||
story = []
|
||||
|
||||
title = 'DIGITALE SIGNATUREN' if language == 'de' else 'DIGITAL SIGNATURES'
|
||||
story.append(Paragraph(title, self.styles['Heading1']))
|
||||
|
||||
explanation = (
|
||||
'Die folgenden Pruefpunkte wurden digital signiert. '
|
||||
'Die SHA-256 Hashes dienen als unveraenderlicher Nachweis des Pruefergebnisses.'
|
||||
) if language == 'de' else (
|
||||
'The following audit items have been digitally signed. '
|
||||
'The SHA-256 hashes serve as immutable proof of the audit result.'
|
||||
)
|
||||
story.append(Paragraph(explanation, self.styles['Normal']))
|
||||
story.append(Spacer(1, 5*mm))
|
||||
|
||||
header = ['Anforderung', 'Signiert von', 'Datum', 'SHA-256 (gekuerzt)'] if language == 'de' else \
|
||||
['Requirement', 'Signed by', 'Date', 'SHA-256 (truncated)']
|
||||
|
||||
table_data = [header]
|
||||
for item in signed_items[:50]: # Limit to 50 entries
|
||||
table_data.append([
|
||||
item.requirement_id[:8] + '...',
|
||||
item.signed_by or '-',
|
||||
item.signed_at.strftime('%d.%m.%Y') if item.signed_at else '-',
|
||||
item.signature_hash[:16] + '...' if item.signature_hash else '-',
|
||||
])
|
||||
|
||||
table = Table(table_data, colWidths=[35*mm, 40*mm, 30*mm, 50*mm])
|
||||
table.setStyle(TableStyle([
|
||||
('FONTNAME', (0, 0), (-1, 0), 'Helvetica-Bold'),
|
||||
('FONTNAME', (0, 1), (-1, -1), 'Courier'),
|
||||
('FONTSIZE', (0, 0), (-1, -1), 8),
|
||||
('BACKGROUND', (0, 0), (-1, 0), COLORS['light']),
|
||||
('ALIGN', (0, 0), (-1, -1), 'LEFT'),
|
||||
('BOTTOMPADDING', (0, 0), (-1, -1), 3),
|
||||
('TOPPADDING', (0, 0), (-1, -1), 3),
|
||||
('GRID', (0, 0), (-1, -1), 0.5, COLORS['muted']),
|
||||
]))
|
||||
|
||||
story.append(table)
|
||||
|
||||
return story
|
||||
383
backend/compliance/services/auto_risk_updater.py
Normal file
383
backend/compliance/services/auto_risk_updater.py
Normal file
@@ -0,0 +1,383 @@
|
||||
"""
|
||||
Automatic Risk Update Service for Compliance Framework.
|
||||
|
||||
This service processes CI/CD security scan results and automatically:
|
||||
1. Updates Control status based on scan findings
|
||||
2. Adjusts Risk levels when critical CVEs are found
|
||||
3. Creates Evidence records from scan reports
|
||||
4. Generates alerts for significant findings
|
||||
|
||||
Sprint 6: CI/CD Evidence Collection (2026-01-18)
|
||||
"""
|
||||
|
||||
import logging
|
||||
from datetime import datetime
|
||||
from typing import Dict, List, Optional, Any
|
||||
from dataclasses import dataclass
|
||||
from enum import Enum
|
||||
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from ..db.models import (
|
||||
ControlDB, ControlStatusEnum,
|
||||
EvidenceDB, EvidenceStatusEnum,
|
||||
RiskDB, RiskLevelEnum,
|
||||
)
|
||||
from ..db.repository import ControlRepository, EvidenceRepository, RiskRepository
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class ScanType(str, Enum):
|
||||
"""Types of CI/CD security scans."""
|
||||
SAST = "sast" # Static Application Security Testing
|
||||
DEPENDENCY = "dependency" # Dependency/CVE scanning
|
||||
SECRET = "secret" # Secret detection
|
||||
CONTAINER = "container" # Container image scanning
|
||||
SBOM = "sbom" # Software Bill of Materials
|
||||
|
||||
|
||||
class FindingSeverity(str, Enum):
|
||||
"""Severity levels for security findings."""
|
||||
CRITICAL = "critical"
|
||||
HIGH = "high"
|
||||
MEDIUM = "medium"
|
||||
LOW = "low"
|
||||
INFO = "info"
|
||||
|
||||
|
||||
@dataclass
|
||||
class ScanResult:
|
||||
"""Represents a CI/CD scan result."""
|
||||
scan_type: ScanType
|
||||
tool: str
|
||||
timestamp: datetime
|
||||
commit_sha: str
|
||||
branch: str
|
||||
control_id: str # Mapped Control ID (e.g., SDLC-001)
|
||||
findings: Dict[str, int] # {"critical": 0, "high": 2, ...}
|
||||
raw_report: Optional[Dict] = None
|
||||
ci_job_id: Optional[str] = None
|
||||
|
||||
|
||||
@dataclass
|
||||
class RiskUpdateResult:
|
||||
"""Result of an automatic risk update."""
|
||||
control_id: str
|
||||
control_updated: bool
|
||||
old_status: Optional[str]
|
||||
new_status: Optional[str]
|
||||
evidence_created: bool
|
||||
evidence_id: Optional[str]
|
||||
risks_affected: List[str]
|
||||
alerts_generated: List[str]
|
||||
message: str
|
||||
|
||||
|
||||
# Mapping from Control IDs to scan types
|
||||
CONTROL_SCAN_MAPPING = {
|
||||
"SDLC-001": ScanType.SAST, # SAST Scanning
|
||||
"SDLC-002": ScanType.DEPENDENCY, # Dependency Scanning
|
||||
"SDLC-003": ScanType.SECRET, # Secret Detection
|
||||
"SDLC-006": ScanType.CONTAINER, # Container Scanning
|
||||
"CRA-001": ScanType.SBOM, # SBOM Generation
|
||||
}
|
||||
|
||||
|
||||
class AutoRiskUpdater:
|
||||
"""
|
||||
Automatically updates Controls and Risks based on CI/CD scan results.
|
||||
|
||||
Flow:
|
||||
1. Receive scan result from CI/CD pipeline
|
||||
2. Determine Control status based on findings
|
||||
3. Create Evidence record
|
||||
4. Update linked Risks if necessary
|
||||
5. Generate alerts for critical findings
|
||||
"""
|
||||
|
||||
def __init__(self, db: Session):
|
||||
self.db = db
|
||||
self.control_repo = ControlRepository(db)
|
||||
self.evidence_repo = EvidenceRepository(db)
|
||||
self.risk_repo = RiskRepository(db)
|
||||
|
||||
def process_scan_result(self, scan_result: ScanResult) -> RiskUpdateResult:
|
||||
"""
|
||||
Process a CI/CD scan result and update Compliance status.
|
||||
|
||||
Args:
|
||||
scan_result: The scan result from CI/CD pipeline
|
||||
|
||||
Returns:
|
||||
RiskUpdateResult with details of all updates made
|
||||
"""
|
||||
logger.info(f"Processing {scan_result.scan_type.value} scan for control {scan_result.control_id}")
|
||||
|
||||
# Find the Control
|
||||
control = self.control_repo.get_by_control_id(scan_result.control_id)
|
||||
if not control:
|
||||
logger.warning(f"Control {scan_result.control_id} not found")
|
||||
return RiskUpdateResult(
|
||||
control_id=scan_result.control_id,
|
||||
control_updated=False,
|
||||
old_status=None,
|
||||
new_status=None,
|
||||
evidence_created=False,
|
||||
evidence_id=None,
|
||||
risks_affected=[],
|
||||
alerts_generated=[],
|
||||
message=f"Control {scan_result.control_id} not found"
|
||||
)
|
||||
|
||||
old_status = control.status.value if control.status else "unknown"
|
||||
|
||||
# Determine new Control status based on findings
|
||||
new_status = self._determine_control_status(scan_result.findings)
|
||||
|
||||
# Update Control status
|
||||
control_updated = False
|
||||
if new_status != old_status:
|
||||
control.status = ControlStatusEnum(new_status)
|
||||
control.status_notes = self._generate_status_notes(scan_result)
|
||||
control.updated_at = datetime.utcnow()
|
||||
control_updated = True
|
||||
logger.info(f"Control {scan_result.control_id} status changed: {old_status} -> {new_status}")
|
||||
|
||||
# Create Evidence record
|
||||
evidence = self._create_evidence(control, scan_result)
|
||||
|
||||
# Update linked Risks
|
||||
risks_affected = self._update_linked_risks(control, new_status, scan_result.findings)
|
||||
|
||||
# Generate alerts for critical findings
|
||||
alerts = self._generate_alerts(scan_result, new_status)
|
||||
|
||||
# Commit all changes
|
||||
self.db.commit()
|
||||
|
||||
return RiskUpdateResult(
|
||||
control_id=scan_result.control_id,
|
||||
control_updated=control_updated,
|
||||
old_status=old_status,
|
||||
new_status=new_status,
|
||||
evidence_created=True,
|
||||
evidence_id=evidence.id,
|
||||
risks_affected=risks_affected,
|
||||
alerts_generated=alerts,
|
||||
message=f"Processed {scan_result.scan_type.value} scan successfully"
|
||||
)
|
||||
|
||||
def _determine_control_status(self, findings: Dict[str, int]) -> str:
|
||||
"""
|
||||
Determine Control status based on security findings.
|
||||
|
||||
Rules:
|
||||
- Any CRITICAL findings -> fail
|
||||
- >5 HIGH findings -> fail
|
||||
- 1-5 HIGH findings -> partial
|
||||
- Only MEDIUM/LOW findings -> pass (with notes)
|
||||
- No findings -> pass
|
||||
"""
|
||||
critical = findings.get("critical", 0)
|
||||
high = findings.get("high", 0)
|
||||
medium = findings.get("medium", 0)
|
||||
|
||||
if critical > 0:
|
||||
return ControlStatusEnum.FAIL.value
|
||||
elif high > 5:
|
||||
return ControlStatusEnum.FAIL.value
|
||||
elif high > 0:
|
||||
return ControlStatusEnum.PARTIAL.value
|
||||
elif medium > 10:
|
||||
return ControlStatusEnum.PARTIAL.value
|
||||
else:
|
||||
return ControlStatusEnum.PASS.value
|
||||
|
||||
def _generate_status_notes(self, scan_result: ScanResult) -> str:
|
||||
"""Generate human-readable status notes from scan result."""
|
||||
findings = scan_result.findings
|
||||
parts = []
|
||||
|
||||
if findings.get("critical", 0) > 0:
|
||||
parts.append(f"{findings['critical']} CRITICAL")
|
||||
if findings.get("high", 0) > 0:
|
||||
parts.append(f"{findings['high']} HIGH")
|
||||
if findings.get("medium", 0) > 0:
|
||||
parts.append(f"{findings['medium']} MEDIUM")
|
||||
|
||||
if parts:
|
||||
findings_str = ", ".join(parts)
|
||||
return f"Auto-updated from {scan_result.tool} scan ({scan_result.timestamp.strftime('%Y-%m-%d %H:%M')}): {findings_str} findings"
|
||||
else:
|
||||
return f"Auto-updated from {scan_result.tool} scan ({scan_result.timestamp.strftime('%Y-%m-%d %H:%M')}): No significant findings"
|
||||
|
||||
def _create_evidence(self, control: ControlDB, scan_result: ScanResult) -> EvidenceDB:
|
||||
"""Create an Evidence record from the scan result."""
|
||||
from uuid import uuid4
|
||||
|
||||
evidence = EvidenceDB(
|
||||
id=str(uuid4()),
|
||||
control_id=control.id,
|
||||
evidence_type=f"{scan_result.scan_type.value}_report",
|
||||
title=f"{scan_result.tool} Scan - {scan_result.timestamp.strftime('%Y-%m-%d')}",
|
||||
description=self._generate_status_notes(scan_result),
|
||||
source="ci_pipeline",
|
||||
ci_job_id=scan_result.ci_job_id,
|
||||
status=EvidenceStatusEnum.VALID,
|
||||
valid_from=datetime.utcnow(),
|
||||
collected_at=scan_result.timestamp,
|
||||
)
|
||||
|
||||
self.db.add(evidence)
|
||||
logger.info(f"Created evidence {evidence.id} for control {control.control_id}")
|
||||
|
||||
return evidence
|
||||
|
||||
def _update_linked_risks(
|
||||
self,
|
||||
control: ControlDB,
|
||||
new_status: str,
|
||||
findings: Dict[str, int]
|
||||
) -> List[str]:
|
||||
"""
|
||||
Update Risks that are mitigated by this Control.
|
||||
|
||||
When a Control fails:
|
||||
- Increase residual risk of linked Risks
|
||||
- Update risk status to "open" if was "mitigated"
|
||||
|
||||
When a Control passes:
|
||||
- Decrease residual risk if appropriate
|
||||
"""
|
||||
affected_risks = []
|
||||
|
||||
# Find all Risks that list this Control as a mitigating control
|
||||
all_risks = self.risk_repo.get_all()
|
||||
|
||||
for risk in all_risks:
|
||||
if not risk.mitigating_controls:
|
||||
continue
|
||||
|
||||
mitigating_ids = risk.mitigating_controls
|
||||
if control.control_id not in mitigating_ids:
|
||||
continue
|
||||
|
||||
# This Risk is linked to the affected Control
|
||||
risk_updated = False
|
||||
|
||||
if new_status == ControlStatusEnum.FAIL.value:
|
||||
# Control failed - increase risk
|
||||
if risk.status == "mitigated":
|
||||
risk.status = "open"
|
||||
risk_updated = True
|
||||
|
||||
# Increase residual likelihood if critical findings
|
||||
if findings.get("critical", 0) > 0:
|
||||
old_likelihood = risk.residual_likelihood or risk.likelihood
|
||||
risk.residual_likelihood = min(5, old_likelihood + 1)
|
||||
risk.residual_risk = RiskDB.calculate_risk_level(
|
||||
risk.residual_likelihood,
|
||||
risk.residual_impact or risk.impact
|
||||
)
|
||||
risk_updated = True
|
||||
|
||||
elif new_status == ControlStatusEnum.PASS.value:
|
||||
# Control passed - potentially reduce risk
|
||||
if risk.status == "open":
|
||||
# Check if all mitigating controls are passing
|
||||
all_passing = True
|
||||
for ctrl_id in mitigating_ids:
|
||||
other_ctrl = self.control_repo.get_by_control_id(ctrl_id)
|
||||
if other_ctrl and other_ctrl.status != ControlStatusEnum.PASS:
|
||||
all_passing = False
|
||||
break
|
||||
|
||||
if all_passing:
|
||||
risk.status = "mitigated"
|
||||
risk_updated = True
|
||||
|
||||
if risk_updated:
|
||||
risk.last_assessed_at = datetime.utcnow()
|
||||
risk.updated_at = datetime.utcnow()
|
||||
affected_risks.append(risk.risk_id)
|
||||
logger.info(f"Updated risk {risk.risk_id} due to control {control.control_id} status change")
|
||||
|
||||
return affected_risks
|
||||
|
||||
def _generate_alerts(self, scan_result: ScanResult, new_status: str) -> List[str]:
|
||||
"""
|
||||
Generate alerts for significant findings.
|
||||
|
||||
Alert conditions:
|
||||
- Any CRITICAL findings
|
||||
- Control status changed to FAIL
|
||||
- >10 HIGH findings in one scan
|
||||
"""
|
||||
alerts = []
|
||||
findings = scan_result.findings
|
||||
|
||||
if findings.get("critical", 0) > 0:
|
||||
alert_msg = f"CRITICAL: {findings['critical']} critical vulnerabilities found in {scan_result.tool} scan"
|
||||
alerts.append(alert_msg)
|
||||
logger.warning(alert_msg)
|
||||
|
||||
if new_status == ControlStatusEnum.FAIL.value:
|
||||
alert_msg = f"Control {scan_result.control_id} status changed to FAIL"
|
||||
alerts.append(alert_msg)
|
||||
logger.warning(alert_msg)
|
||||
|
||||
if findings.get("high", 0) > 10:
|
||||
alert_msg = f"HIGH: {findings['high']} high-severity findings in {scan_result.tool} scan"
|
||||
alerts.append(alert_msg)
|
||||
logger.warning(alert_msg)
|
||||
|
||||
return alerts
|
||||
|
||||
def process_evidence_collect_request(
|
||||
self,
|
||||
tool: str,
|
||||
control_id: str,
|
||||
evidence_type: str,
|
||||
timestamp: str,
|
||||
commit_sha: str,
|
||||
ci_job_id: Optional[str] = None,
|
||||
findings: Optional[Dict[str, int]] = None,
|
||||
**kwargs
|
||||
) -> RiskUpdateResult:
|
||||
"""
|
||||
Process an evidence collection request from CI/CD.
|
||||
|
||||
This is the main entry point for the /evidence/collect API endpoint.
|
||||
"""
|
||||
# Parse timestamp
|
||||
try:
|
||||
ts = datetime.fromisoformat(timestamp.replace('Z', '+00:00'))
|
||||
except (ValueError, AttributeError):
|
||||
ts = datetime.utcnow()
|
||||
|
||||
# Determine scan type from evidence_type
|
||||
scan_type = ScanType.SAST # Default
|
||||
for ctrl_id, stype in CONTROL_SCAN_MAPPING.items():
|
||||
if ctrl_id == control_id:
|
||||
scan_type = stype
|
||||
break
|
||||
|
||||
# Create ScanResult
|
||||
scan_result = ScanResult(
|
||||
scan_type=scan_type,
|
||||
tool=tool,
|
||||
timestamp=ts,
|
||||
commit_sha=commit_sha,
|
||||
branch=kwargs.get("branch", "unknown"),
|
||||
control_id=control_id,
|
||||
findings=findings or {"critical": 0, "high": 0, "medium": 0, "low": 0},
|
||||
ci_job_id=ci_job_id,
|
||||
)
|
||||
|
||||
return self.process_scan_result(scan_result)
|
||||
|
||||
|
||||
def create_auto_risk_updater(db: Session) -> AutoRiskUpdater:
|
||||
"""Factory function for creating AutoRiskUpdater instances."""
|
||||
return AutoRiskUpdater(db)
|
||||
616
backend/compliance/services/export_generator.py
Normal file
616
backend/compliance/services/export_generator.py
Normal file
@@ -0,0 +1,616 @@
|
||||
"""
|
||||
Audit Export Generator.
|
||||
|
||||
Generates ZIP packages for external auditors containing:
|
||||
- Regulations & Requirements
|
||||
- Control Catalogue with status
|
||||
- Evidence artifacts
|
||||
- Risk register
|
||||
- Summary reports
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import shutil
|
||||
import tempfile
|
||||
import zipfile
|
||||
from datetime import datetime, date
|
||||
from pathlib import Path
|
||||
from typing import Dict, List, Optional, Any
|
||||
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from ..db.models import (
|
||||
RegulationDB,
|
||||
RequirementDB,
|
||||
ControlDB,
|
||||
ControlMappingDB,
|
||||
EvidenceDB,
|
||||
RiskDB,
|
||||
AuditExportDB,
|
||||
ExportStatusEnum,
|
||||
ControlStatusEnum,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class AuditExportGenerator:
|
||||
"""Generates audit export packages."""
|
||||
|
||||
def __init__(self, db: Session, export_dir: str = "/tmp/compliance_exports"):
|
||||
self.db = db
|
||||
self.export_dir = Path(export_dir)
|
||||
self.export_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
def create_export(
|
||||
self,
|
||||
requested_by: str,
|
||||
export_type: str = "full",
|
||||
included_regulations: Optional[List[str]] = None,
|
||||
included_domains: Optional[List[str]] = None,
|
||||
date_range_start: Optional[date] = None,
|
||||
date_range_end: Optional[date] = None,
|
||||
) -> AuditExportDB:
|
||||
"""
|
||||
Create a new audit export.
|
||||
|
||||
Args:
|
||||
requested_by: User requesting the export
|
||||
export_type: "full", "controls_only", "evidence_only"
|
||||
included_regulations: Filter by regulation codes
|
||||
included_domains: Filter by control domains
|
||||
date_range_start: Evidence collected after this date
|
||||
date_range_end: Evidence collected before this date
|
||||
|
||||
Returns:
|
||||
AuditExportDB record
|
||||
"""
|
||||
# Create export record
|
||||
export_record = AuditExportDB(
|
||||
export_type=export_type,
|
||||
export_name=f"Breakpilot Compliance Export {datetime.now().strftime('%Y-%m-%d %H:%M')}",
|
||||
included_regulations=included_regulations,
|
||||
included_domains=included_domains,
|
||||
date_range_start=date_range_start,
|
||||
date_range_end=date_range_end,
|
||||
requested_by=requested_by,
|
||||
status=ExportStatusEnum.GENERATING,
|
||||
)
|
||||
self.db.add(export_record)
|
||||
self.db.flush()
|
||||
|
||||
try:
|
||||
# Generate the export
|
||||
file_path, file_hash, file_size = self._generate_zip(
|
||||
export_record.id,
|
||||
export_type,
|
||||
included_regulations,
|
||||
included_domains,
|
||||
date_range_start,
|
||||
date_range_end,
|
||||
)
|
||||
|
||||
# Update record with results
|
||||
export_record.file_path = str(file_path)
|
||||
export_record.file_hash = file_hash
|
||||
export_record.file_size_bytes = file_size
|
||||
export_record.status = ExportStatusEnum.COMPLETED
|
||||
export_record.completed_at = datetime.utcnow()
|
||||
|
||||
# Calculate statistics
|
||||
stats = self._calculate_statistics(
|
||||
included_regulations, included_domains
|
||||
)
|
||||
export_record.total_controls = stats["total_controls"]
|
||||
export_record.total_evidence = stats["total_evidence"]
|
||||
export_record.compliance_score = stats["compliance_score"]
|
||||
|
||||
self.db.commit()
|
||||
logger.info(f"Export completed: {file_path}")
|
||||
return export_record
|
||||
|
||||
except Exception as e:
|
||||
export_record.status = ExportStatusEnum.FAILED
|
||||
export_record.error_message = str(e)
|
||||
self.db.commit()
|
||||
logger.error(f"Export failed: {e}")
|
||||
raise
|
||||
|
||||
def _generate_zip(
|
||||
self,
|
||||
export_id: str,
|
||||
export_type: str,
|
||||
included_regulations: Optional[List[str]],
|
||||
included_domains: Optional[List[str]],
|
||||
date_range_start: Optional[date],
|
||||
date_range_end: Optional[date],
|
||||
) -> tuple:
|
||||
"""Generate the actual ZIP file."""
|
||||
timestamp = datetime.now().strftime("%Y-%m-%d_%H%M%S")
|
||||
zip_filename = f"audit_export_{timestamp}.zip"
|
||||
zip_path = self.export_dir / zip_filename
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
temp_path = Path(temp_dir)
|
||||
|
||||
# Create directory structure
|
||||
(temp_path / "regulations").mkdir()
|
||||
(temp_path / "controls").mkdir()
|
||||
(temp_path / "evidence").mkdir()
|
||||
(temp_path / "risks").mkdir()
|
||||
|
||||
# Generate content based on export type
|
||||
if export_type in ["full", "controls_only"]:
|
||||
self._export_regulations(temp_path / "regulations", included_regulations)
|
||||
self._export_controls(temp_path / "controls", included_domains)
|
||||
|
||||
if export_type in ["full", "evidence_only"]:
|
||||
self._export_evidence(
|
||||
temp_path / "evidence",
|
||||
included_domains,
|
||||
date_range_start,
|
||||
date_range_end,
|
||||
)
|
||||
|
||||
if export_type == "full":
|
||||
self._export_risks(temp_path / "risks")
|
||||
|
||||
# Generate summary
|
||||
self._export_summary(
|
||||
temp_path,
|
||||
export_type,
|
||||
included_regulations,
|
||||
included_domains,
|
||||
)
|
||||
|
||||
# Generate README
|
||||
self._export_readme(temp_path)
|
||||
|
||||
# Generate index.html for navigation
|
||||
self._export_index_html(temp_path)
|
||||
|
||||
# Create ZIP
|
||||
with zipfile.ZipFile(zip_path, "w", zipfile.ZIP_DEFLATED) as zf:
|
||||
for file_path in temp_path.rglob("*"):
|
||||
if file_path.is_file():
|
||||
arcname = file_path.relative_to(temp_path)
|
||||
zf.write(file_path, arcname)
|
||||
|
||||
# Calculate hash
|
||||
file_hash = self._calculate_file_hash(zip_path)
|
||||
file_size = zip_path.stat().st_size
|
||||
|
||||
return zip_path, file_hash, file_size
|
||||
|
||||
def _export_regulations(
|
||||
self, output_dir: Path, included_regulations: Optional[List[str]]
|
||||
) -> None:
|
||||
"""Export regulations to JSON files."""
|
||||
query = self.db.query(RegulationDB).filter(RegulationDB.is_active == True)
|
||||
if included_regulations:
|
||||
query = query.filter(RegulationDB.code.in_(included_regulations))
|
||||
|
||||
regulations = query.all()
|
||||
|
||||
for reg in regulations:
|
||||
# Get requirements for this regulation
|
||||
requirements = self.db.query(RequirementDB).filter(
|
||||
RequirementDB.regulation_id == reg.id
|
||||
).all()
|
||||
|
||||
data = {
|
||||
"code": reg.code,
|
||||
"name": reg.name,
|
||||
"full_name": reg.full_name,
|
||||
"type": reg.regulation_type.value if reg.regulation_type else None,
|
||||
"source_url": reg.source_url,
|
||||
"effective_date": reg.effective_date.isoformat() if reg.effective_date else None,
|
||||
"description": reg.description,
|
||||
"requirements": [
|
||||
{
|
||||
"article": r.article,
|
||||
"paragraph": r.paragraph,
|
||||
"title": r.title,
|
||||
"description": r.description,
|
||||
"is_applicable": r.is_applicable,
|
||||
"breakpilot_interpretation": r.breakpilot_interpretation,
|
||||
}
|
||||
for r in requirements
|
||||
],
|
||||
}
|
||||
|
||||
file_path = output_dir / f"{reg.code.lower()}.json"
|
||||
with open(file_path, "w", encoding="utf-8") as f:
|
||||
json.dump(data, f, indent=2, ensure_ascii=False)
|
||||
|
||||
def _export_controls(
|
||||
self, output_dir: Path, included_domains: Optional[List[str]]
|
||||
) -> None:
|
||||
"""Export controls to JSON and generate summary."""
|
||||
query = self.db.query(ControlDB)
|
||||
if included_domains:
|
||||
from ..db.models import ControlDomainEnum
|
||||
domain_enums = [ControlDomainEnum(d) for d in included_domains]
|
||||
query = query.filter(ControlDB.domain.in_(domain_enums))
|
||||
|
||||
controls = query.order_by(ControlDB.control_id).all()
|
||||
|
||||
controls_data = []
|
||||
for ctrl in controls:
|
||||
# Get mappings
|
||||
mappings = self.db.query(ControlMappingDB).filter(
|
||||
ControlMappingDB.control_id == ctrl.id
|
||||
).all()
|
||||
|
||||
# Get requirement references
|
||||
requirement_refs = []
|
||||
for m in mappings:
|
||||
req = self.db.query(RequirementDB).get(m.requirement_id)
|
||||
if req:
|
||||
reg = self.db.query(RegulationDB).get(req.regulation_id)
|
||||
requirement_refs.append({
|
||||
"regulation": reg.code if reg else None,
|
||||
"article": req.article,
|
||||
"paragraph": req.paragraph,
|
||||
"coverage": m.coverage_level,
|
||||
})
|
||||
|
||||
ctrl_data = {
|
||||
"control_id": ctrl.control_id,
|
||||
"domain": ctrl.domain.value if ctrl.domain else None,
|
||||
"type": ctrl.control_type.value if ctrl.control_type else None,
|
||||
"title": ctrl.title,
|
||||
"description": ctrl.description,
|
||||
"pass_criteria": ctrl.pass_criteria,
|
||||
"status": ctrl.status.value if ctrl.status else None,
|
||||
"is_automated": ctrl.is_automated,
|
||||
"automation_tool": ctrl.automation_tool,
|
||||
"owner": ctrl.owner,
|
||||
"last_reviewed": ctrl.last_reviewed_at.isoformat() if ctrl.last_reviewed_at else None,
|
||||
"code_reference": ctrl.code_reference,
|
||||
"mapped_requirements": requirement_refs,
|
||||
}
|
||||
controls_data.append(ctrl_data)
|
||||
|
||||
# Write full catalogue
|
||||
with open(output_dir / "control_catalogue.json", "w", encoding="utf-8") as f:
|
||||
json.dump(controls_data, f, indent=2, ensure_ascii=False)
|
||||
|
||||
# Write summary by domain
|
||||
domain_summary = {}
|
||||
for ctrl in controls_data:
|
||||
domain = ctrl["domain"]
|
||||
if domain not in domain_summary:
|
||||
domain_summary[domain] = {"total": 0, "pass": 0, "partial": 0, "fail": 0}
|
||||
domain_summary[domain]["total"] += 1
|
||||
status = ctrl["status"]
|
||||
if status in domain_summary[domain]:
|
||||
domain_summary[domain][status] += 1
|
||||
|
||||
with open(output_dir / "domain_summary.json", "w", encoding="utf-8") as f:
|
||||
json.dump(domain_summary, f, indent=2, ensure_ascii=False)
|
||||
|
||||
def _export_evidence(
|
||||
self,
|
||||
output_dir: Path,
|
||||
included_domains: Optional[List[str]],
|
||||
date_range_start: Optional[date],
|
||||
date_range_end: Optional[date],
|
||||
) -> None:
|
||||
"""Export evidence metadata and files."""
|
||||
query = self.db.query(EvidenceDB)
|
||||
|
||||
if date_range_start:
|
||||
query = query.filter(EvidenceDB.collected_at >= datetime.combine(date_range_start, datetime.min.time()))
|
||||
if date_range_end:
|
||||
query = query.filter(EvidenceDB.collected_at <= datetime.combine(date_range_end, datetime.max.time()))
|
||||
|
||||
if included_domains:
|
||||
from ..db.models import ControlDomainEnum
|
||||
domain_enums = [ControlDomainEnum(d) for d in included_domains]
|
||||
query = query.join(ControlDB).filter(ControlDB.domain.in_(domain_enums))
|
||||
|
||||
evidence_list = query.all()
|
||||
|
||||
evidence_data = []
|
||||
for ev in evidence_list:
|
||||
ctrl = self.db.query(ControlDB).get(ev.control_id)
|
||||
|
||||
ev_data = {
|
||||
"id": ev.id,
|
||||
"control_id": ctrl.control_id if ctrl else None,
|
||||
"evidence_type": ev.evidence_type,
|
||||
"title": ev.title,
|
||||
"description": ev.description,
|
||||
"artifact_path": ev.artifact_path,
|
||||
"artifact_url": ev.artifact_url,
|
||||
"artifact_hash": ev.artifact_hash,
|
||||
"status": ev.status.value if ev.status else None,
|
||||
"valid_from": ev.valid_from.isoformat() if ev.valid_from else None,
|
||||
"valid_until": ev.valid_until.isoformat() if ev.valid_until else None,
|
||||
"collected_at": ev.collected_at.isoformat() if ev.collected_at else None,
|
||||
"source": ev.source,
|
||||
}
|
||||
evidence_data.append(ev_data)
|
||||
|
||||
# Copy evidence files if they exist
|
||||
if ev.artifact_path and os.path.exists(ev.artifact_path):
|
||||
evidence_subdir = output_dir / ev.evidence_type
|
||||
evidence_subdir.mkdir(exist_ok=True)
|
||||
filename = os.path.basename(ev.artifact_path)
|
||||
shutil.copy2(ev.artifact_path, evidence_subdir / filename)
|
||||
|
||||
with open(output_dir / "evidence_index.json", "w", encoding="utf-8") as f:
|
||||
json.dump(evidence_data, f, indent=2, ensure_ascii=False)
|
||||
|
||||
def _export_risks(self, output_dir: Path) -> None:
|
||||
"""Export risk register."""
|
||||
risks = self.db.query(RiskDB).order_by(RiskDB.risk_id).all()
|
||||
|
||||
risks_data = []
|
||||
for risk in risks:
|
||||
risk_data = {
|
||||
"risk_id": risk.risk_id,
|
||||
"title": risk.title,
|
||||
"description": risk.description,
|
||||
"category": risk.category,
|
||||
"likelihood": risk.likelihood,
|
||||
"impact": risk.impact,
|
||||
"inherent_risk": risk.inherent_risk.value if risk.inherent_risk else None,
|
||||
"mitigating_controls": risk.mitigating_controls,
|
||||
"residual_likelihood": risk.residual_likelihood,
|
||||
"residual_impact": risk.residual_impact,
|
||||
"residual_risk": risk.residual_risk.value if risk.residual_risk else None,
|
||||
"owner": risk.owner,
|
||||
"status": risk.status,
|
||||
"treatment_plan": risk.treatment_plan,
|
||||
}
|
||||
risks_data.append(risk_data)
|
||||
|
||||
with open(output_dir / "risk_register.json", "w", encoding="utf-8") as f:
|
||||
json.dump(risks_data, f, indent=2, ensure_ascii=False)
|
||||
|
||||
def _export_summary(
|
||||
self,
|
||||
output_dir: Path,
|
||||
export_type: str,
|
||||
included_regulations: Optional[List[str]],
|
||||
included_domains: Optional[List[str]],
|
||||
) -> None:
|
||||
"""Generate summary.json with overall statistics."""
|
||||
stats = self._calculate_statistics(included_regulations, included_domains)
|
||||
|
||||
summary = {
|
||||
"export_date": datetime.now().isoformat(),
|
||||
"export_type": export_type,
|
||||
"filters": {
|
||||
"regulations": included_regulations,
|
||||
"domains": included_domains,
|
||||
},
|
||||
"statistics": stats,
|
||||
"organization": "Breakpilot",
|
||||
"version": "1.0.0",
|
||||
}
|
||||
|
||||
with open(output_dir / "summary.json", "w", encoding="utf-8") as f:
|
||||
json.dump(summary, f, indent=2, ensure_ascii=False)
|
||||
|
||||
def _export_readme(self, output_dir: Path) -> None:
|
||||
"""Generate README.md for auditors."""
|
||||
readme = """# Breakpilot Compliance Export
|
||||
|
||||
Dieses Paket enthält die Compliance-Dokumentation von Breakpilot.
|
||||
|
||||
## Struktur
|
||||
|
||||
```
|
||||
├── summary.json # Zusammenfassung und Statistiken
|
||||
├── index.html # HTML-Navigation (im Browser öffnen)
|
||||
├── regulations/ # Verordnungen und Anforderungen
|
||||
│ ├── gdpr.json
|
||||
│ ├── aiact.json
|
||||
│ └── ...
|
||||
├── controls/ # Control Catalogue
|
||||
│ ├── control_catalogue.json
|
||||
│ └── domain_summary.json
|
||||
├── evidence/ # Nachweise
|
||||
│ ├── evidence_index.json
|
||||
│ └── [evidence_type]/
|
||||
└── risks/ # Risikoregister
|
||||
└── risk_register.json
|
||||
```
|
||||
|
||||
## Verwendung
|
||||
|
||||
1. **HTML-Navigation**: Öffnen Sie `index.html` im Browser für eine visuelle Übersicht.
|
||||
2. **JSON-Dateien**: Maschinenlesbare Daten für Import in GRC-Tools.
|
||||
3. **Nachweis-Dateien**: Originale Scan-Reports und Konfigurationen.
|
||||
|
||||
## Kontakt
|
||||
|
||||
Bei Fragen wenden Sie sich an das Breakpilot Security Team.
|
||||
|
||||
---
|
||||
Generiert am: """ + datetime.now().strftime("%Y-%m-%d %H:%M:%S")
|
||||
|
||||
with open(output_dir / "README.md", "w", encoding="utf-8") as f:
|
||||
f.write(readme)
|
||||
|
||||
def _export_index_html(self, output_dir: Path) -> None:
|
||||
"""Generate index.html for browser navigation."""
|
||||
html = """<!DOCTYPE html>
|
||||
<html lang="de">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>Breakpilot Compliance Export</title>
|
||||
<style>
|
||||
body { font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif; max-width: 1200px; margin: 0 auto; padding: 2rem; background: #f5f5f5; }
|
||||
h1 { color: #1a1a1a; border-bottom: 3px solid #0066cc; padding-bottom: 1rem; }
|
||||
h2 { color: #333; margin-top: 2rem; }
|
||||
.card { background: white; border-radius: 8px; padding: 1.5rem; margin: 1rem 0; box-shadow: 0 2px 4px rgba(0,0,0,0.1); }
|
||||
.stats { display: grid; grid-template-columns: repeat(auto-fit, minmax(200px, 1fr)); gap: 1rem; }
|
||||
.stat { background: linear-gradient(135deg, #0066cc, #004499); color: white; padding: 1.5rem; border-radius: 8px; text-align: center; }
|
||||
.stat-value { font-size: 2.5rem; font-weight: bold; }
|
||||
.stat-label { opacity: 0.9; margin-top: 0.5rem; }
|
||||
ul { list-style: none; padding: 0; }
|
||||
li { padding: 0.75rem; border-bottom: 1px solid #eee; }
|
||||
li:last-child { border-bottom: none; }
|
||||
a { color: #0066cc; text-decoration: none; }
|
||||
a:hover { text-decoration: underline; }
|
||||
.footer { margin-top: 3rem; padding-top: 1rem; border-top: 1px solid #ddd; color: #666; font-size: 0.9rem; }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<h1>Breakpilot Compliance Export</h1>
|
||||
|
||||
<div class="stats">
|
||||
<div class="stat">
|
||||
<div class="stat-value" id="score">--%</div>
|
||||
<div class="stat-label">Compliance Score</div>
|
||||
</div>
|
||||
<div class="stat">
|
||||
<div class="stat-value" id="controls">--</div>
|
||||
<div class="stat-label">Controls</div>
|
||||
</div>
|
||||
<div class="stat">
|
||||
<div class="stat-value" id="evidence">--</div>
|
||||
<div class="stat-label">Evidence Items</div>
|
||||
</div>
|
||||
<div class="stat">
|
||||
<div class="stat-value" id="regulations">--</div>
|
||||
<div class="stat-label">Regulations</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="card">
|
||||
<h2>Regulations & Requirements</h2>
|
||||
<ul id="regulations-list">
|
||||
<li>Loading...</li>
|
||||
</ul>
|
||||
</div>
|
||||
|
||||
<div class="card">
|
||||
<h2>Controls by Domain</h2>
|
||||
<ul id="domains-list">
|
||||
<li>Loading...</li>
|
||||
</ul>
|
||||
</div>
|
||||
|
||||
<div class="card">
|
||||
<h2>Export Contents</h2>
|
||||
<ul>
|
||||
<li><a href="summary.json">summary.json</a> - Export metadata and statistics</li>
|
||||
<li><a href="controls/control_catalogue.json">controls/control_catalogue.json</a> - Full control catalogue</li>
|
||||
<li><a href="evidence/evidence_index.json">evidence/evidence_index.json</a> - Evidence index</li>
|
||||
<li><a href="risks/risk_register.json">risks/risk_register.json</a> - Risk register</li>
|
||||
</ul>
|
||||
</div>
|
||||
|
||||
<div class="footer">
|
||||
<p>Generated by Breakpilot Compliance Framework</p>
|
||||
</div>
|
||||
|
||||
<script>
|
||||
// Load summary and populate stats
|
||||
fetch('summary.json')
|
||||
.then(r => r.json())
|
||||
.then(data => {
|
||||
document.getElementById('score').textContent = (data.statistics.compliance_score || 0).toFixed(0) + '%';
|
||||
document.getElementById('controls').textContent = data.statistics.total_controls || 0;
|
||||
document.getElementById('evidence').textContent = data.statistics.total_evidence || 0;
|
||||
document.getElementById('regulations').textContent = data.statistics.total_regulations || 0;
|
||||
})
|
||||
.catch(() => console.log('Could not load summary'));
|
||||
|
||||
// Load regulations list
|
||||
const regsDir = 'regulations/';
|
||||
document.getElementById('regulations-list').innerHTML =
|
||||
'<li><a href="regulations/gdpr.json">GDPR</a> - Datenschutz-Grundverordnung</li>' +
|
||||
'<li><a href="regulations/aiact.json">AI Act</a> - KI-Verordnung</li>' +
|
||||
'<li><a href="regulations/cra.json">CRA</a> - Cyber Resilience Act</li>';
|
||||
|
||||
// Load domain summary
|
||||
fetch('controls/domain_summary.json')
|
||||
.then(r => r.json())
|
||||
.then(data => {
|
||||
const list = document.getElementById('domains-list');
|
||||
list.innerHTML = Object.entries(data).map(([domain, stats]) =>
|
||||
`<li><strong>${domain.toUpperCase()}</strong>: ${stats.pass || 0}/${stats.total} controls passing</li>`
|
||||
).join('');
|
||||
})
|
||||
.catch(() => console.log('Could not load domain summary'));
|
||||
</script>
|
||||
</body>
|
||||
</html>"""
|
||||
|
||||
with open(output_dir / "index.html", "w", encoding="utf-8") as f:
|
||||
f.write(html)
|
||||
|
||||
def _calculate_statistics(
|
||||
self,
|
||||
included_regulations: Optional[List[str]],
|
||||
included_domains: Optional[List[str]],
|
||||
) -> Dict[str, Any]:
|
||||
"""Calculate compliance statistics."""
|
||||
# Count regulations
|
||||
reg_query = self.db.query(RegulationDB).filter(RegulationDB.is_active == True)
|
||||
if included_regulations:
|
||||
reg_query = reg_query.filter(RegulationDB.code.in_(included_regulations))
|
||||
total_regulations = reg_query.count()
|
||||
|
||||
# Count controls
|
||||
ctrl_query = self.db.query(ControlDB)
|
||||
if included_domains:
|
||||
from ..db.models import ControlDomainEnum
|
||||
domain_enums = [ControlDomainEnum(d) for d in included_domains]
|
||||
ctrl_query = ctrl_query.filter(ControlDB.domain.in_(domain_enums))
|
||||
|
||||
total_controls = ctrl_query.count()
|
||||
passing_controls = ctrl_query.filter(ControlDB.status == ControlStatusEnum.PASS).count()
|
||||
partial_controls = ctrl_query.filter(ControlDB.status == ControlStatusEnum.PARTIAL).count()
|
||||
|
||||
# Count evidence
|
||||
total_evidence = self.db.query(EvidenceDB).count()
|
||||
|
||||
# Calculate compliance score
|
||||
if total_controls > 0:
|
||||
score = ((passing_controls + partial_controls * 0.5) / total_controls) * 100
|
||||
else:
|
||||
score = 0
|
||||
|
||||
return {
|
||||
"total_regulations": total_regulations,
|
||||
"total_controls": total_controls,
|
||||
"passing_controls": passing_controls,
|
||||
"partial_controls": partial_controls,
|
||||
"total_evidence": total_evidence,
|
||||
"compliance_score": round(score, 1),
|
||||
}
|
||||
|
||||
def _calculate_file_hash(self, file_path: Path) -> str:
|
||||
"""Calculate SHA-256 hash of file."""
|
||||
sha256 = hashlib.sha256()
|
||||
with open(file_path, "rb") as f:
|
||||
for chunk in iter(lambda: f.read(8192), b""):
|
||||
sha256.update(chunk)
|
||||
return sha256.hexdigest()
|
||||
|
||||
def get_export_status(self, export_id: str) -> Optional[AuditExportDB]:
|
||||
"""Get status of an export."""
|
||||
return self.db.query(AuditExportDB).get(export_id)
|
||||
|
||||
def list_exports(
|
||||
self, limit: int = 20, offset: int = 0
|
||||
) -> List[AuditExportDB]:
|
||||
"""List recent exports."""
|
||||
return (
|
||||
self.db.query(AuditExportDB)
|
||||
.order_by(AuditExportDB.requested_at.desc())
|
||||
.offset(offset)
|
||||
.limit(limit)
|
||||
.all()
|
||||
)
|
||||
622
backend/compliance/services/llm_provider.py
Normal file
622
backend/compliance/services/llm_provider.py
Normal file
@@ -0,0 +1,622 @@
|
||||
"""
|
||||
LLM Provider Abstraction for Compliance AI Features.
|
||||
|
||||
Supports:
|
||||
- Anthropic Claude API (default)
|
||||
- Self-Hosted LLMs (Ollama, vLLM, LocalAI, etc.)
|
||||
- HashiCorp Vault integration for secure API key storage
|
||||
|
||||
Configuration via environment variables:
|
||||
- COMPLIANCE_LLM_PROVIDER: "anthropic" or "self_hosted"
|
||||
- ANTHROPIC_API_KEY: API key for Claude (or loaded from Vault)
|
||||
- ANTHROPIC_MODEL: Model name (default: claude-sonnet-4-20250514)
|
||||
- SELF_HOSTED_LLM_URL: Base URL for self-hosted LLM
|
||||
- SELF_HOSTED_LLM_MODEL: Model name for self-hosted
|
||||
- SELF_HOSTED_LLM_KEY: Optional API key for self-hosted
|
||||
|
||||
Vault Configuration:
|
||||
- VAULT_ADDR: Vault server address (e.g., http://vault:8200)
|
||||
- VAULT_TOKEN: Vault authentication token
|
||||
- USE_VAULT_SECRETS: Set to "true" to enable Vault integration
|
||||
- VAULT_SECRET_PATH: Path to secrets (default: secret/breakpilot/api_keys)
|
||||
"""
|
||||
|
||||
import os
|
||||
import asyncio
|
||||
import logging
|
||||
from abc import ABC, abstractmethod
|
||||
from typing import List, Optional, Dict, Any
|
||||
from dataclasses import dataclass, field
|
||||
from enum import Enum
|
||||
|
||||
import httpx
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Vault Integration
|
||||
# =============================================================================
|
||||
|
||||
class VaultClient:
|
||||
"""
|
||||
HashiCorp Vault client for retrieving secrets.
|
||||
|
||||
Supports KV v2 secrets engine.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
addr: Optional[str] = None,
|
||||
token: Optional[str] = None
|
||||
):
|
||||
self.addr = addr or os.getenv("VAULT_ADDR", "http://localhost:8200")
|
||||
self.token = token or os.getenv("VAULT_TOKEN")
|
||||
self._cache: Dict[str, Any] = {}
|
||||
self._cache_ttl = 300 # 5 minutes cache
|
||||
|
||||
def _get_headers(self) -> Dict[str, str]:
|
||||
"""Get request headers with Vault token."""
|
||||
headers = {"Content-Type": "application/json"}
|
||||
if self.token:
|
||||
headers["X-Vault-Token"] = self.token
|
||||
return headers
|
||||
|
||||
def get_secret(self, path: str, key: str = "value") -> Optional[str]:
|
||||
"""
|
||||
Get a secret from Vault KV v2.
|
||||
|
||||
Args:
|
||||
path: Secret path (e.g., "breakpilot/api_keys/anthropic")
|
||||
key: Key within the secret data (default: "value")
|
||||
|
||||
Returns:
|
||||
Secret value or None if not found
|
||||
"""
|
||||
cache_key = f"{path}:{key}"
|
||||
|
||||
# Check cache first
|
||||
if cache_key in self._cache:
|
||||
return self._cache[cache_key]
|
||||
|
||||
try:
|
||||
# KV v2 uses /data/ in the path
|
||||
full_path = f"{self.addr}/v1/secret/data/{path}"
|
||||
|
||||
response = httpx.get(
|
||||
full_path,
|
||||
headers=self._get_headers(),
|
||||
timeout=10.0
|
||||
)
|
||||
|
||||
if response.status_code == 200:
|
||||
data = response.json()
|
||||
secret_data = data.get("data", {}).get("data", {})
|
||||
secret_value = secret_data.get(key)
|
||||
|
||||
if secret_value:
|
||||
self._cache[cache_key] = secret_value
|
||||
logger.info(f"Successfully loaded secret from Vault: {path}")
|
||||
return secret_value
|
||||
|
||||
elif response.status_code == 404:
|
||||
logger.warning(f"Secret not found in Vault: {path}")
|
||||
else:
|
||||
logger.error(f"Vault error {response.status_code}: {response.text}")
|
||||
|
||||
except httpx.RequestError as e:
|
||||
logger.error(f"Failed to connect to Vault at {self.addr}: {e}")
|
||||
except Exception as e:
|
||||
logger.error(f"Error retrieving secret from Vault: {e}")
|
||||
|
||||
return None
|
||||
|
||||
def get_anthropic_key(self) -> Optional[str]:
|
||||
"""Get Anthropic API key from Vault."""
|
||||
path = os.getenv("VAULT_ANTHROPIC_PATH", "breakpilot/api_keys/anthropic")
|
||||
return self.get_secret(path, "value")
|
||||
|
||||
def is_available(self) -> bool:
|
||||
"""Check if Vault is available and authenticated."""
|
||||
try:
|
||||
response = httpx.get(
|
||||
f"{self.addr}/v1/sys/health",
|
||||
headers=self._get_headers(),
|
||||
timeout=5.0
|
||||
)
|
||||
return response.status_code in (200, 429, 472, 473, 501, 503)
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
# Singleton Vault client
|
||||
_vault_client: Optional[VaultClient] = None
|
||||
|
||||
|
||||
def get_vault_client() -> VaultClient:
|
||||
"""Get shared Vault client instance."""
|
||||
global _vault_client
|
||||
if _vault_client is None:
|
||||
_vault_client = VaultClient()
|
||||
return _vault_client
|
||||
|
||||
|
||||
def get_secret_from_vault_or_env(
|
||||
vault_path: str,
|
||||
env_var: str,
|
||||
vault_key: str = "value"
|
||||
) -> Optional[str]:
|
||||
"""
|
||||
Get a secret, trying Vault first, then falling back to environment variable.
|
||||
|
||||
Args:
|
||||
vault_path: Path in Vault (e.g., "breakpilot/api_keys/anthropic")
|
||||
env_var: Environment variable name as fallback
|
||||
vault_key: Key within Vault secret data
|
||||
|
||||
Returns:
|
||||
Secret value or None
|
||||
"""
|
||||
use_vault = os.getenv("USE_VAULT_SECRETS", "").lower() in ("true", "1", "yes")
|
||||
|
||||
if use_vault:
|
||||
vault = get_vault_client()
|
||||
secret = vault.get_secret(vault_path, vault_key)
|
||||
if secret:
|
||||
return secret
|
||||
logger.info(f"Vault secret not found, falling back to env: {env_var}")
|
||||
|
||||
return os.getenv(env_var)
|
||||
|
||||
|
||||
class LLMProviderType(str, Enum):
|
||||
"""Supported LLM provider types."""
|
||||
ANTHROPIC = "anthropic"
|
||||
SELF_HOSTED = "self_hosted"
|
||||
MOCK = "mock" # For testing
|
||||
|
||||
|
||||
@dataclass
|
||||
class LLMResponse:
|
||||
"""Standard response from LLM."""
|
||||
content: str
|
||||
model: str
|
||||
provider: str
|
||||
usage: Optional[Dict[str, int]] = None
|
||||
raw_response: Optional[Dict[str, Any]] = None
|
||||
|
||||
|
||||
@dataclass
|
||||
class LLMConfig:
|
||||
"""Configuration for LLM provider."""
|
||||
provider_type: LLMProviderType
|
||||
api_key: Optional[str] = None
|
||||
model: str = "claude-sonnet-4-20250514"
|
||||
base_url: Optional[str] = None
|
||||
max_tokens: int = 4096
|
||||
temperature: float = 0.3
|
||||
timeout: float = 60.0
|
||||
|
||||
|
||||
class LLMProvider(ABC):
|
||||
"""Abstract base class for LLM providers."""
|
||||
|
||||
def __init__(self, config: LLMConfig):
|
||||
self.config = config
|
||||
|
||||
@abstractmethod
|
||||
async def complete(
|
||||
self,
|
||||
prompt: str,
|
||||
system_prompt: Optional[str] = None,
|
||||
max_tokens: Optional[int] = None,
|
||||
temperature: Optional[float] = None
|
||||
) -> LLMResponse:
|
||||
"""Generate a completion for the given prompt."""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
async def batch_complete(
|
||||
self,
|
||||
prompts: List[str],
|
||||
system_prompt: Optional[str] = None,
|
||||
max_tokens: Optional[int] = None,
|
||||
rate_limit: float = 1.0
|
||||
) -> List[LLMResponse]:
|
||||
"""Generate completions for multiple prompts with rate limiting."""
|
||||
pass
|
||||
|
||||
@property
|
||||
@abstractmethod
|
||||
def provider_name(self) -> str:
|
||||
"""Return the provider name."""
|
||||
pass
|
||||
|
||||
|
||||
class AnthropicProvider(LLMProvider):
|
||||
"""Claude API Provider using Anthropic's official API."""
|
||||
|
||||
ANTHROPIC_API_URL = "https://api.anthropic.com/v1/messages"
|
||||
|
||||
def __init__(self, config: LLMConfig):
|
||||
super().__init__(config)
|
||||
if not config.api_key:
|
||||
raise ValueError("Anthropic API key is required")
|
||||
self.api_key = config.api_key
|
||||
self.model = config.model or "claude-sonnet-4-20250514"
|
||||
|
||||
@property
|
||||
def provider_name(self) -> str:
|
||||
return "anthropic"
|
||||
|
||||
async def complete(
|
||||
self,
|
||||
prompt: str,
|
||||
system_prompt: Optional[str] = None,
|
||||
max_tokens: Optional[int] = None,
|
||||
temperature: Optional[float] = None
|
||||
) -> LLMResponse:
|
||||
"""Generate completion using Claude API."""
|
||||
|
||||
headers = {
|
||||
"x-api-key": self.api_key,
|
||||
"anthropic-version": "2023-06-01",
|
||||
"content-type": "application/json"
|
||||
}
|
||||
|
||||
messages = [{"role": "user", "content": prompt}]
|
||||
|
||||
payload = {
|
||||
"model": self.model,
|
||||
"max_tokens": max_tokens or self.config.max_tokens,
|
||||
"messages": messages
|
||||
}
|
||||
|
||||
if system_prompt:
|
||||
payload["system"] = system_prompt
|
||||
|
||||
if temperature is not None:
|
||||
payload["temperature"] = temperature
|
||||
elif self.config.temperature is not None:
|
||||
payload["temperature"] = self.config.temperature
|
||||
|
||||
async with httpx.AsyncClient(timeout=self.config.timeout) as client:
|
||||
try:
|
||||
response = await client.post(
|
||||
self.ANTHROPIC_API_URL,
|
||||
headers=headers,
|
||||
json=payload
|
||||
)
|
||||
response.raise_for_status()
|
||||
data = response.json()
|
||||
|
||||
content = ""
|
||||
if data.get("content"):
|
||||
content = data["content"][0].get("text", "")
|
||||
|
||||
return LLMResponse(
|
||||
content=content,
|
||||
model=self.model,
|
||||
provider=self.provider_name,
|
||||
usage=data.get("usage"),
|
||||
raw_response=data
|
||||
)
|
||||
|
||||
except httpx.HTTPStatusError as e:
|
||||
logger.error(f"Anthropic API error: {e.response.status_code} - {e.response.text}")
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Anthropic API request failed: {e}")
|
||||
raise
|
||||
|
||||
async def batch_complete(
|
||||
self,
|
||||
prompts: List[str],
|
||||
system_prompt: Optional[str] = None,
|
||||
max_tokens: Optional[int] = None,
|
||||
rate_limit: float = 1.0
|
||||
) -> List[LLMResponse]:
|
||||
"""Process multiple prompts with rate limiting."""
|
||||
results = []
|
||||
|
||||
for i, prompt in enumerate(prompts):
|
||||
if i > 0:
|
||||
await asyncio.sleep(rate_limit)
|
||||
|
||||
try:
|
||||
result = await self.complete(
|
||||
prompt=prompt,
|
||||
system_prompt=system_prompt,
|
||||
max_tokens=max_tokens
|
||||
)
|
||||
results.append(result)
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to process prompt {i}: {e}")
|
||||
# Append error response
|
||||
results.append(LLMResponse(
|
||||
content=f"Error: {str(e)}",
|
||||
model=self.model,
|
||||
provider=self.provider_name
|
||||
))
|
||||
|
||||
return results
|
||||
|
||||
|
||||
class SelfHostedProvider(LLMProvider):
|
||||
"""Self-Hosted LLM Provider supporting Ollama, vLLM, LocalAI, etc."""
|
||||
|
||||
def __init__(self, config: LLMConfig):
|
||||
super().__init__(config)
|
||||
if not config.base_url:
|
||||
raise ValueError("Base URL is required for self-hosted provider")
|
||||
self.base_url = config.base_url.rstrip("/")
|
||||
self.model = config.model
|
||||
self.api_key = config.api_key
|
||||
|
||||
@property
|
||||
def provider_name(self) -> str:
|
||||
return "self_hosted"
|
||||
|
||||
def _detect_api_format(self) -> str:
|
||||
"""Detect the API format based on URL patterns."""
|
||||
if "11434" in self.base_url or "ollama" in self.base_url.lower():
|
||||
return "ollama"
|
||||
elif "openai" in self.base_url.lower() or "v1" in self.base_url:
|
||||
return "openai"
|
||||
else:
|
||||
return "ollama" # Default to Ollama format
|
||||
|
||||
async def complete(
|
||||
self,
|
||||
prompt: str,
|
||||
system_prompt: Optional[str] = None,
|
||||
max_tokens: Optional[int] = None,
|
||||
temperature: Optional[float] = None
|
||||
) -> LLMResponse:
|
||||
"""Generate completion using self-hosted LLM."""
|
||||
|
||||
api_format = self._detect_api_format()
|
||||
|
||||
headers = {"content-type": "application/json"}
|
||||
if self.api_key:
|
||||
headers["Authorization"] = f"Bearer {self.api_key}"
|
||||
|
||||
if api_format == "ollama":
|
||||
# Ollama API format
|
||||
endpoint = f"{self.base_url}/api/generate"
|
||||
full_prompt = prompt
|
||||
if system_prompt:
|
||||
full_prompt = f"{system_prompt}\n\n{prompt}"
|
||||
|
||||
payload = {
|
||||
"model": self.model,
|
||||
"prompt": full_prompt,
|
||||
"stream": False,
|
||||
"options": {}
|
||||
}
|
||||
|
||||
if max_tokens:
|
||||
payload["options"]["num_predict"] = max_tokens
|
||||
if temperature is not None:
|
||||
payload["options"]["temperature"] = temperature
|
||||
|
||||
else:
|
||||
# OpenAI-compatible format (vLLM, LocalAI, etc.)
|
||||
endpoint = f"{self.base_url}/v1/chat/completions"
|
||||
|
||||
messages = []
|
||||
if system_prompt:
|
||||
messages.append({"role": "system", "content": system_prompt})
|
||||
messages.append({"role": "user", "content": prompt})
|
||||
|
||||
payload = {
|
||||
"model": self.model,
|
||||
"messages": messages,
|
||||
"max_tokens": max_tokens or self.config.max_tokens,
|
||||
"temperature": temperature if temperature is not None else self.config.temperature
|
||||
}
|
||||
|
||||
async with httpx.AsyncClient(timeout=self.config.timeout) as client:
|
||||
try:
|
||||
response = await client.post(endpoint, headers=headers, json=payload)
|
||||
response.raise_for_status()
|
||||
data = response.json()
|
||||
|
||||
# Parse response based on format
|
||||
if api_format == "ollama":
|
||||
content = data.get("response", "")
|
||||
else:
|
||||
# OpenAI format
|
||||
content = data.get("choices", [{}])[0].get("message", {}).get("content", "")
|
||||
|
||||
return LLMResponse(
|
||||
content=content,
|
||||
model=self.model,
|
||||
provider=self.provider_name,
|
||||
usage=data.get("usage"),
|
||||
raw_response=data
|
||||
)
|
||||
|
||||
except httpx.HTTPStatusError as e:
|
||||
logger.error(f"Self-hosted LLM error: {e.response.status_code} - {e.response.text}")
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Self-hosted LLM request failed: {e}")
|
||||
raise
|
||||
|
||||
async def batch_complete(
|
||||
self,
|
||||
prompts: List[str],
|
||||
system_prompt: Optional[str] = None,
|
||||
max_tokens: Optional[int] = None,
|
||||
rate_limit: float = 0.5 # Self-hosted can be faster
|
||||
) -> List[LLMResponse]:
|
||||
"""Process multiple prompts with rate limiting."""
|
||||
results = []
|
||||
|
||||
for i, prompt in enumerate(prompts):
|
||||
if i > 0:
|
||||
await asyncio.sleep(rate_limit)
|
||||
|
||||
try:
|
||||
result = await self.complete(
|
||||
prompt=prompt,
|
||||
system_prompt=system_prompt,
|
||||
max_tokens=max_tokens
|
||||
)
|
||||
results.append(result)
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to process prompt {i}: {e}")
|
||||
results.append(LLMResponse(
|
||||
content=f"Error: {str(e)}",
|
||||
model=self.model,
|
||||
provider=self.provider_name
|
||||
))
|
||||
|
||||
return results
|
||||
|
||||
|
||||
class MockProvider(LLMProvider):
|
||||
"""Mock provider for testing without actual API calls."""
|
||||
|
||||
def __init__(self, config: LLMConfig):
|
||||
super().__init__(config)
|
||||
self.responses: List[str] = []
|
||||
self.call_count = 0
|
||||
|
||||
@property
|
||||
def provider_name(self) -> str:
|
||||
return "mock"
|
||||
|
||||
def set_responses(self, responses: List[str]):
|
||||
"""Set predetermined responses for testing."""
|
||||
self.responses = responses
|
||||
self.call_count = 0
|
||||
|
||||
async def complete(
|
||||
self,
|
||||
prompt: str,
|
||||
system_prompt: Optional[str] = None,
|
||||
max_tokens: Optional[int] = None,
|
||||
temperature: Optional[float] = None
|
||||
) -> LLMResponse:
|
||||
"""Return mock response."""
|
||||
if self.responses:
|
||||
content = self.responses[self.call_count % len(self.responses)]
|
||||
else:
|
||||
content = f"Mock response for: {prompt[:50]}..."
|
||||
|
||||
self.call_count += 1
|
||||
|
||||
return LLMResponse(
|
||||
content=content,
|
||||
model="mock-model",
|
||||
provider=self.provider_name,
|
||||
usage={"input_tokens": len(prompt), "output_tokens": len(content)}
|
||||
)
|
||||
|
||||
async def batch_complete(
|
||||
self,
|
||||
prompts: List[str],
|
||||
system_prompt: Optional[str] = None,
|
||||
max_tokens: Optional[int] = None,
|
||||
rate_limit: float = 0.0
|
||||
) -> List[LLMResponse]:
|
||||
"""Return mock responses for batch."""
|
||||
return [await self.complete(p, system_prompt, max_tokens) for p in prompts]
|
||||
|
||||
|
||||
def get_llm_config() -> LLMConfig:
|
||||
"""
|
||||
Create LLM config from environment variables or Vault.
|
||||
|
||||
Priority for API key:
|
||||
1. Vault (if USE_VAULT_SECRETS=true and Vault is available)
|
||||
2. Environment variable (ANTHROPIC_API_KEY)
|
||||
"""
|
||||
provider_type_str = os.getenv("COMPLIANCE_LLM_PROVIDER", "anthropic")
|
||||
|
||||
try:
|
||||
provider_type = LLMProviderType(provider_type_str)
|
||||
except ValueError:
|
||||
logger.warning(f"Unknown LLM provider: {provider_type_str}, falling back to mock")
|
||||
provider_type = LLMProviderType.MOCK
|
||||
|
||||
# Get API key from Vault or environment
|
||||
api_key = None
|
||||
if provider_type == LLMProviderType.ANTHROPIC:
|
||||
api_key = get_secret_from_vault_or_env(
|
||||
vault_path="breakpilot/api_keys/anthropic",
|
||||
env_var="ANTHROPIC_API_KEY"
|
||||
)
|
||||
elif provider_type == LLMProviderType.SELF_HOSTED:
|
||||
api_key = get_secret_from_vault_or_env(
|
||||
vault_path="breakpilot/api_keys/self_hosted_llm",
|
||||
env_var="SELF_HOSTED_LLM_KEY"
|
||||
)
|
||||
|
||||
# Select model based on provider type
|
||||
if provider_type == LLMProviderType.ANTHROPIC:
|
||||
model = os.getenv("ANTHROPIC_MODEL", "claude-sonnet-4-20250514")
|
||||
elif provider_type == LLMProviderType.SELF_HOSTED:
|
||||
model = os.getenv("SELF_HOSTED_LLM_MODEL", "qwen2.5:14b")
|
||||
else:
|
||||
model = "mock-model"
|
||||
|
||||
return LLMConfig(
|
||||
provider_type=provider_type,
|
||||
api_key=api_key,
|
||||
model=model,
|
||||
base_url=os.getenv("SELF_HOSTED_LLM_URL"),
|
||||
max_tokens=int(os.getenv("COMPLIANCE_LLM_MAX_TOKENS", "4096")),
|
||||
temperature=float(os.getenv("COMPLIANCE_LLM_TEMPERATURE", "0.3")),
|
||||
timeout=float(os.getenv("COMPLIANCE_LLM_TIMEOUT", "60.0"))
|
||||
)
|
||||
|
||||
|
||||
def get_llm_provider(config: Optional[LLMConfig] = None) -> LLMProvider:
|
||||
"""
|
||||
Factory function to get the appropriate LLM provider based on configuration.
|
||||
|
||||
Usage:
|
||||
provider = get_llm_provider()
|
||||
response = await provider.complete("Analyze this requirement...")
|
||||
"""
|
||||
if config is None:
|
||||
config = get_llm_config()
|
||||
|
||||
if config.provider_type == LLMProviderType.ANTHROPIC:
|
||||
if not config.api_key:
|
||||
logger.warning("No Anthropic API key found, using mock provider")
|
||||
return MockProvider(config)
|
||||
return AnthropicProvider(config)
|
||||
|
||||
elif config.provider_type == LLMProviderType.SELF_HOSTED:
|
||||
if not config.base_url:
|
||||
logger.warning("No self-hosted LLM URL found, using mock provider")
|
||||
return MockProvider(config)
|
||||
return SelfHostedProvider(config)
|
||||
|
||||
elif config.provider_type == LLMProviderType.MOCK:
|
||||
return MockProvider(config)
|
||||
|
||||
else:
|
||||
raise ValueError(f"Unsupported LLM provider type: {config.provider_type}")
|
||||
|
||||
|
||||
# Singleton instance for reuse
|
||||
_provider_instance: Optional[LLMProvider] = None
|
||||
|
||||
|
||||
def get_shared_provider() -> LLMProvider:
|
||||
"""Get a shared LLM provider instance."""
|
||||
global _provider_instance
|
||||
if _provider_instance is None:
|
||||
_provider_instance = get_llm_provider()
|
||||
return _provider_instance
|
||||
|
||||
|
||||
def reset_shared_provider():
|
||||
"""Reset the shared provider instance (useful for testing)."""
|
||||
global _provider_instance
|
||||
_provider_instance = None
|
||||
602
backend/compliance/services/pdf_extractor.py
Normal file
602
backend/compliance/services/pdf_extractor.py
Normal file
@@ -0,0 +1,602 @@
|
||||
"""
|
||||
PDF Extractor for BSI-TR-03161 and EU Regulation Documents.
|
||||
|
||||
This module extracts Pruefaspekte (test aspects) from BSI Technical Guidelines
|
||||
and Articles from EU regulations in PDF format.
|
||||
"""
|
||||
|
||||
import re
|
||||
import logging
|
||||
from dataclasses import dataclass, field
|
||||
from typing import List, Optional, Dict, Any
|
||||
from pathlib import Path
|
||||
from enum import Enum
|
||||
|
||||
try:
|
||||
import fitz # PyMuPDF
|
||||
except ImportError:
|
||||
fitz = None
|
||||
logging.warning("PyMuPDF not installed. PDF extraction will not work.")
|
||||
|
||||
|
||||
class RequirementLevel(str, Enum):
|
||||
"""BSI requirement levels (German: Anforderungsstufen)."""
|
||||
MUSS = "MUSS" # MUST - mandatory
|
||||
SOLL = "SOLL" # SHOULD - recommended
|
||||
KANN = "KANN" # MAY - optional
|
||||
DARF_NICHT = "DARF NICHT" # MUST NOT - prohibited
|
||||
|
||||
|
||||
class AspectCategory(str, Enum):
|
||||
"""Categories for BSI-TR Pruefaspekte."""
|
||||
AUTHENTICATION = "authentication"
|
||||
SESSION_MANAGEMENT = "session_management"
|
||||
CRYPTOGRAPHY = "cryptography"
|
||||
INPUT_VALIDATION = "input_validation"
|
||||
SQL_INJECTION = "sql_injection"
|
||||
XSS_PREVENTION = "xss_prevention"
|
||||
CSRF_PROTECTION = "csrf_protection"
|
||||
LOGGING_AUDIT = "logging_audit"
|
||||
ERROR_HANDLING = "error_handling"
|
||||
NETWORK_SECURITY = "network_security"
|
||||
SECURE_STORAGE = "secure_storage"
|
||||
PRIVACY = "privacy"
|
||||
ACCESS_CONTROL = "access_control"
|
||||
DATA_PROTECTION = "data_protection"
|
||||
KEY_MANAGEMENT = "key_management"
|
||||
SECURE_COMMUNICATION = "secure_communication"
|
||||
UPDATE_MECHANISM = "update_mechanism"
|
||||
GENERAL = "general"
|
||||
TEST_ASPECT = "test_aspect"
|
||||
|
||||
|
||||
@dataclass
|
||||
class BSIAspect:
|
||||
"""A single extracted BSI-TR Pruefaspekt (test aspect)."""
|
||||
aspect_id: str # e.g., "O.Auth_1", "T.Sess_2"
|
||||
title: str # Short title
|
||||
full_text: str # Complete requirement text
|
||||
category: AspectCategory # Categorization
|
||||
page_number: int # PDF page where found
|
||||
section: str # Chapter/section number
|
||||
requirement_level: RequirementLevel # MUSS/SOLL/KANN
|
||||
source_document: str # e.g., "BSI-TR-03161-2"
|
||||
context_before: str = "" # Text before the aspect
|
||||
context_after: str = "" # Text after the aspect
|
||||
related_aspects: List[str] = field(default_factory=list) # Related aspect IDs
|
||||
keywords: List[str] = field(default_factory=list) # Extracted keywords
|
||||
|
||||
|
||||
@dataclass
|
||||
class EUArticle:
|
||||
"""A single extracted EU regulation article."""
|
||||
article_number: str # e.g., "Art. 32", "Artikel 5"
|
||||
title: str # Article title
|
||||
full_text: str # Complete article text
|
||||
paragraphs: List[str] # Individual paragraphs
|
||||
page_number: int # PDF page
|
||||
regulation_name: str # e.g., "DSGVO", "AI Act"
|
||||
recitals: List[str] = field(default_factory=list) # Related recitals
|
||||
keywords: List[str] = field(default_factory=list) # Extracted keywords
|
||||
|
||||
|
||||
class BSIPDFExtractor:
|
||||
"""
|
||||
Extracts Pruefaspekte from BSI-TR-03161 PDF documents.
|
||||
|
||||
The BSI-TR-03161 series contains security requirements for mobile applications:
|
||||
- Part 1: General security requirements
|
||||
- Part 2: Web application security (OAuth, Sessions, Input validation, etc.)
|
||||
- Part 3: Backend/server security
|
||||
|
||||
Each document contains hundreds of Pruefaspekte (test aspects) that need to
|
||||
be extracted, categorized, and stored for compliance tracking.
|
||||
"""
|
||||
|
||||
# Regex patterns for BSI-TR aspect identification
|
||||
PATTERNS = {
|
||||
# Primary aspect ID patterns (e.g., O.Auth_1, T.Network_2)
|
||||
'aspect_id': r'(O\.[A-Za-z]+_\d+|T\.[A-Za-z]+_\d+)',
|
||||
|
||||
# Alternative section-based pattern (e.g., "Pruefaspekt 4.2.1")
|
||||
'section_aspect': r'(?:Prüfaspekt|Pruefaspekt|Anforderung)\s+(\d+\.\d+(?:\.\d+)?)',
|
||||
|
||||
# Section number pattern
|
||||
'section': r'(\d+\.\d+(?:\.\d+)?)',
|
||||
|
||||
# Requirement level pattern
|
||||
'requirement': r'\b(MUSS|SOLL|KANN|DARF\s+NICHT|muss|soll|kann|darf\s+nicht)\b',
|
||||
|
||||
# Table header pattern for Pruefaspekte tables
|
||||
'table_header': r'(?:Prüfaspekt|Bezeichnung|ID|Anforderung)',
|
||||
}
|
||||
|
||||
# Category mapping based on aspect ID prefix
|
||||
CATEGORY_MAP = {
|
||||
'O.Auth': AspectCategory.AUTHENTICATION,
|
||||
'O.Sess': AspectCategory.SESSION_MANAGEMENT,
|
||||
'O.Cryp': AspectCategory.CRYPTOGRAPHY,
|
||||
'O.Crypto': AspectCategory.CRYPTOGRAPHY,
|
||||
'O.Input': AspectCategory.INPUT_VALIDATION,
|
||||
'O.SQL': AspectCategory.SQL_INJECTION,
|
||||
'O.XSS': AspectCategory.XSS_PREVENTION,
|
||||
'O.CSRF': AspectCategory.CSRF_PROTECTION,
|
||||
'O.Log': AspectCategory.LOGGING_AUDIT,
|
||||
'O.Audit': AspectCategory.LOGGING_AUDIT,
|
||||
'O.Err': AspectCategory.ERROR_HANDLING,
|
||||
'O.Error': AspectCategory.ERROR_HANDLING,
|
||||
'O.Net': AspectCategory.NETWORK_SECURITY,
|
||||
'O.Network': AspectCategory.NETWORK_SECURITY,
|
||||
'O.Store': AspectCategory.SECURE_STORAGE,
|
||||
'O.Storage': AspectCategory.SECURE_STORAGE,
|
||||
'O.Priv': AspectCategory.PRIVACY,
|
||||
'O.Privacy': AspectCategory.PRIVACY,
|
||||
'O.Data': AspectCategory.DATA_PROTECTION,
|
||||
'O.Access': AspectCategory.ACCESS_CONTROL,
|
||||
'O.Key': AspectCategory.KEY_MANAGEMENT,
|
||||
'O.Comm': AspectCategory.SECURE_COMMUNICATION,
|
||||
'O.TLS': AspectCategory.SECURE_COMMUNICATION,
|
||||
'O.Update': AspectCategory.UPDATE_MECHANISM,
|
||||
'T.': AspectCategory.TEST_ASPECT,
|
||||
}
|
||||
|
||||
# Keywords for category detection when aspect ID is ambiguous
|
||||
CATEGORY_KEYWORDS = {
|
||||
AspectCategory.AUTHENTICATION: [
|
||||
'authentifizierung', 'authentication', 'login', 'anmeldung',
|
||||
'passwort', 'password', 'credential', 'oauth', 'oidc', 'token',
|
||||
'bearer', 'jwt', 'session', 'multi-faktor', 'mfa', '2fa'
|
||||
],
|
||||
AspectCategory.SESSION_MANAGEMENT: [
|
||||
'session', 'sitzung', 'cookie', 'timeout', 'ablauf',
|
||||
'session-id', 'sessionid', 'logout', 'abmeldung'
|
||||
],
|
||||
AspectCategory.CRYPTOGRAPHY: [
|
||||
'verschlüsselung', 'encryption', 'kryptograph', 'cryptograph',
|
||||
'aes', 'rsa', 'hash', 'signatur', 'signature', 'zertifikat',
|
||||
'certificate', 'tls', 'ssl', 'hmac', 'pbkdf', 'argon'
|
||||
],
|
||||
AspectCategory.INPUT_VALIDATION: [
|
||||
'eingabevalidierung', 'input validation', 'validierung',
|
||||
'eingabeprüfung', 'sanitiz', 'whitelist', 'blacklist',
|
||||
'filter', 'escape', 'encoding'
|
||||
],
|
||||
AspectCategory.SQL_INJECTION: [
|
||||
'sql injection', 'sql-injection', 'prepared statement',
|
||||
'parameterisiert', 'parameterized', 'orm', 'database'
|
||||
],
|
||||
AspectCategory.XSS_PREVENTION: [
|
||||
'xss', 'cross-site scripting', 'script injection',
|
||||
'html encoding', 'output encoding', 'csp', 'content-security'
|
||||
],
|
||||
AspectCategory.CSRF_PROTECTION: [
|
||||
'csrf', 'cross-site request', 'token', 'anti-csrf',
|
||||
'state parameter', 'same-site', 'samesite'
|
||||
],
|
||||
AspectCategory.LOGGING_AUDIT: [
|
||||
'logging', 'protokollierung', 'audit', 'nachvollziehbar',
|
||||
'traceability', 'log', 'event', 'monitoring'
|
||||
],
|
||||
AspectCategory.ERROR_HANDLING: [
|
||||
'fehlerbehandlung', 'error handling', 'exception',
|
||||
'fehlermeldung', 'error message', 'stack trace'
|
||||
],
|
||||
}
|
||||
|
||||
def __init__(self, logger: Optional[logging.Logger] = None):
|
||||
"""Initialize the PDF extractor."""
|
||||
self.logger = logger or logging.getLogger(__name__)
|
||||
|
||||
if fitz is None:
|
||||
raise ImportError(
|
||||
"PyMuPDF is required for PDF extraction. "
|
||||
"Install it with: pip install PyMuPDF"
|
||||
)
|
||||
|
||||
def extract_from_file(self, pdf_path: str, source_name: Optional[str] = None) -> List[BSIAspect]:
|
||||
"""
|
||||
Extract all Pruefaspekte from a BSI-TR PDF file.
|
||||
|
||||
Args:
|
||||
pdf_path: Path to the PDF file
|
||||
source_name: Optional source document name (auto-detected if not provided)
|
||||
|
||||
Returns:
|
||||
List of extracted BSIAspect objects
|
||||
"""
|
||||
path = Path(pdf_path)
|
||||
if not path.exists():
|
||||
raise FileNotFoundError(f"PDF file not found: {pdf_path}")
|
||||
|
||||
source = source_name or path.stem
|
||||
self.logger.info(f"Extracting aspects from: {source}")
|
||||
|
||||
doc = fitz.open(pdf_path)
|
||||
aspects = []
|
||||
|
||||
for page_num in range(len(doc)):
|
||||
page = doc[page_num]
|
||||
text = page.get_text()
|
||||
|
||||
# Extract aspects from this page
|
||||
page_aspects = self._extract_aspects_from_text(
|
||||
text=text,
|
||||
page_num=page_num + 1,
|
||||
source_document=source
|
||||
)
|
||||
aspects.extend(page_aspects)
|
||||
|
||||
doc.close()
|
||||
|
||||
# Post-process: deduplicate and enrich
|
||||
aspects = self._deduplicate_aspects(aspects)
|
||||
aspects = self._enrich_aspects(aspects)
|
||||
|
||||
self.logger.info(f"Extracted {len(aspects)} unique aspects from {source}")
|
||||
return aspects
|
||||
|
||||
def extract_all_documents(self, docs_dir: str) -> Dict[str, List[BSIAspect]]:
|
||||
"""
|
||||
Extract aspects from all BSI-TR PDFs in a directory.
|
||||
|
||||
Args:
|
||||
docs_dir: Directory containing BSI-TR PDF files
|
||||
|
||||
Returns:
|
||||
Dictionary mapping document names to their extracted aspects
|
||||
"""
|
||||
docs_path = Path(docs_dir)
|
||||
results = {}
|
||||
|
||||
# Look for BSI-TR PDFs
|
||||
patterns = ["BSI-TR-03161*.pdf", "bsi-tr-03161*.pdf"]
|
||||
|
||||
for pattern in patterns:
|
||||
for pdf_file in docs_path.glob(pattern):
|
||||
try:
|
||||
aspects = self.extract_from_file(str(pdf_file))
|
||||
results[pdf_file.stem] = aspects
|
||||
except Exception as e:
|
||||
self.logger.error(f"Failed to extract from {pdf_file}: {e}")
|
||||
|
||||
return results
|
||||
|
||||
def _extract_aspects_from_text(
|
||||
self,
|
||||
text: str,
|
||||
page_num: int,
|
||||
source_document: str
|
||||
) -> List[BSIAspect]:
|
||||
"""Extract all Pruefaspekte from a page's text."""
|
||||
aspects = []
|
||||
|
||||
# Find all aspect IDs on this page
|
||||
for match in re.finditer(self.PATTERNS['aspect_id'], text, re.IGNORECASE):
|
||||
aspect_id = match.group(1).upper()
|
||||
|
||||
# Extract context around the match
|
||||
start = max(0, match.start() - 200)
|
||||
end = min(len(text), match.end() + 1000)
|
||||
context = text[start:end]
|
||||
|
||||
# Determine category from aspect ID
|
||||
category = self._determine_category(aspect_id, context)
|
||||
|
||||
# Extract requirement level
|
||||
req_level = self._extract_requirement_level(context)
|
||||
|
||||
# Extract title (text immediately after aspect ID)
|
||||
title = self._extract_title(context, aspect_id)
|
||||
|
||||
# Extract section number
|
||||
section = self._extract_section(context)
|
||||
|
||||
# Extract full requirement text
|
||||
full_text = self._extract_full_text(context, aspect_id)
|
||||
|
||||
aspects.append(BSIAspect(
|
||||
aspect_id=aspect_id,
|
||||
title=title,
|
||||
full_text=full_text,
|
||||
category=category,
|
||||
page_number=page_num,
|
||||
section=section,
|
||||
requirement_level=req_level,
|
||||
source_document=source_document,
|
||||
context_before=text[start:match.start()].strip()[-100:],
|
||||
context_after=text[match.end():end].strip()[:200],
|
||||
))
|
||||
|
||||
# Also look for section-based aspects
|
||||
for match in re.finditer(self.PATTERNS['section_aspect'], text, re.IGNORECASE):
|
||||
section_id = match.group(1)
|
||||
aspect_id = f"SEC_{section_id.replace('.', '_')}"
|
||||
|
||||
# Check if we already have this as an O.* aspect
|
||||
if any(a.section == section_id for a in aspects):
|
||||
continue
|
||||
|
||||
start = max(0, match.start() - 100)
|
||||
end = min(len(text), match.end() + 800)
|
||||
context = text[start:end]
|
||||
|
||||
category = self._determine_category_from_keywords(context)
|
||||
req_level = self._extract_requirement_level(context)
|
||||
|
||||
aspects.append(BSIAspect(
|
||||
aspect_id=aspect_id,
|
||||
title=f"Prüfaspekt {section_id}",
|
||||
full_text=context.strip(),
|
||||
category=category,
|
||||
page_number=page_num,
|
||||
section=section_id,
|
||||
requirement_level=req_level,
|
||||
source_document=source_document,
|
||||
))
|
||||
|
||||
return aspects
|
||||
|
||||
def _determine_category(self, aspect_id: str, context: str) -> AspectCategory:
|
||||
"""Determine the category of an aspect based on its ID and context."""
|
||||
# First try to match by aspect ID prefix
|
||||
for prefix, category in self.CATEGORY_MAP.items():
|
||||
if aspect_id.upper().startswith(prefix.upper()):
|
||||
return category
|
||||
|
||||
# Fall back to keyword-based detection
|
||||
return self._determine_category_from_keywords(context)
|
||||
|
||||
def _determine_category_from_keywords(self, text: str) -> AspectCategory:
|
||||
"""Determine category based on keywords in the text."""
|
||||
text_lower = text.lower()
|
||||
|
||||
category_scores = {}
|
||||
for category, keywords in self.CATEGORY_KEYWORDS.items():
|
||||
score = sum(1 for kw in keywords if kw in text_lower)
|
||||
if score > 0:
|
||||
category_scores[category] = score
|
||||
|
||||
if category_scores:
|
||||
return max(category_scores, key=category_scores.get)
|
||||
|
||||
return AspectCategory.GENERAL
|
||||
|
||||
def _extract_requirement_level(self, text: str) -> RequirementLevel:
|
||||
"""Extract the requirement level from text."""
|
||||
match = re.search(self.PATTERNS['requirement'], text, re.IGNORECASE)
|
||||
if match:
|
||||
level = match.group(1).upper()
|
||||
if 'DARF' in level and 'NICHT' in level:
|
||||
return RequirementLevel.DARF_NICHT
|
||||
elif level == 'MUSS':
|
||||
return RequirementLevel.MUSS
|
||||
elif level == 'SOLL':
|
||||
return RequirementLevel.SOLL
|
||||
elif level == 'KANN':
|
||||
return RequirementLevel.KANN
|
||||
|
||||
return RequirementLevel.SOLL # Default
|
||||
|
||||
def _extract_title(self, context: str, aspect_id: str) -> str:
|
||||
"""Extract the title/short description of an aspect."""
|
||||
# Look for text immediately after the aspect ID
|
||||
pattern = rf'{re.escape(aspect_id)}\s*[:\-–]?\s*([^\n]+)'
|
||||
match = re.search(pattern, context, re.IGNORECASE)
|
||||
|
||||
if match:
|
||||
title = match.group(1).strip()
|
||||
# Clean up the title
|
||||
title = re.sub(r'\s+', ' ', title)
|
||||
# Truncate if too long
|
||||
if len(title) > 200:
|
||||
title = title[:197] + "..."
|
||||
return title
|
||||
|
||||
return aspect_id
|
||||
|
||||
def _extract_section(self, context: str) -> str:
|
||||
"""Extract the section number from context."""
|
||||
match = re.search(self.PATTERNS['section'], context)
|
||||
return match.group(1) if match else ""
|
||||
|
||||
def _extract_full_text(self, context: str, aspect_id: str) -> str:
|
||||
"""Extract the complete requirement text."""
|
||||
# Find the aspect ID and get text until the next aspect or section
|
||||
pattern = rf'{re.escape(aspect_id)}[^\n]*\n(.*?)(?=\n\s*(?:O\.[A-Z]|T\.[A-Z]|\d+\.\d+\s|\Z))'
|
||||
match = re.search(pattern, context, re.IGNORECASE | re.DOTALL)
|
||||
|
||||
if match:
|
||||
full_text = match.group(0).strip()
|
||||
else:
|
||||
# Fall back to context
|
||||
full_text = context.strip()
|
||||
|
||||
# Clean up
|
||||
full_text = re.sub(r'\s+', ' ', full_text)
|
||||
return full_text
|
||||
|
||||
def _deduplicate_aspects(self, aspects: List[BSIAspect]) -> List[BSIAspect]:
|
||||
"""Remove duplicate aspects, keeping the one with more context."""
|
||||
seen = {}
|
||||
|
||||
for aspect in aspects:
|
||||
key = aspect.aspect_id
|
||||
if key not in seen:
|
||||
seen[key] = aspect
|
||||
else:
|
||||
# Keep the one with longer full_text
|
||||
if len(aspect.full_text) > len(seen[key].full_text):
|
||||
seen[key] = aspect
|
||||
|
||||
return list(seen.values())
|
||||
|
||||
def _enrich_aspects(self, aspects: List[BSIAspect]) -> List[BSIAspect]:
|
||||
"""Enrich aspects with additional metadata."""
|
||||
aspect_ids = {a.aspect_id for a in aspects}
|
||||
|
||||
for aspect in aspects:
|
||||
# Find related aspects mentioned in the full text
|
||||
for other_id in aspect_ids:
|
||||
if other_id != aspect.aspect_id and other_id in aspect.full_text:
|
||||
aspect.related_aspects.append(other_id)
|
||||
|
||||
# Extract keywords based on category
|
||||
aspect.keywords = self._extract_keywords(aspect)
|
||||
|
||||
return aspects
|
||||
|
||||
def _extract_keywords(self, aspect: BSIAspect) -> List[str]:
|
||||
"""Extract relevant keywords from an aspect."""
|
||||
keywords = []
|
||||
text_lower = aspect.full_text.lower()
|
||||
|
||||
# Add keywords based on category
|
||||
if aspect.category in self.CATEGORY_KEYWORDS:
|
||||
for kw in self.CATEGORY_KEYWORDS[aspect.category]:
|
||||
if kw in text_lower:
|
||||
keywords.append(kw)
|
||||
|
||||
return list(set(keywords))[:10] # Limit to 10 keywords
|
||||
|
||||
def get_statistics(self, aspects: List[BSIAspect]) -> Dict[str, Any]:
|
||||
"""Get statistics about extracted aspects."""
|
||||
stats = {
|
||||
"total_aspects": len(aspects),
|
||||
"by_category": {},
|
||||
"by_requirement_level": {},
|
||||
"by_source": {},
|
||||
"unique_sections": set(),
|
||||
}
|
||||
|
||||
for aspect in aspects:
|
||||
# By category
|
||||
cat = aspect.category.value
|
||||
stats["by_category"][cat] = stats["by_category"].get(cat, 0) + 1
|
||||
|
||||
# By requirement level
|
||||
level = aspect.requirement_level.value
|
||||
stats["by_requirement_level"][level] = stats["by_requirement_level"].get(level, 0) + 1
|
||||
|
||||
# By source
|
||||
src = aspect.source_document
|
||||
stats["by_source"][src] = stats["by_source"].get(src, 0) + 1
|
||||
|
||||
# Unique sections
|
||||
if aspect.section:
|
||||
stats["unique_sections"].add(aspect.section)
|
||||
|
||||
stats["unique_sections"] = len(stats["unique_sections"])
|
||||
return stats
|
||||
|
||||
|
||||
class EURegulationExtractor:
|
||||
"""
|
||||
Extracts Articles from EU Regulation PDF documents.
|
||||
|
||||
Handles documents like GDPR, AI Act, CRA, etc. in their official formats.
|
||||
"""
|
||||
|
||||
PATTERNS = {
|
||||
'article_de': r'Artikel\s+(\d+)',
|
||||
'article_en': r'Article\s+(\d+)',
|
||||
'paragraph': r'\((\d+)\)',
|
||||
'recital': r'Erwägungsgrund\s+(\d+)|Recital\s+(\d+)',
|
||||
}
|
||||
|
||||
def __init__(self, logger: Optional[logging.Logger] = None):
|
||||
self.logger = logger or logging.getLogger(__name__)
|
||||
|
||||
def extract_from_file(
|
||||
self,
|
||||
pdf_path: str,
|
||||
regulation_name: str,
|
||||
language: str = "de"
|
||||
) -> List[EUArticle]:
|
||||
"""Extract all articles from an EU regulation PDF."""
|
||||
if fitz is None:
|
||||
raise ImportError("PyMuPDF is required for PDF extraction.")
|
||||
|
||||
path = Path(pdf_path)
|
||||
if not path.exists():
|
||||
raise FileNotFoundError(f"PDF file not found: {pdf_path}")
|
||||
|
||||
doc = fitz.open(pdf_path)
|
||||
articles = []
|
||||
|
||||
article_pattern = (
|
||||
self.PATTERNS['article_de'] if language == "de"
|
||||
else self.PATTERNS['article_en']
|
||||
)
|
||||
|
||||
for page_num in range(len(doc)):
|
||||
page = doc[page_num]
|
||||
text = page.get_text()
|
||||
|
||||
# Find article starts
|
||||
for match in re.finditer(article_pattern, text):
|
||||
article_num = match.group(1)
|
||||
|
||||
# Extract article content
|
||||
start = match.start()
|
||||
# Find next article or end of page
|
||||
next_match = re.search(article_pattern, text[match.end():])
|
||||
end = match.end() + next_match.start() if next_match else len(text)
|
||||
|
||||
article_text = text[start:end].strip()
|
||||
|
||||
# Extract paragraphs
|
||||
paragraphs = self._extract_paragraphs(article_text)
|
||||
|
||||
# Extract title
|
||||
title = self._extract_article_title(article_text, article_num)
|
||||
|
||||
articles.append(EUArticle(
|
||||
article_number=f"Art. {article_num}",
|
||||
title=title,
|
||||
full_text=article_text,
|
||||
paragraphs=paragraphs,
|
||||
page_number=page_num + 1,
|
||||
regulation_name=regulation_name,
|
||||
))
|
||||
|
||||
doc.close()
|
||||
return self._deduplicate_articles(articles)
|
||||
|
||||
def _extract_paragraphs(self, text: str) -> List[str]:
|
||||
"""Extract numbered paragraphs from article text."""
|
||||
paragraphs = []
|
||||
matches = list(re.finditer(self.PATTERNS['paragraph'], text))
|
||||
|
||||
for i, match in enumerate(matches):
|
||||
start = match.start()
|
||||
end = matches[i + 1].start() if i + 1 < len(matches) else len(text)
|
||||
para_text = text[start:end].strip()
|
||||
if para_text:
|
||||
paragraphs.append(para_text)
|
||||
|
||||
return paragraphs
|
||||
|
||||
def _extract_article_title(self, text: str, article_num: str) -> str:
|
||||
"""Extract the title of an article."""
|
||||
# Look for title after "Artikel X"
|
||||
pattern = rf'Artikel\s+{article_num}\s*\n\s*([^\n]+)'
|
||||
match = re.search(pattern, text)
|
||||
|
||||
if match:
|
||||
return match.group(1).strip()
|
||||
|
||||
return f"Artikel {article_num}"
|
||||
|
||||
def _deduplicate_articles(self, articles: List[EUArticle]) -> List[EUArticle]:
|
||||
"""Remove duplicate articles."""
|
||||
seen = {}
|
||||
|
||||
for article in articles:
|
||||
key = article.article_number
|
||||
if key not in seen:
|
||||
seen[key] = article
|
||||
else:
|
||||
if len(article.full_text) > len(seen[key].full_text):
|
||||
seen[key] = article
|
||||
|
||||
return list(seen.values())
|
||||
876
backend/compliance/services/regulation_scraper.py
Normal file
876
backend/compliance/services/regulation_scraper.py
Normal file
@@ -0,0 +1,876 @@
|
||||
"""
|
||||
Compliance Regulation Scraper Service.
|
||||
|
||||
Extracts requirements and audit aspects from:
|
||||
- EU-Lex regulations (GDPR, AI Act, CRA, NIS2, etc.)
|
||||
- BSI Technical Guidelines (TR-03161)
|
||||
- German laws (TDDDG, etc.)
|
||||
|
||||
Similar pattern to edu-search and zeugnisse-crawler.
|
||||
"""
|
||||
|
||||
import logging
|
||||
import re
|
||||
import asyncio
|
||||
from datetime import datetime
|
||||
from typing import Dict, List, Any, Optional
|
||||
from enum import Enum
|
||||
import hashlib
|
||||
|
||||
import httpx
|
||||
from bs4 import BeautifulSoup
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from ..db.models import (
|
||||
RegulationDB,
|
||||
RequirementDB,
|
||||
RegulationTypeEnum,
|
||||
)
|
||||
from ..db.repository import (
|
||||
RegulationRepository,
|
||||
RequirementRepository,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class SourceType(str, Enum):
|
||||
EUR_LEX = "eur_lex"
|
||||
BSI_PDF = "bsi_pdf"
|
||||
GESETZE_IM_INTERNET = "gesetze_im_internet"
|
||||
MANUAL = "manual"
|
||||
|
||||
|
||||
class ScraperStatus(str, Enum):
|
||||
IDLE = "idle"
|
||||
RUNNING = "running"
|
||||
COMPLETED = "completed"
|
||||
ERROR = "error"
|
||||
|
||||
|
||||
class RegulationScraperService:
|
||||
"""
|
||||
Scrapes and extracts requirements from regulatory sources.
|
||||
|
||||
Supported sources:
|
||||
- EUR-Lex: https://eur-lex.europa.eu/eli/reg/{year}/{number}/oj/eng
|
||||
- BSI: Local PDF parsing
|
||||
- Gesetze-im-Internet: German law portal
|
||||
"""
|
||||
|
||||
# EUR-Lex patterns for article extraction
|
||||
ARTICLE_PATTERN = re.compile(
|
||||
r'Article\s+(\d+[a-z]?)\s*\n\s*(.+?)(?=\nArticle\s+\d|$)',
|
||||
re.DOTALL | re.IGNORECASE
|
||||
)
|
||||
|
||||
# BSI TR pattern for test aspects
|
||||
BSI_ASPECT_PATTERN = re.compile(
|
||||
r'(O\.[A-Za-z_]+[\d]*)\s+(.+?)(?=\nO\.|$)',
|
||||
re.DOTALL
|
||||
)
|
||||
|
||||
# Known regulation URLs - All 19 regulations from seed data
|
||||
KNOWN_SOURCES = {
|
||||
# A. Datenschutz & Datenuebermittlung
|
||||
"GDPR": {
|
||||
"url": "https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=CELEX:32016R0679",
|
||||
"type": SourceType.EUR_LEX,
|
||||
"regulation_type": RegulationTypeEnum.EU_REGULATION,
|
||||
},
|
||||
"EPRIVACY": {
|
||||
"url": "https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=CELEX:32002L0058",
|
||||
"type": SourceType.EUR_LEX,
|
||||
"regulation_type": RegulationTypeEnum.EU_DIRECTIVE,
|
||||
},
|
||||
"SCC": {
|
||||
"url": "https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=CELEX:32021D0914",
|
||||
"type": SourceType.EUR_LEX,
|
||||
"regulation_type": RegulationTypeEnum.EU_REGULATION,
|
||||
},
|
||||
"DPF": {
|
||||
"url": "https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=CELEX:32023D1795",
|
||||
"type": SourceType.EUR_LEX,
|
||||
"regulation_type": RegulationTypeEnum.EU_REGULATION,
|
||||
},
|
||||
# B. KI-Regulierung
|
||||
"AIACT": {
|
||||
"url": "https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=OJ:L_202401689",
|
||||
"type": SourceType.EUR_LEX,
|
||||
"regulation_type": RegulationTypeEnum.EU_REGULATION,
|
||||
},
|
||||
# C. Cybersecurity & Produktsicherheit
|
||||
"CRA": {
|
||||
"url": "https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=OJ:L_202402847",
|
||||
"type": SourceType.EUR_LEX,
|
||||
"regulation_type": RegulationTypeEnum.EU_REGULATION,
|
||||
},
|
||||
"NIS2": {
|
||||
"url": "https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=CELEX:32022L2555",
|
||||
"type": SourceType.EUR_LEX,
|
||||
"regulation_type": RegulationTypeEnum.EU_DIRECTIVE,
|
||||
},
|
||||
"EUCSA": {
|
||||
"url": "https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=CELEX:32019R0881",
|
||||
"type": SourceType.EUR_LEX,
|
||||
"regulation_type": RegulationTypeEnum.EU_REGULATION,
|
||||
},
|
||||
# D. Datenoekonomie & Interoperabilitaet
|
||||
"DATAACT": {
|
||||
"url": "https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=CELEX:32023R2854",
|
||||
"type": SourceType.EUR_LEX,
|
||||
"regulation_type": RegulationTypeEnum.EU_REGULATION,
|
||||
},
|
||||
"DGA": {
|
||||
"url": "https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=CELEX:32022R0868",
|
||||
"type": SourceType.EUR_LEX,
|
||||
"regulation_type": RegulationTypeEnum.EU_REGULATION,
|
||||
},
|
||||
# E. Plattform-Pflichten
|
||||
"DSA": {
|
||||
"url": "https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=CELEX:32022R2065",
|
||||
"type": SourceType.EUR_LEX,
|
||||
"regulation_type": RegulationTypeEnum.EU_REGULATION,
|
||||
},
|
||||
# F. Barrierefreiheit
|
||||
"EAA": {
|
||||
"url": "https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=CELEX:32019L0882",
|
||||
"type": SourceType.EUR_LEX,
|
||||
"regulation_type": RegulationTypeEnum.EU_DIRECTIVE,
|
||||
},
|
||||
# G. IP & Urheberrecht
|
||||
"DSM": {
|
||||
"url": "https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=CELEX:32019L0790",
|
||||
"type": SourceType.EUR_LEX,
|
||||
"regulation_type": RegulationTypeEnum.EU_DIRECTIVE,
|
||||
},
|
||||
# H. Produkthaftung
|
||||
"PLD": {
|
||||
"url": "https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=CELEX:32024L2853",
|
||||
"type": SourceType.EUR_LEX,
|
||||
"regulation_type": RegulationTypeEnum.EU_DIRECTIVE,
|
||||
},
|
||||
"GPSR": {
|
||||
"url": "https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=CELEX:32023R0988",
|
||||
"type": SourceType.EUR_LEX,
|
||||
"regulation_type": RegulationTypeEnum.EU_REGULATION,
|
||||
},
|
||||
# I. BSI-Standards (Deutschland)
|
||||
"BSI-TR-03161-1": {
|
||||
"url": "/docs/BSI-TR-03161-1.pdf",
|
||||
"type": SourceType.BSI_PDF,
|
||||
"regulation_type": RegulationTypeEnum.BSI_STANDARD,
|
||||
},
|
||||
"BSI-TR-03161-2": {
|
||||
"url": "/docs/BSI-TR-03161-2.pdf",
|
||||
"type": SourceType.BSI_PDF,
|
||||
"regulation_type": RegulationTypeEnum.BSI_STANDARD,
|
||||
},
|
||||
"BSI-TR-03161-3": {
|
||||
"url": "/docs/BSI-TR-03161-3.pdf",
|
||||
"type": SourceType.BSI_PDF,
|
||||
"regulation_type": RegulationTypeEnum.BSI_STANDARD,
|
||||
},
|
||||
}
|
||||
|
||||
def __init__(self, db: Session):
|
||||
self.db = db
|
||||
self.reg_repo = RegulationRepository(db)
|
||||
self.req_repo = RequirementRepository(db)
|
||||
self.status = ScraperStatus.IDLE
|
||||
self.current_source: Optional[str] = None
|
||||
self.last_error: Optional[str] = None
|
||||
self.stats = {
|
||||
"sources_processed": 0,
|
||||
"requirements_extracted": 0,
|
||||
"errors": 0,
|
||||
"last_run": None,
|
||||
}
|
||||
|
||||
async def get_status(self) -> Dict[str, Any]:
|
||||
"""Get current scraper status."""
|
||||
return {
|
||||
"status": self.status.value,
|
||||
"current_source": self.current_source,
|
||||
"last_error": self.last_error,
|
||||
"stats": self.stats,
|
||||
"known_sources": list(self.KNOWN_SOURCES.keys()),
|
||||
}
|
||||
|
||||
async def scrape_all(self) -> Dict[str, Any]:
|
||||
"""Scrape all known regulation sources."""
|
||||
self.status = ScraperStatus.RUNNING
|
||||
self.stats["last_run"] = datetime.utcnow().isoformat()
|
||||
|
||||
results = {
|
||||
"success": [],
|
||||
"failed": [],
|
||||
"skipped": [],
|
||||
}
|
||||
|
||||
for code, source_info in self.KNOWN_SOURCES.items():
|
||||
try:
|
||||
self.current_source = code
|
||||
|
||||
# Check if already scraped recently
|
||||
existing = self.reg_repo.get_by_code(code)
|
||||
if existing and existing.requirements:
|
||||
results["skipped"].append({
|
||||
"code": code,
|
||||
"reason": "already_has_requirements",
|
||||
"requirement_count": len(existing.requirements),
|
||||
})
|
||||
continue
|
||||
|
||||
# Scrape based on source type
|
||||
if source_info["type"] == SourceType.EUR_LEX:
|
||||
count = await self._scrape_eurlex(code, source_info)
|
||||
elif source_info["type"] == SourceType.BSI_PDF:
|
||||
count = await self._scrape_bsi_pdf(code, source_info)
|
||||
else:
|
||||
results["skipped"].append({
|
||||
"code": code,
|
||||
"reason": "unknown_source_type",
|
||||
})
|
||||
continue
|
||||
|
||||
results["success"].append({
|
||||
"code": code,
|
||||
"requirements_extracted": count,
|
||||
})
|
||||
self.stats["sources_processed"] += 1
|
||||
self.stats["requirements_extracted"] += count
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error scraping {code}: {e}")
|
||||
results["failed"].append({
|
||||
"code": code,
|
||||
"error": str(e),
|
||||
})
|
||||
self.stats["errors"] += 1
|
||||
self.last_error = str(e)
|
||||
|
||||
self.status = ScraperStatus.COMPLETED
|
||||
self.current_source = None
|
||||
return results
|
||||
|
||||
async def scrape_single(self, code: str, force: bool = False) -> Dict[str, Any]:
|
||||
"""Scrape a single regulation source."""
|
||||
if code not in self.KNOWN_SOURCES:
|
||||
raise ValueError(f"Unknown regulation code: {code}")
|
||||
|
||||
source_info = self.KNOWN_SOURCES[code]
|
||||
self.status = ScraperStatus.RUNNING
|
||||
self.current_source = code
|
||||
|
||||
try:
|
||||
# Check existing
|
||||
existing = self.reg_repo.get_by_code(code)
|
||||
if existing and existing.requirements and not force:
|
||||
self.status = ScraperStatus.COMPLETED
|
||||
return {
|
||||
"code": code,
|
||||
"status": "skipped",
|
||||
"reason": "already_has_requirements",
|
||||
"requirement_count": len(existing.requirements),
|
||||
}
|
||||
|
||||
# Delete existing requirements if force
|
||||
if existing and force:
|
||||
for req in existing.requirements:
|
||||
self.db.delete(req)
|
||||
self.db.commit()
|
||||
|
||||
# Scrape
|
||||
if source_info["type"] == SourceType.EUR_LEX:
|
||||
count = await self._scrape_eurlex(code, source_info)
|
||||
elif source_info["type"] == SourceType.BSI_PDF:
|
||||
count = await self._scrape_bsi_pdf(code, source_info)
|
||||
else:
|
||||
raise ValueError(f"Unknown source type: {source_info['type']}")
|
||||
|
||||
self.status = ScraperStatus.COMPLETED
|
||||
return {
|
||||
"code": code,
|
||||
"status": "success",
|
||||
"requirements_extracted": count,
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
self.status = ScraperStatus.ERROR
|
||||
self.last_error = str(e)
|
||||
raise
|
||||
finally:
|
||||
self.current_source = None
|
||||
|
||||
async def _scrape_eurlex(self, code: str, source_info: Dict) -> int:
|
||||
"""Scrape EUR-Lex regulation page."""
|
||||
url = source_info["url"]
|
||||
logger.info(f"Scraping EUR-Lex: {code} from {url}")
|
||||
|
||||
async with httpx.AsyncClient(timeout=60.0) as client:
|
||||
response = await client.get(url, follow_redirects=True)
|
||||
response.raise_for_status()
|
||||
|
||||
html = response.text
|
||||
soup = BeautifulSoup(html, 'html.parser')
|
||||
|
||||
# Get or create regulation
|
||||
regulation = self.reg_repo.get_by_code(code)
|
||||
if not regulation:
|
||||
regulation = RegulationDB(
|
||||
code=code,
|
||||
name=code,
|
||||
regulation_type=source_info["regulation_type"],
|
||||
source_url=url,
|
||||
is_active=True,
|
||||
)
|
||||
self.db.add(regulation)
|
||||
self.db.commit()
|
||||
self.db.refresh(regulation)
|
||||
|
||||
# Extract articles
|
||||
requirements_created = 0
|
||||
|
||||
# Find all article elements (EUR-Lex structure varies)
|
||||
articles = soup.find_all('div', class_='eli-subdivision')
|
||||
if not articles:
|
||||
articles = soup.find_all('p', class_='oj-ti-art')
|
||||
|
||||
for article_elem in articles:
|
||||
try:
|
||||
# Extract article number and title
|
||||
article_id = article_elem.get('id', '')
|
||||
if not article_id:
|
||||
title_elem = article_elem.find(['span', 'p'], class_=['oj-ti-art', 'eli-title'])
|
||||
if title_elem:
|
||||
text = title_elem.get_text(strip=True)
|
||||
match = re.search(r'Article\s+(\d+[a-z]?)', text, re.IGNORECASE)
|
||||
if match:
|
||||
article_id = f"art_{match.group(1)}"
|
||||
|
||||
if not article_id:
|
||||
continue
|
||||
|
||||
# Extract article text
|
||||
article_text = article_elem.get_text(separator='\n', strip=True)
|
||||
|
||||
# Parse article number and title
|
||||
lines = article_text.split('\n')
|
||||
article_num = None
|
||||
title = None
|
||||
|
||||
for line in lines[:3]:
|
||||
art_match = re.search(r'Article\s+(\d+[a-z]?)', line, re.IGNORECASE)
|
||||
if art_match:
|
||||
article_num = f"Art. {art_match.group(1)}"
|
||||
elif not article_num:
|
||||
continue
|
||||
elif not title and len(line) > 3 and not line.startswith('Article'):
|
||||
title = line[:200]
|
||||
break
|
||||
|
||||
if not article_num:
|
||||
continue
|
||||
|
||||
# Check if requirement already exists
|
||||
existing = self.db.query(RequirementDB).filter(
|
||||
RequirementDB.regulation_id == regulation.id,
|
||||
RequirementDB.article == article_num
|
||||
).first()
|
||||
|
||||
if existing:
|
||||
continue
|
||||
|
||||
# Create requirement
|
||||
requirement = RequirementDB(
|
||||
regulation_id=regulation.id,
|
||||
article=article_num,
|
||||
title=title or f"{code} {article_num}",
|
||||
requirement_text=article_text[:5000], # Limit length
|
||||
source_section=article_id,
|
||||
is_applicable=True,
|
||||
priority=2,
|
||||
)
|
||||
self.db.add(requirement)
|
||||
requirements_created += 1
|
||||
|
||||
except Exception as e:
|
||||
logger.warning(f"Error parsing article in {code}: {e}")
|
||||
continue
|
||||
|
||||
# Alternative: extract from raw text with regex
|
||||
if requirements_created == 0:
|
||||
text = soup.get_text()
|
||||
matches = self.ARTICLE_PATTERN.findall(text)
|
||||
|
||||
for art_num, art_text in matches[:50]: # Limit to 50 articles
|
||||
article_num = f"Art. {art_num}"
|
||||
|
||||
existing = self.db.query(RequirementDB).filter(
|
||||
RequirementDB.regulation_id == regulation.id,
|
||||
RequirementDB.article == article_num
|
||||
).first()
|
||||
|
||||
if existing:
|
||||
continue
|
||||
|
||||
# Extract first line as title
|
||||
lines = art_text.strip().split('\n')
|
||||
title = lines[0][:200] if lines else f"{code} {article_num}"
|
||||
|
||||
requirement = RequirementDB(
|
||||
regulation_id=regulation.id,
|
||||
article=article_num,
|
||||
title=title,
|
||||
requirement_text=art_text[:5000],
|
||||
is_applicable=True,
|
||||
priority=2,
|
||||
)
|
||||
self.db.add(requirement)
|
||||
requirements_created += 1
|
||||
|
||||
# Fallback: If scraping failed (e.g., WAF protection), use seed requirements
|
||||
if requirements_created == 0:
|
||||
logger.info(f"Scraping returned 0 results for {code}, using seed requirements")
|
||||
seed_reqs = self._get_eurlex_seed_requirements(code)
|
||||
for seed in seed_reqs:
|
||||
existing = self.db.query(RequirementDB).filter(
|
||||
RequirementDB.regulation_id == regulation.id,
|
||||
RequirementDB.article == seed["article"]
|
||||
).first()
|
||||
|
||||
if existing:
|
||||
continue
|
||||
|
||||
requirement = RequirementDB(
|
||||
regulation_id=regulation.id,
|
||||
article=seed["article"],
|
||||
title=seed["title"],
|
||||
description=seed.get("description"),
|
||||
requirement_text=seed.get("requirement_text"),
|
||||
is_applicable=True,
|
||||
priority=seed.get("priority", 2),
|
||||
)
|
||||
self.db.add(requirement)
|
||||
requirements_created += 1
|
||||
|
||||
self.db.commit()
|
||||
logger.info(f"Extracted {requirements_created} requirements from {code}")
|
||||
return requirements_created
|
||||
|
||||
def _get_eurlex_seed_requirements(self, code: str) -> List[Dict[str, Any]]:
|
||||
"""
|
||||
Returns seed requirements for EUR-Lex regulations when scraping fails.
|
||||
|
||||
These are the key articles relevant for Breakpilot compliance.
|
||||
"""
|
||||
if code == "NIS2":
|
||||
return [
|
||||
{"article": "Art. 6", "title": "Risikobewertung", "description": "Risikobewertung fuer Cybersicherheit", "requirement_text": "Einrichtungen muessen eine Risikobewertung fuer Cybersicherheit durchfuehren.", "priority": 1},
|
||||
{"article": "Art. 7", "title": "Nationale Cybersicherheitsstrategie", "description": "Umsetzung nationaler Vorgaben", "requirement_text": "Einhaltung der nationalen Cybersicherheitsstrategie.", "priority": 2},
|
||||
{"article": "Art. 20", "title": "Governance", "description": "Leitungsorgane muessen Cybersicherheit beaufsichtigen", "requirement_text": "Leitungsorgane muessen Cybersicherheitsmassnahmen genehmigen und deren Umsetzung beaufsichtigen.", "priority": 1},
|
||||
{"article": "Art. 21", "title": "Risikomanagementmassnahmen", "description": "Technische und organisatorische Massnahmen", "requirement_text": "Geeignete und verhaeltnismaessige technische, operative und organisatorische Massnahmen zur Beherrschung von Cybersicherheitsrisiken.", "priority": 1},
|
||||
{"article": "Art. 21(2)(a)", "title": "Risikoanalyse und Sicherheitskonzepte", "description": "Konzepte fuer Risikoanalyse", "requirement_text": "Konzepte fuer die Risikoanalyse und Sicherheit von Informationssystemen.", "priority": 1},
|
||||
{"article": "Art. 21(2)(b)", "title": "Bewertung von Sicherheitsvorfaellen", "description": "Incident Handling", "requirement_text": "Bewertung der Wirksamkeit von Risikomanagementmassnahmen.", "priority": 1},
|
||||
{"article": "Art. 21(2)(c)", "title": "Business Continuity", "description": "Betriebskontinuitaet sicherstellen", "requirement_text": "Aufrechterhaltung des Betriebs, Backup-Management und Krisenmanagement.", "priority": 1},
|
||||
{"article": "Art. 21(2)(d)", "title": "Lieferkettensicherheit", "description": "Sicherheit in der Lieferkette", "requirement_text": "Sicherheit der Lieferkette einschliesslich Beziehungen zu Lieferanten.", "priority": 1},
|
||||
{"article": "Art. 21(2)(e)", "title": "Sicherheit bei Entwicklung", "description": "Sichere Entwicklung", "requirement_text": "Sicherheit bei Erwerb, Entwicklung und Wartung von Systemen.", "priority": 1},
|
||||
{"article": "Art. 21(2)(f)", "title": "Schwachstellenmanagement", "description": "Umgang mit Schwachstellen", "requirement_text": "Konzepte zur Bewertung der Wirksamkeit von Massnahmen.", "priority": 1},
|
||||
{"article": "Art. 21(2)(g)", "title": "Cyberhygiene und Schulungen", "description": "Grundlegende Cyberhygiene-Praktiken", "requirement_text": "Grundlegende Cyberhygiene-Praktiken und Schulungen.", "priority": 1},
|
||||
{"article": "Art. 21(2)(h)", "title": "Kryptografie", "description": "Einsatz von Verschluesselung", "requirement_text": "Konzepte und Verfahren fuer Kryptografie und Verschluesselung.", "priority": 1},
|
||||
{"article": "Art. 21(2)(i)", "title": "Personalsicherheit", "description": "HR-Security", "requirement_text": "Sicherheit des Personals, Zugangskontrollen und Asset-Management.", "priority": 1},
|
||||
{"article": "Art. 21(2)(j)", "title": "MFA und sichere Authentifizierung", "description": "Multi-Faktor-Authentifizierung", "requirement_text": "Multi-Faktor-Authentifizierung und sichere Kommunikation.", "priority": 1},
|
||||
{"article": "Art. 23", "title": "Meldepflichten", "description": "Meldung von Sicherheitsvorfaellen", "requirement_text": "Erhebliche Sicherheitsvorfaelle muessen den zustaendigen Behoerden gemeldet werden.", "priority": 1},
|
||||
{"article": "Art. 24", "title": "Europaeische Schwachstellendatenbank", "description": "CVE-Datenbank nutzen", "requirement_text": "Nutzung der europaeischen Schwachstellendatenbank.", "priority": 2},
|
||||
]
|
||||
|
||||
elif code == "DATAACT":
|
||||
return [
|
||||
{"article": "Art. 3", "title": "Datenzugang fuer Nutzer", "description": "Nutzer koennen auf ihre Daten zugreifen", "requirement_text": "Daten, die durch Nutzung vernetzter Produkte generiert werden, muessen dem Nutzer zugaenglich gemacht werden.", "priority": 1},
|
||||
{"article": "Art. 4", "title": "Recht auf Datenzugang", "description": "Unentgeltlicher Zugang", "requirement_text": "Nutzer haben das Recht auf unentgeltlichen Zugang zu ihren Daten.", "priority": 1},
|
||||
{"article": "Art. 5", "title": "Recht auf Datenweitergabe", "description": "Daten an Dritte weitergeben", "requirement_text": "Nutzer koennen verlangen, dass Daten an Dritte weitergegeben werden.", "priority": 1},
|
||||
{"article": "Art. 6", "title": "Pflichten des Dateninhabers", "description": "Daten zeitnah bereitstellen", "requirement_text": "Dateninhaber muessen Daten unverzueglich und in geeignetem Format bereitstellen.", "priority": 1},
|
||||
{"article": "Art. 8", "title": "Faire Vertragsbedingungen", "description": "Keine unfairen Klauseln", "requirement_text": "Vertragsbedingungen fuer Datenzugang muessen fair und nicht-diskriminierend sein.", "priority": 2},
|
||||
{"article": "Art. 14", "title": "Cloud-Switching", "description": "Wechsel zwischen Cloud-Anbietern", "requirement_text": "Unterstuetzung beim Wechsel zwischen Cloud-Diensten und Datenportabilitaet.", "priority": 1},
|
||||
{"article": "Art. 23", "title": "Technische Schutzmassnahmen", "description": "Schutz nicht-personenbezogener Daten", "requirement_text": "Angemessene technische Schutzmassnahmen fuer nicht-personenbezogene Daten.", "priority": 1},
|
||||
{"article": "Art. 25", "title": "Geschaeftsgeheimnisse", "description": "Schutz von Geschaeftsgeheimnissen", "requirement_text": "Massnahmen zum Schutz von Geschaeftsgeheimnissen bei Datenzugang.", "priority": 2},
|
||||
]
|
||||
|
||||
elif code == "DGA":
|
||||
return [
|
||||
{"article": "Art. 5", "title": "Bedingungen fuer Weiterverwendung", "description": "Weiterverwendung oeffentlicher Daten", "requirement_text": "Bedingungen fuer die Weiterverwendung geschuetzter Daten oeffentlicher Stellen.", "priority": 2},
|
||||
{"article": "Art. 7", "title": "Technische Anforderungen", "description": "Sichere Verarbeitungsumgebungen", "requirement_text": "Sichere Verarbeitungsumgebungen fuer Zugang zu geschuetzten Daten.", "priority": 1},
|
||||
{"article": "Art. 10", "title": "Datenvermittlungsdienste", "description": "Registrierung von Vermittlungsdiensten", "requirement_text": "Datenvermittlungsdienste muessen registriert und reguliert werden.", "priority": 2},
|
||||
{"article": "Art. 12", "title": "Bedingungen fuer Datenvermittlung", "description": "Neutralitaet wahren", "requirement_text": "Datenvermittler muessen neutral handeln und duerfen Daten nicht fuer eigene Zwecke nutzen.", "priority": 1},
|
||||
{"article": "Art. 16", "title": "Datenaltruismus", "description": "Freiwillige Datenspende", "requirement_text": "Registrierung als Organisation fuer Datenaltruismus moeglich.", "priority": 3},
|
||||
{"article": "Art. 21", "title": "Einwilligungsformular", "description": "Europaeisches Einwilligungsformular", "requirement_text": "Verwendung des europaeischen Einwilligungsformulars fuer Datenaltruismus.", "priority": 3},
|
||||
]
|
||||
|
||||
elif code == "DSA":
|
||||
return [
|
||||
{"article": "Art. 6", "title": "Haftungsausschluss Hosting", "description": "Bedingungen fuer Haftungsausschluss", "requirement_text": "Hosting-Dienste haften nicht, wenn sie keine Kenntnis von rechtswidrigen Inhalten haben.", "priority": 1},
|
||||
{"article": "Art. 11", "title": "Kontaktstelle", "description": "Behoerdenkontakt", "requirement_text": "Anbieter muessen eine Kontaktstelle fuer Behoerden benennen.", "priority": 2},
|
||||
{"article": "Art. 12", "title": "Rechtsvertreter", "description": "Vertreter in der EU", "requirement_text": "Nicht-EU-Anbieter muessen einen Rechtsvertreter in der EU benennen.", "priority": 2},
|
||||
{"article": "Art. 13", "title": "AGB-Transparenz", "description": "Transparente Nutzungsbedingungen", "requirement_text": "AGB muessen klar, verstaendlich und leicht zugaenglich sein.", "priority": 1},
|
||||
{"article": "Art. 14", "title": "Transparenzberichte", "description": "Jaehrliche Berichte", "requirement_text": "Jaehrliche Transparenzberichte ueber Content-Moderation veroeffentlichen.", "priority": 2},
|
||||
{"article": "Art. 16", "title": "Melde- und Abhilfeverfahren", "description": "Notice and Action", "requirement_text": "Leicht zugaengliches System fuer Meldung rechtswidriger Inhalte.", "priority": 1},
|
||||
{"article": "Art. 17", "title": "Begruendungspflicht", "description": "Entscheidungen begruenden", "requirement_text": "Nutzer muessen ueber Content-Moderation-Entscheidungen informiert werden.", "priority": 1},
|
||||
{"article": "Art. 20", "title": "Internes Beschwerdemanagement", "description": "Beschwerden bearbeiten", "requirement_text": "Internes System zur Bearbeitung von Beschwerden ueber Content-Moderation.", "priority": 1},
|
||||
{"article": "Art. 26", "title": "Werbetransparenz", "description": "Werbung kennzeichnen", "requirement_text": "Online-Werbung muss klar als solche erkennbar sein.", "priority": 1},
|
||||
{"article": "Art. 27", "title": "Empfehlungssysteme", "description": "Algorithmen erklaeren", "requirement_text": "Transparenz ueber Parameter von Empfehlungsalgorithmen.", "priority": 2},
|
||||
]
|
||||
|
||||
elif code == "EUCSA":
|
||||
return [
|
||||
{"article": "Art. 46", "title": "Cybersicherheitszertifizierung", "description": "EU-Zertifizierungsrahmen", "requirement_text": "Freiwillige europaeische Zertifizierung fuer IKT-Produkte und -Dienste.", "priority": 2},
|
||||
{"article": "Art. 51", "title": "Sicherheitsziele", "description": "Ziele der Zertifizierung", "requirement_text": "Schutz von Daten vor unbefugtem Zugriff, Manipulation und Zerstoerung.", "priority": 1},
|
||||
{"article": "Art. 52", "title": "Vertrauenswuerdigkeitsstufen", "description": "Basic, Substantial, High", "requirement_text": "Drei Stufen: Basic, Substantial, High - je nach Risiko.", "priority": 1},
|
||||
{"article": "Art. 54", "title": "Konformitaetsbewertung", "description": "Selbstbewertung oder Drittbewertung", "requirement_text": "Je nach Stufe Selbstbewertung oder unabhaengige Bewertung.", "priority": 2},
|
||||
{"article": "Art. 56", "title": "Zertifizierungsstellen", "description": "Akkreditierte Stellen", "requirement_text": "Zertifizierung durch akkreditierte Konformitaetsbewertungsstellen.", "priority": 2},
|
||||
]
|
||||
|
||||
elif code == "EAA":
|
||||
return [
|
||||
{"article": "Art. 3", "title": "Barrierefreiheitsanforderungen", "description": "Produkte barrierefrei gestalten", "requirement_text": "Produkte und Dienstleistungen muessen die Barrierefreiheitsanforderungen erfuellen.", "priority": 1},
|
||||
{"article": "Art. 4", "title": "Bestehende Rechtsvorschriften", "description": "Verhaeltnis zu anderen Vorschriften", "requirement_text": "Ergaenzung zu bestehenden Barrierefreiheitsvorschriften.", "priority": 3},
|
||||
{"article": "Art. 13", "title": "Konformitaetsvermutung", "description": "Harmonisierte Normen", "requirement_text": "Konformitaet bei Einhaltung harmonisierter Normen vermutet.", "priority": 2},
|
||||
{"article": "Art. 14", "title": "Gemeinsame technische Spezifikationen", "description": "Falls keine Normen existieren", "requirement_text": "EU-Kommission kann technische Spezifikationen festlegen.", "priority": 3},
|
||||
{"article": "Anhang I", "title": "Barrierefreiheitsanforderungen fuer Produkte", "description": "WCAG-konforme Webseiten", "requirement_text": "Webseiten, Apps und E-Books muessen WCAG 2.1 Level AA erfuellen.", "priority": 1},
|
||||
]
|
||||
|
||||
elif code == "DSM":
|
||||
return [
|
||||
{"article": "Art. 3", "title": "Text and Data Mining (Forschung)", "description": "TDM fuer Forschung erlaubt", "requirement_text": "Text- und Data-Mining fuer wissenschaftliche Forschung ist erlaubt.", "priority": 2},
|
||||
{"article": "Art. 4", "title": "Text and Data Mining (Allgemein)", "description": "TDM-Ausnahme", "requirement_text": "TDM erlaubt, wenn Rechteinhaber nicht widersprochen haben.", "priority": 1},
|
||||
{"article": "Art. 15", "title": "Leistungsschutzrecht Presse", "description": "Verguetung fuer Presseverleger", "requirement_text": "Online-Nutzung von Presseveroeffentlichungen erfordert Lizenz.", "priority": 2},
|
||||
{"article": "Art. 17", "title": "Upload-Filter", "description": "Plattformhaftung fuer Uploads", "requirement_text": "Plattformen haften fuer urheberrechtsverletzende Uploads ihrer Nutzer.", "priority": 1},
|
||||
{"article": "Art. 17(7)", "title": "Overblocking verhindern", "description": "Legitime Nutzung schuetzen", "requirement_text": "Massnahmen duerfen nicht zu ungerechtfertigter Sperrung fuehren.", "priority": 1},
|
||||
]
|
||||
|
||||
elif code == "PLD":
|
||||
return [
|
||||
{"article": "Art. 4", "title": "Produktbegriff", "description": "Software als Produkt", "requirement_text": "Software gilt als Produkt im Sinne der Produkthaftung.", "priority": 1},
|
||||
{"article": "Art. 6", "title": "Fehlerhaftes Produkt", "description": "Definition Produktfehler", "requirement_text": "Ein Produkt ist fehlerhaft, wenn es nicht die erwartete Sicherheit bietet.", "priority": 1},
|
||||
{"article": "Art. 7", "title": "KI-Systeme", "description": "Haftung fuer KI", "requirement_text": "Haftung gilt auch fuer durch KI verursachte Schaeden.", "priority": 1},
|
||||
{"article": "Art. 9", "title": "Haftung des Herstellers", "description": "Verschuldensunabhaengige Haftung", "requirement_text": "Hersteller haften verschuldensunabhaengig fuer Produktfehler.", "priority": 1},
|
||||
{"article": "Art. 10", "title": "Softwareaktualisierungen", "description": "Pflicht zu Updates", "requirement_text": "Fehlende Sicherheitsupdates koennen Haftung begruenden.", "priority": 1},
|
||||
]
|
||||
|
||||
elif code == "GPSR":
|
||||
return [
|
||||
{"article": "Art. 5", "title": "Allgemeine Sicherheitsanforderung", "description": "Produkte muessen sicher sein", "requirement_text": "Nur sichere Produkte duerfen in Verkehr gebracht werden.", "priority": 1},
|
||||
{"article": "Art. 8", "title": "Pflichten der Hersteller", "description": "Sicherheitsbewertung durchfuehren", "requirement_text": "Hersteller muessen Risikoanalyse und Sicherheitsbewertung durchfuehren.", "priority": 1},
|
||||
{"article": "Art. 9", "title": "Technische Dokumentation", "description": "Dokumentationspflicht", "requirement_text": "Technische Dokumentation zur Konformitaet erstellen und aufbewahren.", "priority": 1},
|
||||
{"article": "Art. 10", "title": "EU-Konformitaetserklaerung", "description": "CE-Kennzeichnung", "requirement_text": "Konformitaetserklaerung und CE-Kennzeichnung erforderlich.", "priority": 1},
|
||||
{"article": "Art. 14", "title": "Produktrueckrufe", "description": "Rueckrufverfahren", "requirement_text": "Bei Sicherheitsrisiken muessen Produkte zurueckgerufen werden.", "priority": 1},
|
||||
]
|
||||
|
||||
elif code == "CRA":
|
||||
return [
|
||||
{"article": "Art. 5", "title": "Wesentliche Anforderungen", "description": "Cybersicherheit bei Entwurf", "requirement_text": "Produkte muessen so entworfen werden, dass sie ein angemessenes Cybersicherheitsniveau gewaehrleisten.", "priority": 1},
|
||||
{"article": "Art. 6", "title": "Sicherheitsupdates", "description": "Updates bereitstellen", "requirement_text": "Hersteller muessen Sicherheitsupdates fuer die erwartete Produktlebensdauer bereitstellen.", "priority": 1},
|
||||
{"article": "Art. 10", "title": "Schwachstellenbehandlung", "description": "Vulnerability Handling", "requirement_text": "Hersteller muessen ein koordiniertes Schwachstellenmanagement implementieren.", "priority": 1},
|
||||
{"article": "Art. 11", "title": "Meldepflicht", "description": "Schwachstellen melden", "requirement_text": "Aktiv ausgenutzte Schwachstellen muessen innerhalb von 24 Stunden gemeldet werden.", "priority": 1},
|
||||
{"article": "Art. 13", "title": "SBOM", "description": "Software Bill of Materials", "requirement_text": "Eine SBOM muss fuer das Produkt erstellt und gepflegt werden.", "priority": 1},
|
||||
{"article": "Art. 15", "title": "Support-Zeitraum", "description": "Mindest-Support-Dauer", "requirement_text": "Mindestens 5 Jahre Support oder erwartete Produktlebensdauer.", "priority": 1},
|
||||
{"article": "Anhang I.1", "title": "Sichere Standardkonfiguration", "description": "Secure by Default", "requirement_text": "Produkte muessen mit sicheren Standardeinstellungen ausgeliefert werden.", "priority": 1},
|
||||
{"article": "Anhang I.2", "title": "Schutz vor unbefugtem Zugriff", "description": "Access Control", "requirement_text": "Mechanismen zum Schutz vor unbefugtem Zugriff implementieren.", "priority": 1},
|
||||
{"article": "Anhang I.3", "title": "Datenintegritaet", "description": "Integritaetsschutz", "requirement_text": "Schutz der Integritaet von Daten und Konfiguration.", "priority": 1},
|
||||
{"article": "Anhang I.4", "title": "Verfuegbarkeit", "description": "Resilienz", "requirement_text": "Schutz vor DoS-Angriffen und Sicherstellung der Verfuegbarkeit.", "priority": 1},
|
||||
]
|
||||
|
||||
elif code == "EPRIVACY":
|
||||
return [
|
||||
{"article": "Art. 5", "title": "Vertraulichkeit der Kommunikation", "description": "Kommunikation schuetzen", "requirement_text": "Vertraulichkeit der Kommunikation und Verkehrsdaten gewaehrleisten.", "priority": 1},
|
||||
{"article": "Art. 6", "title": "Verkehrsdaten", "description": "Umgang mit Verkehrsdaten", "requirement_text": "Verkehrsdaten muessen nach Abschluss geloescht oder anonymisiert werden.", "priority": 1},
|
||||
{"article": "Art. 9", "title": "Standortdaten", "description": "Nur mit Einwilligung", "requirement_text": "Standortdaten nur mit ausdruecklicher Einwilligung verarbeiten.", "priority": 1},
|
||||
{"article": "Art. 13", "title": "Unerbetene Nachrichten", "description": "Opt-in fuer Marketing", "requirement_text": "Direktwerbung per E-Mail nur mit vorheriger Einwilligung.", "priority": 1},
|
||||
]
|
||||
|
||||
elif code == "SCC":
|
||||
return [
|
||||
{"article": "Klausel 8", "title": "Datenschutzgarantien", "description": "Garantien dokumentieren", "requirement_text": "Datenimporteur muss angemessene Datenschutzgarantien gewaehrleisten.", "priority": 1},
|
||||
{"article": "Klausel 10", "title": "Betroffenenrechte", "description": "Rechte durchsetzen", "requirement_text": "Betroffene koennen ihre Rechte auch gegenueber Datenimporteur geltend machen.", "priority": 1},
|
||||
{"article": "Klausel 14", "title": "Lokale Rechtsvorschriften", "description": "Rechtslage pruefen", "requirement_text": "Parteien muessen pruefen, ob lokale Gesetze die Einhaltung verhindern.", "priority": 1},
|
||||
{"article": "Klausel 15", "title": "Behoerdenzugriff", "description": "Transparenz bei Anfragen", "requirement_text": "Datenimporteur muss ueber Behoerdenanfragen informieren.", "priority": 1},
|
||||
]
|
||||
|
||||
elif code == "DPF":
|
||||
return [
|
||||
{"article": "Prinzip 1", "title": "Notice", "description": "Informationspflicht", "requirement_text": "Betroffene muessen ueber Datenverarbeitung informiert werden.", "priority": 1},
|
||||
{"article": "Prinzip 2", "title": "Choice", "description": "Wahlmoeglichkeit", "requirement_text": "Betroffene muessen der Weitergabe widersprechen koennen.", "priority": 1},
|
||||
{"article": "Prinzip 4", "title": "Security", "description": "Sicherheitsmassnahmen", "requirement_text": "Angemessene Sicherheitsmassnahmen zum Schutz der Daten.", "priority": 1},
|
||||
{"article": "Prinzip 5", "title": "Data Integrity", "description": "Datenintegritaet", "requirement_text": "Daten muessen richtig, vollstaendig und aktuell sein.", "priority": 1},
|
||||
{"article": "Prinzip 6", "title": "Access", "description": "Auskunftsrecht", "requirement_text": "Betroffene haben Recht auf Zugang zu ihren Daten.", "priority": 1},
|
||||
]
|
||||
|
||||
return []
|
||||
|
||||
async def _scrape_bsi_pdf(self, code: str, source_info: Dict) -> int:
|
||||
"""
|
||||
Scrape BSI Technical Guideline PDF.
|
||||
|
||||
Note: Full PDF parsing requires PyMuPDF or pdfplumber.
|
||||
This is a placeholder that creates seed requirements.
|
||||
"""
|
||||
logger.info(f"Processing BSI TR: {code}")
|
||||
|
||||
# Get or create regulation
|
||||
regulation = self.reg_repo.get_by_code(code)
|
||||
if not regulation:
|
||||
regulation = RegulationDB(
|
||||
code=code,
|
||||
name=f"BSI {code}",
|
||||
full_name=f"BSI Technical Guideline {code}",
|
||||
regulation_type=source_info["regulation_type"],
|
||||
local_pdf_path=source_info["url"],
|
||||
is_active=True,
|
||||
)
|
||||
self.db.add(regulation)
|
||||
self.db.commit()
|
||||
self.db.refresh(regulation)
|
||||
|
||||
# Known BSI TR-03161 test aspects (Pruefaspekte)
|
||||
# These are the key security requirements from the TR
|
||||
bsi_aspects = self._get_bsi_aspects(code)
|
||||
|
||||
requirements_created = 0
|
||||
for aspect in bsi_aspects:
|
||||
existing = self.db.query(RequirementDB).filter(
|
||||
RequirementDB.regulation_id == regulation.id,
|
||||
RequirementDB.article == aspect["id"]
|
||||
).first()
|
||||
|
||||
if existing:
|
||||
continue
|
||||
|
||||
requirement = RequirementDB(
|
||||
regulation_id=regulation.id,
|
||||
article=aspect["id"],
|
||||
title=aspect["title"],
|
||||
description=aspect.get("description"),
|
||||
requirement_text=aspect.get("requirement_text"),
|
||||
breakpilot_interpretation=aspect.get("interpretation"),
|
||||
is_applicable=aspect.get("is_applicable", True),
|
||||
priority=aspect.get("priority", 2),
|
||||
source_page=aspect.get("page"),
|
||||
source_section=aspect.get("section"),
|
||||
)
|
||||
self.db.add(requirement)
|
||||
requirements_created += 1
|
||||
|
||||
self.db.commit()
|
||||
logger.info(f"Created {requirements_created} BSI requirements from {code}")
|
||||
return requirements_created
|
||||
|
||||
def _get_bsi_aspects(self, code: str) -> List[Dict[str, Any]]:
|
||||
"""
|
||||
Returns comprehensive BSI TR-03161 test aspects (Pruefaspekte).
|
||||
|
||||
These are the actual test aspects from BSI TR-03161:
|
||||
- Part 1: Allgemeine Anforderungen (~45 Aspekte)
|
||||
- Part 2: Web-Anwendungen (~40 Aspekte)
|
||||
- Part 3: Hintergrundsysteme (~35 Aspekte)
|
||||
|
||||
Total: ~120 Pruefaspekte
|
||||
"""
|
||||
if code == "BSI-TR-03161-1":
|
||||
# Teil 1: Allgemeine Anforderungen
|
||||
return [
|
||||
# Zweckbindung & Datenminimierung
|
||||
{"id": "O.Purp_1", "title": "Zweckbindung", "description": "Anwendungszweck klar definiert", "requirement_text": "Die Anwendung muss einen klar definierten und dokumentierten Zweck haben.", "priority": 1, "section": "4.1"},
|
||||
{"id": "O.Purp_2", "title": "Zweckdokumentation", "description": "Zweck fuer Nutzer einsehbar", "requirement_text": "Der Zweck muss fuer Nutzer transparent und einsehbar dokumentiert sein.", "priority": 2, "section": "4.1"},
|
||||
{"id": "O.Data_1", "title": "Datenminimierung", "description": "Nur notwendige Daten erheben", "requirement_text": "Es duerfen nur die fuer den definierten Zweck erforderlichen Daten erhoben werden.", "priority": 1, "section": "4.2"},
|
||||
{"id": "O.Data_2", "title": "Datenerforderlichkeit", "description": "Erforderlichkeit pruefen", "requirement_text": "Vor jeder Datenerhebung muss die Erforderlichkeit geprueft und dokumentiert werden.", "priority": 1, "section": "4.2"},
|
||||
{"id": "O.Data_3", "title": "Datenkategorien", "description": "Datenkategorien klassifizieren", "requirement_text": "Alle verarbeiteten Datenkategorien muessen klassifiziert und dokumentiert sein.", "priority": 2, "section": "4.2"},
|
||||
{"id": "O.Data_4", "title": "Besondere Kategorien", "description": "Art. 9 DSGVO Daten identifizieren", "requirement_text": "Besondere Kategorien personenbezogener Daten (Art. 9 DSGVO) muessen identifiziert und besonders geschuetzt werden.", "priority": 1, "section": "4.2"},
|
||||
|
||||
# Authentifizierung
|
||||
{"id": "O.Auth_1", "title": "Authentifizierungsmechanismus", "description": "Sichere Authentifizierung", "requirement_text": "Die Anwendung muss sichere Authentifizierungsmechanismen implementieren.", "priority": 1, "section": "4.3"},
|
||||
{"id": "O.Auth_2", "title": "Passwortrichtlinie", "description": "Starke Passwoerter erzwingen", "requirement_text": "Passwortrichtlinien muessen Mindestlaenge (12 Zeichen), Komplexitaet und Historie durchsetzen.", "priority": 1, "section": "4.3"},
|
||||
{"id": "O.Auth_3", "title": "Passwort-Hashing", "description": "Sichere Hash-Algorithmen", "requirement_text": "Passwoerter muessen mit aktuellen Algorithmen (bcrypt, Argon2) gehasht werden.", "priority": 1, "section": "4.3"},
|
||||
{"id": "O.Auth_4", "title": "Multi-Faktor-Authentifizierung", "description": "MFA fuer sensitive Bereiche", "requirement_text": "Fuer administrative und sensitive Funktionen muss MFA verfuegbar sein.", "priority": 1, "section": "4.3"},
|
||||
{"id": "O.Auth_5", "title": "Brute-Force-Schutz", "description": "Rate Limiting bei Login", "requirement_text": "Nach mehreren fehlgeschlagenen Anmeldeversuchen muss Account-Lockout oder Rate-Limiting greifen.", "priority": 1, "section": "4.3"},
|
||||
{"id": "O.Auth_6", "title": "Sichere Passwort-Wiederherstellung", "description": "Reset-Prozess absichern", "requirement_text": "Der Passwort-Reset-Prozess muss gegen Enumeration und Manipulation geschuetzt sein.", "priority": 1, "section": "4.3"},
|
||||
|
||||
# Autorisierung
|
||||
{"id": "O.Authz_1", "title": "Zugriffskontrolle", "description": "Rollenbasierte Zugriffskontrolle", "requirement_text": "Ein rollenbasiertes Zugriffskonzept (RBAC) muss implementiert sein.", "priority": 1, "section": "4.4"},
|
||||
{"id": "O.Authz_2", "title": "Least Privilege", "description": "Minimale Rechte", "requirement_text": "Benutzer sollen nur die minimal notwendigen Berechtigungen erhalten.", "priority": 1, "section": "4.4"},
|
||||
{"id": "O.Authz_3", "title": "Rechtetrennung", "description": "Funktionale Trennung", "requirement_text": "Administrative und operative Rollen muessen getrennt sein.", "priority": 1, "section": "4.4"},
|
||||
{"id": "O.Authz_4", "title": "Autorisierungspruefung", "description": "Serverseitige Pruefung", "requirement_text": "Jede Ressource muss serverseitig auf Zugriffsberechtigung geprueft werden.", "priority": 1, "section": "4.4"},
|
||||
|
||||
# Kryptografie
|
||||
{"id": "O.Cryp_1", "title": "TLS-Verschluesselung", "description": "TLS 1.2+ fuer Transport", "requirement_text": "Alle Daten muessen bei der Uebertragung mit TLS 1.2 oder hoeher verschluesselt werden.", "priority": 1, "section": "4.5"},
|
||||
{"id": "O.Cryp_2", "title": "Verschluesselung at Rest", "description": "Sensible Daten verschluesseln", "requirement_text": "Sensible Daten muessen bei der Speicherung verschluesselt werden (AES-256 oder vergleichbar).", "priority": 1, "section": "4.5"},
|
||||
{"id": "O.Cryp_3", "title": "HSTS", "description": "HTTP Strict Transport Security", "requirement_text": "HSTS-Header muessen gesetzt sein um HTTPS zu erzwingen.", "priority": 1, "section": "4.5"},
|
||||
{"id": "O.Cryp_4", "title": "Zertifikatvalidierung", "description": "Zertifikate pruefen", "requirement_text": "TLS-Zertifikate muessen vollstaendig validiert werden (Chain, Revocation, Hostname).", "priority": 1, "section": "4.5"},
|
||||
{"id": "O.Cryp_5", "title": "Key Management", "description": "Sichere Schluesselverwaltung", "requirement_text": "Kryptographische Schluessel muessen sicher generiert, gespeichert und rotiert werden.", "priority": 1, "section": "4.5"},
|
||||
{"id": "O.Cryp_6", "title": "Aktuelle Algorithmen", "description": "Keine veralteten Algorithmen", "requirement_text": "Es duerfen nur aktuelle, als sicher geltende kryptographische Algorithmen verwendet werden.", "priority": 1, "section": "4.5"},
|
||||
|
||||
# Datenschutz
|
||||
{"id": "O.Priv_1", "title": "Datenschutzerklaerung", "description": "Transparente Information", "requirement_text": "Eine vollstaendige Datenschutzerklaerung muss vor Nutzung einsehbar sein.", "priority": 1, "section": "4.6"},
|
||||
{"id": "O.Priv_2", "title": "Einwilligung", "description": "Wirksame Einwilligung", "requirement_text": "Einwilligungen muessen freiwillig, informiert, spezifisch und dokumentiert sein.", "priority": 1, "section": "4.6"},
|
||||
{"id": "O.Priv_3", "title": "Betroffenenrechte", "description": "Auskunft, Loeschung, etc.", "requirement_text": "Technische Prozesse fuer Betroffenenrechte (Art. 15-21 DSGVO) muessen implementiert sein.", "priority": 1, "section": "4.6"},
|
||||
{"id": "O.Priv_4", "title": "Loeschkonzept", "description": "Aufbewahrungsfristen", "requirement_text": "Ein dokumentiertes Loeschkonzept mit definierten Aufbewahrungsfristen muss umgesetzt sein.", "priority": 1, "section": "4.6"},
|
||||
{"id": "O.Priv_5", "title": "Datenschutz durch Technik", "description": "Privacy by Design", "requirement_text": "Datenschutz muss bereits bei der Entwicklung beruecksichtigt werden (Art. 25 DSGVO).", "priority": 1, "section": "4.6"},
|
||||
|
||||
# Logging & Audit
|
||||
{"id": "O.Log_1", "title": "Security Logging", "description": "Sicherheitsereignisse protokollieren", "requirement_text": "Sicherheitsrelevante Ereignisse (Login, Fehler, Zugriffsverletzungen) muessen protokolliert werden.", "priority": 1, "section": "4.7"},
|
||||
{"id": "O.Log_2", "title": "Audit Trail", "description": "Nachvollziehbarkeit", "requirement_text": "Aenderungen an personenbezogenen Daten muessen nachvollziehbar protokolliert werden.", "priority": 1, "section": "4.7"},
|
||||
{"id": "O.Log_3", "title": "Log-Integritaet", "description": "Logs vor Manipulation schuetzen", "requirement_text": "Logs muessen vor unbefugter Aenderung oder Loeschung geschuetzt sein.", "priority": 2, "section": "4.7"},
|
||||
{"id": "O.Log_4", "title": "Keine PII in Logs", "description": "Keine personenbezogenen Daten loggen", "requirement_text": "Logs duerfen keine personenbezogenen Daten im Klartext enthalten.", "priority": 1, "section": "4.7"},
|
||||
|
||||
# Software-Entwicklung
|
||||
{"id": "O.Dev_1", "title": "Secure SDLC", "description": "Sicherer Entwicklungsprozess", "requirement_text": "Ein dokumentierter sicherer Entwicklungsprozess (Secure SDLC) muss etabliert sein.", "priority": 1, "section": "4.8"},
|
||||
{"id": "O.Dev_2", "title": "Code Review", "description": "Sicherheits-Review von Code", "requirement_text": "Sicherheitsrelevanter Code muss vor Release einem Review unterzogen werden.", "priority": 2, "section": "4.8"},
|
||||
{"id": "O.Dev_3", "title": "Dependency Management", "description": "Abhaengigkeiten pruefen", "requirement_text": "Externe Abhaengigkeiten muessen auf bekannte Schwachstellen geprueft werden.", "priority": 1, "section": "4.8"},
|
||||
{"id": "O.Dev_4", "title": "Penetration Testing", "description": "Regelmaessige Sicherheitstests", "requirement_text": "Regelmaessige Penetrationstests oder Schwachstellenscans muessen durchgefuehrt werden.", "priority": 2, "section": "4.8"},
|
||||
|
||||
# Betrieb
|
||||
{"id": "O.Ops_1", "title": "Patch Management", "description": "Zeitnahes Patchen", "requirement_text": "Sicherheitspatches muessen zeitnah (kritisch: 24-72h) eingespielt werden.", "priority": 1, "section": "4.9"},
|
||||
{"id": "O.Ops_2", "title": "Backup", "description": "Regelmaessige Datensicherung", "requirement_text": "Regelmaessige, getestete Backups muessen vorhanden sein.", "priority": 1, "section": "4.9"},
|
||||
{"id": "O.Ops_3", "title": "Incident Response", "description": "Vorfallsmanagement", "requirement_text": "Ein dokumentierter Incident-Response-Prozess muss etabliert sein.", "priority": 1, "section": "4.9"},
|
||||
{"id": "O.Ops_4", "title": "Monitoring", "description": "Systemueberwachung", "requirement_text": "Kritische Systeme und Dienste muessen kontinuierlich ueberwacht werden.", "priority": 2, "section": "4.9"},
|
||||
|
||||
# Dokumentation
|
||||
{"id": "O.Doc_1", "title": "Technische Dokumentation", "description": "Systemarchitektur dokumentiert", "requirement_text": "Die Systemarchitektur und Datenflüsse muessen dokumentiert sein.", "priority": 2, "section": "4.10"},
|
||||
{"id": "O.Doc_2", "title": "Verarbeitungsverzeichnis", "description": "Art. 30 DSGVO", "requirement_text": "Ein vollstaendiges Verzeichnis von Verarbeitungstaetigkeiten muss gefuehrt werden.", "priority": 1, "section": "4.10"},
|
||||
{"id": "O.Doc_3", "title": "TOMs", "description": "Technisch-organisatorische Massnahmen", "requirement_text": "Technisch-organisatorische Massnahmen (Art. 32 DSGVO) muessen dokumentiert sein.", "priority": 1, "section": "4.10"},
|
||||
]
|
||||
|
||||
elif code == "BSI-TR-03161-2":
|
||||
# Teil 2: Web-Anwendungen
|
||||
return [
|
||||
# Session Management
|
||||
{"id": "O.Sess_1", "title": "Session-Timeout", "description": "Automatische Sitzungsbeendigung", "requirement_text": "Sessions muessen nach Inaktivitaet automatisch beendet werden (max. 30 Min).", "priority": 1, "section": "5.1"},
|
||||
{"id": "O.Sess_2", "title": "Session-ID Sicherheit", "description": "Sichere Session-IDs", "requirement_text": "Session-IDs muessen kryptographisch sicher generiert werden (min. 128 Bit Entropie).", "priority": 1, "section": "5.1"},
|
||||
{"id": "O.Sess_3", "title": "Session-Regeneration", "description": "ID nach Login erneuern", "requirement_text": "Nach erfolgreicher Authentifizierung muss eine neue Session-ID generiert werden.", "priority": 1, "section": "5.1"},
|
||||
{"id": "O.Sess_4", "title": "Secure Cookie Flags", "description": "HttpOnly, Secure, SameSite", "requirement_text": "Session-Cookies muessen mit Secure, HttpOnly und SameSite-Flags gesetzt werden.", "priority": 1, "section": "5.1"},
|
||||
{"id": "O.Sess_5", "title": "Session-Binding", "description": "Session an Client binden", "requirement_text": "Sessions sollten an Client-Eigenschaften (User-Agent, IP) gebunden werden.", "priority": 2, "section": "5.1"},
|
||||
{"id": "O.Sess_6", "title": "Logout-Funktionalitaet", "description": "Vollstaendiges Logout", "requirement_text": "Beim Logout muss die Session vollstaendig invalidiert werden.", "priority": 1, "section": "5.1"},
|
||||
|
||||
# Eingabevalidierung
|
||||
{"id": "O.Input_1", "title": "Serverseitige Validierung", "description": "Alle Eingaben serverseitig pruefen", "requirement_text": "Alle Benutzereingaben muessen serverseitig validiert werden.", "priority": 1, "section": "5.2"},
|
||||
{"id": "O.Input_2", "title": "Whitelist-Validierung", "description": "Erlaubte Zeichen definieren", "requirement_text": "Eingabevalidierung sollte auf Whitelist-Basis erfolgen.", "priority": 1, "section": "5.2"},
|
||||
{"id": "O.Input_3", "title": "Encoding", "description": "Korrekte Zeichenkodierung", "requirement_text": "Einheitliche Zeichenkodierung (UTF-8) muss durchgesetzt werden.", "priority": 2, "section": "5.2"},
|
||||
{"id": "O.Input_4", "title": "Datei-Upload Validierung", "description": "Uploads pruefen", "requirement_text": "Datei-Uploads muessen auf Typ, Groesse und Inhalt validiert werden.", "priority": 1, "section": "5.2"},
|
||||
|
||||
# Injection-Schutz
|
||||
{"id": "O.SQL_1", "title": "SQL-Injection Schutz", "description": "Prepared Statements", "requirement_text": "SQL-Anfragen muessen parametrisiert sein (Prepared Statements).", "priority": 1, "section": "5.3"},
|
||||
{"id": "O.SQL_2", "title": "ORM Nutzung", "description": "Abstraktionsschicht nutzen", "requirement_text": "Es sollte ein ORM oder Query Builder verwendet werden.", "priority": 2, "section": "5.3"},
|
||||
{"id": "O.Cmd_1", "title": "Command Injection Schutz", "description": "Keine Shell-Befehle mit Eingaben", "requirement_text": "Benutzereingaben duerfen nicht in Shell-Befehlen verwendet werden.", "priority": 1, "section": "5.3"},
|
||||
{"id": "O.LDAP_1", "title": "LDAP Injection Schutz", "description": "LDAP-Queries absichern", "requirement_text": "LDAP-Queries muessen gegen Injection geschuetzt sein.", "priority": 1, "section": "5.3"},
|
||||
{"id": "O.XML_1", "title": "XML Injection Schutz", "description": "XXE verhindern", "requirement_text": "XML-Parser muessen gegen XXE-Angriffe konfiguriert sein.", "priority": 1, "section": "5.3"},
|
||||
|
||||
# XSS-Schutz
|
||||
{"id": "O.XSS_1", "title": "Output Encoding", "description": "Kontextabhaengiges Escaping", "requirement_text": "Ausgaben muessen kontextabhaengig (HTML, JS, CSS, URL) escaped werden.", "priority": 1, "section": "5.4"},
|
||||
{"id": "O.XSS_2", "title": "Content Security Policy", "description": "CSP-Header setzen", "requirement_text": "Ein restriktiver Content-Security-Policy-Header muss gesetzt sein.", "priority": 1, "section": "5.4"},
|
||||
{"id": "O.XSS_3", "title": "DOM-basiertes XSS", "description": "DOM-Manipulation absichern", "requirement_text": "JavaScript-DOM-Manipulationen muessen sicher implementiert sein.", "priority": 1, "section": "5.4"},
|
||||
{"id": "O.XSS_4", "title": "Template-Engine Escaping", "description": "Auto-Escaping aktivieren", "requirement_text": "Template-Engines muessen mit aktiviertem Auto-Escaping verwendet werden.", "priority": 1, "section": "5.4"},
|
||||
|
||||
# CSRF-Schutz
|
||||
{"id": "O.CSRF_1", "title": "Anti-CSRF Token", "description": "Token bei State-Changes", "requirement_text": "Zustandsaendernde Anfragen muessen mit Anti-CSRF-Token geschuetzt sein.", "priority": 1, "section": "5.5"},
|
||||
{"id": "O.CSRF_2", "title": "SameSite Cookie", "description": "SameSite-Attribut setzen", "requirement_text": "Cookies sollten das SameSite-Attribut (Strict oder Lax) haben.", "priority": 1, "section": "5.5"},
|
||||
{"id": "O.CSRF_3", "title": "Referer-Pruefung", "description": "Origin validieren", "requirement_text": "Bei kritischen Aktionen sollte der Origin/Referer-Header geprueft werden.", "priority": 2, "section": "5.5"},
|
||||
|
||||
# Security Headers
|
||||
{"id": "O.Head_1", "title": "X-Content-Type-Options", "description": "nosniff setzen", "requirement_text": "Der X-Content-Type-Options: nosniff Header muss gesetzt sein.", "priority": 1, "section": "5.6"},
|
||||
{"id": "O.Head_2", "title": "X-Frame-Options", "description": "Clickjacking-Schutz", "requirement_text": "X-Frame-Options oder CSP frame-ancestors muss Clickjacking verhindern.", "priority": 1, "section": "5.6"},
|
||||
{"id": "O.Head_3", "title": "X-XSS-Protection", "description": "Browser XSS-Filter", "requirement_text": "X-XSS-Protection sollte aktiviert sein (oder CSP nutzen).", "priority": 2, "section": "5.6"},
|
||||
{"id": "O.Head_4", "title": "Referrer-Policy", "description": "Referrer einschraenken", "requirement_text": "Eine restriktive Referrer-Policy sollte gesetzt sein.", "priority": 2, "section": "5.6"},
|
||||
{"id": "O.Head_5", "title": "Permissions-Policy", "description": "Browser-Features einschraenken", "requirement_text": "Nicht benoetigte Browser-APIs sollten per Permissions-Policy deaktiviert werden.", "priority": 3, "section": "5.6"},
|
||||
|
||||
# Fehlerbehandlung
|
||||
{"id": "O.Err_1", "title": "Generische Fehlermeldungen", "description": "Keine technischen Details", "requirement_text": "Fehlermeldungen an Benutzer duerfen keine technischen Details enthalten.", "priority": 1, "section": "5.7"},
|
||||
{"id": "O.Err_2", "title": "Custom Error Pages", "description": "Eigene Fehlerseiten", "requirement_text": "Standard-Fehlerseiten des Servers muessen durch eigene ersetzt werden.", "priority": 2, "section": "5.7"},
|
||||
{"id": "O.Err_3", "title": "Exception Handling", "description": "Alle Exceptions abfangen", "requirement_text": "Unbehandelte Exceptions muessen abgefangen und geloggt werden.", "priority": 1, "section": "5.7"},
|
||||
|
||||
# API-Sicherheit
|
||||
{"id": "O.API_1", "title": "API-Authentifizierung", "description": "API-Keys oder OAuth", "requirement_text": "APIs muessen authentifiziert werden (API-Keys, OAuth, JWT).", "priority": 1, "section": "5.8"},
|
||||
{"id": "O.API_2", "title": "Rate Limiting", "description": "Anfragen begrenzen", "requirement_text": "APIs muessen Rate-Limiting implementieren.", "priority": 1, "section": "5.8"},
|
||||
{"id": "O.API_3", "title": "Input-Validierung API", "description": "Request-Body validieren", "requirement_text": "API-Request-Bodies muessen gegen ein Schema validiert werden.", "priority": 1, "section": "5.8"},
|
||||
{"id": "O.API_4", "title": "Versionierung", "description": "API-Versionen", "requirement_text": "APIs sollten versioniert sein um Breaking Changes zu vermeiden.", "priority": 3, "section": "5.8"},
|
||||
|
||||
# Client-Sicherheit
|
||||
{"id": "O.JS_1", "title": "JavaScript Sicherheit", "description": "Sichere JS-Praktiken", "requirement_text": "JavaScript muss sicher implementiert sein (kein eval, innerHTML mit Vorsicht).", "priority": 1, "section": "5.9"},
|
||||
{"id": "O.JS_2", "title": "Third-Party Scripts", "description": "Externe Scripts absichern", "requirement_text": "Third-Party Scripts muessen mit SRI oder CSP abgesichert werden.", "priority": 1, "section": "5.9"},
|
||||
{"id": "O.Store_1", "title": "Lokale Speicherung", "description": "LocalStorage sicher nutzen", "requirement_text": "Sensible Daten duerfen nicht im LocalStorage/SessionStorage gespeichert werden.", "priority": 1, "section": "5.9"},
|
||||
]
|
||||
|
||||
elif code == "BSI-TR-03161-3":
|
||||
# Teil 3: Hintergrundsysteme (Backend)
|
||||
return [
|
||||
# Systemarchitektur
|
||||
{"id": "O.Arch_1", "title": "Defense in Depth", "description": "Mehrschichtige Sicherheit", "requirement_text": "Eine mehrschichtige Sicherheitsarchitektur (Defense in Depth) muss implementiert sein.", "priority": 1, "section": "6.1"},
|
||||
{"id": "O.Arch_2", "title": "Segmentierung", "description": "Netzwerksegmentierung", "requirement_text": "Das Netzwerk muss segmentiert sein (DMZ, interne Zonen).", "priority": 1, "section": "6.1"},
|
||||
{"id": "O.Arch_3", "title": "Microservices Isolation", "description": "Services isolieren", "requirement_text": "Microservices sollten minimal gekoppelt und isoliert sein.", "priority": 2, "section": "6.1"},
|
||||
{"id": "O.Arch_4", "title": "Zero Trust", "description": "Kein implizites Vertrauen", "requirement_text": "Interne Kommunikation sollte nach Zero-Trust-Prinzipien abgesichert sein.", "priority": 2, "section": "6.1"},
|
||||
|
||||
# Datenspeicherung
|
||||
{"id": "O.DB_1", "title": "Datenbank-Sicherheit", "description": "DB abhaerten", "requirement_text": "Datenbanken muessen gehaertet sein (keine Default-Credentials, minimale Rechte).", "priority": 1, "section": "6.2"},
|
||||
{"id": "O.DB_2", "title": "Verschluesselung in DB", "description": "Sensible Felder verschluesseln", "requirement_text": "Sensible Daten sollten in der Datenbank verschluesselt gespeichert werden.", "priority": 1, "section": "6.2"},
|
||||
{"id": "O.DB_3", "title": "Backup-Verschluesselung", "description": "Backups verschluesseln", "requirement_text": "Datenbank-Backups muessen verschluesselt sein.", "priority": 1, "section": "6.2"},
|
||||
{"id": "O.DB_4", "title": "Zugriffskontrolle DB", "description": "DB-Zugriffe beschraenken", "requirement_text": "Der Datenbankzugriff muss auf notwendige Dienste beschraenkt sein.", "priority": 1, "section": "6.2"},
|
||||
{"id": "O.Store_2", "title": "Dateispeicher-Sicherheit", "description": "Uploads isolieren", "requirement_text": "Hochgeladene Dateien muessen isoliert und mit Malware-Scanning verarbeitet werden.", "priority": 1, "section": "6.2"},
|
||||
|
||||
# Container & Infrastruktur
|
||||
{"id": "O.Cont_1", "title": "Container-Sicherheit", "description": "Images scannen", "requirement_text": "Container-Images muessen auf Schwachstellen gescannt werden.", "priority": 1, "section": "6.3"},
|
||||
{"id": "O.Cont_2", "title": "Rootless Container", "description": "Nicht als Root laufen", "requirement_text": "Container sollten nicht als Root-User ausgefuehrt werden.", "priority": 1, "section": "6.3"},
|
||||
{"id": "O.Cont_3", "title": "Image-Herkunft", "description": "Vertrauenswuerdige Images", "requirement_text": "Es duerfen nur Images aus vertrauenswuerdigen Quellen verwendet werden.", "priority": 1, "section": "6.3"},
|
||||
{"id": "O.Cont_4", "title": "Read-Only Filesystem", "description": "Unveraenderliches Dateisystem", "requirement_text": "Container sollten mit Read-Only Root-Filesystem laufen wo moeglich.", "priority": 2, "section": "6.3"},
|
||||
{"id": "O.Cont_5", "title": "Resource Limits", "description": "CPU/Memory begrenzen", "requirement_text": "Container muessen Resource-Limits (CPU, Memory) konfiguriert haben.", "priority": 2, "section": "6.3"},
|
||||
|
||||
# Secrets Management
|
||||
{"id": "O.Sec_1", "title": "Secrets Management", "description": "Zentrale Secrets-Verwaltung", "requirement_text": "Sensible Konfiguration (Passwoerter, Keys) muss zentral und sicher verwaltet werden.", "priority": 1, "section": "6.4"},
|
||||
{"id": "O.Sec_2", "title": "Keine Hardcoded Secrets", "description": "Secrets nicht im Code", "requirement_text": "Secrets duerfen nicht im Quellcode oder in Git-Repositories stehen.", "priority": 1, "section": "6.4"},
|
||||
{"id": "O.Sec_3", "title": "Secret Rotation", "description": "Regelmaessige Rotation", "requirement_text": "Secrets und API-Keys sollten regelmaessig rotiert werden.", "priority": 2, "section": "6.4"},
|
||||
{"id": "O.Sec_4", "title": "Vault Integration", "description": "Secrets-Vault nutzen", "requirement_text": "Ein Secrets-Management-System (HashiCorp Vault o.ae.) sollte verwendet werden.", "priority": 2, "section": "6.4"},
|
||||
|
||||
# Kommunikation
|
||||
{"id": "O.Comm_1", "title": "Service-to-Service TLS", "description": "Interne Verschluesselung", "requirement_text": "Auch interne Service-Kommunikation sollte verschluesselt sein (mTLS).", "priority": 2, "section": "6.5"},
|
||||
{"id": "O.Comm_2", "title": "Message Queue Sicherheit", "description": "Queue-Zugriff absichern", "requirement_text": "Message Queues muessen authentifiziert und autorisiert werden.", "priority": 1, "section": "6.5"},
|
||||
{"id": "O.Comm_3", "title": "API Gateway", "description": "Zentraler Zugangspunkt", "requirement_text": "Ein API Gateway sollte als zentraler Zugangspunkt dienen.", "priority": 2, "section": "6.5"},
|
||||
|
||||
# Monitoring & Logging
|
||||
{"id": "O.Mon_1", "title": "Zentrale Logs", "description": "Log-Aggregation", "requirement_text": "Logs aller Services muessen zentral aggregiert werden.", "priority": 1, "section": "6.6"},
|
||||
{"id": "O.Mon_2", "title": "Security Monitoring", "description": "Anomalie-Erkennung", "requirement_text": "Sicherheitsrelevante Ereignisse muessen ueberwacht und alarmiert werden.", "priority": 1, "section": "6.6"},
|
||||
{"id": "O.Mon_3", "title": "Metriken", "description": "Performance-Monitoring", "requirement_text": "System-Metriken (CPU, Memory, Latenz) sollten erfasst und ueberwacht werden.", "priority": 2, "section": "6.6"},
|
||||
{"id": "O.Mon_4", "title": "Alerting", "description": "Alarmierung konfigurieren", "requirement_text": "Kritische Schwellwerte muessen definiert und alarmiert werden.", "priority": 1, "section": "6.6"},
|
||||
|
||||
# CI/CD Sicherheit
|
||||
{"id": "O.CI_1", "title": "Pipeline-Sicherheit", "description": "CI/CD absichern", "requirement_text": "CI/CD-Pipelines muessen abgesichert sein (Secrets, Zugriffsrechte).", "priority": 1, "section": "6.7"},
|
||||
{"id": "O.CI_2", "title": "SAST/DAST", "description": "Automatisierte Security-Tests", "requirement_text": "Statische und dynamische Sicherheitsanalysen sollten in die Pipeline integriert sein.", "priority": 2, "section": "6.7"},
|
||||
{"id": "O.CI_3", "title": "Dependency Scanning", "description": "Abhaengigkeiten pruefen", "requirement_text": "Abhaengigkeiten muessen automatisiert auf Schwachstellen geprueft werden.", "priority": 1, "section": "6.7"},
|
||||
{"id": "O.CI_4", "title": "SBOM", "description": "Software Bill of Materials", "requirement_text": "Ein SBOM (Software Bill of Materials) sollte generiert und gepflegt werden.", "priority": 2, "section": "6.7"},
|
||||
|
||||
# Disaster Recovery
|
||||
{"id": "O.DR_1", "title": "Backup-Strategie", "description": "3-2-1 Backup-Regel", "requirement_text": "Backups sollten der 3-2-1-Regel folgen (3 Kopien, 2 Medien, 1 offsite).", "priority": 1, "section": "6.8"},
|
||||
{"id": "O.DR_2", "title": "Recovery-Tests", "description": "Restore regelmaessig testen", "requirement_text": "Die Wiederherstellung aus Backups muss regelmaessig getestet werden.", "priority": 1, "section": "6.8"},
|
||||
{"id": "O.DR_3", "title": "RTO/RPO", "description": "Recovery-Ziele definieren", "requirement_text": "Recovery Time Objective (RTO) und Recovery Point Objective (RPO) muessen definiert sein.", "priority": 2, "section": "6.8"},
|
||||
]
|
||||
|
||||
return []
|
||||
|
||||
def get_known_sources(self) -> List[Dict[str, Any]]:
|
||||
"""Get list of known regulation sources with metadata."""
|
||||
sources = []
|
||||
for code, info in self.KNOWN_SOURCES.items():
|
||||
# Check database for existing data
|
||||
regulation = self.reg_repo.get_by_code(code)
|
||||
requirement_count = 0
|
||||
if regulation:
|
||||
requirement_count = self.db.query(RequirementDB).filter(
|
||||
RequirementDB.regulation_id == regulation.id
|
||||
).count()
|
||||
|
||||
sources.append({
|
||||
"code": code,
|
||||
"url": info["url"],
|
||||
"source_type": info["type"].value,
|
||||
"regulation_type": info["regulation_type"].value,
|
||||
"has_data": regulation is not None,
|
||||
"requirement_count": requirement_count,
|
||||
})
|
||||
|
||||
return sources
|
||||
442
backend/compliance/services/report_generator.py
Normal file
442
backend/compliance/services/report_generator.py
Normal file
@@ -0,0 +1,442 @@
|
||||
"""
|
||||
Compliance Report Generator Service.
|
||||
|
||||
Generates periodic compliance reports (weekly, monthly, quarterly, yearly).
|
||||
Reports include:
|
||||
- Compliance score trends
|
||||
- Control status summary
|
||||
- Risk assessment summary
|
||||
- Evidence coverage
|
||||
- Action items / recommendations
|
||||
"""
|
||||
|
||||
import logging
|
||||
from datetime import datetime, date, timedelta
|
||||
from typing import Dict, List, Any, Optional
|
||||
from enum import Enum
|
||||
|
||||
from sqlalchemy.orm import Session
|
||||
from sqlalchemy import func
|
||||
|
||||
from ..db.models import (
|
||||
RegulationDB,
|
||||
RequirementDB,
|
||||
ControlDB,
|
||||
ControlMappingDB,
|
||||
EvidenceDB,
|
||||
RiskDB,
|
||||
AuditExportDB,
|
||||
ControlStatusEnum,
|
||||
RiskLevelEnum,
|
||||
EvidenceStatusEnum,
|
||||
)
|
||||
from ..db.repository import (
|
||||
RegulationRepository,
|
||||
ControlRepository,
|
||||
EvidenceRepository,
|
||||
RiskRepository,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class ReportPeriod(str, Enum):
|
||||
WEEKLY = "weekly"
|
||||
MONTHLY = "monthly"
|
||||
QUARTERLY = "quarterly"
|
||||
YEARLY = "yearly"
|
||||
|
||||
|
||||
class ComplianceReportGenerator:
|
||||
"""Generates compliance reports for different time periods."""
|
||||
|
||||
def __init__(self, db: Session):
|
||||
self.db = db
|
||||
self.reg_repo = RegulationRepository(db)
|
||||
self.ctrl_repo = ControlRepository(db)
|
||||
self.evidence_repo = EvidenceRepository(db)
|
||||
self.risk_repo = RiskRepository(db)
|
||||
|
||||
def generate_report(
|
||||
self,
|
||||
period: ReportPeriod,
|
||||
as_of_date: Optional[date] = None,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Generate a compliance report for the specified period.
|
||||
|
||||
Args:
|
||||
period: Report period (weekly, monthly, quarterly, yearly)
|
||||
as_of_date: Report date (defaults to today)
|
||||
|
||||
Returns:
|
||||
Complete report dictionary
|
||||
"""
|
||||
if as_of_date is None:
|
||||
as_of_date = date.today()
|
||||
|
||||
# Calculate date ranges
|
||||
date_range = self._get_date_range(period, as_of_date)
|
||||
|
||||
report = {
|
||||
"report_metadata": {
|
||||
"generated_at": datetime.utcnow().isoformat(),
|
||||
"period": period.value,
|
||||
"as_of_date": as_of_date.isoformat(),
|
||||
"date_range_start": date_range["start"].isoformat(),
|
||||
"date_range_end": date_range["end"].isoformat(),
|
||||
"report_title": self._get_report_title(period, as_of_date),
|
||||
},
|
||||
"executive_summary": self._generate_executive_summary(),
|
||||
"compliance_score": self._generate_compliance_score_section(),
|
||||
"regulations_coverage": self._generate_regulations_coverage(),
|
||||
"controls_summary": self._generate_controls_summary(),
|
||||
"risks_summary": self._generate_risks_summary(),
|
||||
"evidence_summary": self._generate_evidence_summary(),
|
||||
"action_items": self._generate_action_items(),
|
||||
"trends": self._generate_trends_placeholder(period),
|
||||
}
|
||||
|
||||
return report
|
||||
|
||||
def _get_date_range(self, period: ReportPeriod, as_of: date) -> Dict[str, date]:
|
||||
"""Calculate date range for the reporting period."""
|
||||
if period == ReportPeriod.WEEKLY:
|
||||
# Last 7 days
|
||||
start = as_of - timedelta(days=7)
|
||||
elif period == ReportPeriod.MONTHLY:
|
||||
# Last 30 days
|
||||
start = as_of - timedelta(days=30)
|
||||
elif period == ReportPeriod.QUARTERLY:
|
||||
# Last 90 days
|
||||
start = as_of - timedelta(days=90)
|
||||
elif period == ReportPeriod.YEARLY:
|
||||
# Last 365 days
|
||||
start = as_of - timedelta(days=365)
|
||||
else:
|
||||
start = as_of - timedelta(days=30)
|
||||
|
||||
return {"start": start, "end": as_of}
|
||||
|
||||
def _get_report_title(self, period: ReportPeriod, as_of: date) -> str:
|
||||
"""Generate report title based on period."""
|
||||
titles = {
|
||||
ReportPeriod.WEEKLY: f"Woechentlicher Compliance-Report KW{as_of.isocalendar()[1]} {as_of.year}",
|
||||
ReportPeriod.MONTHLY: f"Monatlicher Compliance-Report {as_of.strftime('%B %Y')}",
|
||||
ReportPeriod.QUARTERLY: f"Quartals-Compliance-Report Q{(as_of.month - 1) // 3 + 1}/{as_of.year}",
|
||||
ReportPeriod.YEARLY: f"Jaehrlicher Compliance-Report {as_of.year}",
|
||||
}
|
||||
return titles.get(period, f"Compliance Report {as_of.isoformat()}")
|
||||
|
||||
def _generate_executive_summary(self) -> Dict[str, Any]:
|
||||
"""Generate executive summary section."""
|
||||
stats = self.ctrl_repo.get_statistics()
|
||||
risk_matrix = self.risk_repo.get_matrix_data()
|
||||
|
||||
total_controls = stats.get("total", 0)
|
||||
score = stats.get("compliance_score", 0)
|
||||
|
||||
# Determine overall status
|
||||
if score >= 80:
|
||||
status = "GREEN"
|
||||
status_text = "Guter Compliance-Stand"
|
||||
elif score >= 60:
|
||||
status = "YELLOW"
|
||||
status_text = "Verbesserungsbedarf"
|
||||
else:
|
||||
status = "RED"
|
||||
status_text = "Kritischer Handlungsbedarf"
|
||||
|
||||
high_critical_risks = (
|
||||
risk_matrix["by_level"].get("critical", 0) +
|
||||
risk_matrix["by_level"].get("high", 0)
|
||||
)
|
||||
|
||||
return {
|
||||
"overall_status": status,
|
||||
"status_text": status_text,
|
||||
"compliance_score": score,
|
||||
"total_controls": total_controls,
|
||||
"high_critical_risks": high_critical_risks,
|
||||
"key_findings": self._generate_key_findings(stats, risk_matrix),
|
||||
}
|
||||
|
||||
def _generate_key_findings(
|
||||
self,
|
||||
ctrl_stats: Dict[str, Any],
|
||||
risk_matrix: Dict[str, Any],
|
||||
) -> List[str]:
|
||||
"""Generate key findings for executive summary."""
|
||||
findings = []
|
||||
|
||||
# Control status findings
|
||||
by_status = ctrl_stats.get("by_status", {})
|
||||
passed = by_status.get("pass", 0)
|
||||
failed = by_status.get("fail", 0)
|
||||
planned = by_status.get("planned", 0)
|
||||
|
||||
if failed > 0:
|
||||
findings.append(f"{failed} Controls im Status 'Fail' - sofortige Massnahmen erforderlich")
|
||||
|
||||
if planned > 5:
|
||||
findings.append(f"{planned} Controls noch nicht implementiert")
|
||||
|
||||
# Risk findings
|
||||
critical = risk_matrix["by_level"].get("critical", 0)
|
||||
high = risk_matrix["by_level"].get("high", 0)
|
||||
|
||||
if critical > 0:
|
||||
findings.append(f"{critical} kritische Risiken identifiziert - Eskalation empfohlen")
|
||||
|
||||
if high > 3:
|
||||
findings.append(f"{high} hohe Risiken - priorisierte Behandlung erforderlich")
|
||||
|
||||
if not findings:
|
||||
findings.append("Keine kritischen Befunde - Compliance-Status stabil")
|
||||
|
||||
return findings
|
||||
|
||||
def _generate_compliance_score_section(self) -> Dict[str, Any]:
|
||||
"""Generate compliance score section with breakdown."""
|
||||
stats = self.ctrl_repo.get_statistics()
|
||||
|
||||
by_domain = stats.get("by_domain", {})
|
||||
domain_scores = {}
|
||||
|
||||
controls = self.ctrl_repo.get_all()
|
||||
domain_stats = {}
|
||||
|
||||
for ctrl in controls:
|
||||
domain = ctrl.domain.value if ctrl.domain else "unknown"
|
||||
if domain not in domain_stats:
|
||||
domain_stats[domain] = {"total": 0, "pass": 0, "partial": 0}
|
||||
|
||||
domain_stats[domain]["total"] += 1
|
||||
if ctrl.status == ControlStatusEnum.PASS:
|
||||
domain_stats[domain]["pass"] += 1
|
||||
elif ctrl.status == ControlStatusEnum.PARTIAL:
|
||||
domain_stats[domain]["partial"] += 1
|
||||
|
||||
for domain, ds in domain_stats.items():
|
||||
if ds["total"] > 0:
|
||||
score = ((ds["pass"] + ds["partial"] * 0.5) / ds["total"]) * 100
|
||||
domain_scores[domain] = round(score, 1)
|
||||
else:
|
||||
domain_scores[domain] = 0
|
||||
|
||||
return {
|
||||
"overall_score": stats.get("compliance_score", 0),
|
||||
"by_domain": domain_scores,
|
||||
"domain_labels": {
|
||||
"gov": "Governance",
|
||||
"priv": "Datenschutz",
|
||||
"iam": "Identity & Access",
|
||||
"crypto": "Kryptografie",
|
||||
"sdlc": "Secure Development",
|
||||
"ops": "Operations",
|
||||
"ai": "KI-spezifisch",
|
||||
"cra": "Supply Chain",
|
||||
"aud": "Audit",
|
||||
},
|
||||
}
|
||||
|
||||
def _generate_regulations_coverage(self) -> Dict[str, Any]:
|
||||
"""Generate regulations coverage section."""
|
||||
regulations = self.reg_repo.get_all()
|
||||
coverage = []
|
||||
|
||||
for reg in regulations:
|
||||
# Count requirements for this regulation
|
||||
req_count = self.db.query(func.count(RequirementDB.id)).filter(
|
||||
RequirementDB.regulation_id == reg.id
|
||||
).scalar() or 0
|
||||
|
||||
# Count mapped controls
|
||||
mapped_controls = self.db.query(func.count(ControlMappingDB.id)).join(
|
||||
RequirementDB
|
||||
).filter(
|
||||
RequirementDB.regulation_id == reg.id
|
||||
).scalar() or 0
|
||||
|
||||
coverage.append({
|
||||
"code": reg.code,
|
||||
"name": reg.name,
|
||||
"requirements": req_count,
|
||||
"mapped_controls": mapped_controls,
|
||||
"coverage_status": "covered" if mapped_controls > 0 else "pending",
|
||||
})
|
||||
|
||||
return {
|
||||
"total_regulations": len(regulations),
|
||||
"covered_regulations": len([c for c in coverage if c["coverage_status"] == "covered"]),
|
||||
"details": coverage,
|
||||
}
|
||||
|
||||
def _generate_controls_summary(self) -> Dict[str, Any]:
|
||||
"""Generate controls summary section."""
|
||||
stats = self.ctrl_repo.get_statistics()
|
||||
due_for_review = self.ctrl_repo.get_due_for_review()
|
||||
|
||||
return {
|
||||
"total": stats.get("total", 0),
|
||||
"by_status": stats.get("by_status", {}),
|
||||
"by_domain": stats.get("by_domain", {}),
|
||||
"due_for_review": len(due_for_review),
|
||||
"review_items": [
|
||||
{
|
||||
"control_id": c.control_id,
|
||||
"title": c.title,
|
||||
"last_reviewed": c.last_reviewed_at.isoformat() if c.last_reviewed_at else None,
|
||||
}
|
||||
for c in due_for_review[:10] # Top 10
|
||||
],
|
||||
}
|
||||
|
||||
def _generate_risks_summary(self) -> Dict[str, Any]:
|
||||
"""Generate risks summary section."""
|
||||
matrix = self.risk_repo.get_matrix_data()
|
||||
risks = self.risk_repo.get_all()
|
||||
|
||||
# Group by category
|
||||
by_category = {}
|
||||
for risk in risks:
|
||||
cat = risk.category or "other"
|
||||
if cat not in by_category:
|
||||
by_category[cat] = 0
|
||||
by_category[cat] += 1
|
||||
|
||||
# High priority risks
|
||||
high_priority = [
|
||||
{
|
||||
"risk_id": r.risk_id,
|
||||
"title": r.title,
|
||||
"inherent_risk": r.inherent_risk.value if r.inherent_risk else None,
|
||||
"owner": r.owner,
|
||||
"status": r.status,
|
||||
}
|
||||
for r in risks
|
||||
if r.inherent_risk in [RiskLevelEnum.CRITICAL, RiskLevelEnum.HIGH]
|
||||
]
|
||||
|
||||
return {
|
||||
"total_risks": matrix["total_risks"],
|
||||
"by_level": matrix["by_level"],
|
||||
"by_category": by_category,
|
||||
"high_priority_risks": high_priority,
|
||||
"risk_matrix": matrix["matrix"],
|
||||
}
|
||||
|
||||
def _generate_evidence_summary(self) -> Dict[str, Any]:
|
||||
"""Generate evidence summary section."""
|
||||
stats = self.evidence_repo.get_statistics()
|
||||
all_evidence = self.evidence_repo.get_all(limit=100)
|
||||
|
||||
# Find controls without evidence
|
||||
controls = self.ctrl_repo.get_all()
|
||||
controls_with_evidence = set()
|
||||
|
||||
for evidence in all_evidence:
|
||||
control = self.db.query(ControlDB).filter(
|
||||
ControlDB.id == evidence.control_id
|
||||
).first()
|
||||
if control:
|
||||
controls_with_evidence.add(control.control_id)
|
||||
|
||||
controls_without_evidence = [
|
||||
c.control_id for c in controls
|
||||
if c.control_id not in controls_with_evidence
|
||||
]
|
||||
|
||||
return {
|
||||
"total_evidence": stats.get("total", 0),
|
||||
"by_type": stats.get("by_type", {}),
|
||||
"by_status": stats.get("by_status", {}),
|
||||
"coverage_percent": stats.get("coverage_percent", 0),
|
||||
"controls_without_evidence": controls_without_evidence[:20], # Top 20
|
||||
}
|
||||
|
||||
def _generate_action_items(self) -> List[Dict[str, Any]]:
|
||||
"""Generate action items based on current status."""
|
||||
action_items = []
|
||||
|
||||
# Check for failed controls
|
||||
failed_controls = self.ctrl_repo.get_all(status=ControlStatusEnum.FAIL)
|
||||
for ctrl in failed_controls[:5]:
|
||||
action_items.append({
|
||||
"priority": "high",
|
||||
"category": "control_remediation",
|
||||
"title": f"Control {ctrl.control_id} beheben",
|
||||
"description": f"Control '{ctrl.title}' ist im Status 'Fail'. Sofortige Massnahmen erforderlich.",
|
||||
"owner": ctrl.owner,
|
||||
"due_date": (date.today() + timedelta(days=7)).isoformat(),
|
||||
})
|
||||
|
||||
# Check for critical/high risks
|
||||
critical_risks = self.risk_repo.get_all(min_risk_level=RiskLevelEnum.HIGH)
|
||||
for risk in critical_risks[:5]:
|
||||
if risk.status == "open":
|
||||
action_items.append({
|
||||
"priority": "high" if risk.inherent_risk == RiskLevelEnum.CRITICAL else "medium",
|
||||
"category": "risk_treatment",
|
||||
"title": f"Risiko {risk.risk_id} behandeln",
|
||||
"description": f"Risiko '{risk.title}' hat Status 'open' und Level '{risk.inherent_risk.value}'.",
|
||||
"owner": risk.owner,
|
||||
"due_date": (date.today() + timedelta(days=14)).isoformat(),
|
||||
})
|
||||
|
||||
# Check for overdue reviews
|
||||
due_for_review = self.ctrl_repo.get_due_for_review()
|
||||
if len(due_for_review) > 5:
|
||||
action_items.append({
|
||||
"priority": "medium",
|
||||
"category": "review",
|
||||
"title": f"{len(due_for_review)} Control-Reviews ueberfaellig",
|
||||
"description": "Mehrere Controls muessen reviewed werden.",
|
||||
"owner": "Compliance Officer",
|
||||
"due_date": (date.today() + timedelta(days=30)).isoformat(),
|
||||
})
|
||||
|
||||
return action_items
|
||||
|
||||
def _generate_trends_placeholder(self, period: ReportPeriod) -> Dict[str, Any]:
|
||||
"""
|
||||
Generate trends section.
|
||||
|
||||
Note: Full trend analysis requires historical data storage.
|
||||
This is a placeholder for future implementation.
|
||||
"""
|
||||
return {
|
||||
"note": "Trend-Analyse erfordert historische Daten. Feature in Entwicklung.",
|
||||
"period": period.value,
|
||||
"compliance_score_trend": "stable", # Placeholder
|
||||
"risk_trend": "stable", # Placeholder
|
||||
"recommendations": [
|
||||
"Historische Score-Snapshots aktivieren fuer Trend-Analyse",
|
||||
"Regelmaessige Report-Generierung einrichten",
|
||||
],
|
||||
}
|
||||
|
||||
def generate_summary_report(self) -> Dict[str, Any]:
|
||||
"""Generate a quick summary report (for dashboard)."""
|
||||
stats = self.ctrl_repo.get_statistics()
|
||||
risk_matrix = self.risk_repo.get_matrix_data()
|
||||
evidence_stats = self.evidence_repo.get_statistics()
|
||||
|
||||
return {
|
||||
"generated_at": datetime.utcnow().isoformat(),
|
||||
"compliance_score": stats.get("compliance_score", 0),
|
||||
"controls": {
|
||||
"total": stats.get("total", 0),
|
||||
"passing": stats.get("by_status", {}).get("pass", 0),
|
||||
"failing": stats.get("by_status", {}).get("fail", 0),
|
||||
},
|
||||
"risks": {
|
||||
"total": risk_matrix["total_risks"],
|
||||
"critical": risk_matrix["by_level"].get("critical", 0),
|
||||
"high": risk_matrix["by_level"].get("high", 0),
|
||||
},
|
||||
"evidence": {
|
||||
"total": evidence_stats.get("total", 0),
|
||||
"coverage": evidence_stats.get("coverage_percent", 0),
|
||||
},
|
||||
}
|
||||
488
backend/compliance/services/seeder.py
Normal file
488
backend/compliance/services/seeder.py
Normal file
@@ -0,0 +1,488 @@
|
||||
"""
|
||||
Compliance Seeder Service.
|
||||
|
||||
Seeds the database with initial regulations, controls, and requirements.
|
||||
"""
|
||||
|
||||
import logging
|
||||
from typing import Dict, List, Optional
|
||||
from datetime import datetime
|
||||
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from ..db.models import (
|
||||
RegulationDB,
|
||||
RequirementDB,
|
||||
ControlDB,
|
||||
ControlMappingDB,
|
||||
RiskDB,
|
||||
ServiceModuleDB,
|
||||
ModuleRegulationMappingDB,
|
||||
StatementOfApplicabilityDB,
|
||||
RegulationTypeEnum,
|
||||
ControlTypeEnum,
|
||||
ControlDomainEnum,
|
||||
ControlStatusEnum,
|
||||
RiskLevelEnum,
|
||||
ServiceTypeEnum,
|
||||
RelevanceLevelEnum,
|
||||
)
|
||||
from ..data.regulations import REGULATIONS_SEED
|
||||
from ..data.controls import CONTROLS_SEED
|
||||
from ..data.requirements import REQUIREMENTS_SEED
|
||||
from ..data.risks import RISKS_SEED
|
||||
from ..data.service_modules import BREAKPILOT_SERVICES
|
||||
from ..data.iso27001_annex_a import ISO27001_ANNEX_A_CONTROLS
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class ComplianceSeeder:
|
||||
"""Seeds the compliance database with initial data."""
|
||||
|
||||
def __init__(self, db: Session):
|
||||
self.db = db
|
||||
self._regulation_map: Dict[str, str] = {} # code -> id
|
||||
self._module_map: Dict[str, str] = {} # name -> id
|
||||
|
||||
def seed_all(self, force: bool = False) -> Dict[str, int]:
|
||||
"""
|
||||
Seed all compliance data.
|
||||
|
||||
Args:
|
||||
force: If True, re-seed even if data exists
|
||||
|
||||
Returns:
|
||||
Dictionary with counts of seeded items
|
||||
"""
|
||||
results = {
|
||||
"regulations": 0,
|
||||
"controls": 0,
|
||||
"requirements": 0,
|
||||
"mappings": 0,
|
||||
"risks": 0,
|
||||
"service_modules": 0,
|
||||
"module_regulation_mappings": 0,
|
||||
"soa_entries": 0,
|
||||
}
|
||||
|
||||
# Check if already seeded
|
||||
existing_regulations = self.db.query(RegulationDB).count()
|
||||
if existing_regulations > 0 and not force:
|
||||
logger.info(f"Database already has {existing_regulations} regulations, skipping seed")
|
||||
return results
|
||||
|
||||
try:
|
||||
# Seed in order (regulations first, then controls, then requirements, then risks, then service modules)
|
||||
results["regulations"] = self._seed_regulations()
|
||||
results["controls"] = self._seed_controls()
|
||||
results["requirements"] = self._seed_requirements()
|
||||
results["mappings"] = self._seed_default_mappings()
|
||||
results["risks"] = self._seed_risks()
|
||||
results["service_modules"] = self._seed_service_modules()
|
||||
results["module_regulation_mappings"] = self._seed_module_regulation_mappings()
|
||||
results["soa_entries"] = self._seed_soa()
|
||||
|
||||
self.db.commit()
|
||||
logger.info(f"Seeding completed: {results}")
|
||||
return results
|
||||
|
||||
except Exception as e:
|
||||
self.db.rollback()
|
||||
logger.error(f"Seeding failed: {e}")
|
||||
raise
|
||||
|
||||
def _seed_regulations(self) -> int:
|
||||
"""Seed regulations from REGULATIONS_SEED."""
|
||||
count = 0
|
||||
for reg_data in REGULATIONS_SEED:
|
||||
# Check if regulation already exists
|
||||
existing = self.db.query(RegulationDB).filter(
|
||||
RegulationDB.code == reg_data["code"]
|
||||
).first()
|
||||
|
||||
if existing:
|
||||
self._regulation_map[reg_data["code"]] = existing.id
|
||||
continue
|
||||
|
||||
regulation = RegulationDB(
|
||||
code=reg_data["code"],
|
||||
name=reg_data["name"],
|
||||
full_name=reg_data.get("full_name"),
|
||||
regulation_type=RegulationTypeEnum(reg_data["regulation_type"]),
|
||||
source_url=reg_data.get("source_url"),
|
||||
local_pdf_path=reg_data.get("local_pdf_path"),
|
||||
effective_date=reg_data.get("effective_date"),
|
||||
description=reg_data.get("description"),
|
||||
is_active=reg_data.get("is_active", True),
|
||||
)
|
||||
self.db.add(regulation)
|
||||
self.db.flush() # Get the ID
|
||||
self._regulation_map[reg_data["code"]] = regulation.id
|
||||
count += 1
|
||||
|
||||
return count
|
||||
|
||||
def _seed_controls(self) -> int:
|
||||
"""Seed controls from CONTROLS_SEED."""
|
||||
count = 0
|
||||
for ctrl_data in CONTROLS_SEED:
|
||||
# Check if control already exists
|
||||
existing = self.db.query(ControlDB).filter(
|
||||
ControlDB.control_id == ctrl_data["control_id"]
|
||||
).first()
|
||||
|
||||
if existing:
|
||||
continue
|
||||
|
||||
control = ControlDB(
|
||||
control_id=ctrl_data["control_id"],
|
||||
domain=ControlDomainEnum(ctrl_data["domain"]),
|
||||
control_type=ControlTypeEnum(ctrl_data["control_type"]),
|
||||
title=ctrl_data["title"],
|
||||
description=ctrl_data.get("description"),
|
||||
pass_criteria=ctrl_data["pass_criteria"],
|
||||
implementation_guidance=ctrl_data.get("implementation_guidance"),
|
||||
code_reference=ctrl_data.get("code_reference"),
|
||||
is_automated=ctrl_data.get("is_automated", False),
|
||||
automation_tool=ctrl_data.get("automation_tool"),
|
||||
owner=ctrl_data.get("owner"),
|
||||
review_frequency_days=ctrl_data.get("review_frequency_days", 90),
|
||||
status=ControlStatusEnum.PLANNED, # All start as planned
|
||||
)
|
||||
self.db.add(control)
|
||||
count += 1
|
||||
|
||||
return count
|
||||
|
||||
def _seed_requirements(self) -> int:
|
||||
"""Seed requirements from REQUIREMENTS_SEED."""
|
||||
count = 0
|
||||
for req_data in REQUIREMENTS_SEED:
|
||||
# Get regulation ID
|
||||
regulation_code = req_data["regulation_code"]
|
||||
regulation_id = self._regulation_map.get(regulation_code)
|
||||
|
||||
if not regulation_id:
|
||||
# Try to find in database
|
||||
regulation = self.db.query(RegulationDB).filter(
|
||||
RegulationDB.code == regulation_code
|
||||
).first()
|
||||
if regulation:
|
||||
regulation_id = regulation.id
|
||||
self._regulation_map[regulation_code] = regulation_id
|
||||
else:
|
||||
logger.warning(f"Regulation {regulation_code} not found, skipping requirement")
|
||||
continue
|
||||
|
||||
# Check if requirement already exists
|
||||
existing = self.db.query(RequirementDB).filter(
|
||||
RequirementDB.regulation_id == regulation_id,
|
||||
RequirementDB.article == req_data["article"],
|
||||
RequirementDB.paragraph == req_data.get("paragraph"),
|
||||
).first()
|
||||
|
||||
if existing:
|
||||
continue
|
||||
|
||||
requirement = RequirementDB(
|
||||
regulation_id=regulation_id,
|
||||
article=req_data["article"],
|
||||
paragraph=req_data.get("paragraph"),
|
||||
title=req_data["title"],
|
||||
description=req_data.get("description"),
|
||||
requirement_text=req_data.get("requirement_text"),
|
||||
breakpilot_interpretation=req_data.get("breakpilot_interpretation"),
|
||||
is_applicable=req_data.get("is_applicable", True),
|
||||
applicability_reason=req_data.get("applicability_reason"),
|
||||
priority=req_data.get("priority", 2),
|
||||
)
|
||||
self.db.add(requirement)
|
||||
count += 1
|
||||
|
||||
return count
|
||||
|
||||
def _seed_default_mappings(self) -> int:
|
||||
"""Create default mappings between requirements and controls."""
|
||||
# Define default mappings based on domain/regulation relationships
|
||||
mapping_rules = [
|
||||
# GDPR Privacy mappings
|
||||
("GDPR", "Art. 5", ["PRIV-001", "PRIV-003", "PRIV-006", "PRIV-007"]),
|
||||
("GDPR", "Art. 25", ["PRIV-003", "PRIV-007"]),
|
||||
("GDPR", "Art. 28", ["PRIV-005"]),
|
||||
("GDPR", "Art. 30", ["PRIV-001"]),
|
||||
("GDPR", "Art. 32", ["CRYPTO-001", "CRYPTO-002", "CRYPTO-003", "IAM-001", "OPS-002"]),
|
||||
("GDPR", "Art. 35", ["PRIV-002", "AI-005"]),
|
||||
# AI Act mappings
|
||||
("AIACT", "Art. 9", ["AI-001", "AI-004", "AI-005"]),
|
||||
("AIACT", "Art. 13", ["AI-002", "AI-003"]),
|
||||
("AIACT", "Art. 14", ["AI-003"]),
|
||||
("AIACT", "Art. 15", ["AI-004", "SDLC-001", "SDLC-002"]),
|
||||
("AIACT", "Art. 50", ["AI-002"]),
|
||||
# CRA mappings
|
||||
("CRA", "Art. 10", ["SDLC-001", "SDLC-002", "SDLC-006"]),
|
||||
("CRA", "Art. 11", ["GOV-005", "OPS-003"]),
|
||||
("CRA", "Art. 13", ["CRA-001", "SDLC-005"]),
|
||||
("CRA", "Art. 14", ["CRA-003", "OPS-004"]),
|
||||
("CRA", "Art. 15", ["CRA-004"]),
|
||||
# BSI-TR mappings
|
||||
("BSI-TR-03161-1", "O.Arch_1", ["GOV-001", "GOV-002", "GOV-004"]),
|
||||
("BSI-TR-03161-1", "O.Auth_1", ["IAM-001", "IAM-002", "IAM-004"]),
|
||||
("BSI-TR-03161-1", "O.Cryp_1", ["CRYPTO-001", "CRYPTO-002", "CRYPTO-003", "CRYPTO-004"]),
|
||||
("BSI-TR-03161-1", "O.Data_1", ["CRYPTO-001", "CRYPTO-002", "PRIV-007"]),
|
||||
("BSI-TR-03161-2", "O.Auth_2", ["IAM-004"]),
|
||||
("BSI-TR-03161-2", "O.Source_1", ["SDLC-001", "SDLC-004"]),
|
||||
("BSI-TR-03161-3", "O.Back_1", ["CRYPTO-002"]),
|
||||
("BSI-TR-03161-3", "O.Ops_1", ["OPS-001", "OPS-002", "OPS-005"]),
|
||||
]
|
||||
|
||||
count = 0
|
||||
for reg_code, article_prefix, control_ids in mapping_rules:
|
||||
# Find requirements matching this regulation and article
|
||||
requirements = self.db.query(RequirementDB).join(RegulationDB).filter(
|
||||
RegulationDB.code == reg_code,
|
||||
RequirementDB.article.like(f"{article_prefix}%"),
|
||||
).all()
|
||||
|
||||
for req in requirements:
|
||||
for control_id in control_ids:
|
||||
# Find control
|
||||
control = self.db.query(ControlDB).filter(
|
||||
ControlDB.control_id == control_id
|
||||
).first()
|
||||
|
||||
if not control:
|
||||
continue
|
||||
|
||||
# Check if mapping exists
|
||||
existing = self.db.query(ControlMappingDB).filter(
|
||||
ControlMappingDB.requirement_id == req.id,
|
||||
ControlMappingDB.control_id == control.id,
|
||||
).first()
|
||||
|
||||
if existing:
|
||||
continue
|
||||
|
||||
mapping = ControlMappingDB(
|
||||
requirement_id=req.id,
|
||||
control_id=control.id,
|
||||
coverage_level="full",
|
||||
)
|
||||
self.db.add(mapping)
|
||||
count += 1
|
||||
|
||||
return count
|
||||
|
||||
def seed_regulations_only(self) -> int:
|
||||
"""Seed only regulations (useful for incremental updates)."""
|
||||
count = self._seed_regulations()
|
||||
self.db.commit()
|
||||
return count
|
||||
|
||||
def seed_controls_only(self) -> int:
|
||||
"""Seed only controls (useful for incremental updates)."""
|
||||
count = self._seed_controls()
|
||||
self.db.commit()
|
||||
return count
|
||||
|
||||
def _seed_risks(self) -> int:
|
||||
"""Seed risks from RISKS_SEED."""
|
||||
count = 0
|
||||
for risk_data in RISKS_SEED:
|
||||
# Check if risk already exists
|
||||
existing = self.db.query(RiskDB).filter(
|
||||
RiskDB.risk_id == risk_data["risk_id"]
|
||||
).first()
|
||||
|
||||
if existing:
|
||||
continue
|
||||
|
||||
# Calculate inherent risk level
|
||||
inherent_risk = RiskDB.calculate_risk_level(
|
||||
risk_data["likelihood"],
|
||||
risk_data["impact"]
|
||||
)
|
||||
|
||||
risk = RiskDB(
|
||||
risk_id=risk_data["risk_id"],
|
||||
title=risk_data["title"],
|
||||
description=risk_data.get("description"),
|
||||
category=risk_data["category"],
|
||||
likelihood=risk_data["likelihood"],
|
||||
impact=risk_data["impact"],
|
||||
inherent_risk=inherent_risk,
|
||||
mitigating_controls=risk_data.get("mitigating_controls", []),
|
||||
owner=risk_data.get("owner"),
|
||||
treatment_plan=risk_data.get("treatment_plan"),
|
||||
status="open",
|
||||
)
|
||||
self.db.add(risk)
|
||||
count += 1
|
||||
|
||||
return count
|
||||
|
||||
def seed_risks_only(self) -> int:
|
||||
"""Seed only risks (useful for incremental updates)."""
|
||||
count = self._seed_risks()
|
||||
self.db.commit()
|
||||
return count
|
||||
|
||||
def _seed_service_modules(self) -> int:
|
||||
"""Seed service modules from BREAKPILOT_SERVICES."""
|
||||
count = 0
|
||||
for service_data in BREAKPILOT_SERVICES:
|
||||
# Check if service already exists
|
||||
existing = self.db.query(ServiceModuleDB).filter(
|
||||
ServiceModuleDB.name == service_data["name"]
|
||||
).first()
|
||||
|
||||
if existing:
|
||||
self._module_map[service_data["name"]] = existing.id
|
||||
continue
|
||||
|
||||
module = ServiceModuleDB(
|
||||
name=service_data["name"],
|
||||
display_name=service_data["display_name"],
|
||||
description=service_data.get("description"),
|
||||
service_type=ServiceTypeEnum(service_data["service_type"]),
|
||||
port=service_data.get("port"),
|
||||
technology_stack=service_data.get("technology_stack", []),
|
||||
repository_path=service_data.get("repository_path"),
|
||||
docker_image=service_data.get("docker_image"),
|
||||
data_categories=service_data.get("data_categories", []),
|
||||
processes_pii=service_data.get("processes_pii", False),
|
||||
processes_health_data=service_data.get("processes_health_data", False),
|
||||
ai_components=service_data.get("ai_components", False),
|
||||
is_active=True,
|
||||
criticality=service_data.get("criticality", "medium"),
|
||||
owner_team=service_data.get("owner_team"),
|
||||
)
|
||||
self.db.add(module)
|
||||
self.db.flush() # Get the ID
|
||||
self._module_map[service_data["name"]] = module.id
|
||||
count += 1
|
||||
|
||||
return count
|
||||
|
||||
def _seed_module_regulation_mappings(self) -> int:
|
||||
"""Create mappings between service modules and regulations."""
|
||||
count = 0
|
||||
for service_data in BREAKPILOT_SERVICES:
|
||||
# Get module ID
|
||||
module_id = self._module_map.get(service_data["name"])
|
||||
if not module_id:
|
||||
# Try to find in database
|
||||
module = self.db.query(ServiceModuleDB).filter(
|
||||
ServiceModuleDB.name == service_data["name"]
|
||||
).first()
|
||||
if module:
|
||||
module_id = module.id
|
||||
self._module_map[service_data["name"]] = module_id
|
||||
else:
|
||||
logger.warning(f"Module {service_data['name']} not found, skipping regulation mappings")
|
||||
continue
|
||||
|
||||
# Process regulation mappings
|
||||
regulations = service_data.get("regulations", [])
|
||||
for reg_mapping in regulations:
|
||||
# Find regulation by code
|
||||
regulation_code = reg_mapping["code"]
|
||||
regulation_id = self._regulation_map.get(regulation_code)
|
||||
|
||||
if not regulation_id:
|
||||
regulation = self.db.query(RegulationDB).filter(
|
||||
RegulationDB.code == regulation_code
|
||||
).first()
|
||||
if regulation:
|
||||
regulation_id = regulation.id
|
||||
self._regulation_map[regulation_code] = regulation_id
|
||||
else:
|
||||
logger.warning(f"Regulation {regulation_code} not found, skipping mapping for {service_data['name']}")
|
||||
continue
|
||||
|
||||
# Check if mapping exists
|
||||
existing = self.db.query(ModuleRegulationMappingDB).filter(
|
||||
ModuleRegulationMappingDB.module_id == module_id,
|
||||
ModuleRegulationMappingDB.regulation_id == regulation_id,
|
||||
).first()
|
||||
|
||||
if existing:
|
||||
continue
|
||||
|
||||
mapping = ModuleRegulationMappingDB(
|
||||
module_id=module_id,
|
||||
regulation_id=regulation_id,
|
||||
relevance_level=RelevanceLevelEnum(reg_mapping["relevance"]),
|
||||
notes=reg_mapping.get("notes"),
|
||||
)
|
||||
self.db.add(mapping)
|
||||
count += 1
|
||||
|
||||
return count
|
||||
|
||||
def seed_service_modules_only(self) -> int:
|
||||
"""Seed only service modules (useful for incremental updates)."""
|
||||
results = {
|
||||
"service_modules": 0,
|
||||
"module_regulation_mappings": 0,
|
||||
}
|
||||
|
||||
# Ensure regulations are loaded first
|
||||
if not self._regulation_map:
|
||||
self._seed_regulations()
|
||||
|
||||
results["service_modules"] = self._seed_service_modules()
|
||||
results["module_regulation_mappings"] = self._seed_module_regulation_mappings()
|
||||
|
||||
self.db.commit()
|
||||
logger.info(f"Service modules seeding completed: {results}")
|
||||
return results["service_modules"] + results["module_regulation_mappings"]
|
||||
|
||||
def _seed_soa(self) -> int:
|
||||
"""
|
||||
Seed Statement of Applicability (SoA) entries from ISO 27001:2022 Annex A.
|
||||
|
||||
Creates SoA entries for all 93 Annex A controls.
|
||||
This is MANDATORY for ISO 27001 certification.
|
||||
"""
|
||||
count = 0
|
||||
for annex_control in ISO27001_ANNEX_A_CONTROLS:
|
||||
control_id = annex_control["control_id"]
|
||||
|
||||
# Check if SoA entry already exists
|
||||
existing = self.db.query(StatementOfApplicabilityDB).filter(
|
||||
StatementOfApplicabilityDB.annex_a_control == control_id
|
||||
).first()
|
||||
|
||||
if existing:
|
||||
continue
|
||||
|
||||
# Create SoA entry
|
||||
soa_entry = StatementOfApplicabilityDB(
|
||||
annex_a_control=control_id,
|
||||
annex_a_title=annex_control["title"],
|
||||
annex_a_category=annex_control["category"],
|
||||
is_applicable=annex_control.get("default_applicable", True),
|
||||
applicability_justification=annex_control.get("description", ""),
|
||||
implementation_status="planned",
|
||||
implementation_notes=annex_control.get("implementation_guidance", ""),
|
||||
breakpilot_control_ids=annex_control.get("breakpilot_controls", []),
|
||||
evidence_description="",
|
||||
risk_assessment_notes="",
|
||||
)
|
||||
self.db.add(soa_entry)
|
||||
count += 1
|
||||
|
||||
logger.info(f"Seeded {count} SoA entries from ISO 27001:2022 Annex A")
|
||||
return count
|
||||
|
||||
def seed_soa_only(self) -> int:
|
||||
"""
|
||||
Seed only SoA entries (useful for incremental updates).
|
||||
|
||||
Creates all 93 ISO 27001:2022 Annex A control entries in the SoA.
|
||||
"""
|
||||
count = self._seed_soa()
|
||||
self.db.commit()
|
||||
logger.info(f"SoA seeding completed: {count} entries")
|
||||
return count
|
||||
1
backend/compliance/tests/__init__.py
Normal file
1
backend/compliance/tests/__init__.py
Normal file
@@ -0,0 +1 @@
|
||||
# Compliance Module Tests
|
||||
591
backend/compliance/tests/test_audit_routes.py
Normal file
591
backend/compliance/tests/test_audit_routes.py
Normal file
@@ -0,0 +1,591 @@
|
||||
"""
|
||||
Unit tests for Compliance Audit Routes (Sprint 3).
|
||||
|
||||
Tests all audit session and sign-off endpoints.
|
||||
|
||||
Run with: pytest backend/compliance/tests/test_audit_routes.py -v
|
||||
"""
|
||||
|
||||
import pytest
|
||||
import hashlib
|
||||
from datetime import datetime
|
||||
from unittest.mock import MagicMock, patch
|
||||
from uuid import uuid4
|
||||
|
||||
from fastapi.testclient import TestClient
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
# Import the app and dependencies
|
||||
import sys
|
||||
sys.path.insert(0, '/Users/benjaminadmin/Projekte/breakpilot-pwa/backend')
|
||||
|
||||
from compliance.db.models import (
|
||||
AuditSessionDB, AuditSignOffDB, AuditResultEnum, AuditSessionStatusEnum,
|
||||
RequirementDB, RegulationDB
|
||||
)
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Test Fixtures
|
||||
# ============================================================================
|
||||
|
||||
@pytest.fixture
|
||||
def mock_db():
|
||||
"""Create a mock database session."""
|
||||
return MagicMock(spec=Session)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def sample_regulation():
|
||||
"""Create a sample regulation for testing."""
|
||||
return RegulationDB(
|
||||
id=str(uuid4()),
|
||||
code="GDPR",
|
||||
name="General Data Protection Regulation",
|
||||
full_name="Regulation (EU) 2016/679",
|
||||
is_active=True,
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def sample_requirement(sample_regulation):
|
||||
"""Create a sample requirement for testing."""
|
||||
return RequirementDB(
|
||||
id=str(uuid4()),
|
||||
regulation_id=sample_regulation.id,
|
||||
regulation=sample_regulation,
|
||||
article="Art. 32",
|
||||
title="Security of processing",
|
||||
description="Implement appropriate technical measures",
|
||||
implementation_status="not_started",
|
||||
priority=1,
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def sample_session():
|
||||
"""Create a sample audit session for testing."""
|
||||
session_id = str(uuid4())
|
||||
return AuditSessionDB(
|
||||
id=session_id,
|
||||
name="Q1 2026 Compliance Audit",
|
||||
description="Quarterly compliance review",
|
||||
auditor_name="Dr. Thomas Mueller",
|
||||
auditor_email="mueller@audit.de",
|
||||
auditor_organization="Audit GmbH",
|
||||
status=AuditSessionStatusEnum.DRAFT,
|
||||
regulation_ids=["GDPR", "AIACT"],
|
||||
total_items=100,
|
||||
completed_items=0,
|
||||
compliant_count=0,
|
||||
non_compliant_count=0,
|
||||
created_at=datetime.utcnow(),
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def sample_signoff(sample_session, sample_requirement):
|
||||
"""Create a sample sign-off for testing."""
|
||||
return AuditSignOffDB(
|
||||
id=str(uuid4()),
|
||||
session_id=sample_session.id,
|
||||
requirement_id=sample_requirement.id,
|
||||
result=AuditResultEnum.COMPLIANT,
|
||||
notes="All checks passed",
|
||||
signature_hash=None,
|
||||
signed_at=None,
|
||||
signed_by=None,
|
||||
created_at=datetime.utcnow(),
|
||||
)
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Test: Audit Session Creation
|
||||
# ============================================================================
|
||||
|
||||
class TestCreateAuditSession:
|
||||
"""Tests for POST /audit/sessions endpoint."""
|
||||
|
||||
def test_create_session_valid_data_returns_session(self, mock_db, sample_regulation):
|
||||
"""Creating a session with valid data should return the session."""
|
||||
# Arrange
|
||||
mock_db.query.return_value.filter.return_value.all.return_value = [(sample_regulation.id,)]
|
||||
mock_db.query.return_value.count.return_value = 50
|
||||
|
||||
request_data = {
|
||||
"name": "Test Audit Session",
|
||||
"description": "Test description",
|
||||
"auditor_name": "Test Auditor",
|
||||
"auditor_email": "auditor@test.de",
|
||||
"regulation_codes": ["GDPR"],
|
||||
}
|
||||
|
||||
# The session should be created with correct data
|
||||
assert request_data["name"] == "Test Audit Session"
|
||||
assert request_data["auditor_name"] == "Test Auditor"
|
||||
|
||||
def test_create_session_minimal_data_returns_session(self):
|
||||
"""Creating a session with minimal data should work."""
|
||||
request_data = {
|
||||
"name": "Minimal Audit",
|
||||
"auditor_name": "Auditor",
|
||||
}
|
||||
|
||||
assert "name" in request_data
|
||||
assert "auditor_name" in request_data
|
||||
assert "description" not in request_data or request_data.get("description") is None
|
||||
|
||||
def test_create_session_with_multiple_regulations(self):
|
||||
"""Creating a session with multiple regulations should filter correctly."""
|
||||
request_data = {
|
||||
"name": "Multi-Regulation Audit",
|
||||
"auditor_name": "Auditor",
|
||||
"regulation_codes": ["GDPR", "AIACT", "CRA"],
|
||||
}
|
||||
|
||||
assert len(request_data["regulation_codes"]) == 3
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Test: Audit Session List
|
||||
# ============================================================================
|
||||
|
||||
class TestListAuditSessions:
|
||||
"""Tests for GET /audit/sessions endpoint."""
|
||||
|
||||
def test_list_sessions_returns_all(self, mock_db, sample_session):
|
||||
"""Listing sessions without filter should return all sessions."""
|
||||
mock_db.query.return_value.order_by.return_value.all.return_value = [sample_session]
|
||||
|
||||
sessions = [sample_session]
|
||||
assert len(sessions) == 1
|
||||
assert sessions[0].name == "Q1 2026 Compliance Audit"
|
||||
|
||||
def test_list_sessions_filter_by_status_draft(self, mock_db, sample_session):
|
||||
"""Filtering by draft status should only return draft sessions."""
|
||||
sample_session.status = AuditSessionStatusEnum.DRAFT
|
||||
|
||||
assert sample_session.status == AuditSessionStatusEnum.DRAFT
|
||||
|
||||
def test_list_sessions_filter_by_status_in_progress(self, sample_session):
|
||||
"""Filtering by in_progress status should only return in_progress sessions."""
|
||||
sample_session.status = AuditSessionStatusEnum.IN_PROGRESS
|
||||
|
||||
assert sample_session.status == AuditSessionStatusEnum.IN_PROGRESS
|
||||
|
||||
def test_list_sessions_invalid_status_raises_error(self):
|
||||
"""Filtering by invalid status should raise an error."""
|
||||
invalid_status = "invalid_status"
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
AuditSessionStatusEnum(invalid_status)
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Test: Audit Session Get
|
||||
# ============================================================================
|
||||
|
||||
class TestGetAuditSession:
|
||||
"""Tests for GET /audit/sessions/{session_id} endpoint."""
|
||||
|
||||
def test_get_session_existing_returns_details(self, sample_session):
|
||||
"""Getting an existing session should return full details."""
|
||||
assert sample_session.id is not None
|
||||
assert sample_session.name == "Q1 2026 Compliance Audit"
|
||||
assert sample_session.auditor_name == "Dr. Thomas Mueller"
|
||||
|
||||
def test_get_session_includes_statistics(self, sample_session, sample_signoff):
|
||||
"""Getting a session should include statistics."""
|
||||
# Simulate statistics calculation
|
||||
signoffs = [sample_signoff]
|
||||
compliant = sum(1 for s in signoffs if s.result == AuditResultEnum.COMPLIANT)
|
||||
|
||||
assert compliant == 1
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Test: Audit Session Lifecycle
|
||||
# ============================================================================
|
||||
|
||||
class TestAuditSessionLifecycle:
|
||||
"""Tests for session status transitions."""
|
||||
|
||||
def test_start_session_from_draft_success(self, sample_session):
|
||||
"""Starting a draft session should change status to in_progress."""
|
||||
assert sample_session.status == AuditSessionStatusEnum.DRAFT
|
||||
|
||||
sample_session.status = AuditSessionStatusEnum.IN_PROGRESS
|
||||
sample_session.started_at = datetime.utcnow()
|
||||
|
||||
assert sample_session.status == AuditSessionStatusEnum.IN_PROGRESS
|
||||
assert sample_session.started_at is not None
|
||||
|
||||
def test_start_session_from_completed_fails(self, sample_session):
|
||||
"""Starting a completed session should fail."""
|
||||
sample_session.status = AuditSessionStatusEnum.COMPLETED
|
||||
|
||||
# Can only start from DRAFT
|
||||
assert sample_session.status != AuditSessionStatusEnum.DRAFT
|
||||
|
||||
def test_complete_session_from_in_progress_success(self, sample_session):
|
||||
"""Completing an in_progress session should succeed."""
|
||||
sample_session.status = AuditSessionStatusEnum.IN_PROGRESS
|
||||
|
||||
sample_session.status = AuditSessionStatusEnum.COMPLETED
|
||||
sample_session.completed_at = datetime.utcnow()
|
||||
|
||||
assert sample_session.status == AuditSessionStatusEnum.COMPLETED
|
||||
assert sample_session.completed_at is not None
|
||||
|
||||
def test_archive_session_from_completed_success(self, sample_session):
|
||||
"""Archiving a completed session should succeed."""
|
||||
sample_session.status = AuditSessionStatusEnum.COMPLETED
|
||||
|
||||
sample_session.status = AuditSessionStatusEnum.ARCHIVED
|
||||
|
||||
assert sample_session.status == AuditSessionStatusEnum.ARCHIVED
|
||||
|
||||
def test_archive_session_from_in_progress_fails(self, sample_session):
|
||||
"""Archiving an in_progress session should fail."""
|
||||
sample_session.status = AuditSessionStatusEnum.IN_PROGRESS
|
||||
|
||||
# Can only archive from COMPLETED
|
||||
assert sample_session.status != AuditSessionStatusEnum.COMPLETED
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Test: Audit Session Delete
|
||||
# ============================================================================
|
||||
|
||||
class TestDeleteAuditSession:
|
||||
"""Tests for DELETE /audit/sessions/{session_id} endpoint."""
|
||||
|
||||
def test_delete_draft_session_success(self, sample_session):
|
||||
"""Deleting a draft session should succeed."""
|
||||
sample_session.status = AuditSessionStatusEnum.DRAFT
|
||||
|
||||
assert sample_session.status in [
|
||||
AuditSessionStatusEnum.DRAFT,
|
||||
AuditSessionStatusEnum.ARCHIVED
|
||||
]
|
||||
|
||||
def test_delete_archived_session_success(self, sample_session):
|
||||
"""Deleting an archived session should succeed."""
|
||||
sample_session.status = AuditSessionStatusEnum.ARCHIVED
|
||||
|
||||
assert sample_session.status in [
|
||||
AuditSessionStatusEnum.DRAFT,
|
||||
AuditSessionStatusEnum.ARCHIVED
|
||||
]
|
||||
|
||||
def test_delete_in_progress_session_fails(self, sample_session):
|
||||
"""Deleting an in_progress session should fail."""
|
||||
sample_session.status = AuditSessionStatusEnum.IN_PROGRESS
|
||||
|
||||
assert sample_session.status not in [
|
||||
AuditSessionStatusEnum.DRAFT,
|
||||
AuditSessionStatusEnum.ARCHIVED
|
||||
]
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Test: Audit Checklist
|
||||
# ============================================================================
|
||||
|
||||
class TestGetAuditChecklist:
|
||||
"""Tests for GET /audit/checklist/{session_id} endpoint."""
|
||||
|
||||
def test_checklist_returns_paginated_items(self, sample_session, sample_requirement):
|
||||
"""Checklist should return paginated items."""
|
||||
page = 1
|
||||
page_size = 50
|
||||
|
||||
# Simulate pagination
|
||||
offset = (page - 1) * page_size
|
||||
assert offset == 0
|
||||
|
||||
def test_checklist_includes_signoff_status(self, sample_requirement, sample_signoff):
|
||||
"""Checklist items should include sign-off status."""
|
||||
signoff_map = {sample_signoff.requirement_id: sample_signoff}
|
||||
|
||||
signoff = signoff_map.get(sample_requirement.id)
|
||||
if signoff:
|
||||
current_result = signoff.result.value
|
||||
else:
|
||||
current_result = "pending"
|
||||
|
||||
assert current_result in ["compliant", "pending"]
|
||||
|
||||
def test_checklist_filter_by_status(self, sample_signoff):
|
||||
"""Filtering checklist by status should work."""
|
||||
status_filter = "compliant"
|
||||
sample_signoff.result = AuditResultEnum.COMPLIANT
|
||||
|
||||
assert sample_signoff.result.value == status_filter
|
||||
|
||||
def test_checklist_search_by_title(self, sample_requirement):
|
||||
"""Searching checklist by title should work."""
|
||||
search_term = "Security"
|
||||
sample_requirement.title = "Security of processing"
|
||||
|
||||
assert search_term.lower() in sample_requirement.title.lower()
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Test: Sign-off
|
||||
# ============================================================================
|
||||
|
||||
class TestSignOff:
|
||||
"""Tests for PUT /audit/checklist/{session_id}/items/{requirement_id}/sign-off endpoint."""
|
||||
|
||||
def test_signoff_compliant_creates_record(self, sample_session, sample_requirement):
|
||||
"""Signing off as compliant should create a sign-off record."""
|
||||
signoff = AuditSignOffDB(
|
||||
id=str(uuid4()),
|
||||
session_id=sample_session.id,
|
||||
requirement_id=sample_requirement.id,
|
||||
result=AuditResultEnum.COMPLIANT,
|
||||
notes="All requirements met",
|
||||
)
|
||||
|
||||
assert signoff.result == AuditResultEnum.COMPLIANT
|
||||
assert signoff.notes == "All requirements met"
|
||||
|
||||
def test_signoff_with_signature_creates_hash(self, sample_session, sample_requirement):
|
||||
"""Signing off with signature should create SHA-256 hash."""
|
||||
result = AuditResultEnum.COMPLIANT
|
||||
timestamp = datetime.utcnow().isoformat()
|
||||
data = f"{result.value}|{sample_requirement.id}|{sample_session.auditor_name}|{timestamp}"
|
||||
signature_hash = hashlib.sha256(data.encode()).hexdigest()
|
||||
|
||||
assert len(signature_hash) == 64 # SHA-256 produces 64 hex chars
|
||||
assert signature_hash.isalnum()
|
||||
|
||||
def test_signoff_non_compliant_increments_count(self, sample_session):
|
||||
"""Non-compliant sign-off should increment non_compliant_count."""
|
||||
initial_count = sample_session.non_compliant_count
|
||||
|
||||
sample_session.non_compliant_count += 1
|
||||
|
||||
assert sample_session.non_compliant_count == initial_count + 1
|
||||
|
||||
def test_signoff_updates_completion_items(self, sample_session):
|
||||
"""Sign-off should increment completed_items."""
|
||||
initial_completed = sample_session.completed_items
|
||||
|
||||
sample_session.completed_items += 1
|
||||
|
||||
assert sample_session.completed_items == initial_completed + 1
|
||||
|
||||
def test_signoff_auto_starts_session(self, sample_session):
|
||||
"""First sign-off should auto-start a draft session."""
|
||||
assert sample_session.status == AuditSessionStatusEnum.DRAFT
|
||||
|
||||
# First sign-off should trigger auto-start
|
||||
sample_session.status = AuditSessionStatusEnum.IN_PROGRESS
|
||||
sample_session.started_at = datetime.utcnow()
|
||||
|
||||
assert sample_session.status == AuditSessionStatusEnum.IN_PROGRESS
|
||||
|
||||
def test_signoff_update_existing_record(self, sample_signoff):
|
||||
"""Updating an existing sign-off should work."""
|
||||
sample_signoff.result = AuditResultEnum.NON_COMPLIANT
|
||||
sample_signoff.notes = "Updated: needs improvement"
|
||||
sample_signoff.updated_at = datetime.utcnow()
|
||||
|
||||
assert sample_signoff.result == AuditResultEnum.NON_COMPLIANT
|
||||
assert "Updated" in sample_signoff.notes
|
||||
|
||||
def test_signoff_invalid_result_raises_error(self):
|
||||
"""Sign-off with invalid result should raise an error."""
|
||||
invalid_result = "super_compliant"
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
AuditResultEnum(invalid_result)
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Test: Get Sign-off
|
||||
# ============================================================================
|
||||
|
||||
class TestGetSignOff:
|
||||
"""Tests for GET /audit/checklist/{session_id}/items/{requirement_id} endpoint."""
|
||||
|
||||
def test_get_signoff_existing_returns_details(self, sample_signoff):
|
||||
"""Getting an existing sign-off should return its details."""
|
||||
assert sample_signoff.id is not None
|
||||
assert sample_signoff.result == AuditResultEnum.COMPLIANT
|
||||
|
||||
def test_get_signoff_includes_signature_info(self, sample_signoff):
|
||||
"""Sign-off response should include signature information."""
|
||||
# Without signature
|
||||
assert sample_signoff.signature_hash is None
|
||||
assert sample_signoff.signed_at is None
|
||||
|
||||
# With signature
|
||||
sample_signoff.signature_hash = "abc123"
|
||||
sample_signoff.signed_at = datetime.utcnow()
|
||||
sample_signoff.signed_by = "Test Auditor"
|
||||
|
||||
assert sample_signoff.signature_hash == "abc123"
|
||||
assert sample_signoff.signed_by == "Test Auditor"
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Test: AuditResultEnum Values
|
||||
# ============================================================================
|
||||
|
||||
class TestAuditResultEnum:
|
||||
"""Tests for AuditResultEnum values."""
|
||||
|
||||
def test_compliant_value(self):
|
||||
"""Compliant enum should have correct value."""
|
||||
assert AuditResultEnum.COMPLIANT.value == "compliant"
|
||||
|
||||
def test_compliant_with_notes_value(self):
|
||||
"""Compliant with notes enum should have correct value."""
|
||||
assert AuditResultEnum.COMPLIANT_WITH_NOTES.value == "compliant_notes"
|
||||
|
||||
def test_non_compliant_value(self):
|
||||
"""Non-compliant enum should have correct value."""
|
||||
assert AuditResultEnum.NON_COMPLIANT.value == "non_compliant"
|
||||
|
||||
def test_not_applicable_value(self):
|
||||
"""Not applicable enum should have correct value."""
|
||||
assert AuditResultEnum.NOT_APPLICABLE.value == "not_applicable"
|
||||
|
||||
def test_pending_value(self):
|
||||
"""Pending enum should have correct value."""
|
||||
assert AuditResultEnum.PENDING.value == "pending"
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Test: AuditSessionStatusEnum Values
|
||||
# ============================================================================
|
||||
|
||||
class TestAuditSessionStatusEnum:
|
||||
"""Tests for AuditSessionStatusEnum values."""
|
||||
|
||||
def test_draft_value(self):
|
||||
"""Draft enum should have correct value."""
|
||||
assert AuditSessionStatusEnum.DRAFT.value == "draft"
|
||||
|
||||
def test_in_progress_value(self):
|
||||
"""In progress enum should have correct value."""
|
||||
assert AuditSessionStatusEnum.IN_PROGRESS.value == "in_progress"
|
||||
|
||||
def test_completed_value(self):
|
||||
"""Completed enum should have correct value."""
|
||||
assert AuditSessionStatusEnum.COMPLETED.value == "completed"
|
||||
|
||||
def test_archived_value(self):
|
||||
"""Archived enum should have correct value."""
|
||||
assert AuditSessionStatusEnum.ARCHIVED.value == "archived"
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Test: Completion Percentage Calculation
|
||||
# ============================================================================
|
||||
|
||||
class TestCompletionPercentage:
|
||||
"""Tests for completion percentage calculation."""
|
||||
|
||||
def test_completion_percentage_zero_items(self, sample_session):
|
||||
"""Completion percentage with zero total items should be 0."""
|
||||
sample_session.total_items = 0
|
||||
sample_session.completed_items = 0
|
||||
|
||||
percentage = 0.0 if sample_session.total_items == 0 else (
|
||||
sample_session.completed_items / sample_session.total_items * 100
|
||||
)
|
||||
|
||||
assert percentage == 0.0
|
||||
|
||||
def test_completion_percentage_partial(self, sample_session):
|
||||
"""Completion percentage should calculate correctly."""
|
||||
sample_session.total_items = 100
|
||||
sample_session.completed_items = 50
|
||||
|
||||
percentage = sample_session.completed_items / sample_session.total_items * 100
|
||||
|
||||
assert percentage == 50.0
|
||||
|
||||
def test_completion_percentage_complete(self, sample_session):
|
||||
"""Completion percentage at 100% should be correct."""
|
||||
sample_session.total_items = 100
|
||||
sample_session.completed_items = 100
|
||||
|
||||
percentage = sample_session.completed_items / sample_session.total_items * 100
|
||||
|
||||
assert percentage == 100.0
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Test: Digital Signature Generation
|
||||
# ============================================================================
|
||||
|
||||
class TestDigitalSignature:
|
||||
"""Tests for digital signature generation."""
|
||||
|
||||
def test_signature_is_sha256(self):
|
||||
"""Signature should be a valid SHA-256 hash."""
|
||||
data = "compliant|req-123|Dr. Mueller|2026-01-18T12:00:00"
|
||||
signature = hashlib.sha256(data.encode()).hexdigest()
|
||||
|
||||
assert len(signature) == 64
|
||||
assert all(c in '0123456789abcdef' for c in signature)
|
||||
|
||||
def test_signature_is_deterministic(self):
|
||||
"""Same input should produce same signature."""
|
||||
data = "compliant|req-123|Dr. Mueller|2026-01-18T12:00:00"
|
||||
signature1 = hashlib.sha256(data.encode()).hexdigest()
|
||||
signature2 = hashlib.sha256(data.encode()).hexdigest()
|
||||
|
||||
assert signature1 == signature2
|
||||
|
||||
def test_signature_changes_with_input(self):
|
||||
"""Different input should produce different signature."""
|
||||
data1 = "compliant|req-123|Dr. Mueller|2026-01-18T12:00:00"
|
||||
data2 = "non_compliant|req-123|Dr. Mueller|2026-01-18T12:00:00"
|
||||
|
||||
signature1 = hashlib.sha256(data1.encode()).hexdigest()
|
||||
signature2 = hashlib.sha256(data2.encode()).hexdigest()
|
||||
|
||||
assert signature1 != signature2
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Test: Statistics Calculation
|
||||
# ============================================================================
|
||||
|
||||
class TestStatisticsCalculation:
|
||||
"""Tests for audit statistics calculation."""
|
||||
|
||||
def test_statistics_counts_by_result(self):
|
||||
"""Statistics should correctly count by result type."""
|
||||
signoffs = [
|
||||
MagicMock(result=AuditResultEnum.COMPLIANT),
|
||||
MagicMock(result=AuditResultEnum.COMPLIANT),
|
||||
MagicMock(result=AuditResultEnum.COMPLIANT_WITH_NOTES),
|
||||
MagicMock(result=AuditResultEnum.NON_COMPLIANT),
|
||||
MagicMock(result=AuditResultEnum.NOT_APPLICABLE),
|
||||
]
|
||||
|
||||
compliant = sum(1 for s in signoffs if s.result == AuditResultEnum.COMPLIANT)
|
||||
compliant_notes = sum(1 for s in signoffs if s.result == AuditResultEnum.COMPLIANT_WITH_NOTES)
|
||||
non_compliant = sum(1 for s in signoffs if s.result == AuditResultEnum.NON_COMPLIANT)
|
||||
not_applicable = sum(1 for s in signoffs if s.result == AuditResultEnum.NOT_APPLICABLE)
|
||||
|
||||
assert compliant == 2
|
||||
assert compliant_notes == 1
|
||||
assert non_compliant == 1
|
||||
assert not_applicable == 1
|
||||
|
||||
def test_statistics_pending_calculation(self):
|
||||
"""Pending count should be total minus reviewed."""
|
||||
total_items = 100
|
||||
reviewed_items = 75
|
||||
|
||||
pending = total_items - reviewed_items
|
||||
|
||||
assert pending == 25
|
||||
434
backend/compliance/tests/test_auto_risk_updater.py
Normal file
434
backend/compliance/tests/test_auto_risk_updater.py
Normal file
@@ -0,0 +1,434 @@
|
||||
"""
|
||||
Tests for the AutoRiskUpdater Service.
|
||||
|
||||
Sprint 6: CI/CD Evidence Collection & Automatic Risk Updates (2026-01-18)
|
||||
"""
|
||||
|
||||
import pytest
|
||||
from datetime import datetime
|
||||
from unittest.mock import MagicMock, patch
|
||||
from uuid import uuid4
|
||||
|
||||
from ..services.auto_risk_updater import (
|
||||
AutoRiskUpdater,
|
||||
ScanType,
|
||||
FindingSeverity,
|
||||
ScanResult,
|
||||
RiskUpdateResult,
|
||||
CONTROL_SCAN_MAPPING,
|
||||
)
|
||||
from ..db.models import (
|
||||
ControlDB, ControlStatusEnum,
|
||||
EvidenceDB, EvidenceStatusEnum,
|
||||
RiskDB, RiskLevelEnum,
|
||||
)
|
||||
|
||||
|
||||
class TestDetermineControlStatus:
|
||||
"""Tests for _determine_control_status method."""
|
||||
|
||||
def test_critical_findings_return_fail(self):
|
||||
"""Any critical finding should result in FAIL status."""
|
||||
db = MagicMock()
|
||||
updater = AutoRiskUpdater(db)
|
||||
|
||||
findings = {"critical": 1, "high": 0, "medium": 0, "low": 0}
|
||||
result = updater._determine_control_status(findings)
|
||||
|
||||
assert result == ControlStatusEnum.FAIL.value
|
||||
|
||||
def test_multiple_critical_findings_return_fail(self):
|
||||
"""Multiple critical findings should result in FAIL status."""
|
||||
db = MagicMock()
|
||||
updater = AutoRiskUpdater(db)
|
||||
|
||||
findings = {"critical": 5, "high": 2, "medium": 10, "low": 50}
|
||||
result = updater._determine_control_status(findings)
|
||||
|
||||
assert result == ControlStatusEnum.FAIL.value
|
||||
|
||||
def test_more_than_5_high_findings_return_fail(self):
|
||||
"""More than 5 HIGH findings should result in FAIL status."""
|
||||
db = MagicMock()
|
||||
updater = AutoRiskUpdater(db)
|
||||
|
||||
findings = {"critical": 0, "high": 6, "medium": 0, "low": 0}
|
||||
result = updater._determine_control_status(findings)
|
||||
|
||||
assert result == ControlStatusEnum.FAIL.value
|
||||
|
||||
def test_exactly_5_high_findings_return_partial(self):
|
||||
"""Exactly 5 HIGH findings should result in PARTIAL status."""
|
||||
db = MagicMock()
|
||||
updater = AutoRiskUpdater(db)
|
||||
|
||||
findings = {"critical": 0, "high": 5, "medium": 0, "low": 0}
|
||||
result = updater._determine_control_status(findings)
|
||||
|
||||
assert result == ControlStatusEnum.PARTIAL.value
|
||||
|
||||
def test_1_to_5_high_findings_return_partial(self):
|
||||
"""1-5 HIGH findings should result in PARTIAL status."""
|
||||
db = MagicMock()
|
||||
updater = AutoRiskUpdater(db)
|
||||
|
||||
for high_count in [1, 2, 3, 4, 5]:
|
||||
findings = {"critical": 0, "high": high_count, "medium": 0, "low": 0}
|
||||
result = updater._determine_control_status(findings)
|
||||
assert result == ControlStatusEnum.PARTIAL.value, f"Failed for {high_count} HIGH findings"
|
||||
|
||||
def test_more_than_10_medium_findings_return_partial(self):
|
||||
"""More than 10 MEDIUM findings should result in PARTIAL status."""
|
||||
db = MagicMock()
|
||||
updater = AutoRiskUpdater(db)
|
||||
|
||||
findings = {"critical": 0, "high": 0, "medium": 11, "low": 0}
|
||||
result = updater._determine_control_status(findings)
|
||||
|
||||
assert result == ControlStatusEnum.PARTIAL.value
|
||||
|
||||
def test_only_medium_and_low_findings_return_pass(self):
|
||||
"""Only MEDIUM (<=10) and LOW findings should result in PASS status."""
|
||||
db = MagicMock()
|
||||
updater = AutoRiskUpdater(db)
|
||||
|
||||
findings = {"critical": 0, "high": 0, "medium": 5, "low": 100}
|
||||
result = updater._determine_control_status(findings)
|
||||
|
||||
assert result == ControlStatusEnum.PASS.value
|
||||
|
||||
def test_no_findings_return_pass(self):
|
||||
"""No findings should result in PASS status."""
|
||||
db = MagicMock()
|
||||
updater = AutoRiskUpdater(db)
|
||||
|
||||
findings = {"critical": 0, "high": 0, "medium": 0, "low": 0}
|
||||
result = updater._determine_control_status(findings)
|
||||
|
||||
assert result == ControlStatusEnum.PASS.value
|
||||
|
||||
def test_empty_findings_return_pass(self):
|
||||
"""Empty findings dict should result in PASS status."""
|
||||
db = MagicMock()
|
||||
updater = AutoRiskUpdater(db)
|
||||
|
||||
findings = {}
|
||||
result = updater._determine_control_status(findings)
|
||||
|
||||
assert result == ControlStatusEnum.PASS.value
|
||||
|
||||
|
||||
class TestGenerateStatusNotes:
|
||||
"""Tests for _generate_status_notes method."""
|
||||
|
||||
def test_notes_include_tool_name(self):
|
||||
"""Status notes should include the scan tool name."""
|
||||
db = MagicMock()
|
||||
updater = AutoRiskUpdater(db)
|
||||
|
||||
scan_result = ScanResult(
|
||||
scan_type=ScanType.SAST,
|
||||
tool="Semgrep",
|
||||
timestamp=datetime(2026, 1, 18, 14, 30),
|
||||
commit_sha="abc123",
|
||||
branch="main",
|
||||
control_id="SDLC-001",
|
||||
findings={"critical": 1, "high": 2, "medium": 0, "low": 0},
|
||||
)
|
||||
|
||||
notes = updater._generate_status_notes(scan_result)
|
||||
|
||||
assert "Semgrep" in notes
|
||||
assert "1 CRITICAL" in notes
|
||||
assert "2 HIGH" in notes
|
||||
|
||||
def test_notes_include_timestamp(self):
|
||||
"""Status notes should include scan timestamp."""
|
||||
db = MagicMock()
|
||||
updater = AutoRiskUpdater(db)
|
||||
|
||||
scan_result = ScanResult(
|
||||
scan_type=ScanType.DEPENDENCY,
|
||||
tool="Trivy",
|
||||
timestamp=datetime(2026, 1, 18, 10, 0),
|
||||
commit_sha="def456",
|
||||
branch="develop",
|
||||
control_id="SDLC-002",
|
||||
findings={"critical": 0, "high": 3, "medium": 5, "low": 10},
|
||||
)
|
||||
|
||||
notes = updater._generate_status_notes(scan_result)
|
||||
|
||||
assert "2026-01-18 10:00" in notes
|
||||
|
||||
def test_notes_for_no_findings(self):
|
||||
"""Status notes for no findings should indicate clean scan."""
|
||||
db = MagicMock()
|
||||
updater = AutoRiskUpdater(db)
|
||||
|
||||
scan_result = ScanResult(
|
||||
scan_type=ScanType.SECRET,
|
||||
tool="Gitleaks",
|
||||
timestamp=datetime(2026, 1, 18, 12, 0),
|
||||
commit_sha="ghi789",
|
||||
branch="main",
|
||||
control_id="SDLC-003",
|
||||
findings={"critical": 0, "high": 0, "medium": 0, "low": 0},
|
||||
)
|
||||
|
||||
notes = updater._generate_status_notes(scan_result)
|
||||
|
||||
assert "No significant findings" in notes
|
||||
|
||||
|
||||
class TestGenerateAlerts:
|
||||
"""Tests for _generate_alerts method."""
|
||||
|
||||
def test_alert_for_critical_findings(self):
|
||||
"""Critical findings should generate an alert."""
|
||||
db = MagicMock()
|
||||
updater = AutoRiskUpdater(db)
|
||||
|
||||
scan_result = ScanResult(
|
||||
scan_type=ScanType.DEPENDENCY,
|
||||
tool="Trivy",
|
||||
timestamp=datetime.utcnow(),
|
||||
commit_sha="abc123",
|
||||
branch="main",
|
||||
control_id="SDLC-002",
|
||||
findings={"critical": 2, "high": 0, "medium": 0, "low": 0},
|
||||
)
|
||||
|
||||
alerts = updater._generate_alerts(scan_result, ControlStatusEnum.FAIL.value)
|
||||
|
||||
assert len(alerts) >= 1
|
||||
assert any("CRITICAL" in alert for alert in alerts)
|
||||
assert any("2 critical" in alert.lower() for alert in alerts)
|
||||
|
||||
def test_alert_for_fail_status(self):
|
||||
"""Control status change to FAIL should generate an alert."""
|
||||
db = MagicMock()
|
||||
updater = AutoRiskUpdater(db)
|
||||
|
||||
scan_result = ScanResult(
|
||||
scan_type=ScanType.SAST,
|
||||
tool="Semgrep",
|
||||
timestamp=datetime.utcnow(),
|
||||
commit_sha="def456",
|
||||
branch="main",
|
||||
control_id="SDLC-001",
|
||||
findings={"critical": 0, "high": 10, "medium": 0, "low": 0},
|
||||
)
|
||||
|
||||
alerts = updater._generate_alerts(scan_result, ControlStatusEnum.FAIL.value)
|
||||
|
||||
assert any("FAIL" in alert for alert in alerts)
|
||||
|
||||
def test_alert_for_many_high_findings(self):
|
||||
"""More than 10 HIGH findings should generate an alert."""
|
||||
db = MagicMock()
|
||||
updater = AutoRiskUpdater(db)
|
||||
|
||||
scan_result = ScanResult(
|
||||
scan_type=ScanType.CONTAINER,
|
||||
tool="Trivy",
|
||||
timestamp=datetime.utcnow(),
|
||||
commit_sha="ghi789",
|
||||
branch="main",
|
||||
control_id="SDLC-006",
|
||||
findings={"critical": 0, "high": 15, "medium": 0, "low": 0},
|
||||
)
|
||||
|
||||
alerts = updater._generate_alerts(scan_result, ControlStatusEnum.FAIL.value)
|
||||
|
||||
assert any("HIGH" in alert and "15" in alert for alert in alerts)
|
||||
|
||||
def test_no_alert_for_pass_with_low_findings(self):
|
||||
"""No alert should be generated for PASS status with only low findings."""
|
||||
db = MagicMock()
|
||||
updater = AutoRiskUpdater(db)
|
||||
|
||||
scan_result = ScanResult(
|
||||
scan_type=ScanType.SAST,
|
||||
tool="Semgrep",
|
||||
timestamp=datetime.utcnow(),
|
||||
commit_sha="jkl012",
|
||||
branch="main",
|
||||
control_id="SDLC-001",
|
||||
findings={"critical": 0, "high": 0, "medium": 5, "low": 20},
|
||||
)
|
||||
|
||||
alerts = updater._generate_alerts(scan_result, ControlStatusEnum.PASS.value)
|
||||
|
||||
assert len(alerts) == 0
|
||||
|
||||
|
||||
class TestControlScanMapping:
|
||||
"""Tests for CONTROL_SCAN_MAPPING constant."""
|
||||
|
||||
def test_sdlc_001_maps_to_sast(self):
|
||||
"""SDLC-001 should map to SAST scan type."""
|
||||
assert CONTROL_SCAN_MAPPING["SDLC-001"] == ScanType.SAST
|
||||
|
||||
def test_sdlc_002_maps_to_dependency(self):
|
||||
"""SDLC-002 should map to DEPENDENCY scan type."""
|
||||
assert CONTROL_SCAN_MAPPING["SDLC-002"] == ScanType.DEPENDENCY
|
||||
|
||||
def test_sdlc_003_maps_to_secret(self):
|
||||
"""SDLC-003 should map to SECRET scan type."""
|
||||
assert CONTROL_SCAN_MAPPING["SDLC-003"] == ScanType.SECRET
|
||||
|
||||
def test_sdlc_006_maps_to_container(self):
|
||||
"""SDLC-006 should map to CONTAINER scan type."""
|
||||
assert CONTROL_SCAN_MAPPING["SDLC-006"] == ScanType.CONTAINER
|
||||
|
||||
def test_cra_001_maps_to_sbom(self):
|
||||
"""CRA-001 should map to SBOM scan type."""
|
||||
assert CONTROL_SCAN_MAPPING["CRA-001"] == ScanType.SBOM
|
||||
|
||||
|
||||
class TestProcessEvidenceCollectRequest:
|
||||
"""Tests for process_evidence_collect_request method."""
|
||||
|
||||
def test_parses_iso_timestamp(self):
|
||||
"""Should correctly parse ISO format timestamps."""
|
||||
db = MagicMock()
|
||||
updater = AutoRiskUpdater(db)
|
||||
|
||||
# Mock the control repo to return None (control not found)
|
||||
updater.control_repo.get_by_control_id = MagicMock(return_value=None)
|
||||
|
||||
result = updater.process_evidence_collect_request(
|
||||
tool="Semgrep",
|
||||
control_id="SDLC-001",
|
||||
evidence_type="ci_semgrep",
|
||||
timestamp="2026-01-18T14:30:00Z",
|
||||
commit_sha="abc123",
|
||||
findings={"critical": 0, "high": 0, "medium": 0, "low": 0},
|
||||
)
|
||||
|
||||
# Control not found, so control_updated should be False
|
||||
assert result.control_updated is False
|
||||
|
||||
def test_handles_invalid_timestamp(self):
|
||||
"""Should handle invalid timestamps gracefully."""
|
||||
db = MagicMock()
|
||||
updater = AutoRiskUpdater(db)
|
||||
updater.control_repo.get_by_control_id = MagicMock(return_value=None)
|
||||
|
||||
# Should not raise exception
|
||||
result = updater.process_evidence_collect_request(
|
||||
tool="Trivy",
|
||||
control_id="SDLC-002",
|
||||
evidence_type="ci_trivy",
|
||||
timestamp="invalid-timestamp",
|
||||
commit_sha="def456",
|
||||
findings={"critical": 0, "high": 0, "medium": 0, "low": 0},
|
||||
)
|
||||
|
||||
assert result is not None
|
||||
|
||||
def test_control_not_found_returns_result(self):
|
||||
"""Should return appropriate result when control is not found."""
|
||||
db = MagicMock()
|
||||
updater = AutoRiskUpdater(db)
|
||||
updater.control_repo.get_by_control_id = MagicMock(return_value=None)
|
||||
|
||||
result = updater.process_evidence_collect_request(
|
||||
tool="Gitleaks",
|
||||
control_id="UNKNOWN-001",
|
||||
evidence_type="ci_gitleaks",
|
||||
timestamp="2026-01-18T10:00:00Z",
|
||||
commit_sha="ghi789",
|
||||
findings={"critical": 0, "high": 0, "medium": 0, "low": 0},
|
||||
)
|
||||
|
||||
assert result.control_id == "UNKNOWN-001"
|
||||
assert result.control_updated is False
|
||||
assert "not found" in result.message
|
||||
|
||||
|
||||
class TestScanResult:
|
||||
"""Tests for ScanResult dataclass."""
|
||||
|
||||
def test_scan_result_creation(self):
|
||||
"""Should create ScanResult with all required fields."""
|
||||
result = ScanResult(
|
||||
scan_type=ScanType.SAST,
|
||||
tool="Semgrep",
|
||||
timestamp=datetime(2026, 1, 18, 14, 0),
|
||||
commit_sha="abc123def456",
|
||||
branch="main",
|
||||
control_id="SDLC-001",
|
||||
findings={"critical": 0, "high": 2, "medium": 5, "low": 10},
|
||||
)
|
||||
|
||||
assert result.scan_type == ScanType.SAST
|
||||
assert result.tool == "Semgrep"
|
||||
assert result.control_id == "SDLC-001"
|
||||
assert result.findings["high"] == 2
|
||||
|
||||
def test_scan_result_optional_fields(self):
|
||||
"""Should handle optional fields correctly."""
|
||||
result = ScanResult(
|
||||
scan_type=ScanType.DEPENDENCY,
|
||||
tool="Trivy",
|
||||
timestamp=datetime.utcnow(),
|
||||
commit_sha="xyz789",
|
||||
branch="develop",
|
||||
control_id="SDLC-002",
|
||||
findings={"critical": 1},
|
||||
raw_report={"vulnerabilities": []},
|
||||
ci_job_id="github-actions-12345",
|
||||
)
|
||||
|
||||
assert result.raw_report is not None
|
||||
assert result.ci_job_id == "github-actions-12345"
|
||||
|
||||
|
||||
class TestRiskUpdateResult:
|
||||
"""Tests for RiskUpdateResult dataclass."""
|
||||
|
||||
def test_risk_update_result_creation(self):
|
||||
"""Should create RiskUpdateResult with all fields."""
|
||||
result = RiskUpdateResult(
|
||||
control_id="SDLC-001",
|
||||
control_updated=True,
|
||||
old_status="pass",
|
||||
new_status="fail",
|
||||
evidence_created=True,
|
||||
evidence_id="ev-12345",
|
||||
risks_affected=["RISK-001", "RISK-002"],
|
||||
alerts_generated=["Critical vulnerability found"],
|
||||
message="Processed successfully",
|
||||
)
|
||||
|
||||
assert result.control_updated is True
|
||||
assert result.old_status == "pass"
|
||||
assert result.new_status == "fail"
|
||||
assert len(result.risks_affected) == 2
|
||||
assert len(result.alerts_generated) == 1
|
||||
|
||||
|
||||
class TestFindingSeverity:
|
||||
"""Tests for FindingSeverity enum."""
|
||||
|
||||
def test_severity_levels(self):
|
||||
"""Should have all expected severity levels."""
|
||||
assert FindingSeverity.CRITICAL.value == "critical"
|
||||
assert FindingSeverity.HIGH.value == "high"
|
||||
assert FindingSeverity.MEDIUM.value == "medium"
|
||||
assert FindingSeverity.LOW.value == "low"
|
||||
assert FindingSeverity.INFO.value == "info"
|
||||
|
||||
|
||||
class TestScanType:
|
||||
"""Tests for ScanType enum."""
|
||||
|
||||
def test_scan_types(self):
|
||||
"""Should have all expected scan types."""
|
||||
assert ScanType.SAST.value == "sast"
|
||||
assert ScanType.DEPENDENCY.value == "dependency"
|
||||
assert ScanType.SECRET.value == "secret"
|
||||
assert ScanType.CONTAINER.value == "container"
|
||||
assert ScanType.SBOM.value == "sbom"
|
||||
696
backend/compliance/tests/test_isms_routes.py
Normal file
696
backend/compliance/tests/test_isms_routes.py
Normal file
@@ -0,0 +1,696 @@
|
||||
"""
|
||||
Unit tests for ISMS (Information Security Management System) API Routes.
|
||||
|
||||
Tests all ISO 27001 certification-related endpoints:
|
||||
- ISMS Scope (Chapter 4.3)
|
||||
- ISMS Policies (Chapter 5.2)
|
||||
- Security Objectives (Chapter 6.2)
|
||||
- Statement of Applicability (SoA)
|
||||
- Audit Findings & CAPA
|
||||
- Management Reviews (Chapter 9.3)
|
||||
- Internal Audits (Chapter 9.2)
|
||||
- Readiness Check
|
||||
|
||||
Run with: pytest backend/compliance/tests/test_isms_routes.py -v
|
||||
"""
|
||||
|
||||
import pytest
|
||||
from datetime import datetime, date
|
||||
from unittest.mock import MagicMock, patch
|
||||
from uuid import uuid4
|
||||
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
import sys
|
||||
sys.path.insert(0, '/Users/benjaminadmin/Projekte/breakpilot-pwa/backend')
|
||||
|
||||
from compliance.db.models import (
|
||||
ISMSScopeDB, ISMSContextDB, ISMSPolicyDB, SecurityObjectiveDB,
|
||||
StatementOfApplicabilityDB, AuditFindingDB, CorrectiveActionDB,
|
||||
ManagementReviewDB, InternalAuditDB, AuditTrailDB, ISMSReadinessCheckDB,
|
||||
ApprovalStatusEnum, FindingTypeEnum, FindingStatusEnum, CAPATypeEnum
|
||||
)
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Test Fixtures
|
||||
# ============================================================================
|
||||
|
||||
@pytest.fixture
|
||||
def mock_db():
|
||||
"""Create a mock database session."""
|
||||
return MagicMock(spec=Session)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def sample_scope():
|
||||
"""Create a sample ISMS scope for testing."""
|
||||
return ISMSScopeDB(
|
||||
id=str(uuid4()),
|
||||
scope_statement="BreakPilot ISMS covers all digital learning platform operations",
|
||||
included_locations=["Frankfurt Office", "AWS eu-central-1"],
|
||||
included_processes=["Software Development", "Data Processing", "Customer Support"],
|
||||
included_services=["BreakPilot PWA", "Consent Service", "AI Assistant"],
|
||||
excluded_items=["Marketing Website"],
|
||||
exclusion_justification="Marketing website is static and contains no user data",
|
||||
status=ApprovalStatusEnum.DRAFT,
|
||||
version="1.0",
|
||||
created_by="admin@breakpilot.de",
|
||||
created_at=datetime.utcnow(),
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def sample_approved_scope(sample_scope):
|
||||
"""Create an approved ISMS scope for testing."""
|
||||
sample_scope.status = ApprovalStatusEnum.APPROVED
|
||||
sample_scope.approved_by = "ceo@breakpilot.de"
|
||||
sample_scope.approved_at = datetime.utcnow()
|
||||
sample_scope.effective_date = date.today()
|
||||
sample_scope.review_date = date(date.today().year + 1, date.today().month, date.today().day)
|
||||
sample_scope.approval_signature = "sha256_signature_hash"
|
||||
return sample_scope
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def sample_policy():
|
||||
"""Create a sample ISMS policy for testing."""
|
||||
return ISMSPolicyDB(
|
||||
id=str(uuid4()),
|
||||
policy_id="POL-ISMS-001",
|
||||
title="Information Security Policy",
|
||||
policy_type="master",
|
||||
description="Master ISMS policy for BreakPilot",
|
||||
policy_text="This policy establishes the framework for information security...",
|
||||
applies_to=["All Employees", "Contractors", "Partners"],
|
||||
review_frequency_months=12,
|
||||
related_controls=["GOV-001", "GOV-002"],
|
||||
authored_by="iso@breakpilot.de",
|
||||
status=ApprovalStatusEnum.DRAFT,
|
||||
version="1.0",
|
||||
created_at=datetime.utcnow(),
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def sample_objective():
|
||||
"""Create a sample security objective for testing."""
|
||||
return SecurityObjectiveDB(
|
||||
id=str(uuid4()),
|
||||
objective_id="OBJ-2026-001",
|
||||
title="Reduce Security Incidents",
|
||||
description="Reduce the number of security incidents by 30% compared to previous year",
|
||||
category="operational",
|
||||
specific="Reduce security incidents from 10 to 7 per year",
|
||||
measurable="Number of security incidents recorded in ticketing system",
|
||||
achievable="Based on trend analysis and planned control improvements",
|
||||
relevant="Directly supports information security goals",
|
||||
time_bound="By end of Q4 2026",
|
||||
kpi_name="Security Incident Count",
|
||||
kpi_target=7.0,
|
||||
kpi_unit="incidents/year",
|
||||
kpi_current=10.0,
|
||||
measurement_frequency="monthly",
|
||||
owner="security@breakpilot.de",
|
||||
target_date=date(2026, 12, 31),
|
||||
related_controls=["OPS-003"],
|
||||
status="active",
|
||||
progress_percentage=0.0,
|
||||
created_at=datetime.utcnow(),
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def sample_soa_entry():
|
||||
"""Create a sample SoA entry for testing."""
|
||||
return StatementOfApplicabilityDB(
|
||||
id=str(uuid4()),
|
||||
annex_a_control="A.5.1",
|
||||
annex_a_title="Policies for information security",
|
||||
annex_a_category="organizational",
|
||||
is_applicable=True,
|
||||
applicability_justification="Required for ISMS governance",
|
||||
implementation_status="implemented",
|
||||
implementation_notes="Implemented via GOV-001, GOV-002 controls",
|
||||
breakpilot_control_ids=["GOV-001", "GOV-002"],
|
||||
coverage_level="full",
|
||||
evidence_description="ISMS Policy v2.0, signed by CEO",
|
||||
version="1.0",
|
||||
created_at=datetime.utcnow(),
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def sample_finding():
|
||||
"""Create a sample audit finding for testing."""
|
||||
return AuditFindingDB(
|
||||
id=str(uuid4()),
|
||||
finding_id="FIND-2026-001",
|
||||
finding_type=FindingTypeEnum.MINOR,
|
||||
iso_chapter="9.2",
|
||||
annex_a_control="A.5.35",
|
||||
title="Internal audit schedule not documented",
|
||||
description="The internal audit schedule for 2026 was not formally documented",
|
||||
objective_evidence="No document found in DMS",
|
||||
impact_description="Cannot demonstrate planned approach to internal audits",
|
||||
owner="iso@breakpilot.de",
|
||||
auditor="external.auditor@cert.de",
|
||||
identified_date=date.today(),
|
||||
due_date=date(2026, 3, 31),
|
||||
status=FindingStatusEnum.OPEN,
|
||||
created_at=datetime.utcnow(),
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def sample_major_finding():
|
||||
"""Create a major finding (certification blocking)."""
|
||||
return AuditFindingDB(
|
||||
id=str(uuid4()),
|
||||
finding_id="FIND-2026-002",
|
||||
finding_type=FindingTypeEnum.MAJOR,
|
||||
iso_chapter="5.2",
|
||||
title="Information Security Policy not approved",
|
||||
description="The ISMS master policy has not been approved by top management",
|
||||
objective_evidence="Policy document shows 'Draft' status",
|
||||
owner="ceo@breakpilot.de",
|
||||
auditor="external.auditor@cert.de",
|
||||
identified_date=date.today(),
|
||||
due_date=date(2026, 2, 28),
|
||||
status=FindingStatusEnum.OPEN,
|
||||
created_at=datetime.utcnow(),
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def sample_capa(sample_finding):
|
||||
"""Create a sample CAPA for testing."""
|
||||
return CorrectiveActionDB(
|
||||
id=str(uuid4()),
|
||||
capa_id="CAPA-2026-001",
|
||||
finding_id=sample_finding.id,
|
||||
capa_type=CAPATypeEnum.CORRECTIVE,
|
||||
title="Create and approve internal audit schedule",
|
||||
description="Create a formal internal audit schedule document and get management approval",
|
||||
expected_outcome="Approved internal audit schedule for 2026",
|
||||
assigned_to="iso@breakpilot.de",
|
||||
planned_start=date.today(),
|
||||
planned_completion=date(2026, 2, 15),
|
||||
effectiveness_criteria="Document approved and distributed to audit team",
|
||||
status="planned",
|
||||
created_at=datetime.utcnow(),
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def sample_management_review():
|
||||
"""Create a sample management review for testing."""
|
||||
return ManagementReviewDB(
|
||||
id=str(uuid4()),
|
||||
review_id="MR-2026-Q1",
|
||||
title="Q1 2026 Management Review",
|
||||
review_date=date(2026, 1, 15),
|
||||
review_period_start=date(2025, 10, 1),
|
||||
review_period_end=date(2025, 12, 31),
|
||||
chairperson="ceo@breakpilot.de",
|
||||
attendees=[
|
||||
{"name": "CEO", "role": "Chairperson"},
|
||||
{"name": "CTO", "role": "Technical Lead"},
|
||||
{"name": "ISO", "role": "ISMS Manager"},
|
||||
],
|
||||
status="draft",
|
||||
created_at=datetime.utcnow(),
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def sample_internal_audit():
|
||||
"""Create a sample internal audit for testing."""
|
||||
return InternalAuditDB(
|
||||
id=str(uuid4()),
|
||||
audit_id="IA-2026-001",
|
||||
title="Annual ISMS Internal Audit 2026",
|
||||
audit_type="full",
|
||||
scope_description="Complete ISMS audit covering all ISO 27001 chapters and Annex A controls",
|
||||
iso_chapters_covered=["4", "5", "6", "7", "8", "9", "10"],
|
||||
annex_a_controls_covered=["A.5", "A.6", "A.7", "A.8"],
|
||||
criteria="ISO 27001:2022, Internal ISMS procedures",
|
||||
planned_date=date(2026, 3, 1),
|
||||
lead_auditor="internal.auditor@breakpilot.de",
|
||||
audit_team=["internal.auditor@breakpilot.de", "qa@breakpilot.de"],
|
||||
status="planned",
|
||||
created_at=datetime.utcnow(),
|
||||
)
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Test: ISMS Scope
|
||||
# ============================================================================
|
||||
|
||||
class TestISMSScope:
|
||||
"""Tests for ISMS Scope endpoints."""
|
||||
|
||||
def test_scope_has_required_fields(self, sample_scope):
|
||||
"""ISMS scope should have all required fields."""
|
||||
assert sample_scope.scope_statement is not None
|
||||
assert sample_scope.status == ApprovalStatusEnum.DRAFT
|
||||
assert sample_scope.created_by is not None
|
||||
|
||||
def test_scope_approval_sets_correct_fields(self, sample_approved_scope):
|
||||
"""Approving scope should set approval fields."""
|
||||
assert sample_approved_scope.status == ApprovalStatusEnum.APPROVED
|
||||
assert sample_approved_scope.approved_by is not None
|
||||
assert sample_approved_scope.approved_at is not None
|
||||
assert sample_approved_scope.effective_date is not None
|
||||
assert sample_approved_scope.review_date is not None
|
||||
assert sample_approved_scope.approval_signature is not None
|
||||
|
||||
def test_scope_can_include_multiple_locations(self, sample_scope):
|
||||
"""Scope should support multiple locations."""
|
||||
assert isinstance(sample_scope.included_locations, list)
|
||||
assert len(sample_scope.included_locations) >= 1
|
||||
|
||||
def test_scope_exclusions_require_justification(self, sample_scope):
|
||||
"""Scope exclusions should have justification (ISO 27001 requirement)."""
|
||||
if sample_scope.excluded_items:
|
||||
assert sample_scope.exclusion_justification is not None
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Test: ISMS Policies
|
||||
# ============================================================================
|
||||
|
||||
class TestISMSPolicy:
|
||||
"""Tests for ISMS Policy endpoints."""
|
||||
|
||||
def test_policy_has_unique_id(self, sample_policy):
|
||||
"""Each policy should have a unique policy_id."""
|
||||
assert sample_policy.policy_id is not None
|
||||
assert sample_policy.policy_id.startswith("POL-")
|
||||
|
||||
def test_master_policy_type_exists(self, sample_policy):
|
||||
"""Master policy type should be 'master'."""
|
||||
assert sample_policy.policy_type == "master"
|
||||
|
||||
def test_policy_has_review_frequency(self, sample_policy):
|
||||
"""Policy should specify review frequency."""
|
||||
assert sample_policy.review_frequency_months > 0
|
||||
assert sample_policy.review_frequency_months <= 36 # Max 3 years
|
||||
|
||||
def test_policy_can_link_to_controls(self, sample_policy):
|
||||
"""Policy should link to related controls."""
|
||||
assert isinstance(sample_policy.related_controls, list)
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Test: Security Objectives
|
||||
# ============================================================================
|
||||
|
||||
class TestSecurityObjective:
|
||||
"""Tests for Security Objectives endpoints."""
|
||||
|
||||
def test_objective_follows_smart_criteria(self, sample_objective):
|
||||
"""Objectives should follow SMART criteria."""
|
||||
# S - Specific
|
||||
assert sample_objective.specific is not None
|
||||
# M - Measurable
|
||||
assert sample_objective.measurable is not None
|
||||
# A - Achievable
|
||||
assert sample_objective.achievable is not None
|
||||
# R - Relevant
|
||||
assert sample_objective.relevant is not None
|
||||
# T - Time-bound
|
||||
assert sample_objective.time_bound is not None
|
||||
|
||||
def test_objective_has_kpi(self, sample_objective):
|
||||
"""Objectives should have measurable KPIs."""
|
||||
assert sample_objective.kpi_name is not None
|
||||
assert sample_objective.kpi_target is not None
|
||||
assert sample_objective.kpi_unit is not None
|
||||
|
||||
def test_objective_progress_calculation(self, sample_objective):
|
||||
"""Objective progress should be calculable."""
|
||||
if sample_objective.kpi_target and sample_objective.kpi_current:
|
||||
# Progress towards reducing incidents (lower is better for this KPI)
|
||||
expected_progress = max(0, min(100,
|
||||
(sample_objective.kpi_target / sample_objective.kpi_current) * 100
|
||||
))
|
||||
assert expected_progress >= 0
|
||||
assert expected_progress <= 100
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Test: Statement of Applicability (SoA)
|
||||
# ============================================================================
|
||||
|
||||
class TestStatementOfApplicability:
|
||||
"""Tests for SoA endpoints."""
|
||||
|
||||
def test_soa_entry_has_annex_a_reference(self, sample_soa_entry):
|
||||
"""SoA entry should reference Annex A control."""
|
||||
assert sample_soa_entry.annex_a_control is not None
|
||||
assert sample_soa_entry.annex_a_control.startswith("A.")
|
||||
|
||||
def test_soa_entry_requires_justification_for_not_applicable(self):
|
||||
"""Non-applicable controls must have justification."""
|
||||
soa_entry = StatementOfApplicabilityDB(
|
||||
id=str(uuid4()),
|
||||
annex_a_control="A.7.2",
|
||||
annex_a_title="Physical entry",
|
||||
annex_a_category="physical",
|
||||
is_applicable=False,
|
||||
applicability_justification="Cloud-only infrastructure, no physical data center",
|
||||
)
|
||||
assert not soa_entry.is_applicable
|
||||
assert soa_entry.applicability_justification is not None
|
||||
|
||||
def test_soa_entry_tracks_implementation_status(self, sample_soa_entry):
|
||||
"""SoA should track implementation status."""
|
||||
valid_statuses = ["planned", "in_progress", "implemented", "not_implemented"]
|
||||
assert sample_soa_entry.implementation_status in valid_statuses
|
||||
|
||||
def test_soa_entry_maps_to_breakpilot_controls(self, sample_soa_entry):
|
||||
"""SoA should map Annex A controls to Breakpilot controls."""
|
||||
assert isinstance(sample_soa_entry.breakpilot_control_ids, list)
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Test: Audit Findings
|
||||
# ============================================================================
|
||||
|
||||
class TestAuditFinding:
|
||||
"""Tests for Audit Finding endpoints."""
|
||||
|
||||
def test_finding_has_classification(self, sample_finding):
|
||||
"""Finding should have Major/Minor/OFI classification."""
|
||||
valid_types = [FindingTypeEnum.MAJOR, FindingTypeEnum.MINOR,
|
||||
FindingTypeEnum.OFI, FindingTypeEnum.POSITIVE]
|
||||
assert sample_finding.finding_type in valid_types
|
||||
|
||||
def test_major_finding_blocks_certification(self, sample_major_finding):
|
||||
"""Major findings should be identified as certification blocking."""
|
||||
assert sample_major_finding.finding_type == FindingTypeEnum.MAJOR
|
||||
# is_blocking is a property, so we check the type
|
||||
is_blocking = (sample_major_finding.finding_type == FindingTypeEnum.MAJOR and
|
||||
sample_major_finding.status != FindingStatusEnum.CLOSED)
|
||||
assert is_blocking == True
|
||||
|
||||
def test_finding_has_objective_evidence(self, sample_finding):
|
||||
"""Findings should have objective evidence."""
|
||||
assert sample_finding.objective_evidence is not None
|
||||
|
||||
def test_finding_has_due_date(self, sample_finding):
|
||||
"""Findings should have a due date for closure."""
|
||||
assert sample_finding.due_date is not None
|
||||
|
||||
def test_finding_lifecycle_statuses(self):
|
||||
"""Finding should follow proper lifecycle."""
|
||||
valid_statuses = [
|
||||
FindingStatusEnum.OPEN,
|
||||
FindingStatusEnum.CORRECTIVE_ACTION_PENDING,
|
||||
FindingStatusEnum.VERIFICATION_PENDING,
|
||||
FindingStatusEnum.CLOSED,
|
||||
]
|
||||
for status in valid_statuses:
|
||||
assert status in FindingStatusEnum
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Test: Corrective Actions (CAPA)
|
||||
# ============================================================================
|
||||
|
||||
class TestCorrectiveAction:
|
||||
"""Tests for CAPA endpoints."""
|
||||
|
||||
def test_capa_links_to_finding(self, sample_capa, sample_finding):
|
||||
"""CAPA should link to a finding."""
|
||||
assert sample_capa.finding_id == sample_finding.id
|
||||
|
||||
def test_capa_has_type(self, sample_capa):
|
||||
"""CAPA should have corrective or preventive type."""
|
||||
valid_types = [CAPATypeEnum.CORRECTIVE, CAPATypeEnum.PREVENTIVE]
|
||||
assert sample_capa.capa_type in valid_types
|
||||
|
||||
def test_capa_has_effectiveness_criteria(self, sample_capa):
|
||||
"""CAPA should define how effectiveness will be verified."""
|
||||
assert sample_capa.effectiveness_criteria is not None
|
||||
|
||||
def test_capa_has_completion_date(self, sample_capa):
|
||||
"""CAPA should have planned completion date."""
|
||||
assert sample_capa.planned_completion is not None
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Test: Management Review
|
||||
# ============================================================================
|
||||
|
||||
class TestManagementReview:
|
||||
"""Tests for Management Review endpoints."""
|
||||
|
||||
def test_review_has_chairperson(self, sample_management_review):
|
||||
"""Management review must have a chairperson (top management)."""
|
||||
assert sample_management_review.chairperson is not None
|
||||
|
||||
def test_review_has_review_period(self, sample_management_review):
|
||||
"""Review should cover a specific period."""
|
||||
assert sample_management_review.review_period_start is not None
|
||||
assert sample_management_review.review_period_end is not None
|
||||
|
||||
def test_review_id_includes_quarter(self, sample_management_review):
|
||||
"""Review ID should indicate the quarter."""
|
||||
assert "Q" in sample_management_review.review_id
|
||||
|
||||
def test_review_tracks_attendees(self, sample_management_review):
|
||||
"""Review should track attendees."""
|
||||
assert sample_management_review.attendees is not None
|
||||
assert len(sample_management_review.attendees) >= 1
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Test: Internal Audit
|
||||
# ============================================================================
|
||||
|
||||
class TestInternalAudit:
|
||||
"""Tests for Internal Audit endpoints."""
|
||||
|
||||
def test_audit_has_scope(self, sample_internal_audit):
|
||||
"""Internal audit should define scope."""
|
||||
assert sample_internal_audit.scope_description is not None
|
||||
|
||||
def test_audit_covers_iso_chapters(self, sample_internal_audit):
|
||||
"""Audit should specify which ISO chapters are covered."""
|
||||
assert sample_internal_audit.iso_chapters_covered is not None
|
||||
assert len(sample_internal_audit.iso_chapters_covered) >= 1
|
||||
|
||||
def test_audit_has_lead_auditor(self, sample_internal_audit):
|
||||
"""Audit should have a lead auditor."""
|
||||
assert sample_internal_audit.lead_auditor is not None
|
||||
|
||||
def test_audit_has_criteria(self, sample_internal_audit):
|
||||
"""Audit should define audit criteria."""
|
||||
assert sample_internal_audit.criteria is not None
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Test: ISMS Readiness Check
|
||||
# ============================================================================
|
||||
|
||||
class TestISMSReadinessCheck:
|
||||
"""Tests for ISMS Readiness Check."""
|
||||
|
||||
def test_readiness_check_identifies_potential_majors(self):
|
||||
"""Readiness check should identify potential major findings."""
|
||||
check = ISMSReadinessCheckDB(
|
||||
id=str(uuid4()),
|
||||
check_date=datetime.utcnow(),
|
||||
triggered_by="admin@breakpilot.de",
|
||||
overall_status="not_ready",
|
||||
certification_possible=False,
|
||||
chapter_4_status="fail",
|
||||
chapter_5_status="fail",
|
||||
chapter_6_status="warning",
|
||||
chapter_7_status="pass",
|
||||
chapter_8_status="pass",
|
||||
chapter_9_status="fail",
|
||||
chapter_10_status="pass",
|
||||
potential_majors=[
|
||||
{"check": "ISMS Scope not approved", "iso_reference": "4.3"},
|
||||
{"check": "Master policy not approved", "iso_reference": "5.2"},
|
||||
{"check": "No internal audit conducted", "iso_reference": "9.2"},
|
||||
],
|
||||
potential_minors=[
|
||||
{"check": "Risk treatment incomplete", "iso_reference": "6.1.2"},
|
||||
],
|
||||
readiness_score=30.0,
|
||||
)
|
||||
|
||||
assert check.certification_possible == False
|
||||
assert len(check.potential_majors) >= 1
|
||||
assert check.readiness_score < 100
|
||||
|
||||
def test_readiness_check_shows_chapter_status(self):
|
||||
"""Readiness check should show status for each ISO chapter."""
|
||||
check = ISMSReadinessCheckDB(
|
||||
id=str(uuid4()),
|
||||
check_date=datetime.utcnow(),
|
||||
triggered_by="admin@breakpilot.de",
|
||||
overall_status="ready",
|
||||
certification_possible=True,
|
||||
chapter_4_status="pass",
|
||||
chapter_5_status="pass",
|
||||
chapter_6_status="pass",
|
||||
chapter_7_status="pass",
|
||||
chapter_8_status="pass",
|
||||
chapter_9_status="pass",
|
||||
chapter_10_status="pass",
|
||||
potential_majors=[],
|
||||
potential_minors=[],
|
||||
readiness_score=100.0,
|
||||
)
|
||||
|
||||
assert check.chapter_4_status == "pass"
|
||||
assert check.chapter_5_status == "pass"
|
||||
assert check.chapter_9_status == "pass"
|
||||
assert check.certification_possible == True
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Test: ISO 27001 Annex A Coverage
|
||||
# ============================================================================
|
||||
|
||||
class TestAnnexACoverage:
|
||||
"""Tests for ISO 27001 Annex A control coverage."""
|
||||
|
||||
def test_annex_a_has_93_controls(self):
|
||||
"""ISO 27001:2022 has exactly 93 controls."""
|
||||
from compliance.data.iso27001_annex_a import ISO27001_ANNEX_A_CONTROLS, ANNEX_A_SUMMARY
|
||||
|
||||
assert len(ISO27001_ANNEX_A_CONTROLS) == 93
|
||||
assert ANNEX_A_SUMMARY["total_controls"] == 93
|
||||
|
||||
def test_annex_a_categories(self):
|
||||
"""Annex A should have 4 control categories."""
|
||||
from compliance.data.iso27001_annex_a import ANNEX_A_SUMMARY
|
||||
|
||||
# A.5 Organizational (37), A.6 People (8), A.7 Physical (14), A.8 Technological (34)
|
||||
assert ANNEX_A_SUMMARY["organizational_controls"] == 37
|
||||
assert ANNEX_A_SUMMARY["people_controls"] == 8
|
||||
assert ANNEX_A_SUMMARY["physical_controls"] == 14
|
||||
assert ANNEX_A_SUMMARY["technological_controls"] == 34
|
||||
|
||||
def test_annex_a_control_structure(self):
|
||||
"""Each Annex A control should have required fields."""
|
||||
from compliance.data.iso27001_annex_a import ISO27001_ANNEX_A_CONTROLS
|
||||
|
||||
for control in ISO27001_ANNEX_A_CONTROLS:
|
||||
assert "control_id" in control
|
||||
assert "title" in control
|
||||
assert "category" in control
|
||||
assert "description" in control
|
||||
assert control["control_id"].startswith("A.")
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Test: Audit Trail
|
||||
# ============================================================================
|
||||
|
||||
class TestAuditTrail:
|
||||
"""Tests for Audit Trail functionality."""
|
||||
|
||||
def test_audit_trail_entry_has_required_fields(self):
|
||||
"""Audit trail entry should have all required fields."""
|
||||
entry = AuditTrailDB(
|
||||
id=str(uuid4()),
|
||||
entity_type="isms_scope",
|
||||
entity_id=str(uuid4()),
|
||||
entity_name="ISMS Scope v1.0",
|
||||
action="approve",
|
||||
performed_by="ceo@breakpilot.de",
|
||||
performed_at=datetime.utcnow(),
|
||||
checksum="sha256_hash",
|
||||
)
|
||||
|
||||
assert entry.entity_type is not None
|
||||
assert entry.entity_id is not None
|
||||
assert entry.action is not None
|
||||
assert entry.performed_by is not None
|
||||
assert entry.performed_at is not None
|
||||
assert entry.checksum is not None
|
||||
|
||||
def test_audit_trail_tracks_changes(self):
|
||||
"""Audit trail should track field changes."""
|
||||
entry = AuditTrailDB(
|
||||
id=str(uuid4()),
|
||||
entity_type="isms_policy",
|
||||
entity_id=str(uuid4()),
|
||||
entity_name="POL-ISMS-001",
|
||||
action="update",
|
||||
field_changed="status",
|
||||
old_value="draft",
|
||||
new_value="approved",
|
||||
change_summary="Policy approved by CEO",
|
||||
performed_by="ceo@breakpilot.de",
|
||||
performed_at=datetime.utcnow(),
|
||||
checksum="sha256_hash",
|
||||
)
|
||||
|
||||
assert entry.field_changed == "status"
|
||||
assert entry.old_value == "draft"
|
||||
assert entry.new_value == "approved"
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Test: Certification Blockers
|
||||
# ============================================================================
|
||||
|
||||
class TestCertificationBlockers:
|
||||
"""Tests for certification blocking scenarios."""
|
||||
|
||||
def test_open_major_blocks_certification(self):
|
||||
"""Open major findings should block certification."""
|
||||
finding = AuditFindingDB(
|
||||
id=str(uuid4()),
|
||||
finding_id="FIND-2026-001",
|
||||
finding_type=FindingTypeEnum.MAJOR,
|
||||
title="Critical finding",
|
||||
description="Test",
|
||||
auditor="auditor@test.de",
|
||||
status=FindingStatusEnum.OPEN,
|
||||
)
|
||||
|
||||
is_blocking = (finding.finding_type == FindingTypeEnum.MAJOR and
|
||||
finding.status != FindingStatusEnum.CLOSED)
|
||||
assert is_blocking == True
|
||||
|
||||
def test_closed_major_allows_certification(self):
|
||||
"""Closed major findings should not block certification."""
|
||||
finding = AuditFindingDB(
|
||||
id=str(uuid4()),
|
||||
finding_id="FIND-2026-001",
|
||||
finding_type=FindingTypeEnum.MAJOR,
|
||||
title="Critical finding",
|
||||
description="Test",
|
||||
auditor="auditor@test.de",
|
||||
status=FindingStatusEnum.CLOSED,
|
||||
closed_date=date.today(),
|
||||
)
|
||||
|
||||
is_blocking = (finding.finding_type == FindingTypeEnum.MAJOR and
|
||||
finding.status != FindingStatusEnum.CLOSED)
|
||||
assert is_blocking == False
|
||||
|
||||
def test_minor_findings_dont_block_certification(self):
|
||||
"""Minor findings should not block certification."""
|
||||
finding = AuditFindingDB(
|
||||
id=str(uuid4()),
|
||||
finding_id="FIND-2026-002",
|
||||
finding_type=FindingTypeEnum.MINOR,
|
||||
title="Minor finding",
|
||||
description="Test",
|
||||
auditor="auditor@test.de",
|
||||
status=FindingStatusEnum.OPEN,
|
||||
)
|
||||
|
||||
is_blocking = (finding.finding_type == FindingTypeEnum.MAJOR and
|
||||
finding.status != FindingStatusEnum.CLOSED)
|
||||
assert is_blocking == False
|
||||
Reference in New Issue
Block a user