Commit Graph

12 Commits

Author SHA1 Message Date
Sharang Parnerkar
883ef702ac tech-debt: mypy --strict config + integration tests for audit routes
Phase 1 Step 4 follow-up addressing the debt flagged in the worked-example
commit (4a91814).

## mypy --strict policy

Adds backend-compliance/mypy.ini declaring the strict-mode scope:

  Fully strict (enforced today):
    - compliance/domain/
    - compliance/schemas/
    - compliance/api/_http_errors.py
    - compliance/api/audit_routes.py        (refactored in Step 4)
    - compliance/services/audit_session_service.py
    - compliance/services/audit_signoff_service.py

  Loose (ignore_errors=True) with a migration path:
    - compliance/db/*                        — SQLAlchemy 1.x Column[] vs
                                               runtime T; unblocks Phase 1
                                               until a Mapped[T] migration.
    - compliance/api/<route>.py              — each route file flips to
                                               strict as its own Step 4
                                               refactor lands.
    - compliance/services/<legacy util>      — 14 utility services
                                               (llm_provider, pdf_extractor,
                                               seeder, ...) that predate the
                                               clean-arch refactor.
    - compliance/tests/                      — excluded (legacy placeholder
                                               style). The new TestClient-
                                               based integration suite is
                                               type-annotated.

The two new service files carry a scoped `# mypy: disable-error-code="arg-type,assignment"`
header for the ORM Column[T] issue — same underlying SQLAlchemy limitation,
narrowly scoped rather than wholesale ignore_errors.

Flow: `cd backend-compliance && mypy compliance/` -> clean on 119 files.
CI yaml updated to use the config instead of ad-hoc package lists.

## Bugs fixed while enabling strict

mypy --strict surfaced two latent bugs in the pre-refactor code. Both
were invisible because the old `compliance/tests/test_audit_routes.py`
is a placeholder suite that asserts on request-data shape and never
calls the handlers:

  - AuditSessionResponse.updated_at is a required field in the schema,
    but the original handler didn't pass it. Fixed in
    AuditSessionService._to_response.

  - PaginationMeta requires has_next + has_prev. The original audit
    checklist handler didn't compute them. Fixed in
    AuditSignOffService.get_checklist.

Both are behavior-preserving at the HTTP level because the old code
would have raised Pydantic ValidationError at response serialization
had the endpoint actually been exercised.

## Integration test suite

Adds backend-compliance/tests/test_audit_routes_integration.py — 26
real TestClient tests against an in-memory sqlite backend (StaticPool).
Replaces the coverage gap left by the placeholder suite.

Covers:
  - Session CRUD + lifecycle transitions (draft -> in_progress -> completed
    -> archived), including the 409 paths for illegal transitions
  - Checklist pagination, filtering, search
  - Sign-off create / update / auto-start-session / count-flipping
  - Sign-off 400 (invalid result), 404 (missing requirement), 409 (completed session)
  - Get-signoff 404 / 200 round-trip

Uses a module-scoped schema fixture + per-test DELETE-sweep so the
suite runs in ~2.3s despite the ~50-table ORM surface.

Verified:
  - 199/199 pytest (173 original + 26 new audit integration) pass
  - tests/contracts/test_openapi_baseline.py green, OpenAPI 360/484 unchanged
  - mypy compliance/ -> Success: no issues found in 119 source files

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 18:39:40 +02:00
Sharang Parnerkar
3320ef94fc refactor: phase 0 guardrails + phase 1 step 2 (models.py split)
Squash of branch refactor/phase0-guardrails-and-models-split — 4 commits,
81 files, 173/173 pytest green, OpenAPI contract preserved (360 paths /
484 operations).

## Phase 0 — Architecture guardrails

Three defense-in-depth layers to keep the architecture rules enforced
regardless of who opens Claude Code in this repo:

  1. .claude/settings.json PreToolUse hook on Write/Edit blocks any file
     that would exceed the 500-line hard cap. Auto-loads in every Claude
     session in this repo.
  2. scripts/githooks/pre-commit (install via scripts/install-hooks.sh)
     enforces the LOC cap locally, freezes migrations/ without
     [migration-approved], and protects guardrail files without
     [guardrail-change].
  3. .gitea/workflows/ci.yaml gains loc-budget + guardrail-integrity +
     sbom-scan (syft+grype) jobs, adds mypy --strict for the new Python
     packages (compliance/{services,repositories,domain,schemas}), and
     tsc --noEmit for admin-compliance + developer-portal.

Per-language conventions documented in AGENTS.python.md, AGENTS.go.md,
AGENTS.typescript.md at the repo root — layering, tooling, and explicit
"what you may NOT do" lists. Root CLAUDE.md is prepended with the six
non-negotiable rules. Each of the 10 services gets a README.md.

scripts/check-loc.sh enforces soft 300 / hard 500 and surfaces the
current baseline of 205 hard + 161 soft violations so Phases 1-4 can
drain it incrementally. CI gates only CHANGED files in PRs so the
legacy baseline does not block unrelated work.

## Deprecation sweep

47 files. Pydantic V1 regex= -> pattern= (2 sites), class Config ->
ConfigDict in source_policy_router.py (schemas.py intentionally skipped;
it is the Phase 1 Step 3 split target). datetime.utcnow() ->
datetime.now(timezone.utc) everywhere including SQLAlchemy default=
callables. All DB columns already declare timezone=True, so this is a
latent-bug fix at the Python side, not a schema change.

DeprecationWarning count dropped from 158 to 35.

## Phase 1 Step 1 — Contract test harness

tests/contracts/test_openapi_baseline.py diffs the live FastAPI /openapi.json
against tests/contracts/openapi.baseline.json on every test run. Fails on
removed paths, removed status codes, or new required request body fields.
Regenerate only via tests/contracts/regenerate_baseline.py after a
consumer-updated contract change. This is the safety harness for all
subsequent refactor commits.

## Phase 1 Step 2 — models.py split (1466 -> 85 LOC shim)

compliance/db/models.py is decomposed into seven sibling aggregate modules
following the existing repo pattern (dsr_models.py, vvt_models.py, ...):

  regulation_models.py       (134) — Regulation, Requirement
  control_models.py          (279) — Control, Mapping, Evidence, Risk
  ai_system_models.py        (141) — AISystem, AuditExport
  service_module_models.py   (176) — ServiceModule, ModuleRegulation, ModuleRisk
  audit_session_models.py    (177) — AuditSession, AuditSignOff
  isms_governance_models.py  (323) — ISMSScope, Context, Policy, Objective, SoA
  isms_audit_models.py       (468) — Finding, CAPA, MgmtReview, InternalAudit,
                                     AuditTrail, Readiness

models.py becomes an 85-line re-export shim in dependency order so
existing imports continue to work unchanged. Schema is byte-identical:
__tablename__, column definitions, relationship strings, back_populates,
cascade directives all preserved.

All new sibling files are under the 500-line hard cap; largest is
isms_audit_models.py at 468. No file in compliance/db/ now exceeds
the hard cap.

## Phase 1 Step 3 — infrastructure only

backend-compliance/compliance/{schemas,domain,repositories}/ packages
are created as landing zones with docstrings. compliance/domain/
exports DomainError / NotFoundError / ConflictError / ValidationError /
PermissionError — the base classes services will use to raise
domain-level errors instead of HTTPException.

PHASE1_RUNBOOK.md at backend-compliance/PHASE1_RUNBOOK.md documents
the nine-step execution plan for Phase 1: snapshot baseline,
characterization tests, split models.py (this commit), split schemas.py
(next), extract services, extract repositories, mypy --strict, coverage.

## Verification

  backend-compliance/.venv-phase1: uv python install 3.12 + pip -r requirements.txt
  PYTHONPATH=. pytest compliance/tests/ tests/contracts/
  -> 173 passed, 0 failed, 35 warnings, OpenAPI 360/484 unchanged

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 13:18:29 +02:00
Sharang Parnerkar
559d7960a2 Replace deploy-hetzner with Coolify webhook deploy
Some checks failed
CI/CD / go-lint (pull_request) Failing after 15s
CI/CD / python-lint (pull_request) Failing after 12s
CI/CD / nodejs-lint (pull_request) Failing after 2s
CI/CD / test-go-ai-compliance (pull_request) Failing after 2s
CI/CD / test-python-backend-compliance (pull_request) Failing after 11s
CI/CD / test-python-document-crawler (pull_request) Failing after 11s
CI/CD / test-python-dsms-gateway (pull_request) Failing after 10s
CI/CD / validate-canonical-controls (pull_request) Failing after 9s
CI/CD / Deploy (pull_request) Has been skipped
Deploy to Coolify / deploy (push) Has been cancelled
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-13 10:39:12 +01:00
Benjamin Admin
050f353192 feat(canonical-controls): Canonical Control Library — rechtssichere Security Controls
All checks were successful
CI/CD / go-lint (push) Has been skipped
CI/CD / python-lint (push) Has been skipped
CI/CD / nodejs-lint (push) Has been skipped
CI/CD / test-go-ai-compliance (push) Successful in 40s
CI/CD / test-python-backend-compliance (push) Successful in 41s
CI/CD / test-python-document-crawler (push) Successful in 26s
CI/CD / test-python-dsms-gateway (push) Successful in 23s
CI/CD / validate-canonical-controls (push) Successful in 18s
CI/CD / deploy-hetzner (push) Successful in 2m26s
Eigenstaendig formulierte Security Controls mit unabhaengiger Taxonomie
und Open-Source-Verankerung (OWASP, NIST, ENISA). Keine BSI-Nomenklatur.

- Migration 044: 5 DB-Tabellen (frameworks, controls, sources, licenses, mappings)
- 10 Seed Controls mit 39 Open-Source-Referenzen
- License Gate: Quellen-Berechtigungspruefung (analysis/excerpt/embeddings/product)
- Too-Close-Detektor: 5 Metriken (exact-phrase, token-overlap, ngram, embedding, LCS)
- REST API: 8 Endpoints unter /v1/canonical/
- Go Loader mit Multi-Index (ID, domain, severity, framework)
- Frontend: Control Library Browser + Provenance Wiki
- CI/CD: validate-controls.py Job (schema, no-leak, open-anchors)
- 67 Tests (8 Go + 59 Python), alle PASS
- MkDocs Dokumentation

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-12 19:55:06 +01:00
Benjamin Admin
0b836f7e2d fix(ci): Run docker compose from helper container with deploy dir mounted
All checks were successful
CI/CD / go-lint (push) Has been skipped
CI/CD / python-lint (push) Has been skipped
CI/CD / nodejs-lint (push) Has been skipped
CI/CD / test-go-ai-compliance (push) Successful in 38s
CI/CD / test-python-backend-compliance (push) Successful in 41s
CI/CD / test-python-document-crawler (push) Successful in 25s
CI/CD / test-python-dsms-gateway (push) Successful in 21s
CI/CD / deploy-hetzner (push) Successful in 1m27s
The runner container has Docker socket but no host filesystem access.
docker compose needs to read YAML files, so run build+deploy inside
a helper container that has both Docker socket and the deploy dir mounted.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-11 18:31:19 +01:00
Benjamin Admin
18d9eec654 fix(ci): Use --entrypoint sh for alpine/git (default entrypoint is git)
Some checks failed
CI/CD / go-lint (push) Has been skipped
CI/CD / python-lint (push) Has been skipped
CI/CD / nodejs-lint (push) Has been skipped
CI/CD / test-go-ai-compliance (push) Successful in 35s
CI/CD / test-python-backend-compliance (push) Successful in 38s
CI/CD / test-python-document-crawler (push) Successful in 27s
CI/CD / test-python-dsms-gateway (push) Successful in 27s
CI/CD / deploy-hetzner (push) Failing after 6s
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-11 18:14:58 +01:00
Benjamin Admin
339505feed fix(ci): Fix Hetzner deploy — host filesystem access + network + dependencies
Some checks failed
CI/CD / go-lint (push) Has been skipped
CI/CD / python-lint (push) Has been skipped
CI/CD / nodejs-lint (push) Has been skipped
CI/CD / test-go-ai-compliance (push) Successful in 37s
CI/CD / test-python-backend-compliance (push) Successful in 36s
CI/CD / test-python-document-crawler (push) Successful in 23s
CI/CD / test-python-dsms-gateway (push) Successful in 21s
CI/CD / deploy-hetzner (push) Failing after 7s
Problems fixed:
1. Deploy step couldn't access /opt/breakpilot-compliance (host path not
   mounted in runner container). Now uses alpine/git helper container with
   host bind-mount for git ops, then docker compose with host paths.
2. breakpilot-network was external:true but Core doesn't run on Hetzner.
   Override in hetzner.yml creates the network automatically.
3. core-health-check blocks startup waiting for Core. Override in
   hetzner.yml makes it exit immediately.
4. RAG ingestion script now respects RAG_URL/QDRANT_URL env vars.
5. RAG workflow discovers network dynamically from running containers.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-11 18:11:05 +01:00
Benjamin Admin
5d99d5d47a feat(ci): Automatisches Deploy auf Hetzner via Gitea Actions
Some checks failed
CI/CD / go-lint (push) Has been skipped
CI/CD / python-lint (push) Has been skipped
CI/CD / nodejs-lint (push) Has been skipped
CI/CD / test-go-ai-compliance (push) Failing after 38s
CI/CD / test-python-backend-compliance (push) Successful in 36s
CI/CD / test-python-document-crawler (push) Successful in 26s
CI/CD / test-python-dsms-gateway (push) Successful in 26s
CI/CD / deploy-hetzner (push) Has been skipped
- Gitea Actions CI um deploy-hetzner Job erweitert
- Automatischer Build + Deploy bei Push auf main (nach Tests)
- docker-compose.hetzner.yml Override (amd64 statt arm64)
- Deploy-Dir: /opt/breakpilot-compliance/
- Baut parallel: admin, backend, ai-sdk, developer-portal
- Health Checks nach Deploy

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-11 00:08:27 +01:00
Benjamin Boenisch
2d909a8f8e fix(ci): update Go to 1.24 for ai-compliance-sdk
All checks were successful
CI / go-lint (push) Has been skipped
CI / python-lint (push) Has been skipped
CI / nodejs-lint (push) Has been skipped
CI / test-go-ai-compliance (push) Successful in 43s
CI / test-python-backend-compliance (push) Successful in 30s
CI / test-python-document-crawler (push) Successful in 26s
CI / test-python-dsms-gateway (push) Successful in 19s
The ai-compliance-sdk go.mod requires go >= 1.24.0 but CI was using
golang:1.23-alpine. Updated both Gitea Actions and Woodpecker pipelines.
Also updated golangci-lint to v1.62 for Go 1.24 compatibility.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-15 17:43:27 +01:00
Benjamin Boenisch
6b60c2b0f7 fix(ci): replace actions/checkout with manual git clone
Some checks failed
CI / go-lint (push) Has been skipped
CI / python-lint (push) Has been skipped
CI / nodejs-lint (push) Has been skipped
CI / test-go-ai-compliance (push) Failing after 3s
CI / test-python-backend-compliance (push) Successful in 30s
CI / test-python-document-crawler (push) Successful in 24s
CI / test-python-dsms-gateway (push) Successful in 19s
The act_runner cannot create /home/act_runner cache dir inside
container images. Replace actions/checkout@v4 with manual
git clone using GITHUB_SERVER_URL and GITHUB_REPOSITORY env vars.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-15 16:58:31 +01:00
Benjamin Boenisch
8776643045 fix(ci): use docker runner label instead of ubuntu-latest
Some checks failed
CI / go-lint (push) Has been skipped
CI / python-lint (push) Has been skipped
CI / nodejs-lint (push) Has been skipped
CI / test-go-ai-compliance (push) Failing after 1s
CI / test-python-backend-compliance (push) Failing after 1s
CI / test-python-document-crawler (push) Failing after 1s
CI / test-python-dsms-gateway (push) Failing after 1s
The Gitea Actions runner on meghsakha uses label "docker".

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-15 16:53:33 +01:00
Benjamin Boenisch
fb625bdb97 ci: add Gitea Actions workflow for external CI
Some checks failed
CI / go-lint (push) Has been cancelled
CI / python-lint (push) Has been cancelled
CI / nodejs-lint (push) Has been cancelled
CI / test-go-ai-compliance (push) Has been cancelled
CI / test-python-backend-compliance (push) Has been cancelled
CI / test-python-document-crawler (push) Has been cancelled
CI / test-python-dsms-gateway (push) Has been cancelled
Adds .gitea/workflows/ci.yaml with lint and test jobs.
Runs on gitea.meghsakha.com with Gitea Actions runner.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-15 16:39:01 +01:00