feat(onboarding): artifact ingest + classifier + suite-integration seams #138
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: sharang/compliance-scanner-agent#138
Reference in New Issue
Block a user
What
Steps 3–4 of the onboarding backend (artifact ingest + classifier), plus the suite-integration seams so onboarding reconciles with sibling products (tramiton / werkpilot / breakpilot-compliance) instead of duplicating them.
Stacked on #134 (
feat/onboarding-foundation) — review that first; this PR's base is that branch, so the diff here is only steps 3–4. Additive; no legacy scan path is touched.Ingest —
compliance-agent/src/ingest/(#120)ingest_all/ingest_artifactnormalize eachArtifactto a working path + metadata (IngestSet). Every blob is SHA-256 hashed into a content-addressed, deduplicated store — and that digest is the reconciliation key against tramiton (a firmware sha256 equals tramiton'sArtifact.sha256).GitOpsreuse (head SHA = content hash); zip archives + mobile packages (APK/AAB/IPA) extracted; firmware stored as a blob; live URL / plaintext / container-ref = metadata only.IngestContextcarries just the three paths it needs (not the wholeAgentConfig), so ingest is unit-testable in isolation.Classify —
compliance-agent/src/classify/(#121)FirmwareDetectorport +TramitonCli— shells out totramiton detect --jsonand parses a mirrored subset of tramiton'sBuildPlan(we deliberately do not depend on the proprietarytramiton-corecrate). A missing binary degrades to "no detection". A deterministicMockFirmwareDetectorbacks the tests so CI never needs the tramiton binary.HeuristicClassifier— artifact-kind priors (.ipa→ iOS, PLC project → PLC, …) + source-marker fingerprinting (package.json/Cargo.toml/AndroidManifest/xcodeproj/.st→ web/backend/mobile/desktop/PLC).classify_targetruns the registry, merges verdicts by target type (max confidence), and ranks them into aClassification(falls back to the declared type when there's no signal).Suite-integration seams —
compliance-core(#135 / #136 / #137)Set up now so the onboarding model doesn't need rework later (facts in the tramiton exploration; see the linked issues):
ExternalRef/ExternalSystem(link a target to tramiton/werkpilot/breakpilot for reconciliation),ComplianceProfile/ComplianceFrameworkwithdefault_compliance_profile(target_type)(firmware → CRA + IEC 62443; web → GDPR + SOC 2).EvidenceProvider(fetch external SBOM/VEX/lock/attestation — tramiton's formats already match ours) andControlsProvider(built-in OSCAL vs breakpilot RAG).TargetTypenow derivesHash;AgentConfiggainsartifact_store_base_path(envARTIFACT_STORE_BASE_PATH).Testing
44 unit tests (23 compliance-core, 8 ingest, 13 classify). Verified against the exact CI commands:
cargo fmt --all --check✓cargo clippy -p compliance-agent -- -D warnings✓cargo clippy -p compliance-dashboard --features server --no-default-features -- -D warnings✓cargo clippy -p compliance-dashboard --features web --no-default-features -- -D warnings✓cargo clippy -p compliance-mcp -- -D warnings✓Next
The feature-flagged pipeline rewrite (#133) wires ingest → classify → matrix into
run_target; then the migration (#132) and API endpoints (#131). Tramiton evidence reconciliation (#135) builds on theEvidenceProviderseam landed here.Refs #118, #120, #121, #135, #136, #137.
9e25622f18to43a1900850Update: firmware detection now uses tramiton-core natively (not the CLI)
Replaced the
tramiton detect --jsonshell-out with a direct git dependency ontramiton-core(tagv0.4.0) — same-company IP, so we use it in-process and the whole tramiton suite becomes available for firmware bare-metal/RTOS.TramitonNativerunstramiton_core::provider::analyzeon a blocking thread and maps itsBuildPlan→ a minimalFirmwareDetection. TheFirmwareDetectorport +MockFirmwareDetectorstay, so unit tests need neither the tramiton sources nor a firmware tree.⚠️ CI action required before this PR can go green: compliance-agent now fetches a private repo (
sharang/tramiton) at build time. Add a repo secretTRAMITON_FETCH_TOKEN— a Gitea PAT for a user with read access tosharang/tramiton. The workflow already:CARGO_NET_GIT_FETCH_WITH_CLI: "true", andhttps://sharang:<token>@gitea.meghsakha.com/via a gitinsteadOfstep.Without the secret, the
Checkjob fails at the first cargo compile (can't fetch tramiton). Alternative if you'd rather not use a git dep in CI: publishtramiton-coreto the Kellnr registry (crates.meghsakha.com) and switch to a registry dependency — ties into #139.Verified locally: tramiton-core v0.4.0 compiles in-tree; classify tests (12) +
cargo fmt --all --check+clippy -p compliance-agent -- -D warningsall pass.sharang/tramiton lists sharang as a Collaborative Owner (tramiton → Settings → Actions), so this repo's Actions can read tramiton. Use ${{ github.token }} in the git credential rewrite instead of a TRAMITON_FETCH_TOKEN PAT — no secret to manage. If cargo's git fetch 403s (feature scoped to uses:-actions only), fall back to a PAT. Refs #118. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>Updated: switched the CI git-auth to the ephemeral Actions token (
${{ github.token }}) instead of a PAT, sincesharang/tramitonlistssharangas a Collaborative Owner. NoTRAMITON_FETCH_TOKENsecret needed. This CI run tests whether that covers cargo's git dependency fetch (vs. onlyuses:actions) — green means the PAT is unnecessary.✅ CI green (run on
cdcbf2c). The PAT (TRAMITON_FETCH_TOKENsecret) authenticates the private tramiton-core git fetch — full build + clippy + audit + tests pass. For the record: the ephemeral Actions token does not work for a cargo git-dependency clone even with a Collaborative Owner grant (verified:remote: Repository not found, twice), and Gitea PATs can't be scoped to a single repo — a read-only Deploy Key would be the least-privilege alternative if we ever want to drop the broad PAT.