feat(onboarding): artifact ingest + classifier + suite-integration seams
CI / Check (pull_request) Failing after 4m27s
CI / Detect Changes (pull_request) Has been skipped
CI / Deploy Agent (pull_request) Has been skipped
CI / Deploy Dashboard (pull_request) Has been skipped
CI / Deploy Docs (pull_request) Has been skipped
CI / Deploy MCP (pull_request) Has been skipped
CI / Check (pull_request) Failing after 4m27s
CI / Detect Changes (pull_request) Has been skipped
CI / Deploy Agent (pull_request) Has been skipped
CI / Deploy Dashboard (pull_request) Has been skipped
CI / Deploy Docs (pull_request) Has been skipped
CI / Deploy MCP (pull_request) Has been skipped
Steps 3-4 of the onboarding plan, plus the sibling-product reconciliation seams. Ingest (compliance-agent/src/ingest, #120): - ingest_all / ingest_artifact normalize each artifact to a working path + metadata. Every blob is SHA-256 hashed (content-addressed store, dedup) — that digest is also the tramiton reconciliation key. - git via GitOps reuse; zip archives + mobile packages extracted; firmware stored as blob; live URL / plaintext / container = metadata only. - IngestContext decoupled from the full AgentConfig (testable in isolation). Classify (compliance-agent/src/classify, #121): - FirmwareDetector port + TramitonCli (shell out `tramiton detect --json`, parse a mirrored BuildPlan subset — no dependency on the proprietary crate) + a deterministic MockFirmwareDetector so CI never needs the binary. - HeuristicClassifier: artifact-kind priors + source-marker fingerprinting (web/backend/mobile/desktop/PLC). - classify_target merges + ranks verdicts into a Classification. Suite-integration seams (compliance-core, #135/#136/#137): - Model: ExternalRef/ExternalSystem (reconcile with tramiton/werkpilot/breakpilot), ComplianceProfile/ComplianceFramework + default_compliance_profile per type. - Ports: EvidenceProvider (fetch external SBOM/VEX/lock/attestation) and ControlsProvider (built-in OSCAL vs breakpilot RAG). - TargetType now derives Hash; AgentConfig gains artifact_store_base_path. 44 unit tests (23 core + 8 ingest + 13 classify). Passes fmt + clippy -D warnings across agent, dashboard (server + web), and mcp. Additive; legacy paths untouched. Refs #118, #120, #121, #135, #136, #137. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5
parent
def7371d6a
commit
4eb4b1f74b
@@ -0,0 +1,246 @@
|
|||||||
|
//! Firmware classification via tramiton.
|
||||||
|
//!
|
||||||
|
//! tramiton is the company's firmware build/repro engine; we do not re-implement
|
||||||
|
//! its detection. This module shells out to `tramiton detect --json` behind a
|
||||||
|
//! [`FirmwareDetector`] port (so a future in-process or cloud impl can slot in)
|
||||||
|
//! and maps the resulting build plan onto a [`TargetType`]. A deterministic
|
||||||
|
//! [`MockFirmwareDetector`] backs the tests so CI never needs the binary.
|
||||||
|
//!
|
||||||
|
//! The parsed structs mirror a *subset* of tramiton's `BuildPlan` JSON — we
|
||||||
|
//! deliberately do not depend on the proprietary `tramiton-core` crate.
|
||||||
|
|
||||||
|
use std::path::Path;
|
||||||
|
|
||||||
|
use serde::Deserialize;
|
||||||
|
|
||||||
|
use compliance_core::error::CoreError;
|
||||||
|
use compliance_core::models::{DetectedFact, TargetType};
|
||||||
|
use compliance_core::traits::ClassifierVerdict;
|
||||||
|
|
||||||
|
/// The top-level `tramiton detect --json` document (fields we use).
|
||||||
|
#[derive(Debug, Clone, Deserialize)]
|
||||||
|
pub struct TramitonDetect {
|
||||||
|
/// The selected build plan, if tramiton could form one.
|
||||||
|
#[serde(default)]
|
||||||
|
pub plan: Option<TramitonBuildPlan>,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The subset of tramiton's `BuildPlan` we consume for classification.
|
||||||
|
#[derive(Debug, Clone, Default, Deserialize)]
|
||||||
|
pub struct TramitonBuildPlan {
|
||||||
|
/// The detecting provider (e.g. `zephyr`, `cmake`, `source-archaeology`).
|
||||||
|
#[serde(default)]
|
||||||
|
pub provider: String,
|
||||||
|
/// Detection confidence: `low` | `medium` | `high`.
|
||||||
|
#[serde(default)]
|
||||||
|
pub confidence: String,
|
||||||
|
/// Build system (kebab-case: `zephyr`, `esp-idf`, `cmake`, `make`, ...).
|
||||||
|
#[serde(default)]
|
||||||
|
pub build_system: String,
|
||||||
|
/// Framework, when known (`zephyr`, `esp-idf`, `mbed`, `bare-metal`, ...).
|
||||||
|
pub framework: Option<String>,
|
||||||
|
/// Target board / MCU / arch.
|
||||||
|
#[serde(default)]
|
||||||
|
pub target: TramitonTarget,
|
||||||
|
/// Unresolved gaps in the plan.
|
||||||
|
#[serde(default)]
|
||||||
|
pub gaps: Vec<String>,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// tramiton's target descriptor.
|
||||||
|
#[derive(Debug, Clone, Default, Deserialize)]
|
||||||
|
pub struct TramitonTarget {
|
||||||
|
/// Board name.
|
||||||
|
pub board: Option<String>,
|
||||||
|
/// MCU part.
|
||||||
|
pub mcu: Option<String>,
|
||||||
|
/// Architecture.
|
||||||
|
pub arch: Option<String>,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A source of tramiton firmware detection.
|
||||||
|
#[allow(async_fn_in_trait)]
|
||||||
|
pub trait FirmwareDetector: Send + Sync {
|
||||||
|
/// Run detection over a path, returning tramiton's build plan if any.
|
||||||
|
async fn detect(&self, path: &Path) -> Result<Option<TramitonBuildPlan>, CoreError>;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Shells out to the `tramiton` CLI. A missing binary or a non-zero exit is
|
||||||
|
/// treated as "no detection" rather than an error, so firmware classification
|
||||||
|
/// degrades gracefully when tramiton is not installed.
|
||||||
|
pub struct TramitonCli {
|
||||||
|
/// The `tramiton` binary to invoke.
|
||||||
|
pub bin: String,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl TramitonCli {
|
||||||
|
/// Construct from `TRAMITON_BIN` (default `tramiton`).
|
||||||
|
pub fn from_env() -> Self {
|
||||||
|
Self {
|
||||||
|
bin: std::env::var("TRAMITON_BIN").unwrap_or_else(|_| "tramiton".to_string()),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl FirmwareDetector for TramitonCli {
|
||||||
|
async fn detect(&self, path: &Path) -> Result<Option<TramitonBuildPlan>, CoreError> {
|
||||||
|
let output = tokio::process::Command::new(&self.bin)
|
||||||
|
.arg("detect")
|
||||||
|
.arg("--json")
|
||||||
|
.arg(path)
|
||||||
|
.output()
|
||||||
|
.await;
|
||||||
|
match output {
|
||||||
|
Ok(o) if o.status.success() => {
|
||||||
|
let parsed: TramitonDetect = serde_json::from_slice(&o.stdout)?;
|
||||||
|
Ok(parsed.plan)
|
||||||
|
}
|
||||||
|
// Non-zero exit: tramiton ran but formed no plan.
|
||||||
|
Ok(_) => Ok(None),
|
||||||
|
// Binary not found / not executable: degrade gracefully.
|
||||||
|
Err(_) => Ok(None),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Map a tramiton build plan to a target type. Framework/build-system signals
|
||||||
|
/// distinguish RTOS from bare-metal from Yocto.
|
||||||
|
pub fn plan_to_target_type(plan: &TramitonBuildPlan) -> TargetType {
|
||||||
|
let framework = plan.framework.as_deref().unwrap_or("").to_lowercase();
|
||||||
|
let build_system = plan.build_system.to_lowercase();
|
||||||
|
let signal = format!(
|
||||||
|
"{framework} {build_system} {}",
|
||||||
|
plan.provider.to_lowercase()
|
||||||
|
);
|
||||||
|
|
||||||
|
const RTOS: [&str; 6] = ["zephyr", "esp-idf", "freertos", "nuttx", "riot", "chibios"];
|
||||||
|
if signal.contains("bitbake") || signal.contains("yocto") || signal.contains("openembedded") {
|
||||||
|
TargetType::EmbeddedLinuxYocto
|
||||||
|
} else if RTOS.iter().any(|k| signal.contains(k)) {
|
||||||
|
TargetType::FirmwareRtos
|
||||||
|
} else {
|
||||||
|
TargetType::FirmwareBareMetal
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Map tramiton's confidence label to a `[0,1]` score.
|
||||||
|
fn confidence_score(label: &str) -> f32 {
|
||||||
|
match label.to_lowercase().as_str() {
|
||||||
|
"high" => 0.9,
|
||||||
|
"medium" => 0.6,
|
||||||
|
"low" => 0.3,
|
||||||
|
_ => 0.4,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Turn a tramiton build plan into a classifier verdict, carrying the MCU / board
|
||||||
|
/// / build-system as facts.
|
||||||
|
pub fn plan_to_verdict(plan: &TramitonBuildPlan) -> ClassifierVerdict {
|
||||||
|
let target_type = plan_to_target_type(plan);
|
||||||
|
let mut facts = vec![DetectedFact::new(
|
||||||
|
"build_system",
|
||||||
|
plan.build_system.clone(),
|
||||||
|
"tramiton",
|
||||||
|
)];
|
||||||
|
if let Some(fw) = &plan.framework {
|
||||||
|
facts.push(DetectedFact::new("framework", fw.clone(), "tramiton"));
|
||||||
|
}
|
||||||
|
if let Some(mcu) = &plan.target.mcu {
|
||||||
|
facts.push(DetectedFact::new("mcu", mcu.clone(), "tramiton"));
|
||||||
|
}
|
||||||
|
if let Some(board) = &plan.target.board {
|
||||||
|
facts.push(DetectedFact::new("board", board.clone(), "tramiton"));
|
||||||
|
}
|
||||||
|
if let Some(arch) = &plan.target.arch {
|
||||||
|
facts.push(DetectedFact::new("arch", arch.clone(), "tramiton"));
|
||||||
|
}
|
||||||
|
ClassifierVerdict {
|
||||||
|
target_type,
|
||||||
|
confidence: confidence_score(&plan.confidence),
|
||||||
|
facts,
|
||||||
|
rationale: format!(
|
||||||
|
"tramiton detected build system '{}'{}",
|
||||||
|
plan.build_system,
|
||||||
|
plan.framework
|
||||||
|
.as_ref()
|
||||||
|
.map(|f| format!(" (framework {f})"))
|
||||||
|
.unwrap_or_default()
|
||||||
|
),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A deterministic [`FirmwareDetector`] for tests — returns a preset plan.
|
||||||
|
pub struct MockFirmwareDetector {
|
||||||
|
/// The plan to return (or `None` for "no detection").
|
||||||
|
pub plan: Option<TramitonBuildPlan>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl FirmwareDetector for MockFirmwareDetector {
|
||||||
|
async fn detect(&self, _path: &Path) -> Result<Option<TramitonBuildPlan>, CoreError> {
|
||||||
|
Ok(self.plan.clone())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
#[allow(clippy::expect_used, clippy::unwrap_used)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
|
||||||
|
fn plan(build_system: &str, framework: Option<&str>) -> TramitonBuildPlan {
|
||||||
|
TramitonBuildPlan {
|
||||||
|
provider: build_system.to_string(),
|
||||||
|
confidence: "high".to_string(),
|
||||||
|
build_system: build_system.to_string(),
|
||||||
|
framework: framework.map(|s| s.to_string()),
|
||||||
|
target: TramitonTarget {
|
||||||
|
mcu: Some("stm32f429".to_string()),
|
||||||
|
..Default::default()
|
||||||
|
},
|
||||||
|
gaps: Vec::new(),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn zephyr_maps_to_rtos() {
|
||||||
|
assert_eq!(
|
||||||
|
plan_to_target_type(&plan("zephyr", Some("zephyr"))),
|
||||||
|
TargetType::FirmwareRtos
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn bare_cmake_maps_to_bare_metal() {
|
||||||
|
assert_eq!(
|
||||||
|
plan_to_target_type(&plan("cmake", Some("bare-metal"))),
|
||||||
|
TargetType::FirmwareBareMetal
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn bitbake_maps_to_yocto() {
|
||||||
|
assert_eq!(
|
||||||
|
plan_to_target_type(&plan("bitbake", None)),
|
||||||
|
TargetType::EmbeddedLinuxYocto
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn verdict_carries_mcu_fact_and_confidence() {
|
||||||
|
let v = plan_to_verdict(&plan("esp-idf", Some("esp-idf")));
|
||||||
|
assert_eq!(v.target_type, TargetType::FirmwareRtos);
|
||||||
|
assert!((v.confidence - 0.9).abs() < f32::EPSILON);
|
||||||
|
assert!(v
|
||||||
|
.facts
|
||||||
|
.iter()
|
||||||
|
.any(|f| f.key == "mcu" && f.value == "stm32f429"));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn detect_json_parses() {
|
||||||
|
let json = r#"{"repo":"/x","detections":[],"plan":{"provider":"zephyr","confidence":"high","build_system":"zephyr","framework":"zephyr","target":{"mcu":"nrf52840","board":"nrf52840dk","arch":"arm"},"gaps":[]}}"#;
|
||||||
|
let parsed: TramitonDetect = serde_json::from_str(json).expect("parse");
|
||||||
|
let plan = parsed.plan.expect("plan present");
|
||||||
|
assert_eq!(plan.target.mcu.as_deref(), Some("nrf52840"));
|
||||||
|
assert_eq!(plan_to_target_type(&plan), TargetType::FirmwareRtos);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,357 @@
|
|||||||
|
//! Heuristic target-type classification from artifact kinds and source markers.
|
||||||
|
//!
|
||||||
|
//! Complements the tramiton firmware detector: this handles web / backend /
|
||||||
|
//! mobile / desktop / PLC by sniffing manifest files and file extensions in the
|
||||||
|
//! ingested code trees, plus strong priors from the artifact kinds themselves
|
||||||
|
//! (a PLC-project artifact is a PLC target; an `.ipa` is an iOS app).
|
||||||
|
|
||||||
|
use std::collections::HashSet;
|
||||||
|
use std::fs;
|
||||||
|
use std::path::Path;
|
||||||
|
|
||||||
|
use compliance_core::error::CoreError;
|
||||||
|
use compliance_core::models::{ArtifactKind, DetectedFact, TargetType};
|
||||||
|
use compliance_core::traits::{ClassificationInput, ClassifierVerdict, TargetClassifier};
|
||||||
|
|
||||||
|
/// Max directory depth scanned for marker files.
|
||||||
|
const SCAN_DEPTH: usize = 2;
|
||||||
|
|
||||||
|
/// Markers collected from a code tree.
|
||||||
|
#[derive(Default)]
|
||||||
|
struct Markers {
|
||||||
|
files: HashSet<String>,
|
||||||
|
dirs: HashSet<String>,
|
||||||
|
exts: HashSet<String>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl Markers {
|
||||||
|
fn has_file(&self, name: &str) -> bool {
|
||||||
|
self.files.contains(name)
|
||||||
|
}
|
||||||
|
fn has_ext(&self, ext: &str) -> bool {
|
||||||
|
self.exts.contains(ext)
|
||||||
|
}
|
||||||
|
fn any_dir_ends_with(&self, suffix: &str) -> bool {
|
||||||
|
self.dirs.iter().any(|d| d.ends_with(suffix))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Recursively collect marker file/dir/extension names up to [`SCAN_DEPTH`].
|
||||||
|
fn collect_markers(root: &Path) -> Markers {
|
||||||
|
let mut m = Markers::default();
|
||||||
|
scan_dir(root, 0, &mut m);
|
||||||
|
m
|
||||||
|
}
|
||||||
|
|
||||||
|
fn scan_dir(dir: &Path, depth: usize, m: &mut Markers) {
|
||||||
|
let Ok(entries) = fs::read_dir(dir) else {
|
||||||
|
return;
|
||||||
|
};
|
||||||
|
for entry in entries.flatten() {
|
||||||
|
let path = entry.path();
|
||||||
|
let name = entry.file_name().to_string_lossy().to_lowercase();
|
||||||
|
if path.is_dir() {
|
||||||
|
m.dirs.insert(name);
|
||||||
|
if depth < SCAN_DEPTH {
|
||||||
|
scan_dir(&path, depth + 1, m);
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
if let Some(ext) = path.extension() {
|
||||||
|
m.exts.insert(ext.to_string_lossy().to_lowercase());
|
||||||
|
}
|
||||||
|
m.files.insert(name);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Whether a `package.json` at `root` looks like a front-end app.
|
||||||
|
fn package_json_is_frontend(root: &Path) -> bool {
|
||||||
|
let Ok(content) = fs::read_to_string(root.join("package.json")) else {
|
||||||
|
return false;
|
||||||
|
};
|
||||||
|
let c = content.to_lowercase();
|
||||||
|
["react", "next", "vue", "@angular", "svelte", "vite"]
|
||||||
|
.iter()
|
||||||
|
.any(|f| c.contains(f))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The heuristic classifier: artifact-kind priors + source-tree markers.
|
||||||
|
pub struct HeuristicClassifier;
|
||||||
|
|
||||||
|
impl HeuristicClassifier {
|
||||||
|
/// Verdicts from the artifact kinds alone (no filesystem needed).
|
||||||
|
fn kind_priors(&self, input: &ClassificationInput<'_>) -> Vec<ClassifierVerdict> {
|
||||||
|
let mut out = Vec::new();
|
||||||
|
for a in input.artifacts {
|
||||||
|
let lower = a.source_ref.to_lowercase();
|
||||||
|
match a.kind {
|
||||||
|
ArtifactKind::PlcProject => out.push(verdict(
|
||||||
|
TargetType::PlcSps,
|
||||||
|
0.85,
|
||||||
|
"PLC project artifact",
|
||||||
|
vec![],
|
||||||
|
)),
|
||||||
|
ArtifactKind::MobilePackage => {
|
||||||
|
let (tt, why) = if lower.ends_with(".ipa") {
|
||||||
|
(TargetType::IosApp, "iOS package (.ipa)")
|
||||||
|
} else {
|
||||||
|
(TargetType::AndroidApp, "Android package (.apk/.aab)")
|
||||||
|
};
|
||||||
|
out.push(verdict(tt, 0.85, why, vec![]));
|
||||||
|
}
|
||||||
|
ArtifactKind::ContainerImage => out.push(verdict(
|
||||||
|
TargetType::BackendService,
|
||||||
|
0.4,
|
||||||
|
"container image",
|
||||||
|
vec![],
|
||||||
|
)),
|
||||||
|
ArtifactKind::FirmwareImage => out.push(verdict(
|
||||||
|
TargetType::FirmwareBareMetal,
|
||||||
|
0.35,
|
||||||
|
"firmware image (pending tramiton detection)",
|
||||||
|
vec![],
|
||||||
|
)),
|
||||||
|
ArtifactKind::LiveUrl if input.artifacts.len() == 1 => {
|
||||||
|
out.push(verdict(TargetType::WebApp, 0.3, "live URL only", vec![]))
|
||||||
|
}
|
||||||
|
_ => {}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
out
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Verdicts from scanning the ingested code trees for manifest markers.
|
||||||
|
fn source_verdicts(&self, input: &ClassificationInput<'_>) -> Vec<ClassifierVerdict> {
|
||||||
|
let mut out = Vec::new();
|
||||||
|
for a in input.artifacts {
|
||||||
|
if !matches!(a.kind, ArtifactKind::GitRepo | ArtifactKind::SourceArchive) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
let Some(path) = input.working_paths.get(&a.id) else {
|
||||||
|
continue;
|
||||||
|
};
|
||||||
|
let m = collect_markers(path);
|
||||||
|
|
||||||
|
// Mobile (checked first — strongest signal).
|
||||||
|
if m.has_file("androidmanifest.xml") || m.has_ext("apk") || m.has_ext("aab") {
|
||||||
|
out.push(verdict(
|
||||||
|
TargetType::AndroidApp,
|
||||||
|
0.8,
|
||||||
|
"Android manifest / gradle",
|
||||||
|
facts_lang("kotlin/java"),
|
||||||
|
));
|
||||||
|
}
|
||||||
|
if m.any_dir_ends_with(".xcodeproj")
|
||||||
|
|| m.has_file("info.plist")
|
||||||
|
|| m.has_file("podfile")
|
||||||
|
|| m.has_ext("ipa")
|
||||||
|
{
|
||||||
|
out.push(verdict(
|
||||||
|
TargetType::IosApp,
|
||||||
|
0.8,
|
||||||
|
"Xcode project / Info.plist",
|
||||||
|
facts_lang("swift/objc"),
|
||||||
|
));
|
||||||
|
}
|
||||||
|
// Desktop.
|
||||||
|
if m.has_ext("sln")
|
||||||
|
|| m.has_ext("csproj")
|
||||||
|
|| m.has_ext("vcxproj")
|
||||||
|
|| m.has_ext("desktop")
|
||||||
|
{
|
||||||
|
out.push(verdict(
|
||||||
|
TargetType::DesktopApp,
|
||||||
|
0.7,
|
||||||
|
"desktop project files",
|
||||||
|
facts_lang("dotnet/native"),
|
||||||
|
));
|
||||||
|
}
|
||||||
|
// PLC.
|
||||||
|
if m.has_ext("st") {
|
||||||
|
out.push(verdict(
|
||||||
|
TargetType::PlcSps,
|
||||||
|
0.8,
|
||||||
|
"Structured Text sources",
|
||||||
|
facts_lang("iec-61131-3"),
|
||||||
|
));
|
||||||
|
}
|
||||||
|
// Web vs backend from package.json.
|
||||||
|
if m.has_file("package.json") {
|
||||||
|
if package_json_is_frontend(path) {
|
||||||
|
out.push(verdict(
|
||||||
|
TargetType::WebApp,
|
||||||
|
0.65,
|
||||||
|
"package.json with a front-end framework",
|
||||||
|
facts_lang("javascript"),
|
||||||
|
));
|
||||||
|
} else {
|
||||||
|
out.push(verdict(
|
||||||
|
TargetType::BackendService,
|
||||||
|
0.55,
|
||||||
|
"package.json (no front-end framework)",
|
||||||
|
facts_lang("javascript"),
|
||||||
|
));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// Backend languages.
|
||||||
|
for (file, lang) in [
|
||||||
|
("cargo.toml", "rust"),
|
||||||
|
("go.mod", "go"),
|
||||||
|
("pom.xml", "java"),
|
||||||
|
("requirements.txt", "python"),
|
||||||
|
("pyproject.toml", "python"),
|
||||||
|
] {
|
||||||
|
if m.has_file(file) {
|
||||||
|
out.push(verdict(
|
||||||
|
TargetType::BackendService,
|
||||||
|
0.6,
|
||||||
|
"backend build manifest",
|
||||||
|
facts_lang(lang),
|
||||||
|
));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// Container-only.
|
||||||
|
if m.has_file("dockerfile") && out.is_empty() {
|
||||||
|
out.push(verdict(
|
||||||
|
TargetType::BackendService,
|
||||||
|
0.4,
|
||||||
|
"Dockerfile",
|
||||||
|
facts_lang("container"),
|
||||||
|
));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
out
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl TargetClassifier for HeuristicClassifier {
|
||||||
|
fn name(&self) -> &str {
|
||||||
|
"heuristic"
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn classify(
|
||||||
|
&self,
|
||||||
|
input: &ClassificationInput<'_>,
|
||||||
|
) -> Result<Vec<ClassifierVerdict>, CoreError> {
|
||||||
|
let mut out = self.kind_priors(input);
|
||||||
|
out.extend(self.source_verdicts(input));
|
||||||
|
Ok(out)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn verdict(
|
||||||
|
target_type: TargetType,
|
||||||
|
confidence: f32,
|
||||||
|
rationale: &str,
|
||||||
|
facts: Vec<DetectedFact>,
|
||||||
|
) -> ClassifierVerdict {
|
||||||
|
ClassifierVerdict {
|
||||||
|
target_type,
|
||||||
|
confidence,
|
||||||
|
facts,
|
||||||
|
rationale: rationale.to_string(),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn facts_lang(lang: &str) -> Vec<DetectedFact> {
|
||||||
|
vec![DetectedFact::new("language", lang, "heuristic")]
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
#[allow(clippy::expect_used, clippy::unwrap_used)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
use compliance_core::models::Artifact;
|
||||||
|
use std::collections::HashMap;
|
||||||
|
use std::path::PathBuf;
|
||||||
|
|
||||||
|
struct Scratch(PathBuf);
|
||||||
|
impl Scratch {
|
||||||
|
fn new() -> Self {
|
||||||
|
let p = std::env::temp_dir().join(format!("cs-classify-{}", uuid::Uuid::new_v4()));
|
||||||
|
fs::create_dir_all(&p).expect("mkdir");
|
||||||
|
Self(p)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
impl Drop for Scratch {
|
||||||
|
fn drop(&mut self) {
|
||||||
|
let _ = fs::remove_dir_all(&self.0);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn classify_tree(setup: impl FnOnce(&Path)) -> Vec<ClassifierVerdict> {
|
||||||
|
let scratch = Scratch::new();
|
||||||
|
setup(&scratch.0);
|
||||||
|
let artifact = Artifact::git_repo("https://git/x", "main");
|
||||||
|
let mut wp = HashMap::new();
|
||||||
|
wp.insert(artifact.id.clone(), scratch.0.clone());
|
||||||
|
let artifacts = vec![artifact];
|
||||||
|
let input = ClassificationInput {
|
||||||
|
artifacts: &artifacts,
|
||||||
|
working_paths: &wp,
|
||||||
|
description: None,
|
||||||
|
};
|
||||||
|
HeuristicClassifier
|
||||||
|
.classify(&input)
|
||||||
|
.await
|
||||||
|
.expect("classify")
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn frontend_package_json_is_webapp() {
|
||||||
|
let v = classify_tree(|root| {
|
||||||
|
fs::write(
|
||||||
|
root.join("package.json"),
|
||||||
|
r#"{"dependencies":{"react":"18"}}"#,
|
||||||
|
)
|
||||||
|
.unwrap();
|
||||||
|
})
|
||||||
|
.await;
|
||||||
|
assert!(v.iter().any(|x| x.target_type == TargetType::WebApp));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn cargo_toml_is_backend() {
|
||||||
|
let v = classify_tree(|root| {
|
||||||
|
fs::write(root.join("Cargo.toml"), "[package]\nname='x'").unwrap();
|
||||||
|
})
|
||||||
|
.await;
|
||||||
|
assert!(v
|
||||||
|
.iter()
|
||||||
|
.any(|x| x.target_type == TargetType::BackendService));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn android_manifest_is_android() {
|
||||||
|
let v = classify_tree(|root| {
|
||||||
|
fs::write(root.join("AndroidManifest.xml"), "<manifest/>").unwrap();
|
||||||
|
})
|
||||||
|
.await;
|
||||||
|
assert!(v.iter().any(|x| x.target_type == TargetType::AndroidApp));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn structured_text_is_plc() {
|
||||||
|
let v = classify_tree(|root| {
|
||||||
|
fs::write(root.join("main.st"), "PROGRAM main END_PROGRAM").unwrap();
|
||||||
|
})
|
||||||
|
.await;
|
||||||
|
assert!(v.iter().any(|x| x.target_type == TargetType::PlcSps));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn ipa_artifact_prior_is_ios() {
|
||||||
|
let artifacts = vec![Artifact::mobile_package("app.ipa")];
|
||||||
|
let wp = HashMap::new();
|
||||||
|
let input = ClassificationInput {
|
||||||
|
artifacts: &artifacts,
|
||||||
|
working_paths: &wp,
|
||||||
|
description: None,
|
||||||
|
};
|
||||||
|
let v = HeuristicClassifier
|
||||||
|
.classify(&input)
|
||||||
|
.await
|
||||||
|
.expect("classify");
|
||||||
|
assert!(v.iter().any(|x| x.target_type == TargetType::IosApp));
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,227 @@
|
|||||||
|
//! Target classification.
|
||||||
|
//!
|
||||||
|
//! Runs the classifier registry over a target's artifacts and their ingested
|
||||||
|
//! working paths, then merges and ranks the verdicts into a [`Classification`].
|
||||||
|
//! The registry is the heuristic classifier (artifact kinds + source markers)
|
||||||
|
//! plus the tramiton firmware detector (behind a [`FirmwareDetector`] port).
|
||||||
|
|
||||||
|
mod firmware;
|
||||||
|
mod language;
|
||||||
|
|
||||||
|
pub use firmware::{
|
||||||
|
FirmwareDetector, MockFirmwareDetector, TramitonBuildPlan, TramitonCli, TramitonDetect,
|
||||||
|
TramitonTarget,
|
||||||
|
};
|
||||||
|
pub use language::HeuristicClassifier;
|
||||||
|
|
||||||
|
use std::collections::HashMap;
|
||||||
|
use std::path::PathBuf;
|
||||||
|
|
||||||
|
use compliance_core::error::CoreError;
|
||||||
|
use compliance_core::models::{
|
||||||
|
ArtifactKind, Classification, DetectedFact, OnboardedTarget, TargetType, TargetTypeCandidate,
|
||||||
|
};
|
||||||
|
use compliance_core::traits::{ClassificationInput, ClassifierVerdict, TargetClassifier};
|
||||||
|
|
||||||
|
use firmware::plan_to_verdict;
|
||||||
|
|
||||||
|
/// Classify a target from its artifacts and their ingested working paths, using
|
||||||
|
/// the heuristic classifier plus the tramiton firmware detector. Verdicts are
|
||||||
|
/// merged (max confidence per target type) and ranked into a [`Classification`].
|
||||||
|
pub async fn classify_target<D: FirmwareDetector>(
|
||||||
|
target: &OnboardedTarget,
|
||||||
|
working_paths: &HashMap<String, PathBuf>,
|
||||||
|
firmware_detector: &D,
|
||||||
|
) -> Result<Classification, CoreError> {
|
||||||
|
let input = ClassificationInput {
|
||||||
|
artifacts: &target.artifacts,
|
||||||
|
working_paths,
|
||||||
|
description: target.description.as_deref(),
|
||||||
|
};
|
||||||
|
|
||||||
|
let mut verdicts = Vec::new();
|
||||||
|
let mut detected_by = Vec::new();
|
||||||
|
|
||||||
|
let heuristic = HeuristicClassifier.classify(&input).await?;
|
||||||
|
if !heuristic.is_empty() {
|
||||||
|
detected_by.push("heuristic".to_string());
|
||||||
|
}
|
||||||
|
verdicts.extend(heuristic);
|
||||||
|
|
||||||
|
// Tramiton firmware detection over firmware / code working paths.
|
||||||
|
let mut tramiton_used = false;
|
||||||
|
for artifact in &target.artifacts {
|
||||||
|
if !matches!(
|
||||||
|
artifact.kind,
|
||||||
|
ArtifactKind::FirmwareImage | ArtifactKind::GitRepo | ArtifactKind::SourceArchive
|
||||||
|
) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
let Some(path) = working_paths.get(&artifact.id) else {
|
||||||
|
continue;
|
||||||
|
};
|
||||||
|
if let Some(plan) = firmware_detector.detect(path).await? {
|
||||||
|
verdicts.push(plan_to_verdict(&plan));
|
||||||
|
tramiton_used = true;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if tramiton_used {
|
||||||
|
detected_by.push("tramiton".to_string());
|
||||||
|
}
|
||||||
|
|
||||||
|
Ok(rank(verdicts, detected_by, target.target_type))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Merge verdicts by target type (keeping the max confidence and its rationale),
|
||||||
|
/// dedupe facts, rank by descending confidence, and assemble a [`Classification`].
|
||||||
|
/// Falls back to the declared type when no verdict is produced.
|
||||||
|
fn rank(
|
||||||
|
verdicts: Vec<ClassifierVerdict>,
|
||||||
|
detected_by: Vec<String>,
|
||||||
|
fallback: TargetType,
|
||||||
|
) -> Classification {
|
||||||
|
let mut best: HashMap<TargetType, (f32, String)> = HashMap::new();
|
||||||
|
let mut facts: Vec<DetectedFact> = Vec::new();
|
||||||
|
for verdict in verdicts {
|
||||||
|
for fact in verdict.facts {
|
||||||
|
if !facts
|
||||||
|
.iter()
|
||||||
|
.any(|e| e.key == fact.key && e.value == fact.value)
|
||||||
|
{
|
||||||
|
facts.push(fact);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
let entry = best
|
||||||
|
.entry(verdict.target_type)
|
||||||
|
.or_insert((0.0, String::new()));
|
||||||
|
if verdict.confidence > entry.0 {
|
||||||
|
*entry = (verdict.confidence, verdict.rationale);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
let mut candidates: Vec<TargetTypeCandidate> = best
|
||||||
|
.into_iter()
|
||||||
|
.map(
|
||||||
|
|(target_type, (confidence, rationale))| TargetTypeCandidate {
|
||||||
|
target_type,
|
||||||
|
confidence,
|
||||||
|
rationale,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
.collect();
|
||||||
|
// Descending confidence; ties broken by type name for deterministic ordering.
|
||||||
|
candidates.sort_by(|a, b| {
|
||||||
|
b.confidence
|
||||||
|
.partial_cmp(&a.confidence)
|
||||||
|
.unwrap_or(std::cmp::Ordering::Equal)
|
||||||
|
.then_with(|| a.target_type.to_string().cmp(&b.target_type.to_string()))
|
||||||
|
});
|
||||||
|
|
||||||
|
let suggested = candidates
|
||||||
|
.first()
|
||||||
|
.map(|c| c.target_type)
|
||||||
|
.unwrap_or(fallback);
|
||||||
|
|
||||||
|
Classification {
|
||||||
|
suggested,
|
||||||
|
candidates,
|
||||||
|
facts,
|
||||||
|
detected_by,
|
||||||
|
detected_at: chrono::Utc::now(),
|
||||||
|
confirmed: false,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
#[allow(clippy::expect_used, clippy::unwrap_used)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
use compliance_core::models::Artifact;
|
||||||
|
use std::fs;
|
||||||
|
use std::path::Path;
|
||||||
|
|
||||||
|
struct Scratch(PathBuf);
|
||||||
|
impl Scratch {
|
||||||
|
fn new() -> Self {
|
||||||
|
let p = std::env::temp_dir().join(format!("cs-classify-mod-{}", uuid::Uuid::new_v4()));
|
||||||
|
fs::create_dir_all(&p).expect("mkdir");
|
||||||
|
Self(p)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
impl Drop for Scratch {
|
||||||
|
fn drop(&mut self) {
|
||||||
|
let _ = fs::remove_dir_all(&self.0);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn no_firmware() -> MockFirmwareDetector {
|
||||||
|
MockFirmwareDetector { plan: None }
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn backend_repo_classifies_as_backend() {
|
||||||
|
let scratch = Scratch::new();
|
||||||
|
fs::write(scratch.0.join("go.mod"), "module x").unwrap();
|
||||||
|
|
||||||
|
let artifact = Artifact::git_repo("https://git/x", "main");
|
||||||
|
let mut wp = HashMap::new();
|
||||||
|
wp.insert(artifact.id.clone(), scratch.0.clone());
|
||||||
|
let mut target = OnboardedTarget::new("x".to_string(), TargetType::WebApp);
|
||||||
|
target.artifacts.push(artifact);
|
||||||
|
|
||||||
|
let c = classify_target(&target, &wp, &no_firmware())
|
||||||
|
.await
|
||||||
|
.expect("classify");
|
||||||
|
assert_eq!(c.suggested, TargetType::BackendService);
|
||||||
|
assert!(c.detected_by.contains(&"heuristic".to_string()));
|
||||||
|
assert!(!c.confirmed);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn firmware_detector_verdict_ranks_top() {
|
||||||
|
let scratch = Scratch::new();
|
||||||
|
fs::write(scratch.0.join("fw.bin"), b"x").unwrap();
|
||||||
|
|
||||||
|
let artifact =
|
||||||
|
Artifact::firmware_image(scratch.0.join("fw.bin").to_string_lossy().to_string());
|
||||||
|
let mut wp = HashMap::new();
|
||||||
|
wp.insert(artifact.id.clone(), scratch.0.clone());
|
||||||
|
let mut target = OnboardedTarget::new("fw".to_string(), TargetType::FirmwareBareMetal);
|
||||||
|
target.artifacts.push(artifact);
|
||||||
|
|
||||||
|
let detector = MockFirmwareDetector {
|
||||||
|
plan: Some(TramitonBuildPlan {
|
||||||
|
provider: "zephyr".to_string(),
|
||||||
|
confidence: "high".to_string(),
|
||||||
|
build_system: "zephyr".to_string(),
|
||||||
|
framework: Some("zephyr".to_string()),
|
||||||
|
target: TramitonTarget {
|
||||||
|
mcu: Some("nrf52840".to_string()),
|
||||||
|
..Default::default()
|
||||||
|
},
|
||||||
|
gaps: vec![],
|
||||||
|
}),
|
||||||
|
};
|
||||||
|
|
||||||
|
let c = classify_target(&target, &wp, &detector)
|
||||||
|
.await
|
||||||
|
.expect("classify");
|
||||||
|
// tramiton's high-confidence RTOS verdict beats the weak firmware prior.
|
||||||
|
assert_eq!(c.suggested, TargetType::FirmwareRtos);
|
||||||
|
assert!(c.detected_by.contains(&"tramiton".to_string()));
|
||||||
|
assert!(c.facts.iter().any(|f| f.key == "mcu"));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn no_signal_falls_back_to_declared_type() {
|
||||||
|
let scratch = Scratch::new();
|
||||||
|
let _ = Path::new(&scratch.0);
|
||||||
|
let target = OnboardedTarget::new("empty".to_string(), TargetType::DesktopApp);
|
||||||
|
let wp = HashMap::new();
|
||||||
|
let c = classify_target(&target, &wp, &no_firmware())
|
||||||
|
.await
|
||||||
|
.expect("classify");
|
||||||
|
assert_eq!(c.suggested, TargetType::DesktopApp);
|
||||||
|
assert!(c.candidates.is_empty());
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -45,6 +45,8 @@ pub fn load_config() -> Result<AgentConfig, AgentError> {
|
|||||||
.unwrap_or_else(|| "0 0 * * * *".to_string()),
|
.unwrap_or_else(|| "0 0 * * * *".to_string()),
|
||||||
git_clone_base_path: env_var_opt("GIT_CLONE_BASE_PATH")
|
git_clone_base_path: env_var_opt("GIT_CLONE_BASE_PATH")
|
||||||
.unwrap_or_else(|| "/tmp/compliance-scanner/repos".to_string()),
|
.unwrap_or_else(|| "/tmp/compliance-scanner/repos".to_string()),
|
||||||
|
artifact_store_base_path: env_var_opt("ARTIFACT_STORE_BASE_PATH")
|
||||||
|
.unwrap_or_else(|| "/data/compliance-scanner/artifacts".to_string()),
|
||||||
ssh_key_path: env_var_opt("SSH_KEY_PATH")
|
ssh_key_path: env_var_opt("SSH_KEY_PATH")
|
||||||
.unwrap_or_else(|| "/data/compliance-scanner/ssh/id_ed25519".to_string()),
|
.unwrap_or_else(|| "/data/compliance-scanner/ssh/id_ed25519".to_string()),
|
||||||
keycloak_url: env_var_opt("KEYCLOAK_URL"),
|
keycloak_url: env_var_opt("KEYCLOAK_URL"),
|
||||||
|
|||||||
@@ -0,0 +1,154 @@
|
|||||||
|
//! Content-addressed blob storage and archive extraction for ingest.
|
||||||
|
//!
|
||||||
|
//! Blobs are stored at `<base>/blobs/<sha[0:2]>/<sha>` and deduplicated by
|
||||||
|
//! digest; per-run working directories live under `<base>/work/`.
|
||||||
|
|
||||||
|
use std::fs::{self, File};
|
||||||
|
use std::io::{self, Read};
|
||||||
|
use std::path::{Path, PathBuf};
|
||||||
|
|
||||||
|
use sha2::{Digest, Sha256};
|
||||||
|
|
||||||
|
use crate::error::AgentError;
|
||||||
|
|
||||||
|
/// Read buffer size for streaming hashes/copies (64 KiB).
|
||||||
|
const BUF_LEN: usize = 64 * 1024;
|
||||||
|
|
||||||
|
/// Stream-hash a file with SHA-256, returning the lowercase-hex digest and the
|
||||||
|
/// byte length. Streams so large firmware images never load fully into memory.
|
||||||
|
pub fn hash_file(path: &Path) -> Result<(String, u64), AgentError> {
|
||||||
|
let mut file = File::open(path)?;
|
||||||
|
let mut hasher = Sha256::new();
|
||||||
|
let mut buf = [0u8; BUF_LEN];
|
||||||
|
let mut total: u64 = 0;
|
||||||
|
loop {
|
||||||
|
let n = file.read(&mut buf)?;
|
||||||
|
if n == 0 {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
hasher.update(&buf[..n]);
|
||||||
|
total += n as u64;
|
||||||
|
}
|
||||||
|
Ok((hex::encode(hasher.finalize()), total))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Copy `src` into the content-addressed blob store under `base`, returning the
|
||||||
|
/// stored path. Idempotent: an already-present blob is not rewritten.
|
||||||
|
pub fn store_file(base: &Path, src: &Path, sha: &str) -> Result<PathBuf, AgentError> {
|
||||||
|
if sha.len() < 2 {
|
||||||
|
return Err(AgentError::Other(format!("invalid content hash '{sha}'")));
|
||||||
|
}
|
||||||
|
let dir = base.join("blobs").join(&sha[0..2]);
|
||||||
|
fs::create_dir_all(&dir)?;
|
||||||
|
let dest = dir.join(sha);
|
||||||
|
if !dest.exists() {
|
||||||
|
fs::copy(src, &dest)?;
|
||||||
|
}
|
||||||
|
Ok(dest)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Extract a zip archive into `dest` (created if needed). `enclosed_name`
|
||||||
|
/// sanitizes each entry path, so this is safe against zip-slip traversal.
|
||||||
|
pub fn extract_zip(archive: &Path, dest: &Path) -> Result<(), AgentError> {
|
||||||
|
let file = File::open(archive)?;
|
||||||
|
let mut zip =
|
||||||
|
zip::ZipArchive::new(file).map_err(|e| AgentError::Other(format!("open zip: {e}")))?;
|
||||||
|
fs::create_dir_all(dest)?;
|
||||||
|
for i in 0..zip.len() {
|
||||||
|
let mut entry = zip
|
||||||
|
.by_index(i)
|
||||||
|
.map_err(|e| AgentError::Other(format!("read zip entry: {e}")))?;
|
||||||
|
// `enclosed_name` returns `None` for traversal-unsafe paths — skip them.
|
||||||
|
let Some(rel) = entry.enclosed_name() else {
|
||||||
|
continue;
|
||||||
|
};
|
||||||
|
let out = dest.join(rel);
|
||||||
|
if entry.is_dir() {
|
||||||
|
fs::create_dir_all(&out)?;
|
||||||
|
} else {
|
||||||
|
if let Some(parent) = out.parent() {
|
||||||
|
fs::create_dir_all(parent)?;
|
||||||
|
}
|
||||||
|
let mut outfile = File::create(&out)?;
|
||||||
|
io::copy(&mut entry, &mut outfile)?;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The working directory for one artifact of a target: `<base>/work/<target>/<artifact>`.
|
||||||
|
pub fn work_dir(base: &Path, target_id: &str, artifact_id: &str) -> PathBuf {
|
||||||
|
base.join("work").join(target_id).join(artifact_id)
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
#[allow(clippy::expect_used, clippy::unwrap_used)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
|
||||||
|
/// A unique scratch directory, removed on drop.
|
||||||
|
struct Scratch(PathBuf);
|
||||||
|
impl Scratch {
|
||||||
|
fn new() -> Self {
|
||||||
|
let p = std::env::temp_dir().join(format!("cs-ingest-{}", uuid::Uuid::new_v4()));
|
||||||
|
fs::create_dir_all(&p).expect("mkdir scratch");
|
||||||
|
Self(p)
|
||||||
|
}
|
||||||
|
fn path(&self) -> &Path {
|
||||||
|
&self.0
|
||||||
|
}
|
||||||
|
}
|
||||||
|
impl Drop for Scratch {
|
||||||
|
fn drop(&mut self) {
|
||||||
|
let _ = fs::remove_dir_all(&self.0);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn hash_is_stable_and_reports_size() {
|
||||||
|
let dir = Scratch::new();
|
||||||
|
let f = dir.path().join("a.bin");
|
||||||
|
fs::write(&f, b"hello world").expect("write");
|
||||||
|
let (sha, size) = hash_file(&f).expect("hash");
|
||||||
|
assert_eq!(size, 11);
|
||||||
|
// Known SHA-256 of "hello world".
|
||||||
|
assert_eq!(
|
||||||
|
sha,
|
||||||
|
"b94d27b9934d3e08a52e52d7da7dabfac484efe37a5380ee9088f7ace2efcde9"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn store_is_content_addressed_and_idempotent() {
|
||||||
|
let base = Scratch::new();
|
||||||
|
let src = base.path().join("src.bin");
|
||||||
|
fs::write(&src, b"payload").expect("write");
|
||||||
|
let (sha, _) = hash_file(&src).expect("hash");
|
||||||
|
let p1 = store_file(base.path(), &src, &sha).expect("store");
|
||||||
|
let p2 = store_file(base.path(), &src, &sha).expect("store again");
|
||||||
|
assert_eq!(p1, p2);
|
||||||
|
assert!(p1.ends_with(&sha));
|
||||||
|
assert!(p1.starts_with(base.path().join("blobs").join(&sha[0..2])));
|
||||||
|
assert_eq!(fs::read(&p1).expect("read"), b"payload");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn extract_zip_writes_entries() {
|
||||||
|
let base = Scratch::new();
|
||||||
|
let archive = base.path().join("a.zip");
|
||||||
|
{
|
||||||
|
let file = File::create(&archive).expect("create");
|
||||||
|
let mut w = zip::ZipWriter::new(file);
|
||||||
|
let opts: zip::write::SimpleFileOptions = Default::default();
|
||||||
|
w.start_file("dir/hello.txt", opts).expect("start");
|
||||||
|
io::Write::write_all(&mut w, b"hi").expect("write");
|
||||||
|
w.finish().expect("finish");
|
||||||
|
}
|
||||||
|
let dest = base.path().join("out");
|
||||||
|
extract_zip(&archive, &dest).expect("extract");
|
||||||
|
assert_eq!(
|
||||||
|
fs::read_to_string(dest.join("dir/hello.txt")).expect("read"),
|
||||||
|
"hi"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,334 @@
|
|||||||
|
//! Artifact ingest.
|
||||||
|
//!
|
||||||
|
//! Normalizes each [`Artifact`] on an [`OnboardedTarget`] into a local working
|
||||||
|
//! path plus recorded metadata (content hash, size, discovered facts) that the
|
||||||
|
//! classifier and scanners consume. Every blob is SHA-256 hashed — that digest
|
||||||
|
//! is also the reconciliation key against sibling products (a firmware sha256
|
||||||
|
//! matches tramiton's `Artifact.sha256`).
|
||||||
|
|
||||||
|
mod blob;
|
||||||
|
|
||||||
|
use std::collections::HashMap;
|
||||||
|
use std::path::{Path, PathBuf};
|
||||||
|
|
||||||
|
use compliance_core::models::{Artifact, ArtifactKind, DetectedFact, OnboardedTarget};
|
||||||
|
use compliance_core::AgentConfig;
|
||||||
|
|
||||||
|
use crate::error::AgentError;
|
||||||
|
use crate::pipeline::git::{GitOps, RepoCredentials};
|
||||||
|
|
||||||
|
/// The paths and identifiers an ingest needs. Decoupled from the full
|
||||||
|
/// [`AgentConfig`] so ingest is testable without a complete config.
|
||||||
|
pub struct IngestContext<'a> {
|
||||||
|
/// Base directory for content-addressed blobs and working dirs.
|
||||||
|
pub artifact_store_base: &'a Path,
|
||||||
|
/// Base directory for git clones.
|
||||||
|
pub git_clone_base: &'a str,
|
||||||
|
/// Default SSH key path (used when an artifact provides none).
|
||||||
|
pub ssh_key_path: &'a str,
|
||||||
|
/// The id of the target these artifacts belong to (namespaces working dirs).
|
||||||
|
pub target_id: &'a str,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl<'a> IngestContext<'a> {
|
||||||
|
/// Build an ingest context from the agent config for a given target.
|
||||||
|
pub fn from_config(config: &'a AgentConfig, target_id: &'a str) -> Self {
|
||||||
|
Self {
|
||||||
|
artifact_store_base: Path::new(&config.artifact_store_base_path),
|
||||||
|
git_clone_base: &config.git_clone_base_path,
|
||||||
|
ssh_key_path: &config.ssh_key_path,
|
||||||
|
target_id,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The result of ingesting one artifact.
|
||||||
|
pub struct IngestedArtifact {
|
||||||
|
/// The artifact this corresponds to ([`Artifact::id`]).
|
||||||
|
pub artifact_id: String,
|
||||||
|
/// The artifact kind.
|
||||||
|
pub kind: ArtifactKind,
|
||||||
|
/// Local working path (clone dir, extracted dir, or blob file). `None` for
|
||||||
|
/// artifacts with no on-disk form (live URL, plaintext, container ref).
|
||||||
|
pub working_path: Option<PathBuf>,
|
||||||
|
/// SHA-256 of the content (blobs) or git head SHA (git repos).
|
||||||
|
pub content_hash: Option<String>,
|
||||||
|
/// Stored blob size in bytes, when applicable.
|
||||||
|
pub size_bytes: Option<u64>,
|
||||||
|
/// Facts discovered during ingest.
|
||||||
|
pub facts: Vec<DetectedFact>,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// All ingested artifacts for a target, keyed by artifact id.
|
||||||
|
pub struct IngestSet {
|
||||||
|
/// The ingested artifacts, keyed by [`Artifact::id`].
|
||||||
|
pub by_artifact: HashMap<String, IngestedArtifact>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl IngestSet {
|
||||||
|
/// The working paths of every ingested artifact that has one — the input the
|
||||||
|
/// classifier expects.
|
||||||
|
pub fn working_paths(&self) -> HashMap<String, PathBuf> {
|
||||||
|
self.by_artifact
|
||||||
|
.iter()
|
||||||
|
.filter_map(|(id, a)| a.working_path.clone().map(|p| (id.clone(), p)))
|
||||||
|
.collect()
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The ingest result for a specific artifact.
|
||||||
|
pub fn get(&self, artifact_id: &str) -> Option<&IngestedArtifact> {
|
||||||
|
self.by_artifact.get(artifact_id)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Ingest every artifact on a target.
|
||||||
|
pub fn ingest_all(
|
||||||
|
target: &OnboardedTarget,
|
||||||
|
ctx: &IngestContext<'_>,
|
||||||
|
) -> Result<IngestSet, AgentError> {
|
||||||
|
let mut by_artifact = HashMap::new();
|
||||||
|
for artifact in &target.artifacts {
|
||||||
|
let ingested = ingest_artifact(artifact, ctx)?;
|
||||||
|
by_artifact.insert(artifact.id.clone(), ingested);
|
||||||
|
}
|
||||||
|
Ok(IngestSet { by_artifact })
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Ingest a single artifact, dispatching on its kind.
|
||||||
|
pub fn ingest_artifact(
|
||||||
|
artifact: &Artifact,
|
||||||
|
ctx: &IngestContext<'_>,
|
||||||
|
) -> Result<IngestedArtifact, AgentError> {
|
||||||
|
match artifact.kind {
|
||||||
|
ArtifactKind::GitRepo => ingest_git(artifact, ctx),
|
||||||
|
ArtifactKind::SourceArchive | ArtifactKind::MobilePackage | ArtifactKind::PlcProject => {
|
||||||
|
ingest_blob(artifact, ctx, true)
|
||||||
|
}
|
||||||
|
ArtifactKind::FirmwareImage => ingest_blob(artifact, ctx, false),
|
||||||
|
ArtifactKind::ContainerImage => Ok(metadata_only(
|
||||||
|
artifact,
|
||||||
|
DetectedFact::new("container_ref", artifact.source_ref.as_str(), "ingest"),
|
||||||
|
)),
|
||||||
|
ArtifactKind::LiveUrl => Ok(metadata_only(
|
||||||
|
artifact,
|
||||||
|
DetectedFact::new("live_url", artifact.source_ref.as_str(), "ingest"),
|
||||||
|
)),
|
||||||
|
ArtifactKind::PlaintextDescription => Ok(metadata_only(
|
||||||
|
artifact,
|
||||||
|
DetectedFact::new(
|
||||||
|
"description_len",
|
||||||
|
artifact.source_ref.len().to_string(),
|
||||||
|
"ingest",
|
||||||
|
),
|
||||||
|
)),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Clone (or fetch) a git artifact, recording the head SHA as the content hash.
|
||||||
|
fn ingest_git(
|
||||||
|
artifact: &Artifact,
|
||||||
|
ctx: &IngestContext<'_>,
|
||||||
|
) -> Result<IngestedArtifact, AgentError> {
|
||||||
|
let creds = credentials_for(artifact, ctx.ssh_key_path);
|
||||||
|
let git_ops = GitOps::new(ctx.git_clone_base, creds);
|
||||||
|
let repo_path = git_ops.clone_or_fetch(&artifact.source_ref, &artifact.id)?;
|
||||||
|
let head = GitOps::get_head_sha(&repo_path).ok();
|
||||||
|
Ok(IngestedArtifact {
|
||||||
|
artifact_id: artifact.id.clone(),
|
||||||
|
kind: artifact.kind,
|
||||||
|
working_path: Some(repo_path),
|
||||||
|
content_hash: head,
|
||||||
|
size_bytes: None,
|
||||||
|
facts: Vec::new(),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Store a blob artifact content-addressed. When `extract` is set and the blob
|
||||||
|
/// is a zip container (source archive, APK/AAB/IPA), also unpack it into a
|
||||||
|
/// working directory; otherwise the working path is the stored blob.
|
||||||
|
fn ingest_blob(
|
||||||
|
artifact: &Artifact,
|
||||||
|
ctx: &IngestContext<'_>,
|
||||||
|
extract: bool,
|
||||||
|
) -> Result<IngestedArtifact, AgentError> {
|
||||||
|
let base = ctx.artifact_store_base;
|
||||||
|
let src = local_source(artifact)?;
|
||||||
|
let (sha, size) = blob::hash_file(&src)?;
|
||||||
|
let stored = blob::store_file(base, &src, &sha)?;
|
||||||
|
|
||||||
|
let mut facts = Vec::new();
|
||||||
|
let working_path = if extract {
|
||||||
|
let dest = blob::work_dir(base, ctx.target_id, &artifact.id);
|
||||||
|
match blob::extract_zip(&stored, &dest) {
|
||||||
|
Ok(()) => dest,
|
||||||
|
Err(e) => {
|
||||||
|
// Not a zip (e.g. a tar.gz source archive) — keep the blob and
|
||||||
|
// note it so later stages can decide what to do.
|
||||||
|
facts.push(DetectedFact::new(
|
||||||
|
"archive_unextracted",
|
||||||
|
e.to_string(),
|
||||||
|
"ingest",
|
||||||
|
));
|
||||||
|
stored.clone()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
stored.clone()
|
||||||
|
};
|
||||||
|
|
||||||
|
Ok(IngestedArtifact {
|
||||||
|
artifact_id: artifact.id.clone(),
|
||||||
|
kind: artifact.kind,
|
||||||
|
working_path: Some(working_path),
|
||||||
|
content_hash: Some(sha),
|
||||||
|
size_bytes: Some(size),
|
||||||
|
facts,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
/// An artifact with no on-disk form: record a single fact, no hash/path.
|
||||||
|
fn metadata_only(artifact: &Artifact, fact: DetectedFact) -> IngestedArtifact {
|
||||||
|
IngestedArtifact {
|
||||||
|
artifact_id: artifact.id.clone(),
|
||||||
|
kind: artifact.kind,
|
||||||
|
working_path: None,
|
||||||
|
content_hash: None,
|
||||||
|
size_bytes: None,
|
||||||
|
facts: vec![fact],
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The local file backing a blob artifact: its `stored_path` if already
|
||||||
|
/// uploaded, else its `source_ref` interpreted as a filesystem path.
|
||||||
|
fn local_source(artifact: &Artifact) -> Result<PathBuf, AgentError> {
|
||||||
|
let path = artifact
|
||||||
|
.stored_path
|
||||||
|
.as_deref()
|
||||||
|
.unwrap_or(artifact.source_ref.as_str());
|
||||||
|
let path = PathBuf::from(path);
|
||||||
|
if !path.exists() {
|
||||||
|
return Err(AgentError::Other(format!(
|
||||||
|
"artifact {} source not found at {}",
|
||||||
|
artifact.id,
|
||||||
|
path.display()
|
||||||
|
)));
|
||||||
|
}
|
||||||
|
Ok(path)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Build git credentials from an artifact's auth plus a default SSH key path.
|
||||||
|
fn credentials_for(artifact: &Artifact, default_ssh_key_path: &str) -> RepoCredentials {
|
||||||
|
let auth = artifact.auth.as_ref();
|
||||||
|
RepoCredentials {
|
||||||
|
ssh_key_path: auth
|
||||||
|
.and_then(|a| a.ssh_key_path.clone())
|
||||||
|
.or_else(|| Some(default_ssh_key_path.to_string())),
|
||||||
|
auth_token: auth.and_then(|a| a.secret.clone()),
|
||||||
|
auth_username: auth.and_then(|a| a.username.clone()),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
#[allow(clippy::expect_used, clippy::unwrap_used)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
use compliance_core::models::{ArtifactAuth, TargetType};
|
||||||
|
|
||||||
|
/// A unique scratch directory, removed on drop.
|
||||||
|
struct Scratch(PathBuf);
|
||||||
|
impl Scratch {
|
||||||
|
fn new() -> Self {
|
||||||
|
let p = std::env::temp_dir().join(format!("cs-ingest-mod-{}", uuid::Uuid::new_v4()));
|
||||||
|
std::fs::create_dir_all(&p).expect("mkdir scratch");
|
||||||
|
Self(p)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
impl Drop for Scratch {
|
||||||
|
fn drop(&mut self) {
|
||||||
|
let _ = std::fs::remove_dir_all(&self.0);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn ctx_for<'a>(store: &'a Path, target_id: &'a str) -> IngestContext<'a> {
|
||||||
|
IngestContext {
|
||||||
|
artifact_store_base: store,
|
||||||
|
git_clone_base: "/tmp/cs-ingest-test-repos",
|
||||||
|
ssh_key_path: "/tmp/cs-ingest-test-ssh",
|
||||||
|
target_id,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn firmware_blob_is_hashed_and_stored() {
|
||||||
|
let scratch = Scratch::new();
|
||||||
|
let store = scratch.0.join("store");
|
||||||
|
let fw = scratch.0.join("fw.bin");
|
||||||
|
std::fs::write(&fw, b"firmware-bytes").expect("write");
|
||||||
|
|
||||||
|
let ctx = ctx_for(&store, "t1");
|
||||||
|
let artifact = Artifact::firmware_image(fw.to_string_lossy().to_string());
|
||||||
|
let out = ingest_artifact(&artifact, &ctx).expect("ingest");
|
||||||
|
|
||||||
|
assert_eq!(out.kind, ArtifactKind::FirmwareImage);
|
||||||
|
assert_eq!(out.size_bytes, Some(14));
|
||||||
|
let sha = out.content_hash.expect("hash");
|
||||||
|
assert_eq!(sha.len(), 64);
|
||||||
|
// working path is the content-addressed blob
|
||||||
|
let wp = out.working_path.expect("working path");
|
||||||
|
assert!(wp.starts_with(store.join("blobs")));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn live_url_has_no_blob() {
|
||||||
|
let scratch = Scratch::new();
|
||||||
|
let store = scratch.0.join("store");
|
||||||
|
let ctx = ctx_for(&store, "t1");
|
||||||
|
let artifact = Artifact::live_url("https://example.com");
|
||||||
|
let out = ingest_artifact(&artifact, &ctx).expect("ingest");
|
||||||
|
assert!(out.working_path.is_none());
|
||||||
|
assert!(out.content_hash.is_none());
|
||||||
|
assert!(out.facts.iter().any(|f| f.key == "live_url"));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn ingest_all_collects_working_paths() {
|
||||||
|
let scratch = Scratch::new();
|
||||||
|
let store = scratch.0.join("store");
|
||||||
|
let fw = scratch.0.join("fw.bin");
|
||||||
|
std::fs::write(&fw, b"abc").expect("write");
|
||||||
|
let ctx = ctx_for(&store, "t1");
|
||||||
|
|
||||||
|
let mut target = OnboardedTarget::new("t".to_string(), TargetType::FirmwareBareMetal);
|
||||||
|
target
|
||||||
|
.artifacts
|
||||||
|
.push(Artifact::firmware_image(fw.to_string_lossy().to_string()));
|
||||||
|
target.artifacts.push(Artifact::live_url("https://x"));
|
||||||
|
|
||||||
|
let set = ingest_all(&target, &ctx).expect("ingest all");
|
||||||
|
assert_eq!(set.by_artifact.len(), 2);
|
||||||
|
// Only the firmware artifact yields a working path.
|
||||||
|
assert_eq!(set.working_paths().len(), 1);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn credentials_prefer_artifact_auth() {
|
||||||
|
let mut artifact = Artifact::git_repo("https://git/x", "main");
|
||||||
|
artifact.auth = Some(ArtifactAuth {
|
||||||
|
method: "token".to_string(),
|
||||||
|
username: Some("bob".to_string()),
|
||||||
|
secret: Some("pat".to_string()),
|
||||||
|
..Default::default()
|
||||||
|
});
|
||||||
|
let creds = credentials_for(&artifact, "/default/ssh/key");
|
||||||
|
assert_eq!(creds.auth_token.as_deref(), Some("pat"));
|
||||||
|
assert_eq!(creds.auth_username.as_deref(), Some("bob"));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn credentials_fall_back_to_default_ssh_key() {
|
||||||
|
let artifact = Artifact::git_repo("git@host:x.git", "main");
|
||||||
|
let creds = credentials_for(&artifact, "/default/ssh/key");
|
||||||
|
assert_eq!(creds.ssh_key_path.as_deref(), Some("/default/ssh/key"));
|
||||||
|
assert!(creds.auth_token.is_none());
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -2,9 +2,11 @@
|
|||||||
|
|
||||||
pub mod agent;
|
pub mod agent;
|
||||||
pub mod api;
|
pub mod api;
|
||||||
|
pub mod classify;
|
||||||
pub mod config;
|
pub mod config;
|
||||||
pub mod database;
|
pub mod database;
|
||||||
pub mod error;
|
pub mod error;
|
||||||
|
pub mod ingest;
|
||||||
pub mod llm;
|
pub mod llm;
|
||||||
pub mod pentest;
|
pub mod pentest;
|
||||||
pub mod pipeline;
|
pub mod pipeline;
|
||||||
|
|||||||
@@ -328,6 +328,7 @@ mod tests {
|
|||||||
scan_schedule: String::new(),
|
scan_schedule: String::new(),
|
||||||
cve_monitor_schedule: String::new(),
|
cve_monitor_schedule: String::new(),
|
||||||
git_clone_base_path: String::new(),
|
git_clone_base_path: String::new(),
|
||||||
|
artifact_store_base_path: String::new(),
|
||||||
ssh_key_path: String::new(),
|
ssh_key_path: String::new(),
|
||||||
keycloak_url: None,
|
keycloak_url: None,
|
||||||
keycloak_realm: None,
|
keycloak_realm: None,
|
||||||
|
|||||||
@@ -44,6 +44,7 @@ impl TestServer {
|
|||||||
scan_schedule: String::new(),
|
scan_schedule: String::new(),
|
||||||
cve_monitor_schedule: String::new(),
|
cve_monitor_schedule: String::new(),
|
||||||
git_clone_base_path: "/tmp/compliance-scanner-tests/repos".into(),
|
git_clone_base_path: "/tmp/compliance-scanner-tests/repos".into(),
|
||||||
|
artifact_store_base_path: "/tmp/compliance-scanner-tests/artifacts".into(),
|
||||||
ssh_key_path: "/tmp/compliance-scanner-tests/ssh/id_ed25519".into(),
|
ssh_key_path: "/tmp/compliance-scanner-tests/ssh/id_ed25519".into(),
|
||||||
github_token: None,
|
github_token: None,
|
||||||
github_webhook_secret: None,
|
github_webhook_secret: None,
|
||||||
|
|||||||
@@ -24,6 +24,9 @@ pub struct AgentConfig {
|
|||||||
pub scan_schedule: String,
|
pub scan_schedule: String,
|
||||||
pub cve_monitor_schedule: String,
|
pub cve_monitor_schedule: String,
|
||||||
pub git_clone_base_path: String,
|
pub git_clone_base_path: String,
|
||||||
|
/// Base directory for content-addressed artifact blobs and per-run working
|
||||||
|
/// dirs (`<base>/blobs/<sha[0:2]>/<sha>`, `<base>/work/<target>/<artifact>/`).
|
||||||
|
pub artifact_store_base_path: String,
|
||||||
pub ssh_key_path: String,
|
pub ssh_key_path: String,
|
||||||
pub keycloak_url: Option<String>,
|
pub keycloak_url: Option<String>,
|
||||||
pub keycloak_realm: Option<String>,
|
pub keycloak_realm: Option<String>,
|
||||||
|
|||||||
@@ -33,9 +33,10 @@ pub use mcp::{McpServerConfig, McpServerStatus, McpTransport};
|
|||||||
pub use mcp_token::{McpToken, McpTokenView};
|
pub use mcp_token::{McpToken, McpTokenView};
|
||||||
pub use notification::{CveNotification, NotificationSeverity, NotificationStatus};
|
pub use notification::{CveNotification, NotificationSeverity, NotificationStatus};
|
||||||
pub use onboarding::{
|
pub use onboarding::{
|
||||||
Artifact, ArtifactAuth, ArtifactKind, Classification, DetectedFact, GitArtifactConfig,
|
default_compliance_profile, Artifact, ArtifactAuth, ArtifactKind, Classification,
|
||||||
IssueTrackerConfig, OnboardedTarget, PlcArtifactConfig, PlcFormat, TargetScanConfig,
|
ComplianceFramework, ComplianceProfile, DetectedFact, ExternalRef, ExternalSystem,
|
||||||
TargetType, TargetTypeCandidate, WebArtifactConfig,
|
GitArtifactConfig, IssueTrackerConfig, OnboardedTarget, PlcArtifactConfig, PlcFormat,
|
||||||
|
TargetScanConfig, TargetType, TargetTypeCandidate, WebArtifactConfig,
|
||||||
};
|
};
|
||||||
pub use pentest::{
|
pub use pentest::{
|
||||||
AttackChainNode, AttackNodeStatus, AuthMode, CodeContextHint, Environment, IdentityProvider,
|
AttackChainNode, AttackNodeStatus, AuthMode, CodeContextHint, Environment, IdentityProvider,
|
||||||
|
|||||||
@@ -22,7 +22,7 @@ use super::scan::ScanType;
|
|||||||
/// Targets look endlessly varied but fall into a small enumerable set classified
|
/// Targets look endlessly varied but fall into a small enumerable set classified
|
||||||
/// by where the analyzable signal lives. This drives the scan-applicability
|
/// by where the analyzable signal lives. This drives the scan-applicability
|
||||||
/// matrix and the onboarding wizard's type selection.
|
/// matrix and the onboarding wizard's type selection.
|
||||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
|
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Serialize, Deserialize)]
|
||||||
#[serde(rename_all = "snake_case")]
|
#[serde(rename_all = "snake_case")]
|
||||||
pub enum TargetType {
|
pub enum TargetType {
|
||||||
/// Browser-facing web application (front end + server).
|
/// Browser-facing web application (front end + server).
|
||||||
@@ -411,6 +411,134 @@ pub struct TargetScanConfig {
|
|||||||
pub issue_tracker: Option<IssueTrackerConfig>,
|
pub issue_tracker: Option<IssueTrackerConfig>,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// A sibling product in the company suite that may already hold authoritative
|
||||||
|
/// data for a target. compliance-scanner reconciles with these rather than
|
||||||
|
/// recomputing what they already know.
|
||||||
|
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
|
||||||
|
#[serde(rename_all = "snake_case")]
|
||||||
|
pub enum ExternalSystem {
|
||||||
|
/// Reproducible-build & firmware compliance engine (build plan, SBOM, VEX,
|
||||||
|
/// attestation).
|
||||||
|
Tramiton,
|
||||||
|
/// Code assistant (downstream remediation consumer).
|
||||||
|
Werkpilot,
|
||||||
|
/// Compliance-controls RAG (atomic controls derived from laws).
|
||||||
|
BreakpilotCompliance,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl std::fmt::Display for ExternalSystem {
|
||||||
|
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||||
|
match self {
|
||||||
|
Self::Tramiton => write!(f, "tramiton"),
|
||||||
|
Self::Werkpilot => write!(f, "werkpilot"),
|
||||||
|
Self::BreakpilotCompliance => write!(f, "breakpilot_compliance"),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A link from this target to a record in a sibling product, used to reconcile
|
||||||
|
/// existing evidence instead of recomputing it.
|
||||||
|
///
|
||||||
|
/// For tramiton, `project_id` is the shared cross-product key and
|
||||||
|
/// `subject_sha256` matches a firmware artifact's [`Artifact::content_hash`]
|
||||||
|
/// (which equals tramiton's `Artifact.sha256`).
|
||||||
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||||
|
pub struct ExternalRef {
|
||||||
|
/// Which sibling product this reference points at.
|
||||||
|
pub system: ExternalSystem,
|
||||||
|
/// The sibling product's project identifier, if known.
|
||||||
|
pub project_id: Option<String>,
|
||||||
|
/// Content digest of the subject artifact (firmware sha256), if known.
|
||||||
|
pub subject_sha256: Option<String>,
|
||||||
|
/// Reconciliation status: `linked` | `reconciled` | `unavailable`.
|
||||||
|
#[serde(default)]
|
||||||
|
pub status: String,
|
||||||
|
/// Opaque, offline-verifiable entitlement grant (e.g. tramiton's signed
|
||||||
|
/// `LicenseGrant`), if the tenant provided one.
|
||||||
|
pub license_grant: Option<String>,
|
||||||
|
/// When evidence was last reconciled from this system.
|
||||||
|
#[serde(default, with = "super::serde_helpers::opt_bson_datetime")]
|
||||||
|
pub last_reconciled_at: Option<DateTime<Utc>>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl ExternalRef {
|
||||||
|
/// A freshly linked (not yet reconciled) reference to a sibling system.
|
||||||
|
pub fn linked(system: ExternalSystem) -> Self {
|
||||||
|
Self {
|
||||||
|
system,
|
||||||
|
project_id: None,
|
||||||
|
subject_sha256: None,
|
||||||
|
status: "linked".to_string(),
|
||||||
|
license_grant: None,
|
||||||
|
last_reconciled_at: None,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A regulatory / standards framework a target must comply with. Drives which
|
||||||
|
/// controls the mapping engine pulls from the [`crate::traits::ControlsProvider`].
|
||||||
|
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
|
||||||
|
#[serde(rename_all = "snake_case")]
|
||||||
|
pub enum ComplianceFramework {
|
||||||
|
/// EU Cyber Resilience Act.
|
||||||
|
Cra,
|
||||||
|
/// IEC 62443 (industrial automation & control systems security).
|
||||||
|
Iec62443,
|
||||||
|
/// EU General Data Protection Regulation.
|
||||||
|
Gdpr,
|
||||||
|
/// SOC 2.
|
||||||
|
Soc2,
|
||||||
|
/// ISO/IEC 27001.
|
||||||
|
Iso27001,
|
||||||
|
/// EU Radio Equipment Directive (RED) cybersecurity articles.
|
||||||
|
RedDirective,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl std::fmt::Display for ComplianceFramework {
|
||||||
|
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||||
|
match self {
|
||||||
|
Self::Cra => write!(f, "cra"),
|
||||||
|
Self::Iec62443 => write!(f, "iec_62443"),
|
||||||
|
Self::Gdpr => write!(f, "gdpr"),
|
||||||
|
Self::Soc2 => write!(f, "soc2"),
|
||||||
|
Self::Iso27001 => write!(f, "iso_27001"),
|
||||||
|
Self::RedDirective => write!(f, "red_directive"),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The compliance scope of a target: which frameworks apply and, optionally, the
|
||||||
|
/// jurisdiction. Captured at onboarding (with per-target-type defaults from
|
||||||
|
/// [`default_compliance_profile`]).
|
||||||
|
#[derive(Debug, Clone, Default, Serialize, Deserialize)]
|
||||||
|
pub struct ComplianceProfile {
|
||||||
|
/// Applicable frameworks.
|
||||||
|
#[serde(default)]
|
||||||
|
pub frameworks: Vec<ComplianceFramework>,
|
||||||
|
/// Free-form jurisdiction (e.g. `eu`, `us`, `de`).
|
||||||
|
pub jurisdiction: Option<String>,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The sensible default compliance scope for a target type. Firmware / PLC /
|
||||||
|
/// embedded default to CRA + IEC 62443; software defaults to GDPR + SOC 2.
|
||||||
|
pub fn default_compliance_profile(target_type: TargetType) -> ComplianceProfile {
|
||||||
|
use ComplianceFramework::{Cra, Gdpr, Iec62443, Soc2};
|
||||||
|
let frameworks = match target_type {
|
||||||
|
TargetType::PlcSps
|
||||||
|
| TargetType::FirmwareBareMetal
|
||||||
|
| TargetType::FirmwareRtos
|
||||||
|
| TargetType::EmbeddedLinuxYocto => vec![Cra, Iec62443],
|
||||||
|
TargetType::WebApp | TargetType::BackendService => vec![Gdpr, Soc2],
|
||||||
|
TargetType::DesktopApp | TargetType::AndroidApp | TargetType::IosApp => {
|
||||||
|
vec![Gdpr, Cra]
|
||||||
|
}
|
||||||
|
};
|
||||||
|
ComplianceProfile {
|
||||||
|
frameworks,
|
||||||
|
jurisdiction: None,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/// A target onboarded for scanning: the unified replacement for the legacy
|
/// A target onboarded for scanning: the unified replacement for the legacy
|
||||||
/// `TrackedRepository` (SAST) and `DastTarget` (DAST) records.
|
/// `TrackedRepository` (SAST) and `DastTarget` (DAST) records.
|
||||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||||
@@ -434,6 +562,12 @@ pub struct OnboardedTarget {
|
|||||||
/// How this target should be scanned.
|
/// How this target should be scanned.
|
||||||
#[serde(default)]
|
#[serde(default)]
|
||||||
pub scan_config: TargetScanConfig,
|
pub scan_config: TargetScanConfig,
|
||||||
|
/// The compliance scope (applicable frameworks / jurisdiction).
|
||||||
|
#[serde(default)]
|
||||||
|
pub compliance_profile: ComplianceProfile,
|
||||||
|
/// Links to sibling products (tramiton, ...) holding reconcilable evidence.
|
||||||
|
#[serde(default)]
|
||||||
|
pub external_refs: Vec<ExternalRef>,
|
||||||
/// Cron schedule for recurring scans, if any.
|
/// Cron schedule for recurring scans, if any.
|
||||||
pub scan_schedule: Option<String>,
|
pub scan_schedule: Option<String>,
|
||||||
/// Whether inbound webhooks are enabled for this target.
|
/// Whether inbound webhooks are enabled for this target.
|
||||||
@@ -472,6 +606,8 @@ impl OnboardedTarget {
|
|||||||
artifacts: Vec::new(),
|
artifacts: Vec::new(),
|
||||||
classification: None,
|
classification: None,
|
||||||
scan_config: TargetScanConfig::default(),
|
scan_config: TargetScanConfig::default(),
|
||||||
|
compliance_profile: default_compliance_profile(target_type),
|
||||||
|
external_refs: Vec::new(),
|
||||||
scan_schedule: None,
|
scan_schedule: None,
|
||||||
webhook_enabled: false,
|
webhook_enabled: false,
|
||||||
webhook_secret: Some(webhook_secret),
|
webhook_secret: Some(webhook_secret),
|
||||||
@@ -584,4 +720,34 @@ mod tests {
|
|||||||
let b = Artifact::firmware_image("fw.bin");
|
let b = Artifact::firmware_image("fw.bin");
|
||||||
assert_ne!(a.id, b.id);
|
assert_ne!(a.id, b.id);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn firmware_default_profile_is_cra_and_62443() {
|
||||||
|
let p = default_compliance_profile(TargetType::FirmwareBareMetal);
|
||||||
|
assert!(p.frameworks.contains(&ComplianceFramework::Cra));
|
||||||
|
assert!(p.frameworks.contains(&ComplianceFramework::Iec62443));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn webapp_default_profile_is_gdpr_and_soc2() {
|
||||||
|
let p = default_compliance_profile(TargetType::WebApp);
|
||||||
|
assert!(p.frameworks.contains(&ComplianceFramework::Gdpr));
|
||||||
|
assert!(p.frameworks.contains(&ComplianceFramework::Soc2));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn new_target_gets_default_profile_and_no_external_refs() {
|
||||||
|
let t = OnboardedTarget::new("fw".to_string(), TargetType::FirmwareRtos);
|
||||||
|
assert!(!t.compliance_profile.frameworks.is_empty());
|
||||||
|
assert!(t.external_refs.is_empty());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn external_ref_linked_defaults() {
|
||||||
|
let r = ExternalRef::linked(ExternalSystem::Tramiton);
|
||||||
|
assert_eq!(r.system, ExternalSystem::Tramiton);
|
||||||
|
assert_eq!(r.status, "linked");
|
||||||
|
assert!(r.project_id.is_none());
|
||||||
|
assert!(r.last_reconciled_at.is_none());
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,45 @@
|
|||||||
|
//! The compliance-controls provider port.
|
||||||
|
//!
|
||||||
|
//! The mapping engine turns findings into compliance status against a corpus of
|
||||||
|
//! controls. That corpus is pluggable: the built-in OSCAL catalog by default, or
|
||||||
|
//! a tenant-owned RAG of atomic controls derived from laws
|
||||||
|
//! (`breakpilot-compliance`) when available. A [`ControlsProvider`] abstracts the
|
||||||
|
//! source so the mapping engine does not hardcode a catalog.
|
||||||
|
|
||||||
|
use crate::error::CoreError;
|
||||||
|
use crate::models::ComplianceFramework;
|
||||||
|
|
||||||
|
/// A control retrieved from a controls corpus.
|
||||||
|
#[derive(Debug, Clone)]
|
||||||
|
pub struct Control {
|
||||||
|
/// Stable control identifier (e.g. an OSCAL control id or a RAG chunk id).
|
||||||
|
pub id: String,
|
||||||
|
/// The framework this control belongs to.
|
||||||
|
pub framework: ComplianceFramework,
|
||||||
|
/// Short human-readable title.
|
||||||
|
pub title: String,
|
||||||
|
/// The control text / requirement.
|
||||||
|
pub text: String,
|
||||||
|
/// Free-form source reference (catalog name, law citation, ...).
|
||||||
|
pub source: Option<String>,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A query for relevant controls.
|
||||||
|
pub struct ControlQuery<'a> {
|
||||||
|
/// Frameworks in scope for the target.
|
||||||
|
pub frameworks: &'a [ComplianceFramework],
|
||||||
|
/// Free-text describing what to map (a finding summary, a component, ...).
|
||||||
|
pub context: &'a str,
|
||||||
|
/// Maximum number of controls to return.
|
||||||
|
pub limit: usize,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A source of compliance controls (built-in OSCAL catalog, breakpilot RAG, ...).
|
||||||
|
#[allow(async_fn_in_trait)]
|
||||||
|
pub trait ControlsProvider: Send + Sync {
|
||||||
|
/// Stable identifier for this provider.
|
||||||
|
fn name(&self) -> &str;
|
||||||
|
|
||||||
|
/// Retrieve the controls most relevant to the query.
|
||||||
|
async fn controls(&self, query: &ControlQuery<'_>) -> Result<Vec<Control>, CoreError>;
|
||||||
|
}
|
||||||
@@ -0,0 +1,58 @@
|
|||||||
|
//! The external-evidence provider port.
|
||||||
|
//!
|
||||||
|
//! A sibling product (tramiton, for firmware) may already hold authoritative
|
||||||
|
//! analysis for an artifact. An [`EvidenceProvider`] lets compliance-scanner
|
||||||
|
//! *reconcile* that evidence — a build plan, an SBOM, a VEX document, a
|
||||||
|
//! reproducible-build lock, an attestation — instead of recomputing it. The key
|
||||||
|
//! used to match is the artifact content digest (a firmware sha256, which equals
|
||||||
|
//! [`crate::models::Artifact::content_hash`]).
|
||||||
|
//!
|
||||||
|
//! Concrete providers live in the agent (a tramiton CLI shell-out today, a cloud
|
||||||
|
//! client later) plus a deterministic mock for tests, so nothing here depends on
|
||||||
|
//! an external binary.
|
||||||
|
|
||||||
|
use std::path::Path;
|
||||||
|
|
||||||
|
use crate::error::CoreError;
|
||||||
|
use crate::models::{Artifact, ExternalSystem};
|
||||||
|
|
||||||
|
/// A single reconcilable evidence document fetched from a sibling product.
|
||||||
|
#[derive(Debug, Clone)]
|
||||||
|
pub struct EvidenceDocument {
|
||||||
|
/// What the document is: `build_plan` | `sbom` | `vex` | `lock` | `attestation`.
|
||||||
|
pub kind: String,
|
||||||
|
/// The document's format (e.g. `cyclonedx-1.5`, `openvex-0.2.0`, `toml`, `json`).
|
||||||
|
pub format: String,
|
||||||
|
/// The raw document payload.
|
||||||
|
pub content: String,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The evidence a provider could return for a target's artifact.
|
||||||
|
#[derive(Debug, Clone, Default)]
|
||||||
|
pub struct ReconciledEvidence {
|
||||||
|
/// The sibling's project identifier, if resolved.
|
||||||
|
pub project_id: Option<String>,
|
||||||
|
/// The subject content digest the evidence pertains to.
|
||||||
|
pub subject_sha256: Option<String>,
|
||||||
|
/// The documents fetched (any of build plan / SBOM / VEX / lock / attestation).
|
||||||
|
pub documents: Vec<EvidenceDocument>,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A source of externally-held, reconcilable evidence for an artifact.
|
||||||
|
#[allow(async_fn_in_trait)]
|
||||||
|
pub trait EvidenceProvider: Send + Sync {
|
||||||
|
/// Which sibling product this provider integrates.
|
||||||
|
fn system(&self) -> ExternalSystem;
|
||||||
|
|
||||||
|
/// Whether this provider can handle the given artifact + working path
|
||||||
|
/// (e.g. tramiton handles firmware images / embedded source trees).
|
||||||
|
fn handles(&self, artifact: &Artifact, working_path: Option<&Path>) -> bool;
|
||||||
|
|
||||||
|
/// Reconcile existing evidence for the artifact, keyed by its content digest.
|
||||||
|
/// Returns `Ok(None)` when the provider has nothing for this artifact.
|
||||||
|
async fn reconcile(
|
||||||
|
&self,
|
||||||
|
artifact: &Artifact,
|
||||||
|
working_path: Option<&Path>,
|
||||||
|
) -> Result<Option<ReconciledEvidence>, CoreError>;
|
||||||
|
}
|
||||||
@@ -1,12 +1,16 @@
|
|||||||
pub mod classifier;
|
pub mod classifier;
|
||||||
|
pub mod controls;
|
||||||
pub mod dast_agent;
|
pub mod dast_agent;
|
||||||
|
pub mod evidence;
|
||||||
pub mod graph_builder;
|
pub mod graph_builder;
|
||||||
pub mod issue_tracker;
|
pub mod issue_tracker;
|
||||||
pub mod pentest_tool;
|
pub mod pentest_tool;
|
||||||
pub mod scanner;
|
pub mod scanner;
|
||||||
|
|
||||||
pub use classifier::{ClassificationInput, ClassifierVerdict, TargetClassifier};
|
pub use classifier::{ClassificationInput, ClassifierVerdict, TargetClassifier};
|
||||||
|
pub use controls::{Control, ControlQuery, ControlsProvider};
|
||||||
pub use dast_agent::{DastAgent, DastContext, DiscoveredEndpoint, EndpointParameter};
|
pub use dast_agent::{DastAgent, DastContext, DiscoveredEndpoint, EndpointParameter};
|
||||||
|
pub use evidence::{EvidenceDocument, EvidenceProvider, ReconciledEvidence};
|
||||||
pub use graph_builder::{LanguageParser, ParseOutput};
|
pub use graph_builder::{LanguageParser, ParseOutput};
|
||||||
pub use issue_tracker::IssueTracker;
|
pub use issue_tracker::IssueTracker;
|
||||||
pub use pentest_tool::{PentestTool, PentestToolContext, PentestToolResult};
|
pub use pentest_tool::{PentestTool, PentestToolContext, PentestToolResult};
|
||||||
|
|||||||
Reference in New Issue
Block a user