feat(werkbank): Mongo-backed job queue with lease + visibility timeout (WB-02)
CI / Check (pull_request) Successful in 5m49s
CI / Detect Changes (pull_request) Has been skipped
CI / Deploy Agent (pull_request) Has been skipped
CI / Deploy Dashboard (pull_request) Has been skipped
CI / Deploy Docs (pull_request) Has been skipped
CI / Deploy MCP (pull_request) Has been skipped
CI / Check (pull_request) Successful in 5m49s
CI / Detect Changes (pull_request) Has been skipped
CI / Deploy Agent (pull_request) Has been skipped
CI / Deploy Dashboard (pull_request) Has been skipped
CI / Deploy Docs (pull_request) Has been skipped
CI / Deploy MCP (pull_request) Has been skipped
The control-plane pull queue behind the Werkbank runner flow (implements sharang/werkbank#3). A JobQueue over a `werkbank_jobs` collection: - enqueue — idempotent by job id (unique index; duplicate is a no-op) - lease — atomic find-and-modify of the oldest queued job the runner can run, matched by executor and by labels (job labels must be a subset of the runner's, empty/absent matches any), returns the job + a lease token, bumps attempts - heartbeat — extends the lease, flips leased→running, surfaces a cancel request; None means the lease was lost (token mismatch / already terminal) - complete — records the terminal result, token-guarded and only from an active state, so it's idempotent - cancel — queued→cancelled outright, in-flight flagged for the next heartbeat - sweep_expired — the visibility timeout: expired leases go back to queued, or to expired once attempts hit max, so a crashed runner's job recovers All transitions are single atomic Mongo updates guarded by the lease token, so two runners can never both own a job. Every op takes an explicit `now` for deterministic tests. Adds JobRecord/LeasedJob/HeartbeatAck to the contract (BSON datetimes so range queries compare correctly) and the werkbank_jobs indexes. Tests: 5 integration against a real Mongo (idempotent enqueue, executor+label matching + FIFO, heartbeat/cancel, token-guarded idempotent complete, sweep requeue→expire; skip cleanly with no Mongo) + 2 unit. clippy + fmt clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5
parent
7b218fffef
commit
25f232774e
@@ -49,5 +49,6 @@ pub use repository::ScanTrigger;
|
||||
pub use sbom::{SbomEntry, VulnRef};
|
||||
pub use scan::{ScanPhase, ScanRun, ScanRunStatus, ScanType};
|
||||
pub use werkbank::{
|
||||
DastCollect, Executor, InputRef, Job, JobCollect, JobResult, JobRuntime, JobStatus, JobType,
|
||||
DastCollect, Executor, HeartbeatAck, InputRef, Job, JobCollect, JobRecord, JobResult,
|
||||
JobRuntime, JobStatus, JobType, LeasedJob,
|
||||
};
|
||||
|
||||
@@ -266,6 +266,89 @@ impl JobResult {
|
||||
}
|
||||
}
|
||||
|
||||
/// A queued job as persisted by the control plane (WB-02): the [`Job`] contract
|
||||
/// plus the queue bookkeeping — status, lease ownership, attempt count, and the
|
||||
/// eventual result. The runner never sees this record; on lease it receives a
|
||||
/// [`LeasedJob`] (the job plus a token it presents to heartbeat/complete).
|
||||
///
|
||||
/// Timestamps persist as native BSON dates so the queue's range queries (lease
|
||||
/// FIFO by `created_at`, visibility-timeout sweep by `lease_expires_at`) compare
|
||||
/// correctly.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct JobRecord {
|
||||
/// The job to run.
|
||||
pub job: Job,
|
||||
/// Current queue state.
|
||||
pub status: JobStatus,
|
||||
/// The lease token held by the current runner (proves lease ownership).
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub lease_token: Option<String>,
|
||||
/// Id of the runner holding the lease.
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub leased_by: Option<String>,
|
||||
/// When the current lease expires — the visibility timeout after which a
|
||||
/// crashed runner's job is swept back to `queued`.
|
||||
#[serde(default, with = "super::serde_helpers::opt_bson_datetime")]
|
||||
pub lease_expires_at: Option<DateTime<Utc>>,
|
||||
/// Last heartbeat from the runner.
|
||||
#[serde(default, with = "super::serde_helpers::opt_bson_datetime")]
|
||||
pub heartbeat_at: Option<DateTime<Utc>>,
|
||||
/// How many times the job has been leased (incremented on each lease).
|
||||
#[serde(default)]
|
||||
pub attempts: u32,
|
||||
/// Set when the control plane requests cancellation; the runner sees it on
|
||||
/// its next heartbeat and aborts.
|
||||
#[serde(default)]
|
||||
pub cancel_requested: bool,
|
||||
/// The result, once the job reaches a terminal state.
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub result: Option<JobResult>,
|
||||
/// When the job was enqueued.
|
||||
#[serde(with = "super::serde_helpers::bson_datetime")]
|
||||
pub created_at: DateTime<Utc>,
|
||||
/// Last modification.
|
||||
#[serde(with = "super::serde_helpers::bson_datetime")]
|
||||
pub updated_at: DateTime<Utc>,
|
||||
}
|
||||
|
||||
impl JobRecord {
|
||||
/// A freshly-enqueued (`queued`) record for a job.
|
||||
pub fn queued(job: Job, now: DateTime<Utc>) -> Self {
|
||||
Self {
|
||||
job,
|
||||
status: JobStatus::Queued,
|
||||
lease_token: None,
|
||||
leased_by: None,
|
||||
lease_expires_at: None,
|
||||
heartbeat_at: None,
|
||||
attempts: 0,
|
||||
cancel_requested: false,
|
||||
result: None,
|
||||
created_at: now,
|
||||
updated_at: now,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// A job handed to a runner on lease: what to run plus the token the runner must
|
||||
/// present to heartbeat and complete it.
|
||||
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub struct LeasedJob {
|
||||
/// The job to execute.
|
||||
pub job: Job,
|
||||
/// The lease token proving ownership (opaque to the runner).
|
||||
pub lease_token: String,
|
||||
}
|
||||
|
||||
/// The runner's view of a heartbeat: whether the control plane has asked the job
|
||||
/// to stop. `None` from the queue means the lease was lost (token mismatch or the
|
||||
/// job already terminal) and the runner should abandon the work.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub struct HeartbeatAck {
|
||||
/// The control plane requested cancellation — the runner should tear down.
|
||||
pub cancelled: bool,
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
#[allow(clippy::expect_used, clippy::unwrap_used)]
|
||||
mod tests {
|
||||
|
||||
Reference in New Issue
Block a user