feat(werkbank): Mongo-backed job queue with lease + visibility timeout (WB-02)
CI / Check (pull_request) Successful in 5m49s
CI / Detect Changes (pull_request) Has been skipped
CI / Deploy Agent (pull_request) Has been skipped
CI / Deploy Dashboard (pull_request) Has been skipped
CI / Deploy Docs (pull_request) Has been skipped
CI / Deploy MCP (pull_request) Has been skipped

The control-plane pull queue behind the Werkbank runner flow (implements
sharang/werkbank#3). A JobQueue over a `werkbank_jobs` collection:

- enqueue — idempotent by job id (unique index; duplicate is a no-op)
- lease — atomic find-and-modify of the oldest queued job the runner can run,
  matched by executor and by labels (job labels must be a subset of the runner's,
  empty/absent matches any), returns the job + a lease token, bumps attempts
- heartbeat — extends the lease, flips leased→running, surfaces a cancel request;
  None means the lease was lost (token mismatch / already terminal)
- complete — records the terminal result, token-guarded and only from an active
  state, so it's idempotent
- cancel — queued→cancelled outright, in-flight flagged for the next heartbeat
- sweep_expired — the visibility timeout: expired leases go back to queued, or to
  expired once attempts hit max, so a crashed runner's job recovers

All transitions are single atomic Mongo updates guarded by the lease token, so two
runners can never both own a job. Every op takes an explicit `now` for
deterministic tests. Adds JobRecord/LeasedJob/HeartbeatAck to the contract (BSON
datetimes so range queries compare correctly) and the werkbank_jobs indexes.

Tests: 5 integration against a real Mongo (idempotent enqueue, executor+label
matching + FIFO, heartbeat/cancel, token-guarded idempotent complete, sweep
requeue→expire; skip cleanly with no Mongo) + 2 unit. clippy + fmt clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Sharang Parnerkar
2026-07-17 11:07:31 +02:00
co-authored by Claude Fable 5
parent 7b218fffef
commit 25f232774e
7 changed files with 697 additions and 1 deletions
+2 -1
View File
@@ -49,5 +49,6 @@ pub use repository::ScanTrigger;
pub use sbom::{SbomEntry, VulnRef};
pub use scan::{ScanPhase, ScanRun, ScanRunStatus, ScanType};
pub use werkbank::{
DastCollect, Executor, InputRef, Job, JobCollect, JobResult, JobRuntime, JobStatus, JobType,
DastCollect, Executor, HeartbeatAck, InputRef, Job, JobCollect, JobRecord, JobResult,
JobRuntime, JobStatus, JobType, LeasedJob,
};
+83
View File
@@ -266,6 +266,89 @@ impl JobResult {
}
}
/// A queued job as persisted by the control plane (WB-02): the [`Job`] contract
/// plus the queue bookkeeping — status, lease ownership, attempt count, and the
/// eventual result. The runner never sees this record; on lease it receives a
/// [`LeasedJob`] (the job plus a token it presents to heartbeat/complete).
///
/// Timestamps persist as native BSON dates so the queue's range queries (lease
/// FIFO by `created_at`, visibility-timeout sweep by `lease_expires_at`) compare
/// correctly.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct JobRecord {
/// The job to run.
pub job: Job,
/// Current queue state.
pub status: JobStatus,
/// The lease token held by the current runner (proves lease ownership).
#[serde(default, skip_serializing_if = "Option::is_none")]
pub lease_token: Option<String>,
/// Id of the runner holding the lease.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub leased_by: Option<String>,
/// When the current lease expires — the visibility timeout after which a
/// crashed runner's job is swept back to `queued`.
#[serde(default, with = "super::serde_helpers::opt_bson_datetime")]
pub lease_expires_at: Option<DateTime<Utc>>,
/// Last heartbeat from the runner.
#[serde(default, with = "super::serde_helpers::opt_bson_datetime")]
pub heartbeat_at: Option<DateTime<Utc>>,
/// How many times the job has been leased (incremented on each lease).
#[serde(default)]
pub attempts: u32,
/// Set when the control plane requests cancellation; the runner sees it on
/// its next heartbeat and aborts.
#[serde(default)]
pub cancel_requested: bool,
/// The result, once the job reaches a terminal state.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub result: Option<JobResult>,
/// When the job was enqueued.
#[serde(with = "super::serde_helpers::bson_datetime")]
pub created_at: DateTime<Utc>,
/// Last modification.
#[serde(with = "super::serde_helpers::bson_datetime")]
pub updated_at: DateTime<Utc>,
}
impl JobRecord {
/// A freshly-enqueued (`queued`) record for a job.
pub fn queued(job: Job, now: DateTime<Utc>) -> Self {
Self {
job,
status: JobStatus::Queued,
lease_token: None,
leased_by: None,
lease_expires_at: None,
heartbeat_at: None,
attempts: 0,
cancel_requested: false,
result: None,
created_at: now,
updated_at: now,
}
}
}
/// A job handed to a runner on lease: what to run plus the token the runner must
/// present to heartbeat and complete it.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct LeasedJob {
/// The job to execute.
pub job: Job,
/// The lease token proving ownership (opaque to the runner).
pub lease_token: String,
}
/// The runner's view of a heartbeat: whether the control plane has asked the job
/// to stop. `None` from the queue means the lease was lost (token mismatch or the
/// job already terminal) and the runner should abandon the work.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
pub struct HeartbeatAck {
/// The control plane requested cancellation — the runner should tear down.
pub cancelled: bool,
}
#[cfg(test)]
#[allow(clippy::expect_used, clippy::unwrap_used)]
mod tests {