Skip to documentation
Deploy on AWS
Menu
Documentation · 15 of 19

Docs/Queue scheduler

JetEncode PostgreSQL Scheduler

This document describes JetEncode's PostgreSQL scheduler. PostgreSQL is the authoritative store for both Job state and queued work.

The scheduler is intentionally JetEncode-specific. It models long-running media stages, local workspace affinity, operator ordering, and future dedicated workers; it is not intended to become a general-purpose queue library.

Persistent model

Scheduling metadata and delivery metadata have different lifetimes and are stored separately.

job_schedule

One row represents the scheduling and execution assignment for a Job:

  • execution_id identifies one workspace/execution generation.
  • priority and position are the scheduling controls.
  • required_worker_uid constrains the initial dispatch when an operator requested a worker.
  • assigned_worker_uid is the worker that owns the Job's local workspace after dispatch.
  • execution_mode is shared for ordinary work and dedicated for a future emergency reservation.
  • reservation_id identifies the current dedicated allocation generation.
  • paused and cancel_requested_at are durable operator state.

Child tasks derive execution assignment from this row. Affinity is not encoded solely in a task queue name or copied onto every child.

queue_tasks

A row represents delivery of one JetEncode stage:

  • state and retry fields: state, available_at, attempt, max_attempts
  • routing fields: placement, pool, and optional target_worker_uid
  • ownership fields: claimed_by, claimed_worker_session_id, claimed_reservation_id, lease_token, and lease_expires_at
  • history fields: start/finish timestamps and the last failure

Task placement is one of:

  • dispatch: may establish the Job's worker assignment
  • job: requires the worker assigned to the Job
  • cluster: may run on any ordinary worker, such as a webhook

pool distinguishes media work from background work without using queue names for correctness. Workers reserve configured concurrency for media tasks and run background work in a separate single slot, so uploads, packaging, webhooks, and maintenance do not consume media capacity.

queue_workers

A stable worker UID is separate from current_session_id, which changes on every process boot. A restarted process therefore fences the previous process even when both use the same configured worker UID.

The row also stores concurrency, lifecycle state (active, draining, or offline), the latest system-information sample, and the worker's static transcoding capability catalog. Repository claims count current-session running tasks before selecting one more task. This is a distributed capacity guard; the worker runtime must still issue one synchronous claim per free execution slot and must not prefetch.

queue_reservations

This is a minimal durable seam for future Emergency/dedicated-worker execution. It stores a stable reservation ID, Job, eventual worker UID, lifecycle state, and last error. It does not provision infrastructure and has no cloud-provider dependencies.

The claim query enforces both sides of a reservation:

  • a dedicated Job is eligible only on its assigned reserved worker;
  • a reserved worker cannot claim an unrelated shared Job, including cluster webhooks.

Every claimed task records the reservation generation. Heartbeat and completion validate it against both the Job and worker, preventing an owner from an old reservation generation from publishing results.

Claim transaction

A claim uses PostgreSQL row locks; process-local mutexes are not part of distributed correctness.

  1. Lock the worker row and validate the current process session.
  2. Count running tasks owned by that session and stop at configured concurrency.
  3. Select one eligible job_schedule row with FOR UPDATE SKIP LOCKED.
  4. Select one runnable task for that execution with FOR UPDATE SKIP LOCKED.
  5. For an unassigned shared dispatch, persist assigned_worker_uid.
  6. Increment the attempt and write a fresh lease token, worker session, reservation generation, lease expiry, and task deadline.
  7. Commit and return exactly one task.

The claim transaction applies worker/reservation eligibility before the deterministic ordering described below.

Scheduling controls

Claim ordering is deterministic among Jobs eligible for the claiming worker:

  1. highest integer priority
  2. sparse manual position
  3. task available_at
  4. task creation time
  5. task ID

There are no weighted lanes. One integer priority is sufficient: higher values run first, zero is the default, and negative values place work below the default. Priority can be supplied when a Job is created or changed while it remains queued.

Run Next is exactly the same operation as Move To Top. It moves the Job to the front of its worker-local queue within its current priority; it does not create a second priority mechanism or override a higher integer priority.

All short scheduling decisions lock the singleton queue_scheduler_state row. This intentionally serializes only claim selection/control updates—not media execution—so a manual reorder and a concurrent claim cannot observe conflicting queue order.

Eligibility is evaluated before ordering. Moving a Job to the top of worker A's queue does not block worker B from claiming compatible work. Existing running work is never preempted, and repository concurrency checks prevent another claim until a slot is free.

Manual position is considered only among Jobs assigned or targeted to the same worker with the same priority; unassigned Jobs form their own pending group. Positions are sparse (normally 1024 apart). Move Up/Down chooses a midpoint, Move Top/Bottom extends the range, and the comparable group is transactionally rebalanced only when no integer gap remains. Job ID remains the final deterministic tie-breaker.

Paused Jobs are ineligible for claims. Resuming sends a PostgreSQL notification, while polling remains authoritative.

Lease ownership and crash recovery

Queue operations lock only the rows they need, always in this canonical order:

  1. worker row, when acting for a worker
  2. scheduler state row, for claims and scheduling controls
  3. Job schedule row
  4. task row

Heartbeat and finish skip the scheduler-state row but retain worker → Job → task ordering.

The operation is accepted only when all of these still match:

  • task state is running;
  • lease token;
  • worker UID and current worker session;
  • execution generation;
  • claimed reservation generation;
  • current Job reservation generation;
  • unexpired lease and execution deadline.

A stale owner cannot complete, fail, heartbeat, or publish descendants.

The recovery loop locks expired work with SKIP LOCKED. It either:

  • marks a cancelled Job's task cancelled;
  • moves a recoverable task to retry_wait; or
  • marks an exhausted task failed and publishes its terminal failure webhook.

An expired webhook does not generate another webhook, which prevents recursive queue growth.

Retry behavior

attempt starts at zero and increments transactionally when claimed. max_attempts includes the first execution.

Retry delay uses a deterministic quartic curve without random jitter:

n = completed attempt - 1
delay = min(n^4 + 15 seconds, 1 hour)

Examples:

  • attempt 1: 15 seconds
  • attempt 2: 16 seconds
  • attempt 3: 31 seconds

A retry is scheduled only while attempt < max_attempts; attempts cannot exceed the configured maximum.

Atomic stage publication

PostgreSQL finalization can transition a parent and insert all child/notification tasks in one transaction. Descendants receive the parent's Job and execution generation. Their dedupe key is a SHA-256 hash of task type, a separator byte, and payload.

This prevents a retried parent from duplicating a previously published child. The PostgreSQL worker buffers calls to jobs.Dispatcher while a handler runs and sends that buffer to this transaction.

Cancellation

Cancelling a Job:

  • records cancel_requested_at on job_schedule;
  • immediately marks pending/retry-wait tasks cancelled;
  • leaves running rows owned so the worker can observe cancellation through heartbeat;
  • causes finalization of that running task to end as cancelled without publishing children.

Completed work is never changed back into a queued or cancelled state.

Runtime storage

Run jetencode migrate before starting the API or workers. Migrations are embedded in the binary, so deployment does not need a separate migrations directory or psql migration container. Queue rows are authoritative; there is no secondary queue backend or dual-write path.

/v1/sysinfo keeps the API process's latest resource sample in memory. Workers persist their latest CPU, memory, and disk sample plus the existing static transcoding capability catalog in PostgreSQL during registration and heartbeat.

Worker runtime

Each configured worker execution slot owns one synchronous loop:

claim one task -> execute it -> heartbeat while running -> fenced finish -> claim again

There is no local reservation buffer or prefetch. On shutdown the worker cancels claim calls first, marks its session draining, lets owned handlers finish with task heartbeats still active, and then marks the session offline. A new process boot receives a new session UUID and fences the previous process.

Every job-result/progress Exec runs in a transaction that first validates the task lease, worker session, execution generation, and reservation generation. Final reporting uses a short cancellation-independent context so normal handler cancellation does not skip reporting, but lease/deadline validation still rejects stale writes.

Handler panics are converted into task failures rather than crashing unrelated worker slots. PostgreSQL maintenance retains its existing interval and policy timeout.

Workers heartbeat their registration and refresh their system-information sample. An active worker whose heartbeat is older than the stale threshold is reported offline and is not accepted for worker-targeted dispatch. Its expired task leases remain recoverable when another worker is available.

Wakeups

Task-producing transactions call pg_notify on jetencode_queue_tasks. Notifications are only an idle-worker wakeup optimization. Queue rows and one-second periodic polling remain authoritative, so missed notifications cannot lose work. LISTEN uses a dedicated PostgreSQL connection rather than consuming a claim/heartbeat pool slot.

Queue management API

JetEncode exposes a job-centric queue API under /v2/queue:

  • GET /v2/queue
  • POST /v2/queue/:id/run-next
  • PATCH /v2/queue/:id/priority
  • POST /v2/queue/:id/move with up, down, top, or bottom
  • POST /v2/queue/:id/pause
  • POST /v2/queue/:id/resume
  • POST /v2/queue/:id/cancel
  • POST /v2/queue/:id/retry

The list returns one representative task per Job plus priority, sparse position, required and assigned worker UIDs, pending wait time, pause state, attempts, and whether terminal failed tasks can be retried. waiting_seconds is present only while the representative task is actually pending. A targeted Job exposes its required worker immediately, before that worker claims it. Internal task history remains available from the existing per-Job task endpoint.

Manual retry is available only for terminal failed tasks in a non-cancelled execution. It resets those failed deliveries to pending attempt zero with the existing task policy and preserves the Job's execution generation and worker assignment. Completed sibling tasks are not replayed. Child dedupe still prevents repeated publication.

Dashboard Queue page

/dashboard/queue presents the scheduler as Jobs rather than internal task rows. Jobs are grouped into Pending/unassigned and one independent section per worker. Targeted but unclaimed Jobs appear under their required worker immediately. The page shows current stage/state, integer priority, worker assignment, and state-specific timing: pending tasks show how long they have waited, retry-wait tasks show the retry countdown/time, and running or failed tasks show no waiting value. It also exposes Run Next/Move To Top, manual moves, priority changes, pause/resume, cancel, and retry when a terminal failure exists.

The page refreshes every five seconds and after each mutation. The page uses the existing embedded vanilla JavaScript and CSS dashboard; no frontend framework or runtime dependency was added.

GET /v2/workers/:uid and /dashboard/workers/:uid expose worker status, media/background capacity and utilization, active tasks, latest system information, and the existing transcoding capability catalog. Hardware probing is intentionally deferred until scheduling needs hardware-aware routing.

Future Emergency execution

The current schema deliberately supports these future transitions without implementing provisioning:

  1. Change a pending Job to execution_mode = dedicated and attach a durable reservation ID.
  2. Leave it ineligible while no worker exists.
  3. Bind a dynamically registered worker UID to that reservation.
  4. Enforce exclusive Job-to-worker and worker-to-Job claims in SQL.
  5. Fence stale reservation generations.
  6. Move the worker to draining after the terminal Job stage.

A future provisioner subsystem can own provider IDs, idempotent create/terminate calls, and provider-specific errors. AWS, DigitalOcean, Firecracker, and other provider logic must remain outside this scheduler package.