Async jobs

mode: async runs a function in the background instead of within the request. The call answers 202 at once with a job, and a worker runs the function later, on its own schedule and for as long as its timeout allows.

Use it when the work:

• takes more than a few seconds (a sync call is capped at 60 s, and a caller shouldn’t be made to hold a request open for anything close to that);
• must survive the caller going away (a browser tab closed, a mobile app suspended);
• can be retried safely if something goes wrong;
• fans out (a batch of emails, a report over many documents, a slow third-party call).

Use a sync function when the caller needs the answer to continue and it’s quick. Use cron when nobody asks for the work: it just has to happen on a schedule.

The cookbook’s export_orders builds a CSV report of the caller’s orders. As the caller it can read only their orders, and it writes the report as a document they own:

examples/config/workshop/main/_functions/export_orders/function.yaml

# An async function: the call answers 202 with a job, and a worker runs the
# function in the background. Use it for work that may take longer than a
# caller should wait, or that must survive the caller going away.
#
# It runs as the customer who asked (ctx.db), so it can only export their
# own orders. `timeout` is how long the job may run (up to 24h).
mode: async
timeout: 5m
invoke: "user != nil"
rate_limit: { per: user, limit: 5, window: 1h }

examples/config/workshop/main/_functions/export_orders/index.ts

// export_orders: build a CSV of the caller's orders and store it as a
// `reports` document; the job's result is the report's id.
//
// A job is delivered at least once: if a worker dies mid-run, another runs
// the job again. So the function must be safe to run twice. The request id
// of the call that queued the job is the same on every attempt, so it makes
// a stable key: the second attempt finds the report the first one wrote and
// returns it instead of writing another.
import { toCsv } from "../lib/csv.ts";
import type { Context, Doc } from "../lib/types.ts";

export default async function handler(ctx: Context) {
  const key = ctx.idempotencyKey ?? ctx.requestId;
  const db = ctx.db("main");
  const reports = db.collection("reports");

  const existing = await reports.list({ where: { export_key: key }, limit: 1 });
  if (existing.items.length > 0) {
    const report = existing.items[0];
    return { report_id: report.id, rows: report.rows as number, reused: true };
  }

  ctx.step("read the orders");
  const orders: Doc[] = [];
  for await (const order of db.collection("orders").iterate({ orderBy: "_meta.created_at" })) {
    orders.push(order);
  }

  ctx.step("write the report", { total: orders.length });
  const report = await reports.create({
    kind: "orders",
    export_key: key,
    rows: orders.length,
    csv: toCsv(orders, ["id", "item", "quantity", "amount", "status"]),
    requested_by: ctx.user!.email,
  });
  ctx.progress(orders.length, "report stored");
  console.log(`exported ${orders.length} orders for ${ctx.user!.email}`);
  return { report_id: report.id, rows: orders.length, reused: false };
}

Starting a job#

curl -X POST https://api.example.com/v1/workshop/main/_func/export_orders \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' -d '{}'
# → 202 {"id": "d3c9ljp8hc2g00b6s1m0", "function": "main/export_orders", "status": "queued",
#         "created_at": "2026-09-30T09:12:41.002Z", "result": null}
# Location: /v1/workshop/main/_jobs/d3c9ljp8hc2g00b6s1m0

A sync function can be started as a job too, by the caller, with Prefer: respond-async.

The call only enqueues the job: 400, 401, 403 and 429 (a rate limit) still apply at that point, exactly as for sync, and the function never runs inline. Keep the id (or the Location header): it is how you find out what happened.

The life of a job#

   POST _func/…          worker claims it             the function ends
 ───────────────▶ queued ───────────────▶ running ───────────────────▶ done
                                              │                          (result: ok, function_error, timeout, …)
                                              └─ worker lost, executor down or busy: the lease expires
                                                 (timeout + 30 s) and another claim retries it: `attempts` grows

Retrying a failed job#

By default a job runs once: if the function throws, times out or crashes, the job ends as done with that result. Add retry to function.yaml and backd tries again after a growing wait:

mode: async
retry:
  attempts: 5      # total attempts, the first included (default: 1, no retry; at most 20)
  backoff: 1m      # the wait after the first failure; it doubles after each one (default 1m)
  max_backoff: 1h  # the longest single wait (default 1h)

With these values the waits are 1m, 2m, 4m and 8m. retry works for every async function, including scheduled runs, and needs mode: async; the waits of one job may add up to at most 24 hours.

• What is retried: an attempt that threw, ran out of time or memory, or whose process died. What isn’t: an error the function chose with ctx.error(...) (it is an answer, not a fault) and an output over max_output (it would only be over it again). Those end the job at once.
• A retry is the same job: same id, same input. Each attempt is its own record in the invocation history, all with the job’s job_id. While a retry is waiting, the job is queued with attempts so far and next_attempt_at.
• After the last attempt the job is done with the last attempt’s result: look at result.status (it isn’t ok), as for any failed job. The log of the worker names the attempts used up.
• A lost worker is not a failed attempt. If a worker dies mid-run, the lease expires and the job is claimed again without using up one of the retry attempts.

A retry runs the function again, so it must be safe to repeat (below): the first attempt may have done part of its work before it failed.

Knowing when a job finished#

backd can queue a function when a job ends (on_complete); otherwise you ask, or the function tells you. Pick what fits:

Poll the job#

GET /v1/{realm}/{database}/_jobs/{id} answers with the job’s status and, once done, its result:

curl https://api.example.com/v1/workshop/main/_jobs/d3c9ljp8hc2g00b6s1m0 -H "Authorization: Bearer $TOKEN"
# → 200 {"id": "…", "function": "main/export_orders", "status": "done", "created_at": "…",
#         "result": {"status": "ok", "output": {"report_id": "d3c9lpr8hc2g00b6s1n0", "rows": 42, "reused": false}, "duration_ms": 118}}

A shell loop that waits for it:

until [ "$(curl -s "$API/v1/workshop/main/_jobs/$ID" -H "Authorization: Bearer $TOKEN" | jq -r .status)" = done ]; do
  sleep 2
done
curl -s "$API/v1/workshop/main/_jobs/$ID" -H "Authorization: Bearer $TOKEN" | jq .result

Poll every second or two, with a limit on how long you are willing to wait. The result has the same shape a sync call’s answer would have:

Wait in the JavaScript client#

import { JobTimeoutError } from 'backd-js'

const job = await backd.db('main').fn('export_orders', {})   // returns at once with a Job handle
await job.status()                                           // 'queued', 'running' or 'done'

try {
  const out = await job.wait({ pollIntervalMs: 1000, timeoutMs: 60_000 })
  console.log(out.report_id, out.rows)
} catch (err) {
  if (err instanceof JobTimeoutError) {
    // not finished yet: it keeps running; check again later with job.wait() or job.status()
  } else {
    throw err   // the job itself failed: the same error a direct call would have thrown
  }
}

wait() resolves with the output when the job succeeds, and throws when it ends any other way, so it reads like a sync call. See the client.

List jobs (administrators)#

To see many jobs at once, or a job you didn’t start, an administrator uses the admin API, backd functions jobs, or client.admin.jobs.list():

backd functions jobs --function workshop/main/export_orders --status done --since 24h
# CREATED                   FUNCTION            STATUS  RESULT  CODE  DURATION  TRIES  NEXT ATTEMPT  SCHEDULED  ID
# 2026-09-30T09:12:41.002Z  main/export_orders  done    ok      -     118ms     1      -             false      d3c9ljp8hc2g00b6s1m0

backd functions jobs --realm workshop --status running    # what is being worked on right now, in every function

It shows each job’s state, how many times a worker started it (TRIES above 1 means a worker was lost, the executor was unavailable, or the function is being retried; NEXT ATTEMPT says when) and how it ended, but never its input or output. It needs an admin API key or a session of a user with an admin role (the admin API).

Reporting progress from a long job#

A job that runs for minutes should say where it is. Two calls on ctx do it, and nothing has to be configured:

export default async function handler(ctx) {
  ctx.step("read the orders");                       // starts a named step
  const orders = await readAll();

  ctx.step("send the invoices", { total: orders.length, message: "starting" });
  for (const [i, order] of orders.entries()) {
    await send(order);
    ctx.progress(i + 1, `invoice ${order.id}`);       // how far the current step got
  }
  return { sent: orders.length };
}

• ctx.step(name, { total?, message? }) starts a step and closes the one running as done. total is how much there is to do, message a short note.
• ctx.progress(current, message?) sets how far the current step got. Called with no step started, it starts one called progress, so reporting never throws for that reason. A bad argument (an empty name, a negative number) is a TypeError.
• A step is n, name, status (running, done, failed, timed_out or cancelled), started_at, ended_at, duration_ms, current, total, message and updated_at. The last one in the list is the current step.
• Names are clipped to 100 characters and messages to 200. Don’t put anything in them you wouldn’t show the job’s caller: see below.

What backd does with them#

• A running job shows its steps live. The runner sends them to backd as they change, coalesced: a progress update goes out at most every 2 seconds, a new step right away (but no faster than every 200 ms, however many steps a loop starts), and the last values when the call ends. Thousands of ctx.progress() calls make a handful of writes, not thousands.
• At most 100 steps are kept per attempt: the first 50 and the latest 50, with steps_omitted counting the ones in between.
• When an attempt ends, the step still running is closed as done (the function returned), failed (any other end), timed_out or cancelled (an administrator cancelled the job). The steps are then stored in the attempt’s invocation record next to its logs.
• A killed run keeps what it reported. A timeout or a memory kill sends no result, so the job’s steps are the last ones it pushed live, closed as timed_out or failed; at most the last two seconds of progress are lost.
• A retry starts again. A new attempt starts with an empty steps list; each attempt’s own steps stay in its record.
• Sync calls, webhooks and internal sync calls keep their steps in memory and store them once, in the invocation record when they end: nothing is written while they run.
• Reporting never fails the function. A write that fails is dropped, and the steps in the final result still arrive.

Where they are shown#

• To the caller: job.steps in the JavaScript client after status() or wait(), and wait({ onProgress }) to follow a job without polling by hand; they are also in GET .../_jobs/{id} as steps and steps_omitted. The caller who started a job (or an API key) can read its steps, so treat step names and messages as part of the function’s output.
• To administrators: the Jobs page of the admin UI shows the current step of each job and, opened, every step (refreshed while the job runs); GET /_admin/jobs/{id} and backd functions jobs --realm R --job ID list them all, and backd functions jobs has a PROGRESS column; the History page and backd functions history show a call’s steps.
• In tests: @backd/functions-testing records them: createContext() returns steps().

Steps are not logs: they are a small, bounded structure (a job’s document holds one list, replaced in place), and the 64 KiB cap on console output doesn’t count them. Console output is still written once, when the call ends.

Cancelling and re-running a job#

An administrator can end a job that hasn’t finished, and queue a finished one again, with the admin API (POST /_admin/jobs/{id}/cancel and …/rerun), backd functions cancel and backd functions rerun, client.admin.jobs.cancel() and .rerun(), or the Jobs page of the admin UI. Both need the functions area, are audited (job.cancel, job.rerun) and apply to function jobs only: backd’s own emails and erasures are neither cancelled nor re-run by hand.

backd functions cancel --realm workshop --job d3c9ljp8hc2g00b6s1m0
# cancelled d3c9ljp8hc2g00b6s1m0 (main/export_orders)

backd functions rerun --realm workshop --job d3c9ljp8hc2g00b6s1m0
# queued as d3c9lq98hc2g00b6s1n0 (main/export_orders)

• Cancel ends a queued job (one waiting for a retry too) or a running one at once: it is done, with the result cancelled, and nobody claims it again. A worker that is running it looks every couple of seconds, notices, and stops the run (the executor kills the process); the attempt shows in the history as cancelled. What the function already did stays done, as when a worker is lost, so a function that changes things must be safe to repeat. A job that has already finished answers 409.
• Re-run queues a new job for a finished one (any result, a cancelled one included): the same function, the same stored input and the same caller, so it behaves as the first run did (a scheduled run’s re-run acts as the function, like the run). It has its own id, origin: admin and rerun_of naming the original, which keeps its own result. A job that hasn’t finished, or whose function no longer exists, answers 409. The input is the one stored with the job, so a re-run is only possible until the job expires.

Be told when it ends: on_complete#

Name an async function in on_complete and backd queues it whenever a job of this function ends, so nothing has to poll:

# export_orders/function.yaml
mode: async
on_complete:
  function: tell_ops     # an async function of the same database
  on: [failed, cancelled] # optional: ok, failed, cancelled; every ending when left out

The notification is a job of its own, run by a worker like any other, so it is visible in the job list (origin on_complete:<database>/<function>) and in the history, and the target’s own retry: applies when it fails. Its input is:

{
  "job_id": "d3c9ljp8hc2g00b6s1m0",
  "function": "main/export_orders",
  "status": "failed",
  "result": { "status": "timeout", "code": null, "message": null, "duration_ms": 900000 },
  "attempts": 1, "scheduled": false, "origin": "http",
  "created_at": "2026-10-05T10:00:00Z", "completed_at": "2026-10-05T10:15:00Z"
}

status is ok, failed (any result but ok: a ctx.error, a timeout, a crash, a failure of backd itself) or cancelled (an administrator cancelled it). result is the job’s result. The job’s own input and output are not sent: read the job by id if the handler needs them.

• It is one notification per finished job, however many attempts the job took: it is queued when the job ends, not on each failed attempt, and its id comes from the job, so it can’t be queued twice. A re-run is a new job and notifies again.
• The handler has no caller. ctx.user is null and ctx.db acts as an anonymous caller; give the handler admin: true if it must write, and declare network: and secrets: for a chat or webhook call (see Network). To call a URL, write that in the handler.
• Not every end is told. A run skipped by overlap: skip never ran and isn’t. A notification is not queued if backd stops in the instant between the job ending and the notification being queued: the job’s result stands, and a handler that must not miss one should also check the job list now and then.
• No chains. The target must be mode: async, a different function, with no on_complete of its own; startup stops otherwise.

Have the function say so#

• Write the result to a collection. export_orders creates a reports document; the app lists reports (or reads one by id) with its normal access rules, and a report existing means the job finished. Add a status field (working, done, failed) if the app should also show progress.

• Call a URL. A last step that calls your own service, or a chat or push service, with the job’s outcome. Declare the host under network and any key under secrets:

  network: [hooks.example.com]
  secrets: [HOOK_TOKEN]
  

  // the job's last step: tell someone. A failed notification must not lose the result, so catch it.
  try {
    await fetch("https://hooks.example.com/job-finished", {
      method: "POST",
      headers: { "content-type": "application/json", authorization: `Bearer ${ctx.secrets.HOOK_TOKEN}` },
      body: JSON.stringify({ report_id: report.id, requested_by: ctx.user!.email }),
    });
  } catch (err) {
    console.error("notification failed", String(err));
  }
  

  A job runs at least once, so the receiver may be told twice: make it ignore a repeat.

• Send an email the same way, through your mail provider’s API.

When a job doesn’t finish#

Designing jobs that are safe to repeat#

Write functions so a second run does no harm:

• Derive a key from the job and check it first. export_orders uses the request id of the call that queued the job, which is the same on every attempt: the second attempt finds the report the first wrote and returns it ("reused": true) instead of writing another.
• Use natural keys and unique indexes (a day, an order_id) so a repeated insert fails cleanly.
• Send an idempotency key to the outside service for external side effects (a charge, an email), for example Stripe’s Idempotency-Key.
• backd’s own Idempotency-Key on the enqueue request protects against a different problem: a client’s retry queuing a second job. Send the same key again and you get the first job back. It can’t help a job that is already running when a worker is lost.

Storage and retention#

• Jobs are stored in the realm’s system database: one of two deliberate exceptions to “backd never stores a function’s input or output” (the other is idempotency), because there is nowhere else to keep them until you read them.
• Only the caller who enqueued a job, any API key, or a user holding an admin role can read it back. Anonymous callers get 401; anyone else gets 404, the same as an unknown id, so a job’s existence isn’t revealed. A scheduled run has no caller, so only API keys read it.
• Retention: a finished job is kept for functions.job_retention in realm.yaml (default 24 hours, minimum 1 hour). A job that never finishes is removed after 48 hours regardless. Async functions need auth: enabled, like secrets and rate_limit.
• Limits: max_output is capped at 15 MiB for async functions (a result is one MongoDB document); timeout up to 24 hours (15 minutes by default).
• Concurrency: a function’s concurrency and functions.max_concurrency don’t apply to jobs (they protect serve’s own executor calls). WORKER_CONCURRENCY (default 10 per worker process) and the executor’s EXECUTOR_MAX_PROCESSES bound async load: a busy executor just leaves the job queued to be retried.
• Secrets and the bundle are resolved when the job actually runs, not when it was queued, so a job that waited always sees the current values.
• Every run (its result too) is recorded in the invocation history with mode: "async" and the job’s id.