Running in production

What runs where#

Functions add up to four kinds of process. All are the same backd binary (the executor’s image adds Deno), each in its own role:

The smallest useful deployment is backd serve --with-worker plus an executor (add egress once a function declares network). The production reference runs them as separate services with the executor on a network of its own, and the project template’s compose.yaml runs the same set for local development.

Sizing#

• Executor: EXECUTOR_MAX_PROCESSES functions run at once (default 64), each using its memory plus about 130 MiB; beyond that calls answer 503 (a busy executor leaves async jobs queued for retry). Set the container’s memory limit for that many.
• Workers: WORKER_CONCURRENCY jobs per worker process (default 10), and the executor’s process limit above bounds the total. Long jobs hold a slot for their whole run: size for the jobs that run at the same time, not the ones per day.
• backd: each function’s concurrency (default 10) and the realm’s functions.max_concurrency cap sync calls per instance; per-caller rate_limit is shared across instances.

What to watch#

• Startup warnings name functions that declare a secret with no value, functions callable anonymously with no rate_limit, scheduled functions when serve runs without --with-worker, and an executor that isn’t configured or doesn’t answer.
• "msg":"function called" log lines carry each call’s function, status and duration: alert on the rate of statuses other than ok.
• Queue health: backd functions jobs --realm <realm> --status queued (or the admin API) that keeps growing, or an old queued job, means no worker is keeping up.
• Schedules: the newest run of every scheduled function is recent and ok (see Checking that a run happened).
• Retention: functions.log_retention and functions.job_retention in realm.yaml decide how long history, logs and finished jobs are kept.

Troubleshooting#

Running the executor#

Functions run in the executor, backd executor: a supervisor that starts one Deno process per call and stops it at its deadline. It runs in its own container, from the backd-executor image (the official Deno image plus backd; build it from docker/executor/Dockerfile). It never reads CONFIG_DIR or MongoDB: it fetches each function’s bundle from backd by its hash, and functions reach data only through backd.

backd talks to the executor, and the executor and functions talk back to backd on an internal listener that must never be published:

The internal listener serves only bundles (to the executor) and the data routes, for functions calling back with their short-lived credentials; sessions and API keys aren’t accepted there, and those credentials aren’t accepted on the public listener.

Run the executor as a non-root user, with a read-only root filesystem, no capabilities and a tmpfs for EXECUTOR_DIR. Size its memory limit for EXECUTOR_MAX_PROCESSES functions (each uses its memory plus about 130 MiB), and its process limit for about 5 per running function.

Running the worker#

backd worker claims and runs async jobs — same binary and image as serve, a different role. It needs BACKD_EXECUTOR_URL and BACKD_EXECUTOR_TOKEN (it calls the executor directly, the same way serve does for sync calls) and BACKD_CALLBACK_URL/BACKD_CALLBACK_KEY (functions it runs still reach data through backd’s API, via whichever instance’s internal listener BACKD_CALLBACK_URL points at — the worker doesn’t need to run one itself). It also needs BACKD_SECRETS_KEY if any async function declares secrets, and reads CONFIG_DIR/MONGO_URI like serve.

Run it as its own deployment (scale workers independently of API instances) with backd worker, or fold it into an API instance with backd serve --with-worker for a smaller deployment — both drive the same claim loop, so several workers (or several --with-worker instances) safely share one queue: each job is claimed by exactly one of them at a time.

Scheduled functions (cron) run only while at least one worker is running.