Operations on backd

Deploying config

Mon, 01 Jan 0001 00:00:00 +0000

Everything backd serves comes from CONFIG_DIR: realms, schemas, indexes, access rules and function code. Every instance of a deployment must run exactly the same config. If two instances ran different rules or different function code, the same request could be allowed on one and refused on the other.

So the config is treated like code:

it lives in one config repository per deployment;
CI builds it at one pinned tag into an immutable artifact: an image with backd and the config inside;
staging and production run that same artifact;
backd checks it (see what backd enforces).

The config repository

backd template project --dir <directory> --realm <realm> scaffolds this layout in one step — a sample database and function with tests, both Dockerfiles, compose.yaml and the CI workflow below — so you don’t have to assemble it by hand; see Starting from a template.

Production deployment

Mon, 01 Jan 0001 00:00:00 +0000

backd deliberately leaves some protections to the infrastructure around it: TLS, request rate limits and sign-up spam, body size limits at the edge, and database security. The repository ships a reference deployment that puts all of these in place. It lives in deploy/production/, and CI tests it end to end on every change.

It uses Docker Compose, but each part maps directly onto other platforms: a Kubernetes ingress, a cloud load balancer, a managed MongoDB.

Metrics

Mon, 01 Jan 0001 00:00:00 +0000

Every backd process can tell Prometheus how it is doing: requests, errors, database latency and, as the next steps land, functions, jobs and email. It is off by default.

Turning it on

Variable	Default	Meaning
`METRICS_ADDR`	empty (off)	Where this process serves `/metrics`, for example `:9090`. A listener of its own: it can’t be `HTTP_ADDR` or `BACKD_INTERNAL_ADDR`.
`METRICS_TOKEN`	empty	At least 32 characters. When set, every request needs `Authorization: Bearer <token>`. Needs `METRICS_ADDR`.

backd serve, backd worker, backd executor and backd egress each serve their own /metrics (the executor and egress read the same two variables); a process that runs both (serve --with-worker) serves one. The listener answers GET /metrics and nothing else (a request without the token gets 401 whatever the path), serves at most four scrapes at once, and is a separate server: a slow or flooding scraper never slows the API. If the port can’t be bound, the process refuses to start, so a mistake shows at once.

Backup and restore

Mon, 01 Jan 0001 00:00:00 +0000

backd keeps all state in MongoDB. The config directory describes the structure (collections, validators, indexes, rules), and provisioning can rebuild it at any time, but not the data. Back up MongoDB, and keep the config directory in version control.

What a backup must include

Database	Holds	Rebuilt from config?
`<realm>__<database>`	your documents	no
`<realm>___system`	the realm’s users (emails, roles, disabled flag), password hashes, sessions, API keys, invitations, failed-login counters, the audit trail, encrypted function secrets, invocation history, async jobs (queued, running or recently done) and idempotency claims (see system databases)	no, except role seeds

What the configuration can and can’t rebuild:

Hardening checklist

Mon, 01 Jan 0001 00:00:00 +0000

Go through this page before any realm faces the internet, and again after changing the configuration or the deployment. The production reference deployment implements the deployment items, and its test checks many of them.

Realms

No realm has auth: disabled. Such a realm is open to anyone who can reach backd. backd logs a WARN line for each one at startup.
The sample realms aren’t deployed as they are. blog, shop and expenses in examples/config, and the realms backd template realm --sample creates, are examples: shop has auth: disabled, blog and expenses let anyone sign up, and expenses has known holes. Start from them, then review every setting.
Each realm’s signup mode is the intended one. open lets anyone create an account (and needs the sign-up rate limit below), invite needs an invitation, closed leaves it to operators.
Every collection that users or anonymous callers need has a reviewed rules.yaml, with a rules.test.yaml that backd rules test --strict runs in CI: see Testing rules. A missing rule denies; a too-broad one (read: "true") publishes a collection. See access rules.
cors.origins lists exactly your web apps’ origins, and no * for realms with users. Browser apps can keep sessions in an HttpOnly cookie (sessions.cookie), which needs this list.
Someone can administer each realm: an admin role (admin: true) held by the right people, the first created with backd bootstrap. Startup warns when nobody holds an admin role.
Administrators hold only what they use: besides one or two people with admin: true, roles for the rest list their areas (admin: [users, invitations], admin: [audit]), so the person who invites teammates doesn’t hold the secrets and the API keys.
If you run functions: the executor and backd’s internal listener (BACKD_INTERNAL_ADDR) are reachable only from each other, never published; BACKD_EXECUTOR_TOKEN, BACKD_CALLBACK_KEY, BACKD_EGRESS_KEY and, if any function declares secrets, BACKD_SECRETS_KEY are random, 32+ characters, kept as secrets (never in CONFIG_DIR) — the production reference’s setup.sh generates all four; admin: true functions and network lists are reviewed.
The executor can reach only backd egress and backd’s internal listener — nothing else, including MongoDB, is on its network. This is layer 3 of the egress design: without it, a function can reach anything its container’s network reaches over raw TCP, bypassing both the per-function allowlist and the egress proxy. The production reference wires this up (its own test proves it from a real function call); if you assemble your own deployment instead, docker/egress-test/ in the repository proves the isolation and is the reference to reproduce.
Every function callable anonymously has a rate_limit. Startup warns, once, for any that don’t — a public function with no per-caller limit can be abused as a relay (e.g. one that sends email).
Every webhook function verifies its sender’s signature itself, before trusting the body. backd only enforces that its invoke rule allows anonymous callers (it has to — a webhook sender has no session or key); it can’t tell a genuine request from a forged one.
BACKD_SECRETS_KEY (the master key) is backed up separately from database backups, never alongside them: losing it loses every stored function secret, and storing it with a backup defeats that backup’s own encryption. See Backup and restore.
functions.log_retention and functions.job_retention (function logs, async jobs) match your privacy policy: both keep a function’s own console output, and a job additionally keeps its input and output, in the realm’s system database until they expire.
Someone reviews the audit trail (backd audit --realm <realm> --since 7d, or the "msg":"audit" log lines in your log pipeline), and audit.retention matches your privacy policy.
Operators’ machines protect the CLI’s sessions (~/.config/backd/credentials): log out with backd logout on shared machines, and prefer short session lifetimes for realms with administrators.

Credentials

Every API key has an expiry (backd apikey create … --expires 90d), is used by one service, lives in a secret store, and is rotated before it expires. See the expiry and rotation policy. Startup warns about keys without an expiry.
API keys never reach browsers or mobile apps.
Internet-facing instances serve no admin API when an internal instance does: BACKD_ADMIN_API=false on the public ones (the production reference runs the split), and the CLI’s BACKD_URL points at the internal instance.
The admin UI, if used, is served only by the internal instance (BACKD_ADMIN_UI=true there, nothing on the public one), on its own origin, over TLS, with the proxy passing its Content-Security-Policy and other headers through unchanged. BACKD_ADMIN_UI_IDLE is short and the realm’s sessions.admin_idle_timeout and admin_max_lifetime are shorter still: the page’s timer is a convenience, the server’s is the limit. See its security review.
Developers who only look hold admin: read, not admin: true, and admin.read_access.users / .data stay false unless they need personal data. See The read-only level.
Admin sessions are short: sessions.admin_idle_timeout and sessions.admin_max_lifetime in each realm’s realm.yaml (for example 1h and 12h), so a stolen administrator’s session is worth little. See Sessions.
Keys are scoped to what their service does (--scopes read:main/posts,call:main/contact): a leaked key then exposes that and nothing more. See Scopes.
Services use data keys; admin keys (--role admin) exist only for management tooling.
Secrets (MongoDB passwords, TLS keys, the CA key, .env) are outside version control and readable only by the services that need them.

The edge

TLS everywhere: the edge terminates TLS with a trusted certificate that renews automatically, and plain HTTP only redirects. MongoDB connections use TLS too (below).
Rate limits: sign-up, login and the account flows (emailed links, password and address changes) have a strict per-address limit, and the rest of /v1/ a general one. Requests over a limit get 429.
Body size: the edge caps request bodies at MAX_BODY_BYTES.
Client addresses: the edge overwrites X-Forwarded-For, and TRUSTED_PROXIES names exactly the edge’s addresses, never whole private ranges. backd’s access log shows real client addresses.
Only the edge is reachable: backd isn’t published directly, and /healthz and /readyz aren’t public.
The admin API is only reachable from your operators’ network: admin.allowed_networks in each realm.yaml (and admin_networks per admin where useful), plus ADMIN_ALLOW_FROM at the edge in the reference.
The metrics port is private (METRICS_ADDR is never published or proxied, and METRICS_TOKEN is set if anything but Prometheus can reach it).
API keys of services with fixed addresses are pinned to them with --networks.

MongoDB

MongoDB requires TLS and authentication, and isn’t reachable from the internet.
backd runs with PROVISION_MODE=verify as a user limited to the configured collections; admin credentials are only available to the provisioning step. See least-privilege provisioning.
Encrypted backups of every data and <realm>___system database run on a schedule, are copied off the server, and a restore has been practised. The decryption key isn’t on the server. See Backup and restore.

Operations

LOG_LEVEL is info (the default). debug logs every access denial with its reason, which is useful while writing rules but noisy and revealing in production.
Logs are collected and kept, and startup WARN lines are reviewed: realms with auth: disabled, realms without rules, API keys without an expiry or about to expire, anonymous-callable functions with no rate_limit, and (if you run functions) an executor that isn’t configured or doesn’t answer.
The container runs hardened: read-only filesystem, no Linux capabilities, CPU and memory limits (they also bound password hashing), and a stop grace period longer than SHUTDOWN_TIMEOUT.
You know how to report and receive security fixes: see SECURITY.md. Only the latest release gets them.

Operations on backd

Deploying config

The config repository#

Production deployment

Metrics

Turning it on#

Backup and restore

What a backup must include#

Hardening checklist

Realms#

Credentials#

The edge#

MongoDB#

Operations#

The config repository

Turning it on

What a backup must include

Realms

Credentials

The edge

MongoDB

Operations