Operate

MongoDB#

backd needs MongoDB 8.2 running as a replica set. A single node is fine. Replica sets support transactions, which atomic batch writes rely on, and every setup backd is tested with uses one. Sharded clusters (through mongos) work too.

At startup, backd asks the server what it is. It refuses to start against a standalone server:

mongodb: the server is a standalone instance; backd needs a replica set (a single node is fine): start mongod with --replSet and run rs.initiate(), then add ?replicaSet=<name> to MONGO_URI

Managed services such as MongoDB Atlas always run replica sets. To run a single node yourself:

docker run -d --name mongo -p 127.0.0.1:27017:27017 docker.io/library/mongo:8.2 --replSet rs0
docker exec mongo mongosh --quiet --eval \
  "rs.initiate({_id: 'rs0', members: [{_id: 0, host: 'localhost:27017'}]})"
export MONGO_URI='mongodb://localhost:27017/?replicaSet=rs0'

• The member’s host is the address clients use to reach it: localhost:27017 for a server on your machine, the service name (such as mongo:27017) inside a compose network.
• replicaSet=rs0 in MONGO_URI names the set. Without it, the driver still discovers the set from the host you give.
• With authentication, a replica set also needs a key file (--keyFile), even with one node. The production reference shows how.
• make dev, make example and the test setups start a single-node replica set and initiate it from the container’s health check.

Upgrading an existing standalone server#

Restart it as a replica set and initiate it once; the data stays in place:

1. Stop backd.
2. Restart mongod with --replSet rs0 (and --keyFile if authentication is on).
3. Run rs.initiate({_id: 'rs0', members: [{_id: 0, host: '<host>:27017'}]}) once, as the admin user if authentication is on.
4. Add replicaSet=rs0 to MONGO_URI and start backd.

Provisioning#

At startup, backd makes sure MongoDB matches the config. What it does depends on PROVISION_MODE.

apply (default)

• Creates each missing collection with its $jsonSchema validator. MongoDB creates the database implicitly.
• Updates (collMod) validators that differ from schema.json.
• Creates indexes declared in indexes.json that don’t exist yet.
• Creates or updates the system database of every realm with auth: enabled.
• Enables pre-change images (changeStreamPreAndPostImages) on collections that declare realtime: true. They are never switched off by a start: see Disabling them.
• Records the config’s fingerprint for each realm (see Deploying config).
• Is idempotent and safe to run on every start.
• If a declared index exists with a different unique setting, apply stops with an error rather than dropping it. Drop the index manually and restart.
• Creating a unique index fails if the collection already holds duplicate values. The error names the index.

verify

• Only compares collections, validators, indexes and pre-change images with the config, including the system databases.

• If anything differs, it refuses to start and lists every difference. For example:

  backd serve: MongoDB does not match the config (run `backd provision`):
    shop__orders.items: collection missing
    shop__orders.clients: validator differs from schema.json
    shop__orders.clients: index (email) missing
    shop__orders.live: pre-change images are not enabled (this collection is realtime)
  

• Then compares the config’s fingerprint with the one provisioning recorded for each realm, and refuses to start if it differs or none was recorded: every instance must run the config that was provisioned (see Deploying config).

• Needs only read/write privileges on the data, and read access to backd___deployment.realms.

Common to both modes

• Validators always use validationLevel: "moderate" and validationAction: "error".
• Nothing is ever dropped. Some databases, collections and indexes exist in MongoDB but not in the config: databases named like <realm>__<database>, collections inside configured databases, or indexes not in indexes.json. backd doesn’t serve those databases and collections, keeps those indexes, and logs a warning for each. The same goes for a <realm>___system database whose realm is gone or has auth: disabled: it is kept and reported, never used.
• Schema keywords that MongoDB can’t enforce are logged at info level (see Schema validation).

backd provision runs apply once and exits.

Both backd provision and backd serve then apply the role assignments seeded in each realm’s realm.yaml.

Realm system databases#

Each realm with auth: enabled has a system database named <realm>___system (three underscores). It holds the realm’s users and credentials, managed through the admin API (for example with backd user); only backd bootstrap writes to it directly. Realm names must be short enough for this name to stay under MongoDB’s 64-character limit, so a realm name can be at most 54 characters.

Each collection has a $jsonSchema validator that checks its required fields and their types. The TTL indexes make MongoDB delete expired sessions, invitations, counters, audit records, invocation records, jobs and idempotency claims automatically. If one of these indexes exists with different options (for example expires_at without TTL, so sessions would never expire), apply and verify fail and name it; drop the index and let backd recreate it.

Treat the system databases as sensitive: they contain password hashes and token hashes. Back them up with the rest of the realm’s data, encrypted: only role seeds can be rebuilt from the config. Tokens and keys are stored only as hashes, so a leaked backup doesn’t reveal usable credentials, but password hashes can still be attacked offline. See Backup and restore.

Least-privilege deployments#

The service’s MongoDB user doesn’t need to create collections, change validators or build indexes. Run provisioning as a separate deploy step instead:

1. Run backd provision with an admin-privileged MONGO_URI.
2. Grant the service’s user find, insert, update, remove and listIndexes on each collection that backd databases --collections prints, and listCollections on their databases. MongoDB privileges can’t match names by pattern, so repeat this step whenever the config changes.
3. Start the service with PROVISION_MODE=verify and that user.

Grant collections rather than MongoDB’s built-in readWrite role on whole databases: readWrite also allows creating, dropping and re-indexing collections, and MongoDB lets insert on a database create any collection in it. The production reference automates all three steps.

backd databases needs only CONFIG_DIR. It prints every data database (<realm>__<database>) and the system database (<realm>___system) of every realm with auth: enabled. With --collections, it prints every collection as <database>.<collection>, including the system collections:

$ CONFIG_DIR=./config backd databases
blog___system
blog__main
shop__orders
$ CONFIG_DIR=./config backd databases --collections
blog___system.api_keys
…
blog__main.posts
shop__orders.items

Upgrading to v0.9.0#

v0.9.0 adds the Dart client and the generators around it. The server changes only add: no new collections, no new configuration, nothing to provision. What to check:

1. Nothing breaks, and nothing needs provisioning. If you run PROVISION_MODE=verify, no backd provision run is needed for this release alone.
2. backd types --lang dart writes a realm’s types.dart (a ShelfRealm accessor and document, input, where/orderBy, function and configuration classes) next to the TypeScript output, and the admin types endpoint takes ?lang=dart. backd openapi keeps feeding openapi-generator for Kotlin, Swift and the rest; make openapi-clients builds all three from the example realms’ documents.
3. The Dart client is package:backd on pub.dev (dart pub add backd), versioned on its own: sign-in with the second-factor, magic-link, provider and consent steps, collections, files with direct uploads and image versions, function jobs, live lists over realtime, push devices, app configuration, organizations and generated realm types (BackdClient<ShelfRealm>). It has no admin API, by design: administration stays in the JavaScript client, the admin UI and the CLI.

Upgrading to v0.8.5#

v0.8.5 adds realtime subscriptions and triggers, both built on MongoDB’s change stream. Nothing is on until a collection declares realtime: or triggers:, and nothing breaks for config that doesn’t. What to check:

1. Run backd provision when you declare realtime: or triggers:, before you start instances with PROVISION_MODE=verify. Provision enables pre-change images on those collections (without them a removal can’t be told to the right subscribers, and verify refuses to start, naming the collection) and creates the new jobs indexes the trigger dispatcher uses (the per-realm backlog count, and serial_key/serial_order for ordered triggers). Dropping realtime: from a collection leaves its images in place; backd provision --disable-preimages removes them.
2. Realtime needs MongoDB 6.0 or later as a replica set, which the change feed requires at startup. The stream is opened on the whole deployment, so the MONGO_URI user needs the changeStream and find actions on every database — for example the built-in readAnyDatabase role; a confined user that can’t get it can’t serve realtime. See the change feed requirements.
3. Split the work per instance. BACKD_EVENTS=false keeps realtime streams off request-serving replicas; BACKD_TRIGGERS=false keeps an instance from holding the trigger dispatcher’s lease. Both default to true, and nothing is read until a collection declares the feature.
4. A trigger’s writes are stamped _meta.chain ({root, depth}) so chains can be bounded (triggers.max_chain, max_chained_per_minute, max_backlog in realm.yaml). A client can’t set it and a function can’t drop it. backd triggers status|gaps|resolve and backd triggers rescan inspect and repair the dispatcher.
5. The JavaScript client is 0.8.5 (npm install backd-js@latest): client.realtime.subscribe() and collection.subscribe() live lists, the generated trigger input types and four typed errors (RealtimeNotEnabledError, TooManySubscriptionsError, InvalidResumeTokenError, SessionEndedError). It keeps working with a v0.8.4 server for what existed there; live lists need a v0.8.5 server.

Upgrading to v0.8.4#

v0.8.4 adds reference expansion, aggregations and full-text search. Nothing is on until a collection declares it, and nothing breaks for config that doesn’t. What to check:

1. Run backd provision when you declare a text index or the indexes an aggregation’s warning asks for, before you start instances with PROVISION_MODE=verify. A text index is created (and rebuilt when its declaration changes, with searches answering 503 index_building meanwhile); a collection with a language_field also gets a server-owned _search_lang in its validator and, once, in the documents that already exist.
2. Two names are now reserved in schema.json: _expanded and _search_lang. A schema that declares a property with either name stops backd at startup, with the file named: rename the property.
3. New, opt-in features: references: in collection.yaml and ?expand= on reads (BACKD_EXPAND_MAX_DEPTH, _PATHS, _ARRAY, _DOCS), aggregate: and GET …/_aggregate (BACKD_AGGREGATE_TIME, BACKD_AGGREGATE_MAX_GROUPS), a text index and $search. backd logs a warning at startup for each group_by field that no index starts with.
4. _batch still only writes, so there is nothing to expand there; ctx.db and ctx.admin.db in functions do expand, aggregate and search.
5. The JavaScript client is 0.8.4 (npm install backd-js@latest): expand(), aggregate(), search(), fillBuckets() and typed _expanded. It keeps working with a v0.8.3 server for what existed there; the new calls need a v0.8.4 server. Types generated by an older backd types keep working; regenerate to get the new ones.

Upgrading to v0.8.3#

v0.8.3 adds organizations and team access. It has no breaking changes: everything is off until realm.yaml declares organizations. What to check:

1. Run backd provision before you start instances with PROVISION_MODE=verify. The release adds the system collections orgs, org_members, org_teams, org_invitations, org_member_seq and org_templates, and indexes on collections that declare org or team. apply creates them at startup; verify refuses to start until a provisioning run did, and the least-privilege user of the production reference needs them granted.
2. Organizations are opt-in. Declare organizations in realm.yaml (roles, team_roles, hooks, require_mfa) and organization: required|optional and team: required|optional in collection.yaml. _meta.org and _meta.team are server-owned. See organizations and the rules cookbook.
3. Rules, functions and the client gain surface: inOrg, hasOrgRole, inTeam and hasTeamRole in rules, ctx.user.orgs and ctx.admin.orgs in functions, and organizations in the JavaScript client and its admin API. Existing rules keep working.
4. The JavaScript client is 0.8.3 (npm install backd-js@latest). It keeps working with a v0.8.2 server for what existed there; the organizations methods need a v0.8.3 server.

Upgrading to v0.8.2#

v0.8.2 is a large minor release, and it has breaking changes, as allowed before a stable release: the first five points need your action, the rest are worth a look. Features are off until your config uses them. What to check:

1. Run backd provision before you start instances with PROVISION_MODE=verify. The release adds system collections (oauth_states, login_codes, auth_challenges, mfa, devices, consent_records, app_config, app_config_history, push_sends) and one index on every data collection, _meta.created_at,id, which serves the new default order of lists. apply creates them at startup; verify refuses to start until a provisioning run did, and the least-privilege user of the production reference needs the new collections granted. The index is built once per collection: on a very large collection do it at a quiet time.
2. Lists without order_by are ordered by _meta.created_at, then id (before: by id). For generated ids that is almost the same; an after cursor is now an opaque value, and the bare id works only with order_by=id. Callers that built after=<id> by hand must add order_by=id. See Querying.
3. A create that sends id to a collection that generates ids is 400 id_not_allowed (before, the field was ignored). Collections that declare custom_id require it.
4. Sensitive account actions ask for a fresh proof of identity, and their password fields are gone. Changing the password or the address, deactivating the account, unlinking a sign-in method, turning the second factor on or off, and creating an API key as an administrator signed in with a session answer 401 reauth_required without an X-Backd-Reauth token from POST /_auth/reauth. The JavaScript client and the admin UI do it for you; direct callers of those endpoints must change. See Fresh proof.
5. email.default_locale and email.locales moved to the top level of realm.yaml (default_locale, locales), shared by email and push. The old keys are refused at startup, with a message that says where they went. The sign-in return parameters of provider flows are now prefixed backd_ (backd_code, backd_error, backd_linked): a page that handles the return by hand must read the new names (backd-js does it for you).
6. New, opt-in features: sign-in with Google, Microsoft or Apple, a second factor (mfa: in realm.yaml; the template’s admin role requires one, so administrators enrol on their next sign-in), magic links and accounts without a password, push notifications, consents and a personal data scope, app configuration, conditional GET and public caching, custom ids, generated TypeScript types and a per-realm OpenAPI document. Several of them seal values with the key of BACKD_SECRETS_KEY: set it before you turn them on.
7. BACKD_DEV=true reloads the whole config live (schemas, rules, indexes, functions and templates), and rewrites backd.d.ts in each functions folder. Production behavior is unchanged.
8. The JavaScript client is 0.8.2, on the same version as the server from now on (npm install backd-js@latest). It adds the second step and magic links, provider sign-in, push, consents, client.config, the cache, typed clients (createClient<Realm>()) and the custom-id errors. It keeps working with a v0.8.1 server for what existed there.

Upgrading to v0.8.1#

v0.8.1 adds image versions, storage quotas and direct uploads in parts, and it has breaking changes: the first two points need your action, the rest are worth a look. Nothing else that worked on v0.8.0 changes. What to check:

1. Move rules.yaml into collection.yaml. The rules are now its rules: section (same keys). A collection directory that still has a rules.yaml does not load, so a forgotten file can’t leave a collection without rules: indent the file’s content two spaces under rules: in collection.yaml and delete it. rules.test.yaml stays where it is. See rules.
2. backd without a command no longer serves. It prints its help and exits 0. The images already run serve; a Dockerfile, compose file or unit that started backd alone must say backd serve.
3. JPEG and PNG originals lose their metadata by default (EXIF with the GPS position, XMP, comments, PNG text), pixels untouched and orientation kept. Set keep_metadata: true on a field to keep it. A picture too damaged to be copied is refused with 422 invalid_image. Files stored before the upgrade are not changed. See Metadata and the GPS risk.
4. Image versions and quotas are opt-in: versions: under a file field (workers make the copies, so run them; ?version= serves them and backd files regenerate remakes them after a declaration changes) and storage.quota in realm.yaml.
5. Direct uploads above 5 GiB need a provider that checks a SHA-256 per part: AWS S3, Cloudflare R2 and MinIO. With DigitalOcean Spaces max_size of a direct field stays at 5 GiB. A file sent in parts has a sha256 that is the digest of its parts’ digests, not of its bytes (why). Workers abort the uploads left unfinished.
6. Sizes accept TB and TiB. GET …/_files/{field} adds multipart_above for the fields that can hold more; the start of a direct upload takes part_sha256 instead of sha256 for such a file.
7. API additions: POST /_admin/config/check (a change checked against the realm’s whole configuration, without applying it) and the image version routes. The admin UI’s schema and rules editors use the first.
8. The JavaScript client is 0.8.0 (npm install backd-js@latest): uploadFile sends large files in parts by itself, plus the version helpers and admin.checkConfig(). It works with a v0.8.0 server for everything that existed before.

Upgrading to v0.8.0#

v0.8.0 adds files, schema checks, on_complete and job steps, and more control over schedules (pausing, timezone, overlap: skip). Configuration that worked on v0.7.0 still loads, except for the one case in point 2; each new feature is off until you use it. What to check:

1. Provision before rolling out, as always. New system collections are created in each realm’s ___system database: file_journal, file_deletions, storage_usage, schema_checks and schedules. PROVISION_MODE=verify refuses to start until backd provision has made them.
2. A file field can’t be declared in schema.json. Files are declared under files: in collection.yaml (backd adds their schema), and startup refuses a collection that has the same property in both, and one with files: in a realm without storage:. If you used a property to hold file details, move it. The tutorial’s assets.file placeholder moved this way.
3. Files are opt-in, and need more than configuration. A realm declares storage: in realm.yaml and sets its two keys as realm secrets (backd secret set, with BACKD_SECRETS_KEY on the instance); without them the realm’s file endpoints answer 503 storage_unavailable and reads of its documents come without links. Run workers (backd worker, or --with-worker): they delete the objects of replaced, removed and abandoned files and finish uploads a crash interrupted. Set BACKD_URL in production, since backd-signed links name it. BACKD_MAX_UPLOAD_BYTES (default 100 MiB) caps proxy uploads.
4. Give each backd instance its own storage prefix, and read Security of files: links and caches outlive later rule changes, and a function can’t declare the storage secrets or BACKD_FILES_LINK_KEY as its own (startup refuses it).
5. Erasing a user now also queues the files of the documents the policy deletes or anonymizes (a remove of a file field). A DELETE, a batch delete and a purge queue the document’s files too. MongoDB’s own retention of a soft-deleted document can’t: backd storage reconcile finds those objects.
6. The admin UI gains the file controls and schema checks. Its Content-Security-Policy now also allows images and direct uploads from the origin of the realms’ file storage (and from no other origin besides its own); a stack without storage has the policy it had.
7. API and CLI additions: the _files routes and ?file_links=true on reads; GET /_admin/storage, POST /_admin/storage/check and /reconcile; /_admin/data-checks; /_admin/schedules; backd storage check|usage|reconcile, backd data check|checks, backd functions schedules|pause|resume. New error codes: too_many_files, upload_mode_mismatch, unsupported_file_type, file_missing, invalid_file_link, upload_mismatch, file_not_uploaded and storage_unavailable. The owned report lists replace as [], no longer null, for a policy that replaces nothing.
8. The tutorial’s stack now runs a MinIO: fetch compose.yaml and nginx.conf again before chapter 13.
9. The JavaScript client is 0.7.0 (npm install backd-js@latest): files(id, field), uploadFile, prepareUpload, fileUrl, downloadFile, deleteFile, fileLinks, admin.storage.status() and reconcile(), admin.dataChecks, admin.schedules, job.steps and wait({ onProgress }). It works with a v0.7.0 server for everything that existed before.

Upgrading to v0.7.0#

v0.7.0 adds the admin web interface, a read-only admin level with admin.read_access, an audited admin data route, a read-only configuration view in the admin API, and a switch that turns the admin API off per instance. Configuration that worked on v0.6.0 still loads; the interface is off until you switch it on. What to check:

1. admin: true and admin API keys now reach documents. The new admin data route (/v1/{realm}/_admin/data/…) lists, reads and changes documents past the collection’s access rules, and it belongs to every admin: true role and every admin API key. Writes are audited without their content; reads are not (they carry the actor in the access log). Before upgrading, look at who holds those roles and keys: give people who don’t need documents an area list (admin: [users, invitations]), which doesn’t include the new data area, and keep admin keys to the tooling that needs them.
2. Two new areas. An area list can now also name data and config; a role that lists areas and doesn’t name them doesn’t get them. config shows what the instance runs for the realm (GET /_admin/config) and every read-only level has it.
3. The read-only level is new. admin: read (or [read, secrets]) reads every area except users and data, which a realm opts in to with admin.read_access.users and admin.read_access.data (both false by default). GET /_admin/whoami tells a client what the signed-in level may do.
4. The admin API can be switched off per instance: BACKD_ADMIN_API=false leaves every /_admin route out (404). The production reference now splits a public backd (no admin API) from an internal backd-admin, and the CLI’s BACKD_URL points at the internal one: if you built on deploy/production, compare your compose file with it.
5. The admin web interface is off by default. BACKD_ADMIN_UI=true serves it at /_ui/ from an instance that also has the admin API (the combination with BACKD_ADMIN_API=false stops startup); BACKD_ADMIN_UI_IDLE (default 30m) signs an idle administrator out. Release images and binaries embed it; a binary built from source needs make ui (Node) first, and without it /_ui/ answers 404 and startup warns. Serve it only from the internal instance, and read its security review and the hardening checklist first.
6. Smaller changes: GET /_admin/users takes q (a search of at most 254 characters), GET /_admin/users/{id}/sessions and DELETE …/sessions/{session_id} list and end a user’s sessions. The example stack serves /_ui/ and has an adminui realm for it.
7. The JavaScript client is 0.6.0 (npm install backd-js@latest): admin.whoami(), admin.config(), admin.data(database, collection), admin.users.sessions() and revokeSession(), the q search on admin.users.list() and the origin filter on admin.jobs.list(). It works with a v0.6.0 server for everything that existed before.

Upgrading to v0.6.0#

v0.6.0 adds soft delete, admin rights per role, Prefer: respond-async and login_throttle. Configuration that worked on v0.5.0 still loads and behaves the same; each new feature is off until you use it. What to check:

1. Provision before rolling out, as always. A collection with soft_delete gets its unique indexes rebuilt with _meta.deleted_at as the last key and, with a retention, a TTL index on _meta.purge_at; PROVISION_MODE=verify refuses to start until backd provision has created them. Collections that don’t opt in are untouched.
2. Turning soft delete on for a collection that already has a unique index: the old index stays (backd never drops one) and also covers deleted documents, so a deleted document’s values stay taken until you drop it. backd provision warns about it by name; drop it once the new index exists.
3. Admin roles: admin: true is what it was. A role can now list areas (users, invitations, apikeys, secrets, audit, functions) and an administrator can no longer hand out, or change a user who holds, rights they don’t have themselves; admin API keys are created and revoked only by full administrators. Keep one role with admin: true: backd bootstrap needs it, and startup warns when a realm has none.
4. restore and purge are new rule keys for collections that soft-delete. They are denied unless declared and write doesn’t cover them; declaring one in a collection without soft_delete stops startup. backd rules test fixtures take restore and purge assertions and _meta.deleted_at.
5. Two query parameters are now meaningful: deleted on list and get, and purge on DELETE. On a collection that doesn’t soft-delete they still answer 400, with a message that says so.
6. Prefer: respond-async is per call. Nothing changes for callers who don’t send it. Browser apps on another origin can now send Idempotency-Key and Prefer and read Preference-Applied and Idempotent-Replayed: the CORS allow and expose lists lacked them.
7. login_throttle is optional; the defaults are the values the throttle always had.
8. The JavaScript client is 0.5.0 (npm install backd-js@latest): deleted, purge and restore on collections, respondAsync on fn(). It works with a v0.5.0 server for everything that existed before.

Upgrading to v0.5.0#

v0.5.0 adds cursor pagination, API key scopes, session cookies, Idempotency-Key on creates, dates stored as dates and backd rules test. Configuration that worked on v0.4.0 still loads; every new feature is off until you use it. What to check:

1. Provision before rolling out, as always: the api_keys collection of each realm’s system database gains the scopes field in its validator, and each collection with a field marked "x-backd-store": "date" gets bsonType: date for it. PROVISION_MODE=verify refuses to start until backd provision has run.
2. API key scopes are opt-in. Existing keys have no scopes and keep full access. A key created with scopes can only do what they grant (read, write or call, on everything, a database or a collection or function), and answers 403 otherwise.
3. x-backd-store: date does not convert existing data. Documents written before keep text in that field until they are written again, and date queries skip them until then; the dates section has a mongosh one-liner to convert in one go.
4. Session cookies are opt-in per realm and per login (sessions.cookie.enabled, then "cookie": true on the login). A cookie login turns on an origin check for writes that carry only the cookie, so list your app’s origin in cors.origins.
5. Idempotency-Key on creates needs authentication. A create or batch with the header answers 400 for anonymous callers and in realms with auth: disabled; without the header nothing changes. Answers are kept for 24 hours in the realm’s existing idempotency collection.
6. The admin user list is cut by MongoDB now (GET /_admin/users returns next_cursor, with after to continue; skip still works). The command line and the JavaScript client already follow it; a script that read the whole list in one call must follow next_cursor for realms with more users than one page.
7. Two limits are stricter, one is new. The login throttle now serializes the password checks of an account across instances, and answers 429 with Retry-After: 1 if it waits more than 5 seconds for the lock. sessions.admin_idle_timeout and sessions.admin_max_lifetime (unset: nothing changes) shorten the sessions of users who hold an admin role. See the hardening checklist.
8. The JavaScript client is 0.4.0 (npm install backd-js@latest): after and iterate follow cursors, cookies: true, scopes when creating keys, and idempotencyKey on create and batch (keyed requests are retried by the client). It is versioned apart from the server, and works with a v0.4.0 server for everything that existed before.
9. If you copied the template workflow (backd template project → .github/workflows/backd.yml), add a step backd rules test (with CONFIG_DIR: ./config) after backd config check, as the template now does: it checks your access rules against the cases in each rules.test.yaml, with no database.

Upgrading to v0.4.0#

v0.4.0 adds email flows, erasure, metrics and the npm client. Configuration that worked on v0.3.0 still loads and nothing new is required, but two behaviors change, so read the first two items.

1. Deleting a user means something else. DELETE /_auth/me (a user deleting their own account) now deactivates it: the account is disabled and its sessions end, but the data and the email address are kept, and an administrator can reactivate it. DELETE /_admin/users/{id} and backd user delete now erase: the user becomes a tombstone (erased-<id>@erased.invalid, so the real address is free again) and a worker applies the collection.yaml policy of each collection that declares one; they answer 202 with an erase job, and backd user delete waits for it. Before, both removed the user and left every document alone. To get that effect now, declare no policy: an erase without policies leaves the documents as they are. A worker must be running (backd worker, or serve --with-worker); see Deleting and erasing users.
2. Provision before rolling out, as always: each realm’s system database gains collections and indexes (email tokens, an index of erase jobs), and PROVISION_MODE=verify refuses to start without them. The worker now claims jobs in every realm with authentication, not only those with functions.
3. Some addresses are refused. Addresses on the reserved .invalid domain, with a comma, semicolon, colon, angle bracket, parenthesis, square bracket, quote or backslash, or longer than 254 characters no longer pass, at sign-up and when looked up. An existing account with such an address can’t sign in by email until an administrator changes the address in the database. Ordinary addresses are not affected.
4. If you copied the production reference: the stricter rate-limit zone now covers password reset, verification, invitations and address changes (nginx/backd.conf.template), and it gained a Prometheus. Run its setup.sh again: it adds BACKD_METRICS_TOKEN to your .env and secrets/metrics-token.
5. The JavaScript client is on npm as backd-js (npm install backd-js). It was never published as @backd/client: if you installed it from a checkout, change the import. Versions of backd-js before 0.2 were an unrelated, older library.
6. Metrics, email and the account settings are all off by default. Turn them on when you want them: METRICS_ADDR, the email section of realm.yaml and account.*.

Upgrading to v0.2.0#

v0.2.0 adds users and access rules and is a breaking release: every realm needs a realm.yaml, and authentication is on by default. To keep an existing realm working as before on a private network, give it a realm.yaml with auth: disabled; to protect it, provision it and give server-side clients API keys.

1. Add a realm.yaml to every realm. To keep a realm working exactly as before, without authentication, put auth: disabled in it. Only do this for realms that stay on a private network: anyone who can reach them can read and modify their data.
2. Or turn authentication on (the default, auth: enabled), declare an admin role in realm.yaml (roles: {admin: {admin: true}}), then:
   • run backd provision (or start backd serve with PROVISION_MODE=apply) to create the realm’s system database;
   • create the realm’s first administrator with backd bootstrap --realm <realm> --email <email>, and log in as them with backd login --realm <realm> --url <server>;
   • give server-side clients an API key: backd apikey create --realm <realm> --name <name> --expires 90d, sent as Authorization: Bearer <key>;
   • add a rules: section to the collection.yaml of collections that users or anonymous callers should reach. Without one, only API keys can.
3. Existing documents don’t have ownership fields. They keep working: _meta.owner reads as null, so owner-based rules don’t match them, and their next write adds _meta.updated_by. API keys can still manage them.
4. Behind a reverse proxy, set TRUSTED_PROXIES so login throttling sees real client addresses.
5. Browser apps on another origin need that origin under cors.origins in realm.yaml.

Health checks#

Use GET /healthz as the liveness probe and GET /readyz as the readiness probe (see HTTP API).

Logging#

Logs are JSON lines on standard error, filtered by LOG_LEVEL.

• At startup, backd logs a WARN line for each realm with auth: disabled (its data is open to anyone who can reach the service), and for each realm with auth: enabled but no rules in any collection (only API keys can reach its data). It warns about realms with auth: enabled that nobody can administer with a session (no admin role, or no user holding one). It also lists API keys that never expire or expire within 14 days (see the expiry policy).
• Each request produces one access-log line ("msg":"request"). It includes the method, route pattern, realm, database, collection, status, duration, bytes, request_id and the client’s address (client, following TRUSTED_PROXIES), plus actor in realms with auth: enabled: user:<id> for a session, key:<name> for an API key, key:<name> as user:<id> for an API key acting on behalf of a user, or anonymous.
• At debug level, requests refused by access rules are logged ("msg":"access denied") with the collection, operation and reason.
• Request bodies, document contents, passwords and tokens are never logged.

Timeouts#

Shutdown#

On SIGINT or SIGTERM, backd:

1. closes its listener at once, so new connections and readiness probes are refused;
2. lets in-flight requests finish, for up to SHUTDOWN_TIMEOUT (default 15s);
3. closes the MongoDB client and exits with status 0.

Requests still running after the timeout are aborted, and backd exits with a non-zero status.

Give your process manager a stop grace period longer than SHUTDOWN_TIMEOUT:

• the included docker-compose.yml uses stop_grace_period: 20s;
• Docker’s default is 10s;
• Kubernetes’ default terminationGracePeriodSeconds is 30s.

backd serves plain HTTP. Terminate TLS in front of it with a reverse proxy, ingress or load balancer. The production reference deployment does this and everything else below. Before going public, go through the hardening checklist. Before exposing a realm, also go through the security checklist.

Rate limiting and abuse#

backd doesn’t limit request rates, except for failed logins. Put a reverse proxy, API gateway or CDN in front of it and limit there (the production reference has a tested nginx configuration):

• Sign-up: in realms with signup: open, limit POST /v1/{realm}/_auth/signup per client address, or use signup: invite.
• Public data: collections whose read rule lets anonymous callers in can be scraped; limit or cache them.
• Everything else: a general per-address limit protects MongoDB from floods.

Password hashing is bounded by PASSWORD_HASH_CONCURRENCY whatever the traffic: excess logins queue, then answer 503 at their deadline, instead of exhausting memory.

Client addresses behind a proxy#

Login throttling counts failures per client address. Behind a proxy, every connection comes from the proxy, so set TRUSTED_PROXIES to the proxies’ exact addresses:

TRUSTED_PROXIES=172.30.80.10

List networks only if every address in them is one of your proxies. Whole private ranges such as 10.0.0.0/8 would let any other container or host in them forge X-Forwarded-For (see client addresses in the production reference).

• X-Forwarded-For is only believed when the connection comes from a trusted proxy. Otherwise clients could pick any address they like.
• The client is the right-most address in X-Forwarded-For that isn’t a trusted proxy. Entries further left were supplied by the client and are ignored.
• The proxy in front of backd must overwrite X-Forwarded-For with the address it sees, unless it trusts a load balancer in front of it. The production reference shows how. The local stacks (make dev, make example) trust all private ranges for convenience: don’t copy that.
• Without TRUSTED_PROXIES, backd uses the connection’s address. Behind a proxy, all clients then share one counter, and a flood of failures from anyone slows down logins for everyone behind that proxy.