Tutorial on backd

1. An empty shelf

Mon, 01 Jan 0001 00:00:00 +0000

A collection, a schema, and a page that lists documents. No users yet: anyone on localhost can read and write, which is exactly what you want while the shape of the data settles.

Warning

auth: disabled means anyone who can reach backd can read and modify its data. Keep this stack on localhost — chapter 2 turns authentication on and the checklist is what you run before anything faces the internet.

The realm

A realm is one application’s world: its users, roles and settings. A database inside it groups collections (main here), and a collection holds documents — the vocabulary the URLs use: /v1/{realm}/{database}/{collection}. They are directories under config/. Create config/shelf/realm.yaml:

2. Members

Mon, 01 Jan 0001 00:00:00 +0000

Chapter 1’s realm was open to anyone on localhost. Now members sign up and sign in — and the app learns who you are: your gallery, your assets, your account page.

The realm grows

Replace config/shelf/realm.yaml:

auth: enabled
signup: open

roles:
 admin:
 description: "Operators: users, invitations and API keys"
 admin: true
 users:
 - operator@shelf.example
 curator:
 description: Publishes assets and manages share links (chapter 5)
 users:
 - curator@shelf.example

auth: enabled — users, sessions and roles exist; anonymous callers get 401.
signup: open — anyone may create an account (alternatives: invite, closed). It is open only for this chapter and the next: chapter 4 closes it, and from then on every new member needs an invitation — which only an existing administrator can create.
roles: — admin is the operator role (admin: true grants the admin API); curator matters from chapter 5. The users: lists are seed assignments: when someone signs up with a listed address, the role attaches. A shortcut for local stacks — never keep seeded users in a real realm.

Apply, and create the three accounts

docker compose restart backd

Open http://localhost:8080 and sign up (any password of 12 or more characters; the demo stack sends no email yet and account.require_verified_email is off, so nothing needs verifying — the verify-email and welcome messages only appear once chapter 10 wires email) as each of:

3. Who may touch what

Mon, 01 Jan 0001 00:00:00 +0000

Until now every signed-in member could do anything to anything. The rules: section of a collection’s collection.yaml gives it four expressions — read, create, update, delete — evaluated per call. Operations without a rule are denied.

The assets rules

Replace config/shelf/main/assets/collection.yaml (chapter 2’s policy stays; the rules go above it):

rules:
 # Everyone in the workspace reads published assets; a member also reads
 # their own drafts.
 read: >
 user != nil
 && (document.published_at != nil || document._meta.owner == user.id)

 # Members write their own assets. HOLE: published_at is the client's word —
 # any member can publish, and pick the date.
 create: user != nil
 update: user != nil && document._meta.owner == user.id
 delete: user != nil && document._meta.owner == user.id

on_owner_delete:
 action: delete

Three names carry the context: user (the caller, or nil when anonymous — always check user != nil first), document (the stored asset), data (the asset being written). changed() lists the fields a write touches, and hasRole(user, 'curator') checks a realm.yaml role. The full language is in Access rules.

4. Invitations

Mon, 01 Jan 0001 00:00:00 +0000

A team’s library shouldn’t be open to whoever finds it. One word in realm.yaml closes the door; invitations are how people get in anyway.

Close the door

Before you change anything, check the three accounts of chapter 2 exist: operator, curator and your member. Then, in config/shelf/realm.yaml, change one line:

signup: invite # was: open

and docker compose restart backd. Existing accounts keep working — this only governs new sign-ups. From now on POST /_auth/signup without a valid invitation token answers 403:

5. The first function

Mon, 01 Jan 0001 00:00:00 +0000

Chapter 3 left two HOLE comments: members could write published_at themselves, and share links trusted the client’s asset_id and token. Rules can’t fix either — “a curator, at the server’s now” and “a token nobody chose” need code. This chapter writes it.

A function is three files

Create config/shelf/main/_functions/publish/ with three files:

function.yaml — the contract: who may call it (invoke), what it may do (admin, calls, network, secrets), how it runs.

examples/config/shelf/main/_functions/publish/function.yaml

6. Reaching the outside

Mon, 01 Jan 0001 00:00:00 +0000

Functions run in a sandbox with no network by default — fetch to anywhere is refused before a packet leaves. A function that needs the outside declares its hosts in function.yaml, and every request exits through the egress proxy that enforces the list.

preview

Pasting a link into the new-asset form is a chore; preview fetches the page’s title and description so the form can prefill itself:

examples/config/shelf/main/_functions/preview/function.yaml

# Fetches a link's title and description for the new-asset form. This is the
# first function that reaches outside: `network` is the whole allowlist —
# api.microlink.io answers, every other host is refused at the egress. It
# touches no collection, so no `admin:` either.
invoke: "user != nil"
network: [api.microlink.io]

Two things to notice:

7. Functions calling functions

Mon, 01 Jan 0001 00:00:00 +0000

publish works. Now the owner should hear about it — and the same “write a notification” will soon be wanted by the digest, the cleanup, the importer. That’s a building block, not an endpoint. internal: true makes it one.

The notifications collection

A member reads their own notifications and can flip exactly one field:

examples/config/shelf/main/notifications/collection.yaml

rules:
 # Notifications are written by functions only — there is no create rule.
 # A member reads their own and may set exactly one field, read_at.
 read: user != nil && document.to_user == user.id
 update: >
 user != nil && document.to_user == user.id
 && all(changed(), # in ['read_at'])

There is deliberately no create rule — nobody may write a notification directly. The only writer is a function.

8. Work that takes time

Mon, 01 Jan 0001 00:00:00 +0000

Fanning a notification out to every member doesn’t belong inside an HTTP request. mode: async moves it to a job: the call returns 202 with a job id immediately, the work runs on the executor, and the caller watches it finish.

The function

Create config/shelf/main/_functions/digest/. This chapter’s function.yaml:

mode: async
admin: true
invoke: "user != nil && hasRole(user, 'admin')"
calls: [notify]
retry:
 attempts: 3
 backoff: 5s
 max_backoff: 1m

(The repository’s file also has schedule: and email: true — chapters 9 and 10 add them.)

9. On a schedule

Mon, 01 Jan 0001 00:00:00 +0000

Some work shouldn’t wait for a person: the morning digest, the nightly sweep. One line in function.yaml and the function runs itself.

schedule

# digest/function.yaml — the same function as chapter 8, plus:
schedule: "@daily"

Cron expressions work too ("0 7 * * *", UTC unless you add a timezone). A scheduled run has no caller: ctx.user is null, ctx.input is null, ctx.db acts as an anonymous caller — and ctx.admin.db is always available, recorded as func:shelf/main/digest. Review a scheduled function as if it declared admin: true (Cron).

10. Email for real

Mon, 01 Jan 0001 00:00:00 +0000

Invitations emailed themselves? Not yet — chapter 4 handed you a token to pass along. This chapter gives the realm an email pipeline: a delivery function, templates for every flow backd sends, hosted pages the links open, and one custom kind of your own.

The realm side

Add to config/shelf/realm.yaml (keeping auth, signup and roles as they are):

email:
 function: mail/email-capture
 from: "Shelf <no-reply@shelf.example>"
 allowed_redirects: [http://localhost:8080]
 redirects:
 verify_email: http://localhost:8080/
 reset_password: http://localhost:8080/
 change_email: http://localhost:8080/
 invitation: http://localhost:8080/

account:
 allow_email_change: true

function: — one function (in the new mail database) delivers every email backd renders. email-capture stores them in the mail/outbox collection instead of sending — dev_only: true, it only runs under BACKD_DEV, which the tutorial’s compose.yaml sets (a real deployment doesn’t, and backd then refuses to start with it configured) (Email).
Templates — email/<kind>/<locale>.{subject.txt,txt,html}, one folder per kind backd sends (verify-email, reset-password, invitation, welcome, account-exists, change-email, email-changed, password-changed) plus custom kinds like shelf-digest. Every locale you list needs every kind — startup fails naming what’s missing.
Pages — pages/<kind>/<locale>.html: the hosted pages the email links open. GET shows a form, POST consumes the token — mail scanners can never spend your links.
allowed_redirects / redirects — after a flow’s page, the reader lands back at the app, and only at origins you list.

On a checkout, backd template realm --realm shelf writes all of email/ and pages/ for you, and backd template email-capture --realm shelf --database mail the delivery function and outbox; the tutorial’s config/ is mounted read-only into the container, so fetch the finished files instead:

11. Being called by the internet

Mon, 01 Jan 0001 00:00:00 +0000

So far everything called your functions for you — a page, a schedule, another function. A webhook reverses it: a partner feed pushes assets into the shelf, signed in to nobody. The function is the whole trust boundary.

The function

examples/config/shelf/main/_functions/import/function.yaml

# A webhook: a partner feed pushes assets into the shelf. The sender has no
# session, so invoke allows anonymous calls and the function itself is the
# only check — it verifies the signature before anything else.
#
# `admin: true` because creating assets is a member's right, not a stranger's.
# The signing secret is set with
# backd secret set --realm shelf --database main --name IMPORT_WEBHOOK_SECRET
mode: webhook
invoke: "true"
admin: true
secrets: [IMPORT_WEBHOOK_SECRET]

mode: webhook — ctx.input is gone; ctx.request holds the raw { body, headers }, and the function returns { status, body } itself.
invoke: "true" — anonymous calls must reach it; the sender has no session and never will.
secrets: — IMPORT_WEBHOOK_SECRET is shared with the sender out of band, stored per-database, never in git.

examples/config/shelf/main/_functions/import/index.ts

12. Running it

Mon, 01 Jan 0001 00:00:00 +0000

The shelf works. The last chapter is about operating it: who acts without a session, who did what, and what to review before a real deployment.

API keys

(This chapter needs the Admin view, so sign in as operator@shelf.example.)

Sessions are for people; API keys are for machines — a CI job that reads the gallery, a monitor that calls a function. In the Admin view, type a name (say ci-reader) and press Create read:main key: it makes one scoped to read-only on main, expiring in 90 days:

13. Files on assets

Mon, 01 Jan 0001 00:00:00 +0000

Until now an asset was a link or a note. Real shelves hold files: a PDF handbook, a picture of the office. backd keeps no files itself: a realm brings its own S3-compatible storage, and the tutorial’s stack gains a MinIO to be it. Every file belongs to a field of a document and is as private as that document: reading a file is reading the asset, and every upload or removal is an update checked before any byte is stored.

14. Big files and thumbnails

Mon, 01 Jan 0001 00:00:00 +0000

A 25 MiB limit and a trip through backd suit a handbook, not a conference recording. This chapter adds attachments: several files, up to a gigabyte each, that the browser sends straight to the bucket. Then a worker makes a small picture of each image, declared in one line.

Attachments, straight to the storage

attachments was declared with files: in the last chapter. What makes it different is one line, upload: direct:

15. Sharing and counting downloads

Mon, 01 Jan 0001 00:00:00 +0000

Shelf can hold files; this chapter makes them travel. Members will see how often an asset’s files were downloaded, a stranger holding a share link can download without an account, and the operator can see what it all costs.

Counting downloads

A plain download link tells backd nothing: the browser fetches the file from the storage. To count, the app asks a function for the link, and the function counts. That is the whole trick: a function is called with POST and answers JSON, so it can’t stream a file itself. It returns the link, and the app opens it.

Tutorial on backd

1. An empty shelf

The realm#

2. Members

The realm grows#

Apply, and create the three accounts#

3. Who may touch what

The assets rules#

4. Invitations

Close the door#

5. The first function

A function is three files#

6. Reaching the outside

preview#

7. Functions calling functions

The notifications collection#

8. Work that takes time

The function#

9. On a schedule

schedule#

10. Email for real

The realm side#

11. Being called by the internet

The function#

12. Running it

API keys#

13. Files on assets

14. Big files and thumbnails

Attachments, straight to the storage#

15. Sharing and counting downloads

Counting downloads#

The realm

The realm grows

Apply, and create the three accounts

The assets rules

Close the door

A function is three files

preview

The notifications collection

The function

schedule

The realm side

The function

API keys

Attachments, straight to the storage

Counting downloads