Files on back(d)

Files in functions

Mon, 01 Jan 0001 00:00:00 +0000

A function reaches the files of a document through the same client it uses for data: files(id, field) on a collection of ctx.db or ctx.admin.db. It makes the same requests as any client, to the _files routes, so the same limits, type detection, journal and usage totals apply.

const photos = ctx.db("main").collection("people").files(personId, "photo");

const doc = await photos.put(bytes, { name: "ada.png", type: "image/png" }); // the updated document
const file = await photos.get(); // { file, response, bytes(), text() }
const link = await photos.link(); // { url, expires_at }
await photos.delete(); // the field's files; delete(fileId) removes one


`put(data, { name, type, ifMatch })`	Stores a file (a string, bytes, a `Blob` or a stream): it replaces a single field’s file and is added to a `multiple` field’s. Resolves with the document, which now holds the file’s details. `type` is a hint: backd detects the real one from the content
`get(fileId?)`	Downloads a file. Without `fileId`, the one a single field holds; a field with several needs one. The bytes come through backd, even where downloads are redirects for everyone else, because a function can’t reach the bucket
`link(fileId?)`	`{ url, expires_at }`, exactly like `?link=json`: for the app to open. A link of the storage is for the bucket; a backd link needs `BACKD_URL`, so it can name backd’s public address
`delete(fileId?, { ifMatch })`	Removes one file, or every file of the field. Resolves with the document

As the caller, or as the function

ctx.db acts as the caller: put and delete are asked of the document’s update rule (with data and changed() as for an upload), get and link of the read rule, and an upload is recorded as theirs. A document the caller can’t read is a NotFoundError.

File fields

Mon, 01 Jan 0001 00:00:00 +0000

A collection says which fields of its documents carry files under files: in its collection.yaml. The realm needs a storage: section.

files:
 avatar:
 max_size: 2MiB
 types: [image/png, image/jpeg, image/webp]
 receipts:
 multiple: true
 max_files: 5
 max_size: 10MiB
 types: [application/pdf, image/*]
 manual:
 max_size: 20MiB
 download: proxy
 cache: 1h

Key	Default	Meaning
`max_size`	required	The largest one file may be (`512KiB`, `10MiB`, `1GiB`). Checked while the bytes stream in
`types`	any	Allowed content types, `image/png` or a family such as `image/`. Checked against the type detected from the file’s content*, never the one the client claims. A family never admits SVG, HTML or XML: name them to accept them
`multiple`	`false`	The field is a list of files instead of one
`max_files`	`10`	With `multiple`: how many files the field may hold (1 to 1000)
`upload`	`proxy`	`proxy` streams the bytes through backd; `direct` has the client send them straight to the bucket with a signed link, which backd verifies afterwards (Direct uploads). `max_size` is at most 5GiB for `direct`, or 4TiB with a provider that takes uploads in parts
`download`	the realm’s	`presigned` redirects to a short-lived link of the storage; `proxy` streams the file through backd
`presigned_ttl`	the realm’s	How long this field’s links last (at most 7 days)
`cache`	none	With `download: proxy`: how long shared caches may keep the file, only when anyone may read the document
`keep_metadata`	`false`	By default the metadata of JPEG and PNG pictures is removed when they are stored. `true` stores them exactly as sent
`max_pixels`	the instance’s	For images: the most pixels (width × height) the field gets versions for. It can only lower `BACKD_IMAGE_MAX_PIXELS`; a higher value stops startup
`versions`	none	Resized copies of an image, declared by name: see Image versions

A field name is lower-case letters, digits and underscores. A file field is declared only here: one that is also in schema.json is a startup error, because backd adds its schema itself, to the validators in MongoDB, to the rules and to the queries.

Uploads and downloads

Mon, 01 Jan 0001 00:00:00 +0000

Every file operation is an operation on a document and follows its access rules: the update rule for uploads and removals, the read rule for downloads. A document the caller can’t read answers 404, never revealing its files. API keys and the admin data route skip rules, as everywhere.

Operation	Request
Upload	`POST /v1/{realm}/{database}/{collection}/{id}/_files/{field}`
Download	`GET …/{id}/_files/{field}/{file id}`
Remove one	`DELETE …/{id}/_files/{field}/{file id}`
Clear the field	`DELETE …/{id}/_files/{field}`

The same four exist under /v1/{realm}/_admin/data/…. There is no PUT: a file is replaced by uploading another.

Rules for files

Mon, 01 Jan 0001 00:00:00 +0000

Files have no rules of their own: they follow the rules of their document.

Operation	Rule
Upload, remove one, clear a field	`update`
Download, `?link=json`, `?file_links=true`	`read`

File fields are known to the rules like any other field of the collection, so a rule can read them from document and data. For a file operation:

document is the document as stored; data is the document as it would be after the change; changed() lists the changed fields, which includes the file field.
The rule runs before any byte is stored: a refused upload costs nothing and leaves nothing behind.
API keys and ctx.admin.db in functions skip rules.

Reserving a field for functions

A field only a function may set, such as a generated thumbnail, is a file field the rule refuses to clients:

Files in apps

Mon, 01 Jan 0001 00:00:00 +0000

An app reaches files through the JavaScript client, which makes the requests described in Uploads and downloads. This page is about what to do with them in a page.

Uploading

uploadFile(id, field, file, { onProgress }) puts a file in a document, and prepareUpload(field, file) makes one for a document that doesn’t exist yet: name its ref in the create. Both work whether the field takes uploads through backd or straight to the bucket; the client asks the field (fileField(field)), which also tells the page max_size, types and max_files to check a file before sending it.

Security of files

Mon, 01 Jan 0001 00:00:00 +0000

The files feature was reviewed as a whole before this page: rules on file operations, storage credentials and the endpoint restriction, both upload modes, type detection, links, proxied downloads, pending uploads and their tokens, the journal and deletion queue, reconcile, erasure and the functions API. This page says what came out of it.

How files are protected

The bucket is private, and only backd reaches it. Files are served through backd, or through a link backd signed for one file. backd connects only to the realm’s declared endpoint (checked after DNS resolution, with no redirects and no proxy); public_endpoint is only written into links and is never connected to.
A file is as private as its document. An upload, a removal and a direct-upload start are update operations; a download, a link and ?file_links=true are read operations; a document the caller can’t read answers 404 for all of them, never revealing its files. The rule runs before any byte is stored, with data as the document will be, and again when the file is attached. See Rules for files.
Clients never write what describes a file. The id, name, size, detected type, SHA-256 and time are backd’s: a client’s write drops them, and a PUT keeps the stored files. The object key is <prefix>/<realm>/<database>/<collection>/<file id>: no name, and no part of a request, ever reaches it.
The type is what the bytes say. It is detected from the content, never the client’s claim, and types is checked against it. A family such as image/* or text/* does not admit active content (SVG, HTML, XML, scripts): a field takes one only when it names it. A field with no types takes anything, and serves it only as a download.
Downloads can’t run in your origin. A presigned link comes from the storage’s origin, never backd’s, and forces attachment. A proxied download carries Content-Disposition: attachment, X-Content-Type-Options: nosniff and Content-Security-Policy: sandbox. Names are cleaned (no path, no control or direction characters, 255 bytes) and quoted, so a name can’t add a header or leave its quotes.
Direct uploads are verified, not trusted. The link has the length, the type and the SHA-256 signed in, so the storage refuses another body; completing then checks the size and the checksum the storage computed (a missing one fails) and detects the type from the first bytes. A mismatch deletes the object. A file sent in parts is verified the same way, part by part and as a whole: every part has its own SHA-256 signed in, and the composite checksum the storage computes is checked against the declared parts. The sha256 recorded for it is that composite’s digest, not the SHA-256 of its bytes.
Tokens are secrets. An upload token is 256 random bits, only its SHA-256 is stored, it is compared in constant time, used once, bound to a collection and field and, when a signed-in user made it, to that user, and never logged. An upload id alone attaches or completes nothing, and every wrong, used, expired or foreign reference answers the same 400.
backd’s own links are an HMAC over the realm, database, collection, document, field, file and expiry, with a per-realm key, verified in constant time: a link can’t be moved to another file or kept past its time. Rotating the key (setting or deleting the secret BACKD_FILES_LINK_KEY) invalidates every one.
Backd’s keys are not a function’s to read. A function can’t declare the realm’s storage secrets or the link key as its own secrets: the configuration is refused at startup.
Cleanup never touches what a document holds. A worker checks that no document references an upload before it deletes it, and reconcile only considers objects named like files, older than 24 hours, with no open upload record, and that no declared field references.
Metadata is removed from pictures. The GPS position, camera and comments of a JPEG, and the text and time of a PNG, are dropped as it is stored, unless the field sets keep_metadata: true: see Metadata and the GPS risk.
Limits bound what a caller can store: max_size and max_files per field, BACKD_MAX_UPLOAD_BYTES for proxy uploads, 20 unused uploads per caller (IPv6 callers counted per /64) and a rate for starting them.

What you are responsible for

The bucket and its key. Keep it private, give the key only what your provider’s page lists, on the realm’s prefix; encryption at rest is the bucket’s setting. Give each backd instance its own prefix: two instances on one prefix collide, and a reconcile on one deletes the other’s files.
Links outlive rules. A storage link and a backd link both work for anyone who has them until they expire, even if the document’s rules change meanwhile. Keep presigned_ttl short on sensitive fields; the default is five minutes.
Caches outlive rules too. With download: proxy and cache, a shared cache may keep a file of a document anyone can read, and keeps it after you tighten the rule. Use cache only for files meant to be public.
Set BACKD_URL in production. backd-signed links name backd’s public address; without it they are built from the request’s Host header, which a caller chooses.
Anonymous uploads cost storage. Where a collection’s create rule admits anonymous callers, each address may hold 20 unused uploads of up to max_size for an hour. Keep max_size small on such fields and let the edge rate-limit by address. Nothing scans files for malware.
A removed file field leaves trusted-looking data. If you take a field out of files: and later declare it again, what the documents still hold in it is read as file details again. Don’t re-use a field name for something else; clear it first.
Functions with ctx.admin.db skip the rules, files included: review them as privileged code.
Backups. MongoDB backups don’t include files: back the bucket up at the provider and run reconcile after a restore.

Files on back(d)

Files in functions

As the caller, or as the function#

File fields

Uploads and downloads

Rules for files

Reserving a field for functions#

Files in apps

Uploading#

Security of files

How files are protected#

What you are responsible for#

As the caller, or as the function

Reserving a field for functions

Uploading

How files are protected

What you are responsible for