14. Big files and thumbnails

A 25 MiB limit and a trip through backd suit a handbook, not a conference recording. This chapter adds attachments: several files, up to a gigabyte each, that the browser sends straight to the bucket. Then a worker makes a small picture of each image, declared in one line.

Attachments, straight to the storage#

attachments was declared with files: in the last chapter. What makes it different is one line, upload: direct:

examples/config/shelf/main/assets/collection.yaml

  # Chapter 14: big attachments go straight from the browser to the bucket.
  attachments:
    multiple: true
    max_files: 5
    upload: direct
    max_size: 1GiB

The browser asks backd to start the upload (declaring the file’s name, size, type and SHA-256), receives a signed link with those signed in, sends the bytes to MinIO, and tells backd to complete: backd checks the size, the checksum the storage computed and the type from the first bytes, and attaches the file. Anything that doesn’t match deletes the object. The client does all of it:

await assets.uploadFile(asset.id, 'attachments', file, { onProgress: ({ loaded, total }) => bar.set(loaded / total) })

The checksum is computed incrementally while the file is read, so a large file isn’t held in memory, and onProgress reports the bytes the browser has sent (through XMLHttpRequest). The bucket must allow the app’s origin: the stack’s MinIO already does (MINIO_API_CORS_ALLOW_ORIGIN). If you ever see upload_failed, it is nearly always CORS.

Try it: My assets → New asset → kind: file, pick two or three big files as Attachments, and watch the bar. Add more to an existing asset with Add an attachment. The sixth is refused: 409 too_many_files, answered before any byte moves.

A thumbnail, declared#

A gallery of full-size images is slow, and a small copy of each picture fixes it. There is nothing to write: the file field declares a version, and a worker makes it.

examples/config/shelf/main/assets/collection.yaml

  # One document or image per asset, streamed through backd.
  file:
    max_size: 25MiB
    types: [application/pdf, image/png, image/jpeg, text/plain]
    # Chapter 14: a small copy of an image for the gallery, made by a worker
    # after the upload. A PDF or a text file has none (its version is skipped).
    versions:
      thumb: { max_width: 256, max_height: 256, fit: cover }

thumb fits the picture in a 256 × 256 box, cropping the excess (cover), and keeps the picture’s format (a PNG stays a PNG). When an upload becomes part of an asset, backd queues a job and a worker (the stack already runs one) reads the original, makes the copy and stores it beside it. The asset records what happened, next to the file’s other details:

"file": {
  "id": "fl_8f3k2n5x0q7w1a9b4c6d", "name": "office.png", "type": "image/png", "size": 1203442,
  "width": 800, "height": 600,
  "versions": { "thumb": { "status": "ready", "type": "image/png", "width": 256, "height": 256, "size": 41022 } }
}

Until the worker is done the status is pending, so the app can say so; a PDF or a text file has no thumbnail, and its version is skipped rather than an error. The tutorial’s backd runs its worker inside (--with-worker), so it needs nothing else. Fetch the finished collection.yaml and restart:

fetch the chapter's files (optional — the listings above are the same files)

for f in main/assets/collection.yaml; do
  mkdir -p "config/shelf/$(dirname "$f")"
  curl -fsSL "https://fernandezvara.github.io/backd/tutorial/config/shelf/$f" -o "config/shelf/$f"
done

docker compose restart backd

The app asks for links with fileLinks: true, which also links every ready version, so a card shows the thumbnail when there is one and the picture when there isn’t:

previewUrl(asset) {
  return asset.file?.versions?.thumb?.url ?? (this.isImage(asset.file) ? asset.file.url : null)
}

After an upload the app looks again every second and a half, for a few seconds, until versionStatus(asset.file, 'thumb') says the worker is done. To link a version by hand, assets.fileUrl(id, 'file', fileId, { version: 'thumb' }) answers the link, or null while the version isn’t ready. The API says the same with 404 version_unavailable, carrying the version’s status and reason.

Try it: My assets → New asset → kind: file, pick a PNG or JPEG, and watch the card swap its picture for the thumbnail. The worker’s memory and how many pictures it handles at once are covered in Image versions.

The details a worker records (width, height, versions) are backd’s: a client that tries to write them has them dropped, so there is no rule to add for them. What a function writes into a field of its own is chapter 15.

You should see#

• An attachment of a few hundred MiB uploads with a progress bar, straight to MinIO (watch docker compose logs minio: the PUT arrives there, not at backd).
• Five attachments fit; the sixth answers 409 too_many_files.
• After a PNG upload, the card shows the thumbnail within a few seconds (docker compose logs backd shows the image versions made line).
• A text file’s versions.thumb.status is skipped; asking for its thumbnail answers 404 version_unavailable.

Next: chapter 15 — sharing files, and counting downloads.